A server daemon I run keeps its database in memory, and after a scheduled time flushes the dirty database files from memory to harddrives. The database files are stored as millions of tiny CDB binary files in 1 directory. I use RAID 0+1 on a promise raid card. When the server kicks off a flush and starts doing a large amount of IO, i get the following errors in dmesg: Dec 10 13:51:39 <kern.crit> bnet kernel: DOH! ata_alloc_request failed! Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:51:39 <kern.crit> bnet last message repeated 4 times Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE g- out of memory i_vn fsat_da_one(raid):_iarni0st_1dre[WquRIestTE(offset=2030 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs Dec 10 13:51:39 <kern.crit> bnet kernel: _done()FA:aILr0UREs1 -d[ oWRIut oTEf (omeffmoseryt= 2in ata0329_rai03d_94init_request Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_rai62d_4, inleit_rengquthest=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290411008, length=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_dFAoInLUe():RE - arou0st 1dof[ mWRemITory E(inoffs aet=t2a_raid0329_init0427392 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE_re(oquffseest=t203290460160, length=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290509312, length=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=20329057 Dec 10 13:51:39 <kern.crit> bnet kernel: 4848, lenFAILgtURh=E - 163ou84t )]of error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290787840, length=16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203675779072, length=16384)]error me= mory5 in ata_r Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:51:39 <kern.crit> bnet kernel: F Dec 10 13:51:39 <kern.crit> bnet kernel: AILURE - out go_fv fmse_mdoornye (i)n: aart0as_1rda[iWdR_IiTnEi(to_frfesqeute=st Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory i2n0 3a8t6a8_5r2a0i4d4_8i,n ilte_nrgetqhu=est Dec 10 13:51:39 <kern.crit> bnet kernel: 16384)]error = 5 Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - ougt_ ovff mse_modroy nien (a)ta:_arra0isd1_di[nWiRtI_TrEe(qofufesestt= Dec 10 13:51:39 <kern.crit> bnet kernel: 203868536832, length=F16A3I84L)U]ReErr o-r =o u5t Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203 8o6f8 5m5e3m2o1r6y, ilne nagttah_=raid_1i6n3i8t Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memorry i=n 5a Dec 10 13:51:39 <kern.crit> bnet kernel: ta_raid_init_request Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=F2A04I0L61U2R78E2 0-8 ,o ulte nogft hm=emory 1in ata Dec 10 13:51:39 <kern.crit> bnet kernel: <snip> Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=12288)]error = 5 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5 Dec 10 13:52:02 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5 Dec 10 13:52:02 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ <crash> Dec 10 13:53:24 <kern.crit> bnet kernel: Copyright (c) 1992-2005 The FreeBSD Project. Dec 10 13:53:24 <kern.crit> bnet kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Dec 10 13:53:24 <kern.crit> bnet kernel: The Regents of the University of California. All rights reserved. Dec 10 13:53:24 <kern.crit> bnet kernel: FreeBSD 6.0-RELEASE #0: Sat Dec 10 02:13:22 UTC 2005 Kernel: http://ender.tog.net/crash/BNET dmesg: http://ender.tog.net/crash/dmesg.today loader.conf: http://ender.tog.net/crash/loader.conf messages: http://ender.tog.net/crash/messages.0 This problem was also repeated on just RAID 0 array. Unfortunatally I do not have a kernel dump file at the moment due to the bnet# dumpon /dev/ar0 dumpon: ioctl(DIOCSKERNELDUMP): Operation not supported by device limitation, but hopefully will have another disk in this server to get a proper kernel dump soon. How-To-Repeat: Run a daemon that uses a lot of disk IO on a raid array.
I see the opener of this PR no longer has a valid email address. I am experiencing the same symptoms on 6.2-Release amd64 + sil raid1. Can I help at all by providing dumps or access to a developer to my system ? If anyone is interested in solving this bug I would be happy to donate hardware (I checked the request list and no developer has asked for an ata raid1 card + hdds to test this with). This issue does not appear to be universal (atleast, it doesn't appear to be fixed yet so I imagine it isn't universal). I get these errors from nightly periodic filesystem checks on a system with multiple jails - the IO of each jail performing its periodics within the same few minutes causes the errors to show up in dmesg and an occasional reboot. Regards, Alex Trull
I get these errors periodically (without a crash so far) on my FreeBSD 6.2 system still. My server is amd64 but using: ar0: 76319MB <nVidia MediaShield RAID1> status: READY It's a Tyan barebones server (Tyan Transport GT20) with an nvidia nforce pro 3400 chipset whose built-in SATA RAID I am using. I probably haven't managed a crash yet because my I/O is so light. So far we have a Silicon Image, a Promise Fasttrak and an nvidia chipset's RAID1 mirroring causing the same problem. Indications are that this is a generic problem with FreeBSD 6's software ATA RAID1 stuff. DOH! ata_alloc_request failed! FAILURE - out of memory in ata_raid_init_request FAILURE - out of memory in ata_raid_init_request FAILURE - out of memory in ata_raid_init_request FAILURE - out of memory in ata_raid_init_request FAILURE - out of memory in ata_raid_init_request FAILURE - out of memory in ata_raid_init_request FAILURE - out of memory in ata_raid_init_request FAILURE - out of memory in ata_raid_init_request FAILURE - out of memory in ata_raid_init_request g_vfs_done():ar0s1e[WRITE(offset=75324096512, length=16384)]error = 5 g_vfs_done():ar0s1e[WRITE(offset=75324112896, length=16384)]error = 5 g_vfs_done():ar0s1e[WRITE(offset=75324129280, length=16384)]error = 5 g_vfs_done():ar0s1e[WRITE(offset=75324145664, length=16384)]error = 5 g_vfs_done():ar0s1e[WRITE(offset=75324162048, length=16384)]error = 5 swap_pager: I/O error - pageout failed; blkno 63550,size 4096, error 5 swap_pager: I/O error - pageout failed; blkno 78419,size 49152, error 5 swap_pager: I/O error - pageout failed; blkno 69285,size 12288, error 5 swap_pager: I/O error - pageout failed; blkno 69875,size 12288, error 5 -- TerraNovaNet Internet Services - Key Largo, FL Voice: (305)453-4011 x101 Fax: (305)451-5991 http://www.terranova.net/ ---------------------------------------------- Life's not fair, but the root password helps.
For bugs matching the following criteria: Status: In Progress Changed: (is less than) 2014-06-01 Reset to default assignee and clear in-progress tags. Mail being skipped
OBE: the old ata driver has been removed.