System Hangs after approx. 1 Week sometimes earlier sometimes later, then, there is no input or output and no logs. The time of the freeze is typically random, it seems not to be bound to a high load or any special circumstances. This is a client machine in a VMWare VCenter 4.0 FreeBSD 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #7: Mon Feb 1 13:42:26 CET 2010 toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB i386 this machine is running mysql, has sometimes 1600+ threads. made a debugging kernel with following additions to generic: options KDB options DDB options INVARIANT_SUPPORT options INVARIANTS options ALT_BREAK_TO_DEBUGGER options DEBUG_VFS_LOCKS options WITNESS but still can't get into kernel debugger (hot key relocated to control-shift-f6, which worked, normally). output of dmesg: FreeBSD Host 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #7: Mon Feb 1 13:42:26 CET 2010 toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB i386 [root@mx5 ~]# uname -a FreeBSD mx5.imp.ch 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #7: Mon Feb 1 13:42:26 CET 2010 toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB i386 [root@mx5 ~]# uname -a FreeBSD mx5.imp.ch 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #7: Mon Feb 1 13:42:26 CET 2010 toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB i386 [root@mx5 ~]# dmesg | more Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RELEASE-p2 #7: Mon Feb 1 13:42:26 CET 2010 toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU X5460 @ 3.16GHz (3158.37-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f1 Stepping = 1 Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS> Features2=0x2201<SSE3,SSSE3,CX16> AMD Features=0x20100000<NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 4026531840 (3840 MB) avail memory = 3932672000 (3750 MB) ACPI APIC Table: <PTLTD APIC > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 <Version 1.1> irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: <INTEL 440BX> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 isab0: <PCI-ISA bridge> at device 7.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] pci0: <bridge> at device 7.3 (no driver attached) vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4860000-0xf4860fff irq 17 at device 16.0 on pci0 mpt0: [ITHREAD] mpt0: MPI Version=1.2.0.0 em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:50:56:85:4a:8b em1: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x1440-0x147f mem 0xf4840000-0xf485ffff,0xf4810000-0xf481ffff irq 19 at device 18.0 on pci0 em1: Memory Access and/or Bus Master bits were not set! em1: [FILTER] em1: Ethernet address: 00:50:56:85:31:fc acpi_acad0: <AC Adapter> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: <Parallel port bus> on ppc0 plip0: <PLIP network interface> on ppbus0 plip0: [ITHREAD] lpt0: <Printer> on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] cpu0: <ACPI CPU> on acpi0 acpi_throttle0: <ACPI CPU Throttling> on cpu0 cpu1: <ACPI CPU> on acpi0 acpi_throttle1: <ACPI CPU Throttling> on cpu1 acpi_throttle1: failed to attach P_CNT device_attach: acpi_throttle1 attach returned 6 cpu2: <ACPI CPU> on acpi0 acpi_throttle2: <ACPI CPU Throttling> on cpu2 acpi_throttle2: failed to attach P_CNT device_attach: acpi_throttle2 attach returned 6 cpu3: <ACPI CPU> on acpi0 acpi_throttle3: <ACPI CPU Throttling> on cpu3 acpi_throttle3: failed to attach P_CNT device_attach: acpi_throttle3 attach returned 6 pmtimer0 on isa0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xcb000-0xcbfff,0xdc000-0xdffff,0xe0000-0xe3fff pnpid ORM0000 on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 10.000 msec Waiting 5 seconds for SCSI devices to settle acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33 fdc0: ready for output in input fdc0: ready for output in input fdc0: ready for output in input fdc0: ready for output in input fdc0: ready for output in input fdc0: ready for output in input fdc0: ready for output in input fdc0: ready for output in input da0 at mpt0 bus 0 target 0 lun 0 da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 20480MB (41943040 512 byte sectors: 255H 63S/T 2610C) da1 at mpt0 bus 0 target 1 lun 0 da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 2048MB (4194304 512 byte sectors: 255H 63S/T 261C) da2 at mpt0 bus 0 target 2 lun 0 da2: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 2048MB (4194304 512 byte sectors: 255H 63S/T 261C) da3 at mpt0 bus 0 target 3 lun 0 da3: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da3: Command Queueing enabled da3: 61440MB (125829120 512 byte sectors: 255H 63S/T 7832C) da4 at mpt0 bus 0 target 4 lun 0 da4: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device da4: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da4: Command Queueing enabled da4: 30720MB (62914560 512 byte sectors: 255H 63S/T 3916C) SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted WARNING: /tmp was not properly dismounted WARNING: /var was not properly dismounted /var: mount pending error: blocks 17900 files 94 WARNING: /idms was not properly dismounted lock order reversal: 1st 0xdb8f55e0 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559 2nd 0xc7dbf400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper(c0c70601,ea0aea74,c08bb055,c08abe7b,c0c7348e,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08abe7b,c0c7348e,c792c008,c792f2d0,ea0aead0,...) at kdb_backtrace+0x29 _witness_debugger(c0c7348e,c7dbf400,c0c94d77,c792f2d0,c0c94a10,...) at _witness_debugger+0x25 witness_checkorder(c7dbf400,9,c0c94a10,11d,0,...) at witness_checkorder+0x839 _sx_xlock(c7dbf400,0,c0c94a10,11d,dbe37364,...) at _sx_xlock+0x85 ufsdirhash_acquire(0,e,c7e09800,db8f5580,dbe37364,...) at ufsdirhash_acquire+0x48 ufsdirhash_remove(c7e58d24,dbe37364,364,ea0aeb60,ea0aeb5c,...) at ufsdirhash_remove+0x14 ufs_dirremove(c7bc3c90,c7fa29f8,500800c,0,c7bc3c90,...) at ufs_dirremove+0xe5 ufs_remove(ea0aec34,c0ca9ea0,0,0,c83ef430,...) at ufs_remove+0x6e VOP_REMOVE_APV(c0d73f80,ea0aec34,c83ef430,ea0aec0c,bfbfef62,...) at VOP_REMOVE_APV+0xf6 kern_unlinkat(c7fbb480,ffffff9c,bfbfef62,0,ea0aec80,...) at kern_unlinkat+0x181 kern_unlink(c7fbb480,bfbfef62,0,ea0aed2c,c0bac083,...) at kern_unlink+0x27 unlink(c7fbb480,ea0aecf8,4,c0c73d2b,c0d52998,...) at unlink+0x22 syscall(ea0aed38) at syscall+0x2a3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (10, FreeBSD ELF32, unlink), eip = 0x2814fa7f, esp = 0xbfbfe51c, ebp = 0xbfbfeda8 --- lock order reversal: 1st 0xc83f8bdc ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1054 2nd 0xc7fa4ce8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2083 KDB: stack backtrace: db_trace_self_wrapper(c0c70601,c77d2814,c08bb055,c08abe7b,c0c7348e,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08abe7b,c0c7348e,c792f268,c792f198,c77d2870,...) at kdb_backtrace+0x29 _witness_debugger(c0c7348e,c7fa4ce8,c0c62468,c792f198,c0c7a714,...) at _witness_debugger+0x25 witness_checkorder(c7fa4ce8,9,c0c7a714,823,0,...) at witness_checkorder+0x839 __lockmgr_args(c7fa4ce8,80100,c7fa4d04,0,0,...) at __lockmgr_args+0x7a7 vop_stdlock(c77d2978,c08badfb,c0c62699,80100,c7fa4c90,...) at vop_stdlock+0x62 VOP_LOCK1_APV(c0d4f640,c77d2978,c7b549a4,c0d8ca80,c7fa4c90,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7fa4c90,80100,c0c7a714,823,8,...) at _vn_lock+0x78 vget(c7fa4c90,80100,c7b54900,15e,c0c625bb,...) at vget+0xb9 devfs_allocv(c7dd5080,c83ffc94,c77d2a10,9d,c0f2dab8,...) at devfs_allocv+0x102 devfs_root(c83ffc94,80000,c77d2c30,42c,0,...) at devfs_root+0x4a vfs_donmount(c7b54900,0,c7c84c80,c7c84c80,bfbfde51,...) at vfs_donmount+0x14c2 nmount(c7b54900,c77d2cf8,c,c7b54900,c0d551d8,...) at nmount+0x75 syscall(c77d2d38) at syscall+0x2a3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (378, FreeBSD ELF32, nmount), eip = 0x280d246b, esp = 0xbfbfde2c, ebp = 0xbfbfe388 --- Limiting closed port RST response from 248 to 200 packets/sec Limiting closed port RST response from 335 to 200 packets/sec Limiting closed port RST response from 305 to 200 packets/sec Limiting closed port RST response from 248 to 200 packets/sec Limiting closed port RST response from 281 to 200 packets/sec Limiting closed port RST response from 323 to 200 packets/sec Limiting closed port RST response from 239 to 200 packets/sec Limiting closed port RST response from 276 to 200 packets/sec Limiting closed port RST response from 263 to 200 packets/sec pid 1873 (sshd) is using legacy pty devices - not logging anymore Limiting closed port RST response from 253 to 200 packets/sec lock order reversal: 1st 0xc9a6e058 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:423 2nd 0xdba7cd20 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559 3rd 0xc7ec8ce8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:544 KDB: stack backtrace: db_trace_self_wrapper(c0c70601,c77da40c,c08bb055,c08abe7b,c0c734a7,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08abe7b,c0c734a7,c792c008,c792f268,c77da468,...) at kdb_backtrace+0x29 _witness_debugger(c0c734a7,c7ec8ce8,c0c66067,c792f268,c0c9280e,...) at _witness_debugger+0x25 witness_checkorder(c7ec8ce8,9,c0c9280e,220,0,...) at witness_checkorder+0x839 __lockmgr_args(c7ec8ce8,80100,c7ec8d04,0,0,...) at __lockmgr_args+0x7a7 ffs_lock(c77da574,c0ef0c28,c7e699a4,80100,c7ec8c90,...) at ffs_lock+0x9e VOP_LOCK1_APV(c0d73f80,c77da574,c77da594,c0d8ca80,c7ec8c90,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c7ec8c90,80100,c0c9280e,220,c795a700,...) at _vn_lock+0x78 ffs_snapshot(c7e4ea10,c7dab920,c0c94215,15f,c0c79a46,...) at ffs_snapshot+0x150b ffs_mount(c7e4ea10,0,c0c79f33,3d2,0,...) at ffs_mount+0x14aa vfs_donmount(c7e69900,1211000,c86ac380,c86ac380,1211000,...) at vfs_donmount+0x1012 nmount(c7e69900,c77dacf8,c,c7e69900,c0d551d8,...) at nmount+0x75 syscall(c77dad38) at syscall+0x2a3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (378, FreeBSD ELF32, nmount), eip = 0x280d746b, esp = 0xbfbfeacc, ebp = 0xbfbfee28 --- lock order reversal: 1st 0xdba7cd20 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559 2nd 0xc905e3dc snaplk (snaplk) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:793 KDB: stack backtrace: db_trace_self_wrapper(c0c70601,c77da40c,c08bb055,c08abe7b,c0c7348e,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08abe7b,c0c7348e,c792c008,c792fa20,c77da468,...) at kdb_backtrace+0x29 _witness_debugger(c0c7348e,c905e3dc,c0c92891,c792fa20,c0c9280e,...) at _witness_debugger+0x25 witness_checkorder(c905e3dc,9,c0c9280e,319,c9a6e074,...) at witness_checkorder+0x839 __lockmgr_args(c905e3dc,80400,c9a6e074,0,0,...) at __lockmgr_args+0x7a7 ffs_lock(c77da574,0,0,80400,c9a6e000,...) at ffs_lock+0x9e VOP_LOCK1_APV(c0d73f80,c77da574,c191a330,c0d8ca80,c9a6e000,...) at VOP_LOCK1_APV+0xb5 _vn_lock(c9a6e000,80400,c0c9280e,319,0,...) at _vn_lock+0x78 ffs_snapshot(c7e4ea10,c7dab920,c0c94215,15f,c0c79a46,...) at ffs_snapshot+0x28c6 ffs_mount(c7e4ea10,0,c0c79f33,3d2,0,...) at ffs_mount+0x14aa vfs_donmount(c7e69900,1211000,c86ac380,c86ac380,1211000,...) at vfs_donmount+0x1012 nmount(c7e69900,c77dacf8,c,c7e69900,c0d551d8,...) at nmount+0x75 syscall(c77dad38) at syscall+0x2a3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (378, FreeBSD ELF32, nmount), eip = 0x280d746b, esp = 0xbfbfeacc, ebp = 0xbfbfee28 --- lock order reversal: 1st 0xc905e3dc snaplk (snaplk) @ /usr/src/sys/kern/vfs_vnops.c:296 2nd 0xc9a6e058 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:1587 KDB: stack backtrace: db_trace_self_wrapper(c0c70601,c77da8b4,c08bb055,c08abe7b,c0c7348e,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c08abe7b,c0c7348e,c792fa20,c792f268,c77da910,...) at kdb_backtrace+0x29 _witness_debugger(c0c7348e,c9a6e058,c0c66067,c792f268,c0c9280e,...) at _witness_debugger+0x25 witness_checkorder(c9a6e058,9,c0c9280e,633,0,...) at witness_checkorder+0x839 __lockmgr_args(c9a6e058,80000,0,0,0,...) at __lockmgr_args+0x7a7 ffs_snapremove(c9a6e000,c7e4ea10,0,c0c7bb58,41d,...) at ffs_snapremove+0x11f softdep_releasefile(c873e15c,c77daa98,2,c7e699a4,c9a6e000,...) at softdep_releasefile+0x3b ufs_inactive(c77daadc,c0ca9c14,c9a6e074,c9a6e074,c9a6e000,...) at ufs_inactive+0x1bc VOP_INACTIVE_APV(c0d73f80,c77daadc,c0c7a714,924,c0d8ca40,...) at VOP_INACTIVE_APV+0xc5 vinactive(c0d73f80,c77dab10,c0c7a714,8aa,c9a6e000,...) at vinactive+0xae vput(c9a6e000,c77dab4c,c0c7bb58,128,0,...) at vput+0x1dd vn_close(c9a6e000,1,c796b400,c7e69900,0,...) at vn_close+0x19a vn_closefile(c8400d20,c7e69900,3,0,c8400d20,...) at vn_closefile+0xe4 _fdrop(c8400d20,c7e69900,c77dac18,c08bae9c,0,c7e699a4,c0ef0bf8,c0d5bd50,c0c67d9b,c873612c,45b,c0c67d9b,c77dac40,c0881d80,c873612c,8,c0c67d9b,45b) at _fdrop+0x43 closef(c8400d20,c7e69900,45b,440,c873612c,...) at closef+0x290 kern_close(c7e69900,4,c77dad2c,c0bac083,c7e69900,...) at kern_close+0x117 close(c7e69900,c77dacf8,4,c0c74283,c0d52928,...) at close+0x1a syscall(c77dad38) at syscall+0x2a3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (6, FreeBSD ELF32, close), eip = 0x2816e283, esp = 0xbfbfeacc, ebp = 0xbfbfee28 ---
Responsible Changed From-To: freebsd-i386->freebsd-bugs I dont think it's i386 specific...
For bugs matching the following criteria: Status: In Progress Changed: (is less than) 2014-06-01 Reset to default assignee and clear in-progress tags. Mail being skipped