Bug 144005 - [hang] System freezes
Summary: [hang] System freezes
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: i386 (show other bugs)
Version: 8.0-RELEASE
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-i386 (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-02-16 17:50 UTC by Daniel Mueller
Modified: 2017-12-31 22:35 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Mueller 2010-02-16 17:50:01 UTC
System Hangs after approx. 1 Week sometimes earlier sometimes later, then, there is no input or output and no logs.

The time of the freeze is typically random, it seems not to be bound to a high load or any special circumstances.

This is a client machine in a VMWare VCenter 4.0

FreeBSD 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #7: Mon Feb  1 13:42:26 CET 2010     toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB  i386

this machine is running mysql, has sometimes 1600+ threads.

made a debugging kernel with following additions to generic:

options     KDB
options     DDB
options     INVARIANT_SUPPORT
options     INVARIANTS
options     ALT_BREAK_TO_DEBUGGER
options     DEBUG_VFS_LOCKS
options     WITNESS

but still can't get into kernel debugger (hot key relocated to control-shift-f6, which worked, normally).

output of dmesg:

FreeBSD Host 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #7: Mon Feb  1 13:42:26 CET 2010     toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB  i386
[root@mx5 ~]# uname -a
FreeBSD mx5.imp.ch 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #7: Mon Feb  1 13:42:26 CET 2010     toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB  i386
[root@mx5 ~]# uname -a
FreeBSD mx5.imp.ch 8.0-RELEASE-p2 FreeBSD 8.0-RELEASE-p2 #7: Mon Feb  1 13:42:26 CET 2010     toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB  i386
[root@mx5 ~]# dmesg | more
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RELEASE-p2 #7: Mon Feb  1 13:42:26 CET 2010
    toor@dev6.imp.ch:/usr/obj/usr/src/sys/GENDDB
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU           X5460  @ 3.16GHz (3158.37-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f1  Stepping = 1
  Features=0xfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
  Features2=0x2201<SSE3,SSSE3,CX16>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 4026531840 (3840 MB)
avail memory = 3932672000 (3750 MB)
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <INTEL 440BX> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xf8000000-0xfbffffff,0xf4000000-0xf47fffff at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1080-0x10ff mem 0xf4860000-0xf4860fff irq 17 at device 16.0 on pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x1400-0x143f mem 0xf4820000-0xf483ffff,0xf4800000-0xf480ffff irq 18 at device 17.0 on pci0
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:50:56:85:4a:8b
em1: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x1440-0x147f mem 0xf4840000-0xf485ffff,0xf4810000-0xf481ffff irq 19 at device 18.0 on pci0
em1: Memory Access and/or Bus Master bits were not set!
em1: [FILTER]
em1: Ethernet address: 00:50:56:85:31:fc
acpi_acad0: <AC Adapter> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model IntelliMouse, device ID 3
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
cpu2: <ACPI CPU> on acpi0
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
cpu3: <ACPI CPU> on acpi0
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xcb000-0xcbfff,0xdc000-0xdffff,0xe0000-0xe3fff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 10.000 msec
Waiting 5 seconds for SCSI devices to settle
acd0: CDROM <VMware Virtual IDE CDROM Drive/00000001> at ata0-master UDMA33
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
fdc0: ready for output in input
da0 at mpt0 bus 0 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 20480MB (41943040 512 byte sectors: 255H 63S/T 2610C)
da1 at mpt0 bus 0 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 2048MB (4194304 512 byte sectors: 255H 63S/T 261C)
da2 at mpt0 bus 0 target 2 lun 0
da2: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da2: Command Queueing enabled
da2: 2048MB (4194304 512 byte sectors: 255H 63S/T 261C)
da3 at mpt0 bus 0 target 3 lun 0
da3: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da3: Command Queueing enabled
da3: 61440MB (125829120 512 byte sectors: 255H 63S/T 7832C)
da4 at mpt0 bus 0 target 4 lun 0
da4: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
da4: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da4: Command Queueing enabled
da4: 30720MB (62914560 512 byte sectors: 255H 63S/T 3916C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 17900 files 94
WARNING: /idms was not properly dismounted
lock order reversal:
 1st 0xdb8f55e0 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559
 2nd 0xc7dbf400 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
db_trace_self_wrapper(c0c70601,ea0aea74,c08bb055,c08abe7b,c0c7348e,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08abe7b,c0c7348e,c792c008,c792f2d0,ea0aead0,...) at kdb_backtrace+0x29
_witness_debugger(c0c7348e,c7dbf400,c0c94d77,c792f2d0,c0c94a10,...) at _witness_debugger+0x25
witness_checkorder(c7dbf400,9,c0c94a10,11d,0,...) at witness_checkorder+0x839
_sx_xlock(c7dbf400,0,c0c94a10,11d,dbe37364,...) at _sx_xlock+0x85
ufsdirhash_acquire(0,e,c7e09800,db8f5580,dbe37364,...) at ufsdirhash_acquire+0x48
ufsdirhash_remove(c7e58d24,dbe37364,364,ea0aeb60,ea0aeb5c,...) at ufsdirhash_remove+0x14
ufs_dirremove(c7bc3c90,c7fa29f8,500800c,0,c7bc3c90,...) at ufs_dirremove+0xe5
ufs_remove(ea0aec34,c0ca9ea0,0,0,c83ef430,...) at ufs_remove+0x6e
VOP_REMOVE_APV(c0d73f80,ea0aec34,c83ef430,ea0aec0c,bfbfef62,...) at VOP_REMOVE_APV+0xf6
kern_unlinkat(c7fbb480,ffffff9c,bfbfef62,0,ea0aec80,...) at kern_unlinkat+0x181
kern_unlink(c7fbb480,bfbfef62,0,ea0aed2c,c0bac083,...) at kern_unlink+0x27
unlink(c7fbb480,ea0aecf8,4,c0c73d2b,c0d52998,...) at unlink+0x22
syscall(ea0aed38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (10, FreeBSD ELF32, unlink), eip = 0x2814fa7f, esp = 0xbfbfe51c, ebp = 0xbfbfeda8 ---
lock order reversal:
 1st 0xc83f8bdc ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1054
 2nd 0xc7fa4ce8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2083
KDB: stack backtrace:
db_trace_self_wrapper(c0c70601,c77d2814,c08bb055,c08abe7b,c0c7348e,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08abe7b,c0c7348e,c792f268,c792f198,c77d2870,...) at kdb_backtrace+0x29
_witness_debugger(c0c7348e,c7fa4ce8,c0c62468,c792f198,c0c7a714,...) at _witness_debugger+0x25
witness_checkorder(c7fa4ce8,9,c0c7a714,823,0,...) at witness_checkorder+0x839
__lockmgr_args(c7fa4ce8,80100,c7fa4d04,0,0,...) at __lockmgr_args+0x7a7
vop_stdlock(c77d2978,c08badfb,c0c62699,80100,c7fa4c90,...) at vop_stdlock+0x62
VOP_LOCK1_APV(c0d4f640,c77d2978,c7b549a4,c0d8ca80,c7fa4c90,...) at VOP_LOCK1_APV+0xb5
_vn_lock(c7fa4c90,80100,c0c7a714,823,8,...) at _vn_lock+0x78
vget(c7fa4c90,80100,c7b54900,15e,c0c625bb,...) at vget+0xb9
devfs_allocv(c7dd5080,c83ffc94,c77d2a10,9d,c0f2dab8,...) at devfs_allocv+0x102
devfs_root(c83ffc94,80000,c77d2c30,42c,0,...) at devfs_root+0x4a
vfs_donmount(c7b54900,0,c7c84c80,c7c84c80,bfbfde51,...) at vfs_donmount+0x14c2
nmount(c7b54900,c77d2cf8,c,c7b54900,c0d551d8,...) at nmount+0x75
syscall(c77d2d38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (378, FreeBSD ELF32, nmount), eip = 0x280d246b, esp = 0xbfbfde2c, ebp = 0xbfbfe388 ---
Limiting closed port RST response from 248 to 200 packets/sec
Limiting closed port RST response from 335 to 200 packets/sec
Limiting closed port RST response from 305 to 200 packets/sec
Limiting closed port RST response from 248 to 200 packets/sec
Limiting closed port RST response from 281 to 200 packets/sec
Limiting closed port RST response from 323 to 200 packets/sec
Limiting closed port RST response from 239 to 200 packets/sec
Limiting closed port RST response from 276 to 200 packets/sec
Limiting closed port RST response from 263 to 200 packets/sec
pid 1873 (sshd) is using legacy pty devices - not logging anymore
Limiting closed port RST response from 253 to 200 packets/sec
lock order reversal:
 1st 0xc9a6e058 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:423
 2nd 0xdba7cd20 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559
 3rd 0xc7ec8ce8 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:544
KDB: stack backtrace:
db_trace_self_wrapper(c0c70601,c77da40c,c08bb055,c08abe7b,c0c734a7,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08abe7b,c0c734a7,c792c008,c792f268,c77da468,...) at kdb_backtrace+0x29
_witness_debugger(c0c734a7,c7ec8ce8,c0c66067,c792f268,c0c9280e,...) at _witness_debugger+0x25
witness_checkorder(c7ec8ce8,9,c0c9280e,220,0,...) at witness_checkorder+0x839
__lockmgr_args(c7ec8ce8,80100,c7ec8d04,0,0,...) at __lockmgr_args+0x7a7
ffs_lock(c77da574,c0ef0c28,c7e699a4,80100,c7ec8c90,...) at ffs_lock+0x9e
VOP_LOCK1_APV(c0d73f80,c77da574,c77da594,c0d8ca80,c7ec8c90,...) at VOP_LOCK1_APV+0xb5
_vn_lock(c7ec8c90,80100,c0c9280e,220,c795a700,...) at _vn_lock+0x78
ffs_snapshot(c7e4ea10,c7dab920,c0c94215,15f,c0c79a46,...) at ffs_snapshot+0x150b
ffs_mount(c7e4ea10,0,c0c79f33,3d2,0,...) at ffs_mount+0x14aa
vfs_donmount(c7e69900,1211000,c86ac380,c86ac380,1211000,...) at vfs_donmount+0x1012
nmount(c7e69900,c77dacf8,c,c7e69900,c0d551d8,...) at nmount+0x75
syscall(c77dad38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (378, FreeBSD ELF32, nmount), eip = 0x280d746b, esp = 0xbfbfeacc, ebp = 0xbfbfee28 ---
lock order reversal:
 1st 0xdba7cd20 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559
 2nd 0xc905e3dc snaplk (snaplk) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:793
KDB: stack backtrace:
db_trace_self_wrapper(c0c70601,c77da40c,c08bb055,c08abe7b,c0c7348e,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08abe7b,c0c7348e,c792c008,c792fa20,c77da468,...) at kdb_backtrace+0x29
_witness_debugger(c0c7348e,c905e3dc,c0c92891,c792fa20,c0c9280e,...) at _witness_debugger+0x25
witness_checkorder(c905e3dc,9,c0c9280e,319,c9a6e074,...) at witness_checkorder+0x839
__lockmgr_args(c905e3dc,80400,c9a6e074,0,0,...) at __lockmgr_args+0x7a7
ffs_lock(c77da574,0,0,80400,c9a6e000,...) at ffs_lock+0x9e
VOP_LOCK1_APV(c0d73f80,c77da574,c191a330,c0d8ca80,c9a6e000,...) at VOP_LOCK1_APV+0xb5
_vn_lock(c9a6e000,80400,c0c9280e,319,0,...) at _vn_lock+0x78
ffs_snapshot(c7e4ea10,c7dab920,c0c94215,15f,c0c79a46,...) at ffs_snapshot+0x28c6
ffs_mount(c7e4ea10,0,c0c79f33,3d2,0,...) at ffs_mount+0x14aa
vfs_donmount(c7e69900,1211000,c86ac380,c86ac380,1211000,...) at vfs_donmount+0x1012
nmount(c7e69900,c77dacf8,c,c7e69900,c0d551d8,...) at nmount+0x75
syscall(c77dad38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (378, FreeBSD ELF32, nmount), eip = 0x280d746b, esp = 0xbfbfeacc, ebp = 0xbfbfee28 ---
lock order reversal:
 1st 0xc905e3dc snaplk (snaplk) @ /usr/src/sys/kern/vfs_vnops.c:296
 2nd 0xc9a6e058 ufs (ufs) @ /usr/src/sys/ufs/ffs/ffs_snapshot.c:1587
KDB: stack backtrace:
db_trace_self_wrapper(c0c70601,c77da8b4,c08bb055,c08abe7b,c0c7348e,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c08abe7b,c0c7348e,c792fa20,c792f268,c77da910,...) at kdb_backtrace+0x29
_witness_debugger(c0c7348e,c9a6e058,c0c66067,c792f268,c0c9280e,...) at _witness_debugger+0x25
witness_checkorder(c9a6e058,9,c0c9280e,633,0,...) at witness_checkorder+0x839
__lockmgr_args(c9a6e058,80000,0,0,0,...) at __lockmgr_args+0x7a7
ffs_snapremove(c9a6e000,c7e4ea10,0,c0c7bb58,41d,...) at ffs_snapremove+0x11f
softdep_releasefile(c873e15c,c77daa98,2,c7e699a4,c9a6e000,...) at softdep_releasefile+0x3b
ufs_inactive(c77daadc,c0ca9c14,c9a6e074,c9a6e074,c9a6e000,...) at ufs_inactive+0x1bc
VOP_INACTIVE_APV(c0d73f80,c77daadc,c0c7a714,924,c0d8ca40,...) at VOP_INACTIVE_APV+0xc5
vinactive(c0d73f80,c77dab10,c0c7a714,8aa,c9a6e000,...) at vinactive+0xae
vput(c9a6e000,c77dab4c,c0c7bb58,128,0,...) at vput+0x1dd
vn_close(c9a6e000,1,c796b400,c7e69900,0,...) at vn_close+0x19a
vn_closefile(c8400d20,c7e69900,3,0,c8400d20,...) at vn_closefile+0xe4
_fdrop(c8400d20,c7e69900,c77dac18,c08bae9c,0,c7e699a4,c0ef0bf8,c0d5bd50,c0c67d9b,c873612c,45b,c0c67d9b,c77dac40,c0881d80,c873612c,8,c0c67d9b,45b) at _fdrop+0x43
closef(c8400d20,c7e69900,45b,440,c873612c,...) at closef+0x290
kern_close(c7e69900,4,c77dad2c,c0bac083,c7e69900,...) at kern_close+0x117
close(c7e69900,c77dacf8,4,c0c74283,c0d52928,...) at close+0x1a
syscall(c77dad38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (6, FreeBSD ELF32, close), eip = 0x2816e283, esp = 0xbfbfeacc, ebp = 0xbfbfee28 ---
Comment 1 Remko Lodder freebsd_committer freebsd_triage 2010-02-24 07:18:29 UTC
Responsible Changed
From-To: freebsd-i386->freebsd-bugs

I dont think it's i386 specific...
Comment 2 Eitan Adler freebsd_committer freebsd_triage 2017-12-31 08:00:25 UTC
For bugs matching the following criteria:

Status: In Progress Changed: (is less than) 2014-06-01

Reset to default assignee and clear in-progress tags.

Mail being skipped