When a single process loads the CPU for a long(*) time, other processes which want to access to the filesystem get stuck in the "ufs" state when trying to do that. Other processes which don't need to access the filesystem (like top, etc.) proceed normally. The owner UID, nice- and idprio- status of the offending (or offended) process do not matter. It seems essential that a single process is working all the time (e.g. two hours of compilation don't show up any errors, because there are many processes). Example top outputs for this situation: last pid: 9798; load averages: 2.00, 1.96, 1.72 up 0+02:29:59 11:02:17 113 processes: 3 running, 109 sleeping, 1 zombie CPU states: 0.0% user, 96.2% nice, 0.4% system, 3.4% interrupt, 0.0% idle Mem: 134M Active, 193M Inact, 88M Wired, 316K Cache, 59M Buf, 70M Free Swap: 4097M Total, 64M Used, 4033M Free, 1% Inuse PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 2661 df 1 139 20 63532K 58468K RUN 107:21 97.46% generic_slave 8035 df 1 96 0 7608K 2752K select 0:00 0.05% mc 702 root 1 96 0 5200K 792K select 0:41 0.00% syslogd 1245 root 1 -4 0 5152K 572K ufs 0:35 0.00% tail 912 squid 1 -4 0 13928K 4288K ufs 0:31 0.00% squid 1163 df 1 -4 0 9716K 2512K ufs 0:27 0.00% fetchmail 2600 root 1 -32 0 5516K 2052K RUN 0:13 0.00% top 1042 mysql 6 20 0 59128K 1956K kserel 0:06 0.00% mysqld 9338 df 1 -4 0 15988K 9812K ufs 0:06 0.00% links 1179 root 1 96 0 5200K 112K select 0:02 0.00% moused last pid: 2739; load averages: 2.00, 1.95, 1.57 up 0+00:31:39 16:46:48 91 processes: 3 running, 87 sleeping, 1 zombie CPU states: 97.8% user, 0.0% nice, 0.4% system, 1.9% interrupt, 0.0% idle Mem: 120M Active, 29M Inact, 32M Wired, 17M Cache, 15M Buf, 287M Free Swap: 4097M Total, 4097M Free PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 1352 df 1 132 0 2248K 896K RUN 25:58 97.41% testcase 1580 df 1 96 0 7608K 2696K select 0:09 0.00% mc 970 squid 1 -4 0 13928K 8108K ufs 0:03 0.00% squid 1316 root 1 -32 0 5516K 1608K RUN 0:03 0.00% top 1238 df 1 -4 0 7668K 1884K ufs 0:02 0.00% fetchmail 1110 mysql 6 20 0 59128K 51600K kserel 0:01 0.00% mysqld 754 root 1 96 0 5200K 1088K select 0:01 0.00% syslogd 1322 root 1 96 0 22656K 16968K select 0:01 0.00% Xorg 868 root 1 8 0 9232K 4748K nanslp 0:00 0.00% httpd 1126 news 1 8 0 5484K 1308K wait 0:00 0.00% sh 1317 root 1 -4 0 5152K 684K ufs 0:00 0.00% tail 1144 news 1 4 4 7816K 3396K sbwait 0:00 0.00% perl5.8.8 1120 news 1 -4 0 150M 12828K ufs 0:00 0.00% innd 1351 df 1 20 0 4032K 1784K pause 0:00 0.00% csh 993 squid 1 -4 0 5640K 824K msgwai 0:00 0.00% diskd 1699 df 1 8 0 5244K 3104K ppwait 0:00 0.00% csh 1182 root 1 8 0 5200K 1100K nanslp 0:00 0.00% cron (*) for values of `long' from 10 minutes to 2 hours for me. Fix: I don't know the fix, but an offending process can be stopped with 'kill -STOP' and continued with 'kill -CONT', which allows other processes to access the filesystem (until another such failure occurs). Periodic stopping and starting processes might count as a lousy workaround. How-To-Repeat: A program to generate the necessary load can be quite simple, like int main(void) { /* Crunch some numbers (really meaningless) */ unsigned u=1; while (1) { u*=0x8088405; } } Compile and run it, run `top', and wait for a long (see above) time. Browse directories with `ls' from time to time on a different terminal. See `ls' hang at some time. View the `top' terminal.
I have the problem with the same symptoms on SMP FreeBSD 6.2-STABLE with dual AMD Opteron(tm) Processor 850. Sometimes (apparently when CPU load suddenly goes up) all processes that interacts with disk gets stuck in "ufs" state, but in my case SIGSTOP/SIGCONT seemingly does not help. uname -a output: FreeBSD serv2.vsi.ru 6.2-STABLE FreeBSD 6.2-STABLE #2: Sat Mar 3 01:59:08 MSK 2007 oleg@serv2.vsi.ru:/usr/obj/usr/src/sys/serv2 i386 dmesg.boot: Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-STABLE #2: Sat Mar 3 01:59:08 MSK 2007 oleg@serv2.vsi.ru:/usr/obj/usr/src/sys/serv2 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Opteron(tm) Processor 850 (2389.26-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x20f51 Stepping = 1 Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2> Features2=0x1<SSE3> AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow> AMD Features2=0x1<LAHF> real memory = 8589934592 (8192 MB) avail memory = 8350457856 (7963 MB) ACPI APIC Table: <PTLTD APIC > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 <Version 1.1> irqs 0-23 on motherboard ioapic1 <Version 1.1> irqs 24-27 on motherboard ioapic2 <Version 1.1> irqs 28-31 on motherboard ioapic3 <Version 1.1> irqs 32-35 on motherboard ioapic4 <Version 1.1> irqs 36-39 on motherboard ioapic5 <Version 1.1> irqs 40-43 on motherboard ioapic6 <Version 1.1> irqs 44-47 on motherboard kbd1 at kbdmux0 acpi0: <PTLTD XSDT> on motherboard acpi0: Power Button (fixed) unknown: I/O range not supported unknown: I/O range not supported Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xf008-0xf00b on acpi0 cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 acpi_button0: <Power Button> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0xf000-0xf07f,0xf080-0xf0ff iomem 0xd8000-0xdbfff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0 pci1: <ACPI PCI bus> on pcib1 ohci0: <OHCI (generic) USB controller> mem 0xfc900000-0xfc900fff irq 19 at device 0.0 on pci1 ohci0: [GIANT-LOCKED] usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: <OHCI (generic) USB controller> on ohci0 usb0: USB revision 1.0 uhub0: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 3 ports with 3 removable, self powered ohci1: <OHCI (generic) USB controller> mem 0xfc901000-0xfc901fff irq 19 at device 0.1 on pci1 ohci1: [GIANT-LOCKED] usb1: OHCI version 1.0, legacy support usb1: SMM does not respond, resetting usb1: <OHCI (generic) USB controller> on ohci1 usb1: USB revision 1.0 uhub1: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub1: 3 ports with 3 removable, self powered pci1: <display, VGA> at device 5.0 (no driver attached) isab0: <PCI-ISA bridge> at device 7.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <AMD 8111 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1000-0x100f at device 7.1 on pci0 ata0: <ATA channel 0> on atapci0 ata1: <ATA channel 1> on atapci0 pci0: <bridge> at device 7.3 (no driver attached) pcib2: <ACPI PCI-PCI bridge> at device 10.0 on pci0 pci2: <ACPI PCI bus> on pcib2 bge0: <Broadcom BCM5704 A3, ASIC rev. 0x2003> mem 0xfe010000-0xfe01ffff,0xfe000000-0xfe00ffff irq 25 at device 2.0 on pci2 miibus0: <MII bus> on bge0 brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge0: Ethernet address: 00:09:3d:13:fd:00 bge1: <Broadcom BCM5704 A3, ASIC rev. 0x2003> mem 0xfe030000-0xfe03ffff,0xfe020000-0xfe02ffff irq 26 at device 2.1 on pci2 miibus1: <MII bus> on bge1 brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge1: Ethernet address: 00:09:3d:13:fd:01 mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0xfe050000-0xfe05ffff,0xfe040000-0xfe04ffff irq 27 at device 4.0 on pci2 mpt0: [GIANT-LOCKED] mpt0: MPI Version=1.2.15.0 mpt0: Capabilities: ( RAID-1E RAID-1 SAFTE ) mpt0: 0 Active Volumes (1 Max) mpt0: 0 Hidden Drive Members (6 Max) pci0: <base peripheral, interrupt controller> at device 10.1 (no driver attached) pcib3: <ACPI PCI-PCI bridge> at device 11.0 on pci0 pci3: <ACPI PCI bus> on pcib3 pci0: <base peripheral, interrupt controller> at device 11.1 (no driver attached) pcib4: <ACPI Host-PCI bridge> iomem 0xfe301000-0xfe301fff,0xfe303000-0xfe303fff,0xfe305000-0xfe305fff,0xfe307000-0xfe307fff on acpi0 pci32: <ACPI PCI bus> on pcib4 pcib5: <ACPI PCI-PCI bridge> mem 0xfe300000-0xfe300fff irq 32 at device 1.0 on pci32 pci33: <ACPI PCI bus> on pcib5 pci32: <base peripheral, interrupt controller> at device 1.1 (no driver attached) pcib6: <ACPI PCI-PCI bridge> mem 0xfe302000-0xfe302fff irq 36 at device 2.0 on pci32 pci34: <ACPI PCI bus> on pcib6 pci32: <base peripheral, interrupt controller> at device 2.1 (no driver attached) pcib7: <ACPI PCI-PCI bridge> mem 0xfe304000-0xfe304fff irq 40 at device 3.0 on pci32 pci35: <ACPI PCI bus> on pcib7 pci32: <base peripheral, interrupt controller> at device 3.1 (no driver attached) pcib8: <ACPI PCI-PCI bridge> mem 0xfe306000-0xfe306fff irq 44 at device 4.0 on pci32 pci36: <ACPI PCI bus> on pcib8 pci32: <base peripheral, interrupt controller> at device 4.1 (no driver attached) atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FAST] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 pmtimer0 on isa0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcafff,0xcb000-0xcefff on isa0 ppc0: parallel port not found. sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1 at port 0x2f8-0x2ff irq 3 on isa0 sio1: type 16550A vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec IP Filter: v4.1.13 initialized. Default = block all, Logging = enabled Waiting 5 seconds for SCSI devices to settle acd0: DMA limited to UDMA33, controller found non-ATA66 cable acd0: DVDROM <MATSHITADVD-ROM SR-8178/PZ21> at ata1-master UDMA33 ses0 at mpt0 bus 0 target 6 lun 0 ses0: <SDR GEM318P 1> Fixed Processor SCSI-2 device ses0: 3.300MB/s transfers ses0: SAF-TE Compliant Device SMP: AP CPU #1 Launched! da0 at mpt0 bus 0 target 0 lun 0 da0: <SEAGATE ST373207LC 0003> Fixed Direct Access SCSI-3 device da0: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing Enabled da0: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C) da1 at mpt0 bus 0 target 2 lun 0 da1: <SEAGATE ST336807LC 0C01> Fixed Direct Access SCSI-3 device da1: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing Enabled da1: 35003MB (71687372 512 byte sectors: 255H 63S/T 4462C) Trying to mount root from ufs:/dev/da0s1a Accounting enabled Any ideas on how to debug this ? -- Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE Phone: +7 4732 539880 Fax: +7 4732 531415 http://www.vsi.ru CenterTelecom Voronezh ISP http://isp.vsi.ru
I believe we have also experienced this bug (or a very similar one) on our 8-core amd64 systems under 6.2-RELEASE-p4. In our case, "top" shows that the system is 100% CPU utilized, with the vast majority of it as "system" time. (Ordinarily the system In the last case, we ended up with about 200 Apache processes that looked like this in ps: UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 25000 27121 26860 1977 -4 5 146324 33732 ufs DN ?? 0:03.75 httpd 25000 27147 37257 1994 -4 5 153748 29280 ufs DN ?? 0:03.72 httpd 25000 27157 36912 1805 -4 5 150756 26592 ufs DN ?? 0:02.91 httpd 25000 27224 27030 1845 -4 5 137536 24804 ufs DN ?? 0:01.25 httpd 25000 27274 26794 1829 -4 5 148140 35416 ufs DN ?? 0:02.90 httpd Once a process gets "stuck" in WCHAN ufs, it's blocked indefinitely, as described here, or at least so slow as to be indistinguishable from stuck. (Typical wait channels for our httpds are accept or kqread, as one would expect.) Each process in this state counts against the load average, so we often see load averages north of 200 when this is occurring. (Typical load average is below 2.) Kill enough processes (or possibly enough to hit the "right" process) and everything picks up again right where it left off. I also have no idea how to debug this. Thanks, Jeff -- Jeff Wheelhouse jdw@wheelhouse.org
This problem experiences on our another AMD64 machine (also with 6-STABLE). When we copy large amount of small files using mc from FTP to local filesystem, after some time mc hangs in "wdrain" state, and all other processes that need to access filesystem are hangs in "ufs" state. There are some debug stuff: uname -a: FreeBSD serv13.vsi.ru 6.2-STABLE FreeBSD 6.2-STABLE #1: Fri Oct 19 16:28:07 MSD 2007 oleg@serv13.vsi.ru:/usr/obj/usr/src/sys/serv13 i386 dmesg: Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-STABLE #1: Fri Oct 19 16:28:07 MSD 2007 oleg@serv13.vsi.ru:/usr/obj/usr/src/sys/serv13 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Dual-Core AMD Opteron(tm) Processor 2212 (2010.31-MHz 686-class CPU) Origin = "AuthenticAMD" Id = 0x40f12 Stepping = 2 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x2001<SSE3,CX16> AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!> AMD Features2=0x1f<LAHF,CMP,SVM,ExtAPIC,CR8> Cores per package: 2 real memory = 3220176896 (3071 MB) avail memory = 3149598720 (3003 MB) ACPI APIC Table: <PTLTD APIC > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 <Version 1.1> irqs 0-23 on motherboard ioapic1 <Version 1.1> irqs 24-47 on motherboard kbd1 at kbdmux0 acpi0: <PTLTD RSDT> on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 acpi_button0: <Power Button> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pci0: <memory, RAM> at device 0.0 (no driver attached) isab0: <PCI-ISA bridge> port 0x1c00-0x1c7f at device 1.0 on pci0 isa0: <ISA bus> on isab0 pci0: <serial bus, SMBus> at device 1.1 (no driver attached) ohci0: <OHCI (generic) USB controller> mem 0xc0040000-0xc0040fff irq 16 at device 2.0 on pci0 ohci0: [GIANT-LOCKED] usb0: OHCI version 1.0, legacy support usb0: SMM does not respond, resetting usb0: <OHCI (generic) USB controller> on ohci0 usb0: USB revision 1.0 uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 10 ports with 10 removable, self powered ehci0: <EHCI (generic) USB 2.0 controller> mem 0xc0041000-0xc00410ff irq 17 at device 2.1 on pci0 ehci0: [GIANT-LOCKED] usb1: EHCI version 1.0 usb1: companion controller, 10 ports each: usb0 usb1: <EHCI (generic) USB 2.0 controller> on ehci0 usb1: USB revision 2.0 uhub1: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub1: 10 ports with 10 removable, self powered atapci0: <nVidia nForce MCP55 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3480-0x348f at device 4.0 on pci0 ata0: <ATA channel 0> on atapci0 ata1: <ATA channel 1> on atapci0 pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <PCI-PCI bridge> at device 4.0 on pci1 pci2: <PCI bus> on pcib2 mly0: <Mylex AcceleRAID 170> mem 0xc0600000-0xc0601fff irq 20 at device 4.1 on pci1 mly0: [GIANT-LOCKED] mly0: AcceleRAID 170 , 1 channel, firmware 6.00-7-00 (20001214), 32MB RAM pci0: <bridge> at device 8.0 (no driver attached) pci0: <bridge> at device 9.0 (no driver attached) pcib3: <ACPI PCI-PCI bridge> at device 13.0 on pci0 pci3: <ACPI PCI bus> on pcib3 pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3 pci4: <ACPI PCI bus> on pcib4 fxp0: <Intel 82559 Pro/100 Ethernet> port 0x4000-0x403f mem 0xc0300000-0xc0300fff,0xc0200000-0xc02fffff irq 22 at device 9.0 on pci4 miibus0: <MII bus> on fxp0 inphy0: <i82555 10/100 media interface> on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto fxp0: Ethernet address: 00:d0:b7:1c:80:7e pcib5: <ACPI PCI-PCI bridge> mem 0xc0100000-0xc010007f irq 21 at device 0.1 on pci3 pci5: <ACPI PCI bus> on pcib5 pcib6: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci128: <ACPI PCI bus> on pcib6 pci128: <memory, RAM> at device 0.0 (no driver attached) pci128: <memory, RAM> at device 1.0 (no driver attached) pci128: <serial bus, SMBus> at device 1.1 (no driver attached) pcib7: <ACPI PCI-PCI bridge> at device 13.0 on pci128 pci129: <ACPI PCI bus> on pcib7 pcib8: <ACPI PCI-PCI bridge> at device 15.0 on pci128 pci130: <ACPI PCI bus> on pcib8 pci130: <display, VGA> at device 0.0 (no driver attached) atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FAST] pmtimer0 on isa0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xcefff,0xcf000-0xd07ff on isa0 ppc0: <Parallel port> at port 0x278-0x27f irq 7 on isa0 ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode ppbus0: <Parallel port bus> on ppc0 plip0: <PLIP network interface> on ppbus0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port ppi0: <Parallel I/O> on ppbus0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec IP Filter: v4.1.13 initialized. Default = block all, Logging = enabled ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, default to accept, logging disabled acd0: CDROM <NEC CD-ROM CD-3002A/C000> at ata0-master UDMA33 da0 at mly0 bus 1 target 0 lun 0 da0: <RAID 7 online > Fixed Direct Access SCSI-3 device da0: 135.168MB/s transfers da0: 34712MB (71090176 512 byte sectors: 255H 63S/T 4425C) da1 at mly0 bus 1 target 1 lun 0 da1: <RAID 7 online > Fixed Direct Access SCSI-3 device da1: 135.168MB/s transfers da1: 34712MB (71090176 512 byte sectors: 255H 63S/T 4425C) SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da1:mly0:1:1:0): Sense Error Code 0x0 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 (da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da0:mly0:1:0:0): Sense Error Code 0x0 Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da1:mly0:1:1:0): Sense Error Code 0x0 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da1:mly0:1:1:0): Sense Error Code 0x0 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da1:mly0:1:1:0): Sense Error Code 0x0 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da1:mly0:1:1:0): Sense Error Code 0x0 (da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0 (da1:mly0:1:1:0): Sense Error Code 0x0 acquiring duplicate lock of same type: "vnode interlock" 1st vnode interlock @ /usr/src/sys/kern/vfs_vnops.c:806 2nd vnode interlock @ /usr/src/sys/kern/vfs_subr.c:2039 KDB: stack backtrace: kdb_backtrace(3,c80e6780,c07a89f0,c07a89f0,c07678a4,...) at kdb_backtrace+0x29 witness_checkorder(c7eb0d04,9,c072db32,7f7) at witness_checkorder+0x578 _mtx_lock_flags(c7eb0d04,0,c072db32,7f7,c8055980,...) at _mtx_lock_flags+0x78 vrefcnt(c7eb0c3c) at vrefcnt+0x20 null_checkvp(c81dac3c,c071c808,215) at null_checkvp+0x56 null_lock(ea872a68) at null_lock+0x66 VOP_LOCK_APV(c0760b20,ea872a68) at VOP_LOCK_APV+0x87 vn_lock(c81dac3c,1002,c80e6780,c81dac3c,c81dae60,...) at vn_lock+0xac nullfs_root(c8209000,2,ea872ae0,c80e6780,0,8,0,c07e5fe0,0,c072d3b0,407) at nullfs_root+0x26 vfs_domount(c80e6780,c8055100,c8055700,d,c8055030,c0797ae0,0,c072d3b0,2bf) at vfs_domount+0x975 vfs_donmount(c80e6780,d,c80ede80,c80ede80,0,...) at vfs_donmount+0x3f9 nmount(c80e6780,ea872d04) at nmount+0x8b syscall(3b,3b,3b,bfbfe5f5,bfbfeea0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (378, FreeBSD ELF32, nmount), eip = 0x280bc11b, esp = 0xbfbfe5bc, ebp = 0xbfbfee38 --- Accounting enabled show pcpu: cpuid = 0 curthread = 0xc7ce3300: pid 20 "swi6: Giant tasq" curpcb = 0xe68dbd90 fpcurthread = none idlethread = 0xc7ce3a80: pid 13 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: show locks: exclusive sleep mutex Giant r = 0 (0xc0797ae0) locked @ /usr/src/sys/kern/kern_intr.c: 681 show alllocks: Process 20 (swi6: Giant tasq) thread 0xc7ce3300 (100008) exclusive sleep mutex Giant r = 0 (0xc0797ae0) locked @ /usr/src/sys/kern/kern_intr.c: 681 After getting a kernel dump I can obtain more information. -- Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE Phone: +7 4732 539880 Fax: +7 4732 531415 http://www.vsi.ru CenterTelecom Voronezh ISP http://isp.vsi.ru
There are all required information from "Debugging Deadlocks" chapter of FreeBSD Developer's Handbook: db> show pcpu cpuid = 1 curthread = 0xc7ce3900: pid 12 "idle: cpu1" curpcb = 0xe68e7d90 fpcurthread = none idlethread = 0xc7ce3900: pid 12 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: exclusive spin mutex sio r = 0 (0xc07f61a0) locked @ /usr/src/sys/dev/sio/sio.c:1390 db> show allpcpu Current CPU: 1 cpuid = 0 curthread = 0xc7ce3a80: pid 13 "idle: cpu0" curpcb = 0xe68ead90 fpcurthread = none idlethread = 0xc7ce3a80: pid 13 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 curthread = 0xc7ce3900: pid 12 "idle: cpu1" curpcb = 0xe68e7d90 fpcurthread = none idlethread = 0xc7ce3900: pid 12 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: exclusive spin mutex sio r = 0 (0xc07f61a0) locked @ /usr/src/sys/dev/sio/sio.c:1390 cpuid = 2 curthread = 0xc7ce3780: pid 11 "idle: cpu2" curpcb = 0xe68e4d90 fpcurthread = none idlethread = 0xc7ce3780: pid 11 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 curthread = 0xc7ce3600: pid 10 "idle: cpu3" curpcb = 0xe68e1d90 fpcurthread = none idlethread = 0xc7ce3600: pid 10 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db> show locks exclusive spin mutex sio r = 0 (0xc07f61a0) locked @ /usr/src/sys/dev/sio/sio.c:1390 db> show alllocks Process 938 (sendmail) thread 0xc82e8300 (100115) shared sx acct_sx r = 0 (0xc0795f00) locked @ /usr/src/sys/kern/kern_acct.c:308 Process 937 (sendmail) thread 0xc83c4780 (100154) shared sx acct_sx r = 0 (0xc0795f00) locked @ /usr/src/sys/kern/kern_acct.c:308 db> show lockedvnods Locked vnodes 0xc7eb1ae0: tag ufs, type VREG usecount 3, writecount 0, refcount 4 mountedhere 0 flags (VV_TEXT) v_object 0xc106d318 ref 1 pages 25 lock type ufs: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc0559c15 at lockmgr+0x4ed #1 0xc0673b06 at ffs_lock+0x76 #2 0xc06e9797 at VOP_LOCK_APV+0x87 #3 0xc05c9444 at vn_lock+0xac #4 0xc05bd2ea at vget+0xbe #5 0xc06725b1 at ffs_sync+0x1c1 #6 0xc05bee5c at sync_fsync+0x164 #7 0xc06e8b27 at VOP_FSYNC_APV+0x9b #8 0xc05bcace at sync_vnode+0x106 #9 0xc05bcd49 at sched_sync+0x1ed #10 0xc0550cc4 at fork_exit+0xa0 #11 0xc06c23bc at fork_trampoline+0x8 ino 70723, on dev da0s1a 0xc817a570: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type syncer: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc0559c15 at lockmgr+0x4ed #1 0xc05b4189 at vop_stdlock+0x21 #2 0xc06e9797 at VOP_LOCK_APV+0x87 #3 0xc05c9444 at vn_lock+0xac #4 0xc05bcaab at sync_vnode+0xe3 #5 0xc05bcd49 at sched_sync+0x1ed #6 0xc0550cc4 at fork_exit+0xa0 #7 0xc06c23bc at fork_trampoline+0x8 0xc8228000: tag ufs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () v_object 0xc814e4a4 ref 0 pages 0 lock type ufs: EXCL (count 1) by thread 0xc8105d80 (pid 939)#0 0xc0559c15 at lockmgr+0x4ed #1 0xc0673b06 at ffs_lock+0x76 #2 0xc06e9797 at VOP_LOCK_APV+0x87 #3 0xc05c9444 at vn_lock+0xac #4 0xc05bd2ea at vget+0xbe #5 0xc05b1a66 at cache_lookup+0x34a #6 0xc05b1f9a at vfs_cache_lookup+0x92 #7 0xc06e7857 at VOP_LOOKUP_APV+0x87 #8 0xc05b61d0 at lookup+0x4a4 #9 0xc05b5a8a at namei+0x39a #10 0xc05c8443 at vn_open_cred+0x26f #11 0xc05c81d2 at vn_open+0x1e #12 0xc05c1879 at kern_open+0x231 #13 0xc05c1612 at open+0x1a #14 0xc06d6d4b at syscall+0x25b #15 0xc06c23af at Xint0x80_syscall+0x1f ino 282624, on dev da0s1e 0xc8302ae0: tag ufs, type VREG usecount 1, writecount 1, refcount 3 mountedhere 0 flags () v_object 0xc8233840 ref 0 pages 10 lock type ufs: EXCL (count 1) by thread 0xc8087a80 (pid 548)#0 0xc0559c15 at lockmgr+0x4ed #1 0xc0673b06 at ffs_lock+0x76 #2 0xc06e9797 at VOP_LOCK_APV+0x87 #3 0xc05c9444 at vn_lock+0xac #4 0xc05c8de0 at vn_write+0x138 #5 0xc058ba28 at dofilewrite+0x7c #6 0xc058b8c7 at kern_writev+0x3b #7 0xc058b870 at writev+0x30 #8 0xc06d6d4b at syscall+0x25b #9 0xc06c23af at Xint0x80_syscall+0x1f ino 259099, on dev da0s1e 0xc830bc3c: tag ufs, type VREG usecount 1, writecount 1, refcount 4 mountedhere 0 flags () v_object 0xc81578c4 ref 0 pages 12 lock type ufs: EXCL (count 1) by thread 0xc83c4780 (pid 937) with 1 pending#0 0xc0559c15 at lockmgr+0x4ed #1 0xc0673b06 at ffs_lock+0x76 #2 0xc06e9797 at VOP_LOCK_APV+0x87 #3 0xc05c9444 at vn_lock+0xac #4 0xc05c87dd at vn_rdwr+0x85 #5 0xc053d3bf at acct_process+0x31f #6 0xc054eb82 at exit1+0x716 #7 0xc054e46c at exit1+0 #8 0xc06d6d4b at syscall+0x25b #9 0xc06c23af at Xint0x80_syscall+0x1f ino 23555, on dev da0s1e 0xc837915c: tag ufs, type VREG usecount 0, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xc8282210 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc83c4480 (pid 936) with 1 pending#0 0xc0559c15 at lockmgr+0x4ed #1 0xc0673b06 at ffs_lock+0x76 #2 0xc06e9797 at VOP_LOCK_APV+0x87 #3 0xc05c9444 at vn_lock+0xac #4 0xc05c8708 at vn_close+0x5c #5 0xc05c9582 at vn_closefile+0xca #6 0xc05488ac at fdrop_locked+0xb0 #7 0xc05487f4 at fdrop+0x24 #8 0xc05472e3 at closef+0x367 #9 0xc0544dd2 at kern_close+0x1aa #10 0xc0544c24 at close+0x10 #11 0xc06d6d4b at syscall+0x25b #12 0xc06c23af at Xint0x80_syscall+0x1f ino 282642, on dev da0s1e 0xc9feb984: tag ufs, type VREG usecount 1, writecount 1, refcount 19 mountedhere 0 flags () v_object 0xc9fee210 ref 0 pages 68 lock type ufs: EXCL (count 1) by thread 0xc8193c00 (pid 844)#0 0xc0559c15 at lockmgr+0x4ed #1 0xc0673b06 at ffs_lock+0x76 #2 0xc06e9797 at VOP_LOCK_APV+0x87 #3 0xc05c9444 at vn_lock+0xac #4 0xc05c8de0 at vn_write+0x138 #5 0xc058ba28 at dofilewrite+0x7c #6 0xc058b8c7 at kern_writev+0x3b #7 0xc058b7ed at write+0x45 #8 0xc06d6d4b at syscall+0x25b #9 0xc06c23af at Xint0x80_syscall+0x1f ino 4098799, on dev da1s1d db> alltrace Tracing command sshd pid 939 tid 100071 td 0xc8105d80 sched_switch(c8105d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c83791b4,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c83791b4,0,c8379224,b7,c072d198,...) at sleepq_wait+0x46 msleep(c83791b4,c07969b0,50,c072b884,0,...) at msleep+0x27d acquire(ea87e820,40,60000,c8105d80,0,...) at acquire+0x76 lockmgr(c83791b4,2002,c8379224,c8105d80) at lockmgr+0x44e ffs_lock(ea87e878) at ffs_lock+0x76 VOP_LOCK_APV(c077a640,ea87e878) at VOP_LOCK_APV+0x87 vn_lock(c837915c,2002,c8105d80,c837915c) at vn_lock+0xac vget(c837915c,2002,c8105d80) at vget+0xbe cache_lookup(c8228000,ea87ebe0,ea87ebf4) at cache_lookup+0x34a vfs_cache_lookup(ea87e9bc) at vfs_cache_lookup+0x92 VOP_LOOKUP_APV(c077a640,ea87e9bc) at VOP_LOOKUP_APV+0x87 lookup(ea87ebcc) at lookup+0x4a4 namei(ea87ebcc) at namei+0x39a vn_open_cred(ea87ebcc,ea87eccc,0,c82fe380,3,...) at vn_open_cred+0x26f vn_open(ea87ebcc,ea87eccc,0,3,c0797aa0,...) at vn_open+0x1e kern_open(c8105d80,280915c0,0,1,0,...) at kern_open+0x231 open(c8105d80,ea87ed04) at open+0x1a syscall(3b,3b,3b,2809d000,804aa41,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x2807ca7f, esp = 0xbfbfec1c, ebp = 0xbfbfecc8 --- Tracing command sendmail pid 938 tid 100115 td 0xc82e8300 sched_switch(c82e8300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c830bc94,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c830bc94,0,c830bd04,b7,c071f3eb,...) at sleepq_wait+0x46 msleep(c830bc94,c07975ec,50,c072b884,0,...) at msleep+0x27d acquire(ea951ac0,40,60000,c82e8300,0,...) at acquire+0x76 lockmgr(c830bc94,3002,c830bd04,c82e8300) at lockmgr+0x44e ffs_lock(ea951b18) at ffs_lock+0x76 VOP_LOCK_APV(c077a640,ea951b18) at VOP_LOCK_APV+0x87 vn_lock(c830bc3c,1002,c82e8300) at vn_lock+0xac vn_rdwr(1,c830bc3c,ea951c34,30,0,0,1,3,c7ce0780,0,0,c82e8300) at vn_rdwr+0x85 acct_process(c82e8300,c0797c20,c07209a0,163,c8364648,...) at acct_process+0x31f exit1(c82e8300,0,ea951d30,c06d6d4b,c82e8300,...) at exit1+0x716 exit1(c82e8300,ea951d04) at exit1 syscall(3b,3b,3b,0,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x282fc0fb, esp = 0xbfbfcb1c, ebp = 0xbfbfcb38 --- Tracing command sendmail pid 937 tid 100154 td 0xc83c4780 sched_switch(c83c4780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(dc011d14,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(dc011d14,0,c830bd04,b7,c071f3eb,...) at sleepq_wait+0x46 msleep(dc011d14,c0796dc4,50,c072c4b4,0,...) at msleep+0x27d acquire(ea9db860,120,60000,c83c4780,0,...) at acquire+0x76 lockmgr(dc011d14,202122,c830bd04,c83c4780) at lockmgr+0x44e getblk(c830bc3c,10,0,4000,0,...) at getblk+0x12f breadn(c830bc3c,10,0,4000,0,...) at breadn+0x31 bread(c830bc3c,10,0,4000,0,ea9db970) at bread+0x20 ffs_balloc_ufs2(c830bc3c,41a60,0,30,c7ce0780,...) at ffs_balloc_ufs2+0x1849 ffs_write(ea9dbb54) at ffs_write+0x2c0 VOP_WRITE_APV(c077a640,ea9dbb54) at VOP_WRITE_APV+0x132 vn_rdwr(1,c830bc3c,ea9dbc34,30,0,0,1,3,c7ce0780,0,0,c83c4780) at vn_rdwr+0x142 acct_process(c83c4780,c0797c20,c07209a0,163,c865b860,...) at acct_process+0x31f exit1(c83c4780,0,ea9dbd30,c06d6d4b,c83c4780,...) at exit1+0x716 exit1(c83c4780,ea9dbd04) at exit1 syscall(3b,3b,3b,0,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x282fc0fb, esp = 0xbfbfbf4c, ebp = 0xbfbfbf68 --- Tracing command sh pid 936 tid 100156 td 0xc83c4480 sched_switch(c83c4480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(dbfd0b84,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(dbfd0b84,0,c817a8f0,b7,c072e794,...) at sleepq_wait+0x46 msleep(dbfd0b84,c0796af4,50,c072c4b4,0,...) at msleep+0x27d acquire(ea9d59bc,120,60000,c83c4480,0,...) at acquire+0x76 lockmgr(dbfd0b84,202122,c817a8f0,c83c4480) at lockmgr+0x44e getblk(c817a828,44ea60,0,4000,0,...) at getblk+0x12f breadn(c817a828,44ea60,0,4000,0,...) at breadn+0x31 bread(c817a828,44ea60,0,4000,0,ea9d5aa8) at bread+0x20 ffs_update(c837915c,0) at ffs_update+0x22c ufs_inactive(ea9d5b04) at ufs_inactive+0x2a6 VOP_INACTIVE_APV(c077a640,ea9d5b04) at VOP_INACTIVE_APV+0x9b vinactive(c837915c,c83c4480) at vinactive+0x8d vput(c837915c,c8227000,c078b740,c837915c,1,...) at vput+0x1d3 vn_close(c837915c,1,c8593780,c83c4480) at vn_close+0x97 vn_closefile(c812e2d0,c83c4480) at vn_closefile+0xca fdrop_locked(c812e2d0,c83c4480,c7ccba0c,0,c072016e,...) at fdrop_locked+0xb0 fdrop(c812e2d0,c83c4480,6ac,c079efa0,0,...) at fdrop+0x24 closef(c812e2d0,c83c4480,0,c83c4480,0,...) at closef+0x367 kern_close(c83c4480,3,ea9d5d30,c06d6d4b,c83c4480,...) at kern_close+0x1aa close(c83c4480,ea9d5d04) at close+0x10 syscall(3b,3b,3b,2808e000,3,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (6, FreeBSD ELF32, close), eip = 0x2806ba9f, esp = 0xbfbfebcc, ebp = 0xbfbfec78 --- Tracing command cron pid 935 tid 100160 td 0xc8366d80 sched_switch(c8366d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8121330) at sleepq_switch+0xc1 sleepq_wait_sig(c8121330) at sleepq_wait_sig+0x1d msleep(c8121330,c81214a0,14c,c0728d39,0) at msleep+0x26e pipe_read(c8750ca8,ea9c9cbc,c810e400,0,c8366d80) at pipe_read+0x377 dofileread(c8366d80,5,c8750ca8,ea9c9cbc,ffffffff,...) at dofileread+0x85 kern_readv(c8366d80,5,ea9c9cbc,805a000,1000,...) at kern_readv+0x36 read(c8366d80,ea9c9d04) at read+0x45 syscall(3b,3b,3b,28171040,28171040,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x2814b953, esp = 0xbfbfe65c, ebp = 0xbfbfe678 --- Tracing command mc pid 844 tid 100148 td 0xc8193c00 sched_switch(c8193c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5d04,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c07e5d04,0,c072be5a,180,c072e794,...) at sleepq_wait+0x46 msleep(c07e5d04,c07e5d20,44,c072be76,0) at msleep+0x27d waitrunningbufspace(dbd8a778,10,0,ea90d9a4,c05acc09,...) at waitrunningbufspace+0x4a bufwrite(dbd502cc) at bufwrite+0x191 bawrite(dbd502cc,c9feba4c,c1e75650,c9feba4c,c9feba4c,...) at bawrite+0x51 cluster_wbuild(c9feb984,4000,8,0,8) at cluster_wbuild+0x61c cluster_write(c9feb984,dbd8a778,40000,0,50) at cluster_write+0x503 ffs_write(ea90dbf4) at ffs_write+0x523 VOP_WRITE_APV(c077a640,ea90dbf4) at VOP_WRITE_APV+0x132 vn_write(c8738af8,ea90dcbc,c810b200,0,c8193c00) at vn_write+0x1ea dofilewrite(c8193c00,7,c8738af8,ea90dcbc,ffffffff,...) at dofilewrite+0x7c kern_writev(c8193c00,7,ea90dcbc,86c4130,570,...) at kern_writev+0x3b write(c8193c00,ea90dd04) at write+0x45 syscall(3b,3b,3b,16a0,7,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (4, FreeBSD ELF32, write), eip = 0x2836d933, esp = 0xbfbfde6c, ebp = 0xbfbfde98 --- Tracing command csh pid 841 tid 100143 td 0xc83c5a80 sched_switch(c83c5a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c852f034) at sleepq_switch+0xc1 sleepq_wait_sig(c852f034) at sleepq_wait_sig+0x1d msleep(c852f034,c852f068,168,c0724047,0) at msleep+0x26e kern_sigsuspend(c83c5a80,2,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c83c5a80,ea9ffd04) at sigsuspend+0x33 syscall(3b,3b,3b,8299008,bfbf1ad0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1abc, ebp = 0xbfbf1ae8 --- Tracing command su pid 840 tid 100146 td 0xc8193d80 sched_switch(c8193d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c852f648) at sleepq_switch+0xc1 sleepq_wait_sig(c852f648) at sleepq_wait_sig+0x1d msleep(c852f648,c852f6b0,15c,c0729071,0) at msleep+0x26e kern_wait(c8193d80,349,ea910c84,2,0) at kern_wait+0x6bb wait4(c8193d80,ea910d04) at wait4+0x2a syscall(3b,3b,3b,bfbfe794,bfbfe810,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x280f451f, esp = 0xbfbfe70c, ebp = 0xbfbfe728 --- Tracing command csh pid 835 tid 100098 td 0xc8102480 sched_switch(c8102480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c82e6034) at sleepq_switch+0xc1 sleepq_wait_sig(c82e6034) at sleepq_wait_sig+0x1d msleep(c82e6034,c82e6068,168,c0724047,0) at msleep+0x26e kern_sigsuspend(c8102480,2,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c8102480,ea84ed04) at sigsuspend+0x33 syscall(3b,3b,3b,81c1308,bfbf1bd0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1bbc, ebp = 0xbfbf1be8 --- Tracing command sshd pid 834 tid 100042 td 0xc7e05480 sched_switch(c7e05480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b kern_select(c7e05480,9,807bb60,807bb70,0,...) at kern_select+0x4bf select(c7e05480,e8386d04) at select+0x44 syscall(3b,3b,3b,0,bfbfe11c,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe0ac, ebp = 0xbfbfe0f8 --- Tracing command sshd pid 831 tid 100074 td 0xc8105900 sched_switch(c8105900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8275370) at sleepq_switch+0xc1 sleepq_wait_sig(c8275370) at sleepq_wait_sig+0x1d msleep(c8275370,c827533c,158,c072b38c,0,...) at msleep+0x26e sbwait(c8275318,ea875c0c,c058ae7c,4,0,...) at sbwait+0x4b soreceive(c82752c8,0,ea875cbc,0,0,0) at soreceive+0x2ee soo_read(c8111c18,ea875cbc,c82e1380,0,c8105900) at soo_read+0x41 dofileread(c8105900,5,c8111c18,ea875cbc,ffffffff,...) at dofileread+0x85 kern_readv(c8105900,5,ea875cbc,bfbfe108,4,...) at kern_readv+0x36 read(c8105900,ea875d04) at read+0x45 syscall(3b,3b,3b,4,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x2834d953, esp = 0xbfbfe0ac, ebp = 0xbfbfe0d8 --- Tracing command httpd pid 830 tid 100087 td 0xc818a780 sched_switch(c818a780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b0900) at sleepq_switch+0xc1 sleepq_wait_sig(c80b0900) at sleepq_wait_sig+0x1d msleep(c80b0900,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b0900,16,ffffffff,ffffffff,ea8c8cb0,...) at lf_setlock+0x181 lf_advlock(ea8c8c90,c84cfd9c,0,0,ea8c8c7c,...) at lf_advlock+0x203 ufs_advlock(ea8c8c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8c8c90) at VOP_ADVLOCK_APV+0xa9 flock(c818a780,ea8c8d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 829 tid 100084 td 0xc818ac00 sched_switch(c818ac00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b0440) at sleepq_switch+0xc1 sleepq_wait_sig(c80b0440) at sleepq_wait_sig+0x1d msleep(c80b0440,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b0440,16,ffffffff,ffffffff,ea8d1cb0,...) at lf_setlock+0x181 lf_advlock(ea8d1c90,c84cfd9c,0,0,ea8d1c7c,...) at lf_advlock+0x203 ufs_advlock(ea8d1c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8d1c90) at VOP_ADVLOCK_APV+0xa9 flock(c818ac00,ea8d1d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 828 tid 100063 td 0xc815ea80 sched_switch(c815ea80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c815b3c0) at sleepq_switch+0xc1 sleepq_wait_sig(c815b3c0) at sleepq_wait_sig+0x1d msleep(c815b3c0,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c815b3c0,16,ffffffff,ffffffff,ea8a3cb0,...) at lf_setlock+0x181 lf_advlock(ea8a3c90,c84cfd9c,0,0,ea8a3c7c,...) at lf_advlock+0x203 ufs_advlock(ea8a3c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8a3c90) at VOP_ADVLOCK_APV+0xa9 flock(c815ea80,ea8a3d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 827 tid 100080 td 0xc818e300 sched_switch(c818e300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8067540) at sleepq_switch+0xc1 sleepq_wait_sig(c8067540) at sleepq_wait_sig+0x1d msleep(c8067540,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c8067540,16,ffffffff,ffffffff,ea8ddcb0,...) at lf_setlock+0x181 lf_advlock(ea8ddc90,c84cfd9c,0,0,ea8ddc7c,...) at lf_advlock+0x203 ufs_advlock(ea8ddc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8ddc90) at VOP_ADVLOCK_APV+0xa9 flock(c818e300,ea8ddd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 826 tid 100043 td 0xc7e05300 sched_switch(c7e05300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8067c80) at sleepq_switch+0xc1 sleepq_wait_sig(c8067c80) at sleepq_wait_sig+0x1d msleep(c8067c80,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c8067c80,16,ffffffff,ffffffff,e8383cb0,...) at lf_setlock+0x181 lf_advlock(e8383c90,c84cfd9c,0,0,e8383c7c,...) at lf_advlock+0x203 ufs_advlock(e8383c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,e8383c90) at VOP_ADVLOCK_APV+0xa9 flock(c7e05300,e8383d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 825 tid 100076 td 0xc8105600 sched_switch(c8105600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b0940) at sleepq_switch+0xc1 sleepq_wait_sig(c80b0940) at sleepq_wait_sig+0x1d msleep(c80b0940,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b0940,16,ffffffff,ffffffff,ea86fcb0,...) at lf_setlock+0x181 lf_advlock(ea86fc90,c84cfd9c,0,0,ea86fc7c,...) at lf_advlock+0x203 ufs_advlock(ea86fc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea86fc90) at VOP_ADVLOCK_APV+0xa9 flock(c8105600,ea86fd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 824 tid 100064 td 0xc815e900 sched_switch(c815e900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b0980) at sleepq_switch+0xc1 sleepq_wait_sig(c80b0980) at sleepq_wait_sig+0x1d msleep(c80b0980,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b0980,16,ffffffff,ffffffff,ea8a0cb0,...) at lf_setlock+0x181 lf_advlock(ea8a0c90,c84cfd9c,0,0,ea8a0c7c,...) at lf_advlock+0x203 ufs_advlock(ea8a0c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8a0c90) at VOP_ADVLOCK_APV+0xa9 flock(c815e900,ea8a0d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 823 tid 100110 td 0xc82e8a80 sched_switch(c82e8a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c812a540) at sleepq_switch+0xc1 sleepq_wait_sig(c812a540) at sleepq_wait_sig+0x1d msleep(c812a540,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c812a540,16,ffffffff,ffffffff,ea960cb0,...) at lf_setlock+0x181 lf_advlock(ea960c90,c84cfd9c,0,0,ea960c7c,...) at lf_advlock+0x203 ufs_advlock(ea960c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea960c90) at VOP_ADVLOCK_APV+0xa9 flock(c82e8a80,ea960d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 822 tid 100105 td 0xc8087900 sched_switch(c8087900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c812bd40) at sleepq_switch+0xc1 sleepq_wait_sig(c812bd40) at sleepq_wait_sig+0x1d msleep(c812bd40,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c812bd40,16,ffffffff,ffffffff,e864dcb0,...) at lf_setlock+0x181 lf_advlock(e864dc90,c84cfd9c,0,0,e864dc7c,...) at lf_advlock+0x203 ufs_advlock(e864dc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,e864dc90) at VOP_ADVLOCK_APV+0xa9 flock(c8087900,e864dd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 821 tid 100039 td 0xc7e05900 sched_switch(c7e05900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b09c0) at sleepq_switch+0xc1 sleepq_wait_sig(c80b09c0) at sleepq_wait_sig+0x1d msleep(c80b09c0,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b09c0,16,ffffffff,ffffffff,e838fcb0,...) at lf_setlock+0x181 lf_advlock(e838fc90,c84cfd9c,0,0,e838fc7c,...) at lf_advlock+0x203 ufs_advlock(e838fc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,e838fc90) at VOP_ADVLOCK_APV+0xa9 flock(c7e05900,e838fd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 820 tid 100089 td 0xc818a480 sched_switch(c818a480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80af540) at sleepq_switch+0xc1 sleepq_wait_sig(c80af540) at sleepq_wait_sig+0x1d msleep(c80af540,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80af540,16,ffffffff,ffffffff,ea8c2cb0,...) at lf_setlock+0x181 lf_advlock(ea8c2c90,c84cfd9c,0,0,ea8c2c7c,...) at lf_advlock+0x203 ufs_advlock(ea8c2c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8c2c90) at VOP_ADVLOCK_APV+0xa9 flock(c818a480,ea8c2d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 819 tid 100092 td 0xc818a000 sched_switch(c818a000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8067480) at sleepq_switch+0xc1 sleepq_wait_sig(c8067480) at sleepq_wait_sig+0x1d msleep(c8067480,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c8067480,16,ffffffff,ffffffff,ea8b9cb0,...) at lf_setlock+0x181 lf_advlock(ea8b9c90,c84cfd9c,0,0,ea8b9c7c,...) at lf_advlock+0x203 ufs_advlock(ea8b9c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8b9c90) at VOP_ADVLOCK_APV+0xa9 flock(c818a000,ea8b9d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 818 tid 100106 td 0xc8087780 sched_switch(c8087780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c815b4c0) at sleepq_switch+0xc1 sleepq_wait_sig(c815b4c0) at sleepq_wait_sig+0x1d msleep(c815b4c0,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c815b4c0,16,ffffffff,ffffffff,e864acb0,...) at lf_setlock+0x181 lf_advlock(e864ac90,c84cfd9c,0,0,e864ac7c,...) at lf_advlock+0x203 ufs_advlock(e864ac90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,e864ac90) at VOP_ADVLOCK_APV+0xa9 flock(c8087780,e864ad04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 817 tid 100079 td 0xc818e480 sched_switch(c818e480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8067240) at sleepq_switch+0xc1 sleepq_wait_sig(c8067240) at sleepq_wait_sig+0x1d msleep(c8067240,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c8067240,16,ffffffff,ffffffff,ea8e0cb0,...) at lf_setlock+0x181 lf_advlock(ea8e0c90,c84cfd9c,0,0,ea8e0c7c,...) at lf_advlock+0x203 ufs_advlock(ea8e0c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8e0c90) at VOP_ADVLOCK_APV+0xa9 flock(c818e480,ea8e0d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 816 tid 100041 td 0xc7e05600 sched_switch(c7e05600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b0a00) at sleepq_switch+0xc1 sleepq_wait_sig(c80b0a00) at sleepq_wait_sig+0x1d msleep(c80b0a00,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b0a00,16,ffffffff,ffffffff,e8389cb0,...) at lf_setlock+0x181 lf_advlock(e8389c90,c84cfd9c,0,0,e8389c7c,...) at lf_advlock+0x203 ufs_advlock(e8389c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,e8389c90) at VOP_ADVLOCK_APV+0xa9 flock(c7e05600,e8389d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 815 tid 100075 td 0xc8105780 sched_switch(c8105780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c815b580) at sleepq_switch+0xc1 sleepq_wait_sig(c815b580) at sleepq_wait_sig+0x1d msleep(c815b580,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c815b580,16,ffffffff,ffffffff,ea872cb0,...) at lf_setlock+0x181 lf_advlock(ea872c90,c84cfd9c,0,0,ea872c7c,...) at lf_advlock+0x203 ufs_advlock(ea872c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea872c90) at VOP_ADVLOCK_APV+0xa9 flock(c8105780,ea872d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 814 tid 100078 td 0xc818e600 sched_switch(c818e600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8067180) at sleepq_switch+0xc1 sleepq_wait_sig(c8067180) at sleepq_wait_sig+0x1d msleep(c8067180,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c8067180,16,ffffffff,ffffffff,ea8e3cb0,...) at lf_setlock+0x181 lf_advlock(ea8e3c90,c84cfd9c,0,0,ea8e3c7c,...) at lf_advlock+0x203 ufs_advlock(ea8e3c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8e3c90) at VOP_ADVLOCK_APV+0xa9 flock(c818e600,ea8e3d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 813 tid 100095 td 0xc8195600 sched_switch(c8195600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b0a40) at sleepq_switch+0xc1 sleepq_wait_sig(c80b0a40) at sleepq_wait_sig+0x1d msleep(c80b0a40,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b0a40,16,ffffffff,ffffffff,ea91fcb0,...) at lf_setlock+0x181 lf_advlock(ea91fc90,c84cfd9c,0,0,ea91fc7c,...) at lf_advlock+0x203 ufs_advlock(ea91fc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea91fc90) at VOP_ADVLOCK_APV+0xa9 flock(c8195600,ea91fd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 812 tid 100120 td 0xc8195a80 sched_switch(c8195a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c815b6c0) at sleepq_switch+0xc1 sleepq_wait_sig(c815b6c0) at sleepq_wait_sig+0x1d msleep(c815b6c0,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c815b6c0,16,ffffffff,ffffffff,ea928cb0,...) at lf_setlock+0x181 lf_advlock(ea928c90,c84cfd9c,0,0,ea928c7c,...) at lf_advlock+0x203 ufs_advlock(ea928c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea928c90) at VOP_ADVLOCK_APV+0xa9 flock(c8195a80,ea928d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 811 tid 100051 td 0xc8087480 sched_switch(c8087480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8066e00) at sleepq_switch+0xc1 sleepq_wait_sig(c8066e00) at sleepq_wait_sig+0x1d msleep(c8066e00,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c8066e00,16,ffffffff,ffffffff,e8644cb0,...) at lf_setlock+0x181 lf_advlock(e8644c90,c84cfd9c,0,0,e8644c7c,...) at lf_advlock+0x203 ufs_advlock(e8644c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,e8644c90) at VOP_ADVLOCK_APV+0xa9 flock(c8087480,e8644d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 810 tid 100117 td 0xc82e8000 sched_switch(c82e8000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b0a80) at sleepq_switch+0xc1 sleepq_wait_sig(c80b0a80) at sleepq_wait_sig+0x1d msleep(c80b0a80,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b0a80,16,ffffffff,ffffffff,ea94bcb0,...) at lf_setlock+0x181 lf_advlock(ea94bc90,c84cfd9c,0,0,ea94bc7c,...) at lf_advlock+0x203 ufs_advlock(ea94bc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea94bc90) at VOP_ADVLOCK_APV+0xa9 flock(c82e8000,ea94bd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 809 tid 100052 td 0xc8087300 sched_switch(c8087300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c815b8c0) at sleepq_switch+0xc1 sleepq_wait_sig(c815b8c0) at sleepq_wait_sig+0x1d msleep(c815b8c0,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c815b8c0,16,ffffffff,ffffffff,e8641cb0,...) at lf_setlock+0x181 lf_advlock(e8641c90,c84cfd9c,0,0,e8641c7c,...) at lf_advlock+0x203 ufs_advlock(e8641c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,e8641c90) at VOP_ADVLOCK_APV+0xa9 flock(c8087300,e8641d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 808 tid 100038 td 0xc7e05a80 sched_switch(c7e05a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8066c80) at sleepq_switch+0xc1 sleepq_wait_sig(c8066c80) at sleepq_wait_sig+0x1d msleep(c8066c80,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c8066c80,16,ffffffff,ffffffff,e8392cb0,...) at lf_setlock+0x181 lf_advlock(e8392c90,c84cfd9c,0,0,e8392c7c,...) at lf_advlock+0x203 ufs_advlock(e8392c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,e8392c90) at VOP_ADVLOCK_APV+0xa9 flock(c7e05a80,e8392d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 807 tid 100070 td 0xc815e000 sched_switch(c815e000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b0ac0) at sleepq_switch+0xc1 sleepq_wait_sig(c80b0ac0) at sleepq_wait_sig+0x1d msleep(c80b0ac0,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b0ac0,16,ffffffff,ffffffff,ea88ecb0,...) at lf_setlock+0x181 lf_advlock(ea88ec90,c84cfd9c,0,0,ea88ec7c,...) at lf_advlock+0x203 ufs_advlock(ea88ec90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea88ec90) at VOP_ADVLOCK_APV+0xa9 flock(c815e000,ea88ed04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 806 tid 100077 td 0xc818e780 sched_switch(c818e780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c815a440) at sleepq_switch+0xc1 sleepq_wait_sig(c815a440) at sleepq_wait_sig+0x1d msleep(c815a440,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c815a440,16,ffffffff,ffffffff,ea8e6cb0,...) at lf_setlock+0x181 lf_advlock(ea8e6c90,c84cfd9c,0,0,ea8e6c7c,...) at lf_advlock+0x203 ufs_advlock(ea8e6c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8e6c90) at VOP_ADVLOCK_APV+0xa9 flock(c818e780,ea8e6d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 805 tid 100083 td 0xc818ad80 sched_switch(c818ad80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c815a000) at sleepq_switch+0xc1 sleepq_wait_sig(c815a000) at sleepq_wait_sig+0x1d msleep(c815a000,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c815a000,16,ffffffff,ffffffff,ea8d4cb0,...) at lf_setlock+0x181 lf_advlock(ea8d4c90,c84cfd9c,0,0,ea8d4c7c,...) at lf_advlock+0x203 ufs_advlock(ea8d4c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8d4c90) at VOP_ADVLOCK_APV+0xa9 flock(c818ad80,ea8d4d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 804 tid 100067 td 0xc815e480 sched_switch(c815e480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80b0b00) at sleepq_switch+0xc1 sleepq_wait_sig(c80b0b00) at sleepq_wait_sig+0x1d msleep(c80b0b00,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c80b0b00,16,ffffffff,ffffffff,ea897cb0,...) at lf_setlock+0x181 lf_advlock(ea897c90,c84cfd9c,0,0,ea897c7c,...) at lf_advlock+0x203 ufs_advlock(ea897c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea897c90) at VOP_ADVLOCK_APV+0xa9 flock(c815e480,ea897d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 803 tid 100112 td 0xc82e8780 sched_switch(c82e8780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8066c40) at sleepq_switch+0xc1 sleepq_wait_sig(c8066c40) at sleepq_wait_sig+0x1d msleep(c8066c40,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c8066c40,16,ffffffff,ffffffff,ea95acb0,...) at lf_setlock+0x181 lf_advlock(ea95ac90,c84cfd9c,0,0,ea95ac7c,...) at lf_advlock+0x203 ufs_advlock(ea95ac90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea95ac90) at VOP_ADVLOCK_APV+0xa9 flock(c82e8780,ea95ad04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 802 tid 100088 td 0xc818a600 sched_switch(c818a600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c815bbc0) at sleepq_switch+0xc1 sleepq_wait_sig(c815bbc0) at sleepq_wait_sig+0x1d msleep(c815bbc0,0,168,c0765c84,0) at msleep+0x26e lf_setlock(c815bbc0,16,ffffffff,ffffffff,ea8c5cb0,...) at lf_setlock+0x181 lf_advlock(ea8c5c90,c84cfd9c,0,0,ea8c5c7c,...) at lf_advlock+0x203 ufs_advlock(ea8c5c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c077a640,ea8c5c90) at VOP_ADVLOCK_APV+0xa9 flock(c818a600,ea8c5d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 801 tid 100082 td 0xc818e000 sched_switch(c818e000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,3b7,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b poll(c818e000,ea8d7d04) at poll+0x3b9 syscall(3b,3b,3b,2,28,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (209, FreeBSD ELF32, poll), eip = 0x282c3a87, esp = 0xbfbfebcc, ebp = 0xbfbfec18 --- Tracing command getty pid 800 tid 100093 td 0xc815ed80 sched_switch(c815ed80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e95410) at sleepq_switch+0xc1 sleepq_wait_sig(c7e95410) at sleepq_wait_sig+0x1d msleep(c7e95410,0,159,c072a5af,0) at msleep+0x26e ttysleep(c7e95400,c7e95410,159,c072a5af,0,...) at ttysleep+0x21 ttread(c7e95400,ea8a9cbc,0) at ttread+0x48f ttyread(c7e8b800,ea8a9cbc,0,c0797ae0,0,...) at ttyread+0x2f giant_read(c7e8b800,ea8a9cbc,0) at giant_read+0x2d devfs_read_f(c83d6048,ea8a9cbc,c7ce0780,0,c815ed80) at devfs_read_f+0x62 dofileread(c815ed80,0,c83d6048,ea8a9cbc,ffffffff,...) at dofileread+0x85 kern_readv(c815ed80,0,ea8a9cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c815ed80,ea8a9d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 799 tid 100123 td 0xc8366900 sched_switch(c8366900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea7c10) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea7c10) at sleepq_wait_sig+0x1d msleep(c7ea7c10,0,159,c072a5af,0) at msleep+0x26e ttysleep(c7ea7c00,c7ea7c10,159,c072a5af,0,...) at ttysleep+0x21 ttread(c7ea7c00,ea9c0cbc,0) at ttread+0x48f ttyread(c7e8c800,ea9c0cbc,0,ea9c0c14,c054002d,...) at ttyread+0x2f scread(c7e8c800,ea9c0cbc,0,c0797ae0,0,...) at scread+0x22 giant_read(c7e8c800,ea9c0cbc,0) at giant_read+0x2d devfs_read_f(c811c2d0,ea9c0cbc,c7ce0780,0,c8366900) at devfs_read_f+0x62 dofileread(c8366900,0,c811c2d0,ea9c0cbc,ffffffff,...) at dofileread+0x85 kern_readv(c8366900,0,ea9c0cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c8366900,ea9c0d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 798 tid 100103 td 0xc8087c00 sched_switch(c8087c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea6c10) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea6c10) at sleepq_wait_sig+0x1d msleep(c7ea6c10,0,159,c072a5af,0) at msleep+0x26e ttysleep(c7ea6c00,c7ea6c10,159,c072a5af,0,...) at ttysleep+0x21 ttread(c7ea6c00,e8653cbc,0) at ttread+0x48f ttyread(c7e8c700,e8653cbc,0,e8653c14,c054002d,...) at ttyread+0x2f scread(c7e8c700,e8653cbc,0,c0797ae0,0,...) at scread+0x22 giant_read(c7e8c700,e8653cbc,0) at giant_read+0x2d devfs_read_f(c8111e10,e8653cbc,c7ce0780,0,c8087c00) at devfs_read_f+0x62 dofileread(c8087c00,0,c8111e10,e8653cbc,ffffffff,...) at dofileread+0x85 kern_readv(c8087c00,0,e8653cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c8087c00,e8653d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 797 tid 100113 td 0xc82e8600 sched_switch(c82e8600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea5810) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea5810) at sleepq_wait_sig+0x1d msleep(c7ea5810,0,159,c072a5af,0) at msleep+0x26e ttysleep(c7ea5800,c7ea5810,159,c072a5af,0,...) at ttysleep+0x21 ttread(c7ea5800,ea957cbc,0) at ttread+0x48f ttyread(c7e8c600,ea957cbc,0,ea957c14,c054002d,...) at ttyread+0x2f scread(c7e8c600,ea957cbc,0,c0797ae0,0,...) at scread+0x22 giant_read(c7e8c600,ea957cbc,0) at giant_read+0x2d devfs_read_f(c811c558,ea957cbc,c7ce0780,0,c82e8600) at devfs_read_f+0x62 dofileread(c82e8600,0,c811c558,ea957cbc,ffffffff,...) at dofileread+0x85 kern_readv(c82e8600,0,ea957cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c82e8600,ea957d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 796 tid 100081 td 0xc818e180 sched_switch(c818e180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea5c10) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea5c10) at sleepq_wait_sig+0x1d msleep(c7ea5c10,0,159,c072a5af,0) at msleep+0x26e ttysleep(c7ea5c00,c7ea5c10,159,c072a5af,0,...) at ttysleep+0x21 ttread(c7ea5c00,ea8dacbc,0) at ttread+0x48f ttyread(c7e8c500,ea8dacbc,0,ea8dac14,c054002d,...) at ttyread+0x2f scread(c7e8c500,ea8dacbc,0,c0797ae0,0,...) at scread+0x22 giant_read(c7e8c500,ea8dacbc,0) at giant_read+0x2d devfs_read_f(c8108ca8,ea8dacbc,c7ce0780,0,c818e180) at devfs_read_f+0x62 dofileread(c818e180,0,c8108ca8,ea8dacbc,ffffffff,...) at dofileread+0x85 kern_readv(c818e180,0,ea8dacbc,bfbfee1f,1,...) at kern_readv+0x36 read(c818e180,ea8dad04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 795 tid 100116 td 0xc82e8180 sched_switch(c82e8180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea8410) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea8410) at sleepq_wait_sig+0x1d msleep(c7ea8410,0,159,c072a5af,0) at msleep+0x26e ttysleep(c7ea8400,c7ea8410,159,c072a5af,0,...) at ttysleep+0x21 ttread(c7ea8400,ea94ecbc,0) at ttread+0x48f ttyread(c7e8c400,ea94ecbc,0,ea94ec14,c054002d,...) at ttyread+0x2f scread(c7e8c400,ea94ecbc,0,c0797ae0,0,...) at scread+0x22 giant_read(c7e8c400,ea94ecbc,0) at giant_read+0x2d devfs_read_f(c811c480,ea94ecbc,c7ce0780,0,c82e8180) at devfs_read_f+0x62 dofileread(c82e8180,0,c811c480,ea94ecbc,ffffffff,...) at dofileread+0x85 kern_readv(c82e8180,0,ea94ecbc,bfbfee1f,1,...) at kern_readv+0x36 read(c82e8180,ea94ed04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 794 tid 100072 td 0xc8105c00 sched_switch(c8105c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e9f410) at sleepq_switch+0xc1 sleepq_wait_sig(c7e9f410) at sleepq_wait_sig+0x1d msleep(c7e9f410,0,159,c072a5af,0) at msleep+0x26e ttysleep(c7e9f400,c7e9f410,159,c072a5af,0,...) at ttysleep+0x21 ttread(c7e9f400,ea87bcbc,0) at ttread+0x48f ttyread(c7e8c300,ea87bcbc,0,ea87bc14,c054002d,...) at ttyread+0x2f scread(c7e8c300,ea87bcbc,0,c0797ae0,0,...) at scread+0x22 giant_read(c7e8c300,ea87bcbc,0) at giant_read+0x2d devfs_read_f(c8111c60,ea87bcbc,c7ce0780,0,c8105c00) at devfs_read_f+0x62 dofileread(c8105c00,0,c8111c60,ea87bcbc,ffffffff,...) at dofileread+0x85 kern_readv(c8105c00,0,ea87bcbc,bfbfee1f,1,...) at kern_readv+0x36 read(c8105c00,ea87bd04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 793 tid 100122 td 0xc8195780 sched_switch(c8195780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea7010) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea7010) at sleepq_wait_sig+0x1d msleep(c7ea7010,0,159,c072a5af,0) at msleep+0x26e ttysleep(c7ea7000,c7ea7010,159,c072a5af,0,...) at ttysleep+0x21 ttread(c7ea7000,ea922cbc,0) at ttread+0x48f ttyread(c7e8c200,ea922cbc,0,ea922c14,c054002d,...) at ttyread+0x2f scread(c7e8c200,ea922cbc,0,c0797ae0,0,...) at scread+0x22 giant_read(c7e8c200,ea922cbc,0) at giant_read+0x2d devfs_read_f(c811cd80,ea922cbc,c7ce0780,0,c8195780) at devfs_read_f+0x62 dofileread(c8195780,0,c811cd80,ea922cbc,ffffffff,...) at dofileread+0x85 kern_readv(c8195780,0,ea922cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c8195780,ea922d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 792 tid 100066 td 0xc815e600 sched_switch(c815e600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e9bc10) at sleepq_switch+0xc1 sleepq_wait_sig(c7e9bc10) at sleepq_wait_sig+0x1d msleep(c7e9bc10,0,159,c072a5af,0) at msleep+0x26e ttysleep(c7e9bc00,c7e9bc10,159,c072a5af,0,...) at ttysleep+0x21 ttread(c7e9bc00,ea89acbc,0) at ttread+0x48f ttyread(c7e8c100,ea89acbc,0,ea89ac14,c054002d,...) at ttyread+0x2f scread(c7e8c100,ea89acbc,0,c0797ae0,0,...) at scread+0x22 giant_read(c7e8c100,ea89acbc,0) at giant_read+0x2d devfs_read_f(c8108240,ea89acbc,c7ce0780,0,c815e600) at devfs_read_f+0x62 dofileread(c815e600,0,c8108240,ea89acbc,ffffffff,...) at dofileread+0x85 kern_readv(c815e600,0,ea89acbc,bfbfee1f,1,...) at kern_readv+0x36 read(c815e600,ea89ad04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command inetd pid 778 tid 100057 td 0xc8102d80 sched_switch(c8102d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b kern_select(c8102d80,7,bfbfe8a0,0,0,...) at kern_select+0x4bf select(c8102d80,ea860d04) at select+0x44 syscall(3b,3b,3b,bfbfe8a0,bfbfee30,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2815a8d3, esp = 0xbfbfdb7c, ebp = 0xbfbfee68 --- Tracing command httpd pid 758 tid 100054 td 0xc8087000 sched_switch(c8087000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07e5744,c07e5744,3e9,e863bb40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07e5744,c07e5720,3e9) at cv_timedwait_sig+0x194 kern_select(c8087000,0,0,0,0,...) at kern_select+0x4a9 select(c8087000,e863bd04) at select+0x44 syscall(3b,3b,3b,0,f4240,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2830c8d3, esp = 0xbfbfec3c, ebp = 0xbfbfec78 --- Tracing command mysqld pid 757 tid 100128 td 0xc82e9480 sched_switch(c82e9480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e066f4) at sleepq_switch+0xc1 sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e msleep(c7e066f4,c8364cf8,168,c0721153,290,...) at msleep+0x244 kse_release(c82e9480,ea972d04) at kse_release+0x1df syscall(3b,3b,3b,8530900,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c76f28, ebp = 0x13c76f64 --- Tracing command mysqld pid 757 tid 100129 td 0xc818ed80 sched_switch(c818ed80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e066f4) at sleepq_switch+0xc1 sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e msleep(c7e066f4,c8364cf8,168,c0721153,291,...) at msleep+0x244 kse_release(c818ed80,ea8f2d04) at kse_release+0x1df syscall(3b,3b,3b,8530a00,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7af28, ebp = 0x13c7af64 --- Tracing command mysqld pid 757 tid 100132 td 0xc82e9a80 sched_switch(c82e9a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e066f4) at sleepq_switch+0xc1 sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e msleep(c7e066f4,c8364cf8,168,c0721153,290,...) at msleep+0x244 kse_release(c82e9a80,ea97ed04) at kse_release+0x1df syscall(3b,3b,3b,8530b00,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7ef28, ebp = 0x13c7ef64 --- Tracing command mysqld pid 757 tid 100131 td 0xc8193a80 sched_switch(c8193a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e066f4) at sleepq_switch+0xc1 sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e msleep(c7e066f4,c8364cf8,168,c0721153,290,...) at msleep+0x244 kse_release(c8193a80,ea90ad04) at kse_release+0x1df syscall(3b,3b,3b,8530000,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x8534f28, ebp = 0x8534f64 --- Tracing command mysqld pid 757 tid 100127 td 0xc83c5180 sched_switch(c83c5180,c82e9480,1) at sched_switch+0x177 mi_switch(1,c82e9480) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b kern_select(c83c5180,f,bfbfebf0,0,0,...) at kern_select+0x4bf select(c83c5180,ea9edd04) at select+0x44 syscall(3b,3b,3b,8536000,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2861b8d3, esp = 0xbfbfeacc, ebp = 0xbfbfeb08 --- Tracing command mysqld pid 757 tid 100134 td 0xc83c5780 sched_switch(c83c5780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(ea9f9c14) at sleepq_switch+0xc1 sleepq_wait_sig(ea9f9c14) at sleepq_wait_sig+0x1d msleep(ea9f9c14,c8364cf8,168,c072114b,0,...) at msleep+0x26e kern_sigtimedwait(c83c5780,26005,0,0,0,ea9f9c80,0) at kern_sigtimedwait+0x30b sigtimedwait(c83c5780,ea9f9d04) at sigtimedwait+0x6b syscall(3b,3b,3b,13c83e00,bf0f5f37,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (345, FreeBSD ELF32, sigtimedwait), eip = 0x284735b7, esp = 0xbf0f5e2c, ebp = 0xbf0f5ee8 --- Tracing command mysqld pid 757 tid 100126 td 0xc8105480 sched_switch(c8105480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8364dc8) at sleepq_switch+0xc1 sleepq_timedwait_sig(c8364dc8) at sleepq_timedwait_sig+0x1e msleep(c8364dc8,c8364cf8,168,c0721148,7531,...) at msleep+0x244 kse_release(c8105480,ea86cd04) at kse_release+0x121 syscall(3b,3b,3b,8530800,81,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0xbf9fef2c, ebp = 0xbf9fef88 --- Tracing command sh pid 727 tid 100111 td 0xc82e8900 sched_switch(c82e8900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c82e7c90) at sleepq_switch+0xc1 sleepq_wait_sig(c82e7c90) at sleepq_wait_sig+0x1d msleep(c82e7c90,c82e7cf8,15c,c0729071,0) at msleep+0x26e kern_wait(c82e8900,ffffffff,ea95dc84,2,0) at kern_wait+0x6bb wait4(c82e8900,ea95dd04) at wait4+0x2a syscall(3b,3b,3b,bfbfe678,2d,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2813d51f, esp = 0xbfbfe62c, ebp = 0xbfbfe648 --- Tracing command cron pid 724 tid 100108 td 0xc82e8d80 sched_switch(c82e8d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07983cc) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07983cc) at sleepq_timedwait_sig+0x1e msleep(c07983cc,0,15c,c0724ee0,ea61,...) at msleep+0x244 kern_nanosleep(c82e8d80,ea966ccc,ea966cc4) at kern_nanosleep+0xab nanosleep(c82e8d80,ea966d04) at nanosleep+0x4f syscall(3b,3b,3b,8050e6c,3c,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 --- Tracing command snmpd pid 716 tid 100101 td 0xc8102000 sched_switch(c8102000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07e5744,c07e5744,1389,ea845b40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07e5744,c07e5720,1389) at cv_timedwait_sig+0x194 kern_select(c8102000,9,bfbfed50,bfbfecd0,bfbfec50,...) at kern_select+0x4a9 select(c8102000,ea845d04) at select+0x44 syscall(3b,3b,3b,bfbfecd0,bfbfec38,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x284598d3, esp = 0xbfbfea4c, ebp = 0xbfbfee68 --- Tracing command cron pid 705 tid 100119 td 0xc8195c00 sched_switch(c8195c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07983cc) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07983cc) at sleepq_timedwait_sig+0x1e msleep(c07983cc,0,15c,c0724ee0,ea61,...) at msleep+0x244 kern_nanosleep(c8195c00,ea92bccc,ea92bcc4) at kern_nanosleep+0xab nanosleep(c8195c00,ea92bd04) at nanosleep+0x4f syscall(3b,3b,3b,805008c,3c,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 --- Tracing command sendmail pid 699 tid 100121 td 0xc8195900 sched_switch(c8195900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8365464) at sleepq_switch+0xc1 sleepq_wait_sig(c8365464) at sleepq_wait_sig+0x1d msleep(c8365464,c8365498,168,c0724047,0) at msleep+0x26e kern_sigsuspend(c8195900,0,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c8195900,ea925d04) at sigsuspend+0x33 syscall(3b,3b,3b,0,bfbfd000,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2827e3fb, esp = 0xbfbfcfec, ebp = 0xbfbfd018 --- Tracing command sendmail pid 693 tid 100056 td 0xc8105000 sched_switch(c8105000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07e5744,c07e5744,1389,ea863b40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07e5744,c07e5720,1389) at cv_timedwait_sig+0x194 kern_select(c8105000,6,bfbfc530,0,0,...) at kern_select+0x4a9 select(c8105000,ea863d04) at select+0x44 syscall(3b,3b,3b,2,5,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x282fc8d3, esp = 0xbfbfc44c, ebp = 0xbfbfd038 --- Tracing command sshd pid 688 tid 100059 td 0xc8102a80 sched_switch(c8102a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b kern_select(c8102a80,7,807b920,0,0,...) at kern_select+0x4bf select(c8102a80,ea85ad04) at select+0x44 syscall(3b,3b,3b,1,a,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe16c, ebp = 0xbfbfee78 --- Tracing command ntpd pid 663 tid 100069 td 0xc815e180 sched_switch(c815e180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b kern_select(c815e180,a,bfbfed60,0,0,...) at kern_select+0x4bf select(c815e180,ea891d04) at select+0x44 syscall(3b,3b,3b,bfbfed60,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2828a8d3, esp = 0xbfbfed1c, ebp = 0xbfbfedf8 --- Tracing command usbd pid 632 tid 100085 td 0xc818aa80 sched_switch(c818aa80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07e5744,c07e5744,7531,ea8ceb40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07e5744,c07e5720,7531) at cv_timedwait_sig+0x194 kern_select(c818aa80,6,bfbfeda0,bfbfed20,0,...) at kern_select+0x4a9 select(c818aa80,ea8ced04) at select+0x44 syscall(3b,3b,3b,bfbfed20,1e,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2813c8d3, esp = 0xbfbfeccc, ebp = 0xbfbfee78 --- Tracing command accounting pid 585 tid 100053 td 0xc8087180 sched_switch(c8087180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c0795f00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c0795f00) at sleepq_wait+0x46 _sx_xlock_hard(c0795f00,c8087180,0,c071f3eb,215) at _sx_xlock_hard+0x195 _sx_xlock(c0795f00,0,c071f3eb,215) at _sx_xlock+0x85 acct_thread(0,e863ed38,0,c053d6c4,0,...) at acct_thread+0xee fork_exit(c053d6c4,0,e863ed38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe863ed6c, ebp = 0 --- Tracing command named pid 568 tid 100086 td 0xc818a900 sched_switch(c818a900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07e5744,c07e5744,7531,ea8cbb40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07e5744,c07e5720,7531) at cv_timedwait_sig+0x194 kern_select(c818a900,1a,bfbfed60,bfbfece0,0,...) at kern_select+0x4a9 select(c818a900,ea8cbd04) at select+0x44 syscall(3b,3b,3b,0,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2835e8d3, esp = 0xbfbfec9c, ebp = 0xbfbfedf8 --- Tracing command syslogd pid 548 tid 100104 td 0xc8087a80 sched_switch(c8087a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(dbfc2388,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(dbfc2388,0,c8302ba8,b7,c072e794,...) at sleepq_wait+0x46 msleep(dbfc2388,c0797a48,50,c072c4b4,0,...) at msleep+0x27d acquire(e865090c,120,60000,c8087a80,0,...) at acquire+0x76 lockmgr(dbfc2388,202122,c8302ba8,c8087a80) at lockmgr+0x44e getblk(c8302ae0,4,0,1800,0,...) at getblk+0x12f breadn(c8302ae0,4,0,1800,0,...) at breadn+0x31 bread(c8302ae0,4,0,1800,0,e8650a2c) at bread+0x20 ffs_balloc_ufs2(c8302ae0,1173b,0,7d,c7ce0780,...) at ffs_balloc_ufs2+0xcd3 ffs_write(e8650c10) at ffs_write+0x2c0 VOP_WRITE_APV(c077a640,e8650c10) at VOP_WRITE_APV+0x132 vn_write(c8111ab0,c8593980,c7ce0780,0,c8087a80) at vn_write+0x1ea dofilewrite(c8087a80,c,c8111ab0,c8593980,ffffffff,...) at dofilewrite+0x7c kern_writev(c8087a80,c,c8593980,c8593980,0,...) at kern_writev+0x3b writev(c8087a80,e8650d04) at writev+0x30 syscall(3b,3b,3b,0,bfbfd050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (121, FreeBSD ELF32, writev), eip = 0x28137f1f, esp = 0xbfbfc96c, ebp = 0xbfbfd078 --- Tracing command ipmon pid 531 tid 100050 td 0xc8087600 sched_switch(c8087600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07983cc) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07983cc) at sleepq_timedwait_sig+0x1e msleep(c07983cc,0,15c,c0724ee0,3e9,...) at msleep+0x244 kern_nanosleep(c8087600,e8647ccc,e8647cc4) at kern_nanosleep+0xab nanosleep(c8087600,e8647d04) at nanosleep+0x4f syscall(3b,3b,3b,2,1,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2812f513, esp = 0xbfbfcd3c, ebp = 0xbfbfcd68 --- Tracing command devd pid 435 tid 100107 td 0xc82e9000 sched_switch(c82e9000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5744) at sleepq_switch+0xc1 sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b kern_select(c82e9000,5,bfbfe9d0,0,0,...) at kern_select+0x4bf select(c82e9000,ea969d04) at select+0x44 syscall(3b,3b,3b,0,8,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x8075917, esp = 0xbfbfe96c, ebp = 0xbfbfee78 --- Tracing command adjkerntz pid 164 tid 100060 td 0xc8102900 sched_switch(c8102900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8101894) at sleepq_switch+0xc1 sleepq_wait_sig(c8101894) at sleepq_wait_sig+0x1d msleep(c8101894,c81018c8,168,c0724047,0) at msleep+0x26e kern_sigsuspend(c8102900,0,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c8102900,ea857d04) at sigsuspend+0x33 syscall(3b,3b,3b,0,ffffc7c0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x280bc3fb, esp = 0xbfbfed9c, ebp = 0xbfbfee78 --- Tracing command schedcpu pid 44 tid 100044 td 0xc7e05180 sched_switch(c7e05180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(e8380d04,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(e8380d04) at sleepq_timedwait+0x4a msleep(e8380d04,0,0,c071cc2c,3e8) at msleep+0x259 schedcpu_thread(0,e8380d38,0,c05768b4,0,...) at schedcpu_thread+0x26 fork_exit(c05768b4,0,e8380d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8380d6c, ebp = 0 --- Tracing command softdepflush pid 43 tid 100045 td 0xc7e05000 sched_switch(c7e05000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07f3254,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c07f3254) at sleepq_timedwait+0x4a msleep(c07f3254,c07f3220,44,c0738a08,3e8) at msleep+0x259 softdep_flush(0,e837dd38,0,c06669b0,0,...) at softdep_flush+0x227 fork_exit(c06669b0,0,e837dd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe837dd6c, ebp = 0 --- Tracing command vnlru pid 42 tid 100046 td 0xc7dbdd80 sched_switch(c7dbdd80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8084860,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c8084860) at sleepq_timedwait+0x4a msleep(c8084860,c07e60e0,250,c072dc00,3e8,c07e6160) at msleep+0x259 vnlru_proc(0,e836dd38,0,c05bb048,0,...) at vnlru_proc+0xdf fork_exit(c05bb048,0,e836dd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe836dd6c, ebp = 0 --- Tracing command syncer pid 41 tid 100047 td 0xc7dbdc00 sched_switch(c7dbdc00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(dbf1b09c,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(dbf1b09c,0,c7eb0ba8,b7,c072db32,...) at sleepq_wait+0x46 msleep(dbf1b09c,c07975ec,50,c072c4b4,0,...) at msleep+0x27d acquire(e836aacc,120,60000,c7dbdc00,0,...) at acquire+0x76 lockmgr(dbf1b09c,202122,c7eb0ba8,c7dbdc00) at lockmgr+0x44e getblk(c7eb0ae0,113b60,0,4000,0,...) at getblk+0x12f breadn(c7eb0ae0,113b60,0,4000,0,...) at breadn+0x31 bread(c7eb0ae0,113b60,0,4000,0,e836abb8) at bread+0x20 ffs_update(c7eb1ae0,0,7,0,0,...) at ffs_update+0x22c ffs_syncvnode(c7eb1ae0,3) at ffs_syncvnode+0x3be ffs_sync(c80cbb90,3,c7dbdc00,c80cbb90,2,...) at ffs_sync+0x209 sync_fsync(e836acbc) at sync_fsync+0x164 VOP_FSYNC_APV(c0770380,e836acbc) at VOP_FSYNC_APV+0x9b sync_vnode(c817a67c,c7dbdc00) at sync_vnode+0x106 sched_sync(0,e836ad38,0,c05bcb5c,0,...) at sched_sync+0x1ed fork_exit(c05bcb5c,0,e836ad38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe836ad6c, ebp = 0 --- Tracing command bufdaemon pid 40 tid 100048 td 0xc7dbda80 sched_switch(c7dbda80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5cc0,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c07e5cc0) at sleepq_timedwait+0x4a msleep(c07e5cc0,c07e5ce0,44,c072c3ea,3e8) at msleep+0x259 buf_daemon(0,e8367d38,0,c05ae534,0,...) at buf_daemon+0x108 fork_exit(c05ae534,0,e8367d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8367d6c, ebp = 0 --- Tracing command pagezero pid 39 tid 100049 td 0xc7dbd900 sched_switch(c7dbd900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07f4224,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c07f4224) at sleepq_timedwait+0x4a msleep(c07f4224,c07f3ca0,200,c073dabc,493e0,c07f3ca0,0,c073da9e,9d) at msleep+0x259 vm_pagezero(0,e8364d38,0,c0697cbc,0,...) at vm_pagezero+0x6c fork_exit(c0697cbc,0,e8364d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8364d6c, ebp = 0 --- Tracing command vmdaemon pid 38 tid 100027 td 0xc7dbd180 sched_switch(c7dbd180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07f3d2c,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c07f3d2c,0,c073d73f,63e,0,...) at sleepq_wait+0x46 msleep(c07f3d2c,c07f3d40,68,c072c3ea,0,...) at msleep+0x27d vm_daemon(0,e8355d38,0,c0696cd0,0,...) at vm_daemon+0x36 fork_exit(c0696cd0,0,e8355d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8355d6c, ebp = 0 --- Tracing command pagedaemon pid 37 tid 100028 td 0xc7dbd000 sched_switch(c7dbd000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07f3ce8,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c07f3ce8) at sleepq_timedwait+0x4a msleep(c07f3ce8,c07f3ca0,44,c072c3ea,1388) at msleep+0x259 vm_pageout(0,e8352d38,0,c0696958,0,...) at vm_pageout+0x260 fork_exit(c0696958,0,e8352d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8352d6c, ebp = 0 --- Tracing command irq7: ppc0 pid 36 tid 100029 td 0xc7ce9d80 fork_trampoline() at fork_trampoline Tracing command fdc0 pid 35 tid 100030 td 0xc7ce9c00 sched_switch(c7ce9c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e1f03c,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c7e1f03c) at sleepq_timedwait+0x4a msleep(c7e1f03c,c7e1f0f0,4c,c071cc2c,3e8) at msleep+0x259 fdc_worker(c7e1f000,c7e1f0f0,0,c073f3ff,4a1) at fdc_worker+0x1fe fdc_thread(c7e1f000,e6929d38,c7e1f000,c06a57f4,0,...) at fdc_thread+0x54 fork_exit(c06a57f4,c7e1f000,e6929d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6929d6c, ebp = 0 --- Tracing command swi0: sio pid 34 tid 100031 td 0xc7ce9a80 sched_switch(c7ce9a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7e89540,e6926d38,c7e89540,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7e89540,e6926d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6926d6c, ebp = 0 --- Tracing command irq1: atkbd0 pid 33 tid 100032 td 0xc7ce9900 sched_switch(c7ce9900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7e846c0,e6923d38,c7e846c0,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7e846c0,e6923d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6923d6c, ebp = 0 --- Tracing command irq22: fxp0 pid 32 tid 100033 td 0xc7ce9780 sched_switch(c7ce9780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7e25010,e6920d38,c7e25010,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7e25010,e6920d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6920d6c, ebp = 0 --- Tracing command irq20: mly0 pid 31 tid 100034 td 0xc7ce9600 sched_switch(c7ce9600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7dfea40,e691dd38,c7dfea40,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7dfea40,e691dd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe691dd6c, ebp = 0 --- Tracing command irq15: ata1 pid 30 tid 100035 td 0xc7ce9480 fork_trampoline() at fork_trampoline Tracing command irq14: ata0 pid 29 tid 100036 td 0xc7e05d80 sched_switch(c7e05d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7e072e0,e8398d38,c7e072e0,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7e072e0,e8398d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8398d6c, ebp = 0 --- Tracing command usb1 pid 28 tid 100037 td 0xc7e05c00 sched_switch(c7e05c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7dada10,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c7dada10) at sleepq_timedwait+0x4a msleep(c7dada10,0,5c,c071b821,ea60,c7debd00) at msleep+0x259 usb_event_thread(c7debd00,e8395d38,c7debd00,c051db30,0,...) at usb_event_thread+0x70 fork_exit(c051db30,c7debd00,e8395d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8395d6c, ebp = 0 --- Tracing command irq17: ehci0 pid 27 tid 100017 td 0xc7ce5900 fork_trampoline() at fork_trampoline Tracing command usbtask pid 26 tid 100018 td 0xc7ce5780 sched_switch(c7ce5780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07953e4,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c07953e4,0,c055d0cc,c0797ae0,c071b806,...) at sleepq_wait+0x46 msleep(c07953e4,0,5c,c071b828,0,...) at msleep+0x27d usb_task_thread(0,e6902d38,0,c051dbc0,0,...) at usb_task_thread+0x47 fork_exit(c051dbc0,0,e6902d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6902d6c, ebp = 0 --- Tracing command usb0 pid 25 tid 100019 td 0xc7ce5600 sched_switch(c7ce5600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7db4210,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c7db4210) at sleepq_timedwait+0x4a msleep(c7db4210,0,5c,c071b821,ea60,c7dea640) at msleep+0x259 usb_event_thread(c7dea640,e68ffd38,c7dea640,c051db30,0,...) at usb_event_thread+0x70 fork_exit(c051db30,c7dea640,e68ffd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68ffd6c, ebp = 0 --- Tracing command irq16: ohci0 pid 24 tid 100020 td 0xc7ce5480 fork_trampoline() at fork_trampoline Tracing command irq9: acpi0 pid 23 tid 100021 td 0xc7ce5300 fork_trampoline() at fork_trampoline Tracing command acpi_task_2 pid 22 tid 100022 td 0xc7ce5180 sched_switch(c7ce5180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d43c00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d43c00,0,c07273e4,49,8,...) at sleepq_wait+0x46 msleep(c7d43c00,c7d43c1c,0,c071cc2c,0) at msleep+0x27d taskqueue_thread_loop(c0791fc8,e68f6d38,c0791fc8,c0585fd4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c0585fd4,c0791fc8,e68f6d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68f6d6c, ebp = 0 --- Tracing command acpi_task_1 pid 9 tid 100023 td 0xc7dbd780 sched_switch(c7dbd780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d43c00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d43c00,0,c07273e4,49,8,...) at sleepq_wait+0x46 msleep(c7d43c00,c7d43c1c,0,c071cc2c,0) at msleep+0x27d taskqueue_thread_loop(c0791fc8,e8361d38,c0791fc8,c0585fd4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c0585fd4,c0791fc8,e8361d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8361d6c, ebp = 0 --- Tracing command acpi_task_0 pid 8 tid 100024 td 0xc7dbd600 sched_switch(c7dbd600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d43c00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d43c00,0,c07273e4,49,c7ddf260,...) at sleepq_wait+0x46 msleep(c7d43c00,c7d43c1c,0,c071cc2c,0) at msleep+0x27d taskqueue_thread_loop(c0791fc8,e835ed38,c0791fc8,c0585fd4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c0585fd4,c0791fc8,e835ed38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe835ed6c, ebp = 0 --- Tracing command swi5: + pid 21 tid 100025 td 0xc7dbd480 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 7 tid 100026 td 0xc7dbd300 sched_switch(c7dbd300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d43d80,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d43d80,0,c07273e4,49,c806aeb0,...) at sleepq_wait+0x46 msleep(c7d43d80,c7d43d9c,0,c071cc2c,0) at msleep+0x27d taskqueue_thread_loop(c079da88,e8358d38,c079da88,c0585fd4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c0585fd4,c079da88,e8358d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8358d6c, ebp = 0 --- Tracing command swi6: Giant taskq pid 20 tid 100008 td 0xc7ce3300 sched_switch(c7ce3300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8b70,e68dbd38,c7cc8b70,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7cc8b70,e68dbd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68dbd6c, ebp = 0 --- Tracing command swi6: task queue pid 19 tid 100009 td 0xc7ce3180 sched_switch(c7ce3180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8b80,e68d8d38,c7cc8b80,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7cc8b80,e68d8d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68d8d6c, ebp = 0 --- Tracing command kqueue taskq pid 6 tid 100010 td 0xc7ce3000 sched_switch(c7ce3000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d72100,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d72100,0,c07273e4,49,8,...) at sleepq_wait+0x46 msleep(c7d72100,c7d7211c,0,c071cc2c,0) at msleep+0x27d taskqueue_thread_loop(c0796120,e68d5d38,c0796120,c0585fd4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c0585fd4,c0796120,e68d5d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68d5d6c, ebp = 0 --- Tracing command swi2: cambio pid 18 tid 100011 td 0xc7ce9300 sched_switch(c7ce9300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8ba0,e6917d38,c7cc8ba0,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7cc8ba0,e6917d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6917d6c, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100012 td 0xc7ce9180 sched_switch(c7ce9180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c078c4c4,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1 sleepq_wait(c078c4c4,0,c055d0cc,c0797ae0,c06fdf9f,...) at sleepq_wait+0x46 msleep(c078c4c4,0,4c,c06fdfba,0) at msleep+0x27d xpt_scanner_thread(0,e6914d38,0,c04441f4,0,...) at xpt_scanner_thread+0x35 fork_exit(c04441f4,0,e6914d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6914d6c, ebp = 0 --- Tracing command yarrow pid 17 tid 100013 td 0xc7ce9000 sched_switch(c7ce9000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07933e0,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c07933e0) at sleepq_timedwait+0x4a msleep(c07933e0,0,0,c071cc2c,64) at msleep+0x259 random_kthread(0,e6911d38,0,c0506b28,0,...) at random_kthread+0x180 fork_exit(c0506b28,0,e6911d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6911d6c, ebp = 0 --- Tracing command g_down pid 4 tid 100014 td 0xc7ce5d80 sched_switch(c7ce5d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c0795888,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c0795888) at sleepq_timedwait+0x4a msleep(c0795888,c0795768,24c,c071cc2c,64) at msleep+0x259 g_io_schedule_down(c7ce5d80) at g_io_schedule_down+0x56 g_down_procbody(0,e690ed38,0,c052fab0,0,...) at g_down_procbody+0x5a fork_exit(c052fab0,0,e690ed38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe690ed6c, ebp = 0 --- Tracing command g_up pid 3 tid 100015 td 0xc7ce5c00 sched_switch(c7ce5c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c0795884,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c0795884) at sleepq_timedwait+0x4a msleep(c0795884,c07957a8,24c,c071cc2c,64) at msleep+0x259 g_io_schedule_up(c7ce5c00) at g_io_schedule_up+0x127 g_up_procbody(0,e690bd38,0,c052fa50,0,...) at g_up_procbody+0x5a fork_exit(c052fa50,0,e690bd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe690bd6c, ebp = 0 --- Tracing command g_event pid 2 tid 100016 td 0xc7ce5a80 sched_switch(c7ce5a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c079587c,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c079587c) at sleepq_timedwait+0x4a msleep(c079587c,0,4c,c071cc2c,64) at msleep+0x259 g_event_procbody(0,e6908d38,0,c052fb10,0,...) at g_event_procbody+0x9e fork_exit(c052fb10,0,e6908d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6908d6c, ebp = 0 --- Tracing command swi3: vm pid 16 tid 100000 td 0xc7ce5000 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 15 tid 100001 td 0xc7ce3d80 sched_switch(c7ce3d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8870,e68f0d38,c7cc8870,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7cc8870,e68f0d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68f0d6c, ebp = 0 --- Tracing command swi1: net pid 14 tid 100002 td 0xc7ce3c00 sched_switch(c7ce3c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8880,e68edd38,c7cc8880,c0551b84,0,...) at ithread_loop+0xda fork_exit(c0551b84,c7cc8880,e68edd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68edd6c, ebp = 0 --- Tracing command idle: cpu0 pid 13 tid 100003 td 0xc7ce3a80 sched_switch(c072811b,6a9,c07e5594,e68eac40,c058a4e6,...) at sched_switch+0x177 (null)(c072813c,50000,c7ccb010,c07981b0,0,...) at 0 __func__.0(c055d930,0,2001,1,94378,...) at 0xc072813c (null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa Tracing command idle: cpu1 pid 12 tid 100004 td 0xc7ce3900 kdb_enter(c074002a) at kdb_enter+0x2b siointr1(c7e95800,c07f61a0,0,c073fe30,56e,...) at siointr1+0xce siointr(c7e95800) at siointr+0x21 intr_execute_handlers(c7cd74c8,e68e7c94,4,e68e7cd8,c06c2713,...) at intr_execute_handlers+0xe1 lapic_handle_intr(38) at lapic_handle_intr+0x2e Xapic_isr1() at Xapic_isr1+0x33 --- interrupt, eip = 0xc06bb645, esp = 0xe68e7cd8, ebp = 0xe68e7cd8 --- acpi_cpu_c1(c0797aa0,2,e68e7cf8,2,c7ce2860,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(e68e7d0c,c0550ee1,c0550e84,e68e7d24,c0550cc4,...) at acpi_cpu_idle+0x15d cpu_idle(c0550e84,e68e7d24,c0550cc4,0,e68e7d38,...) at cpu_idle+0x28 idle_proc(0,e68e7d38,0,c0550e84,0,...) at idle_proc+0x5d fork_exit(c0550e84,0,e68e7d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68e7d6c, ebp = 0 --- Tracing command idle: cpu2 pid 11 tid 100005 td 0xc7ce3780 sched_switch(c079efa0,0,ffc06034) at sched_switch+0x177 mi_switch(c055d321,c07e5590,e68e4c1c,c058ae7c,c079efa0,...) at mi_switch+0x270 end(c072813c,50000,c7ccb010,c07981b0,0,...) at 0xe68e4bf0 __func__.0(c055d930,0,2001,1,94378,...) at 0xc072813c (null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa Tracing command idle: cpu3 pid 10 tid 100006 td 0xc7ce3600 sched_switch(28,12,c06c2aa1,c080fd90,8,...) at sched_switch+0x177 Tracing command init pid 1 tid 100007 td 0xc7ce3480 sched_switch(c7ce3480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ce8000) at sleepq_switch+0xc1 sleepq_wait_sig(c7ce8000) at sleepq_wait_sig+0x1d msleep(c7ce8000,c7ce8068,15c,c0729071,0) at msleep+0x26e kern_wait(c7ce3480,ffffffff,e68dec84,0,0) at kern_wait+0x6bb wait4(c7ce3480,e68ded04) at wait4+0x2a syscall(3b,3b,3b,0,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8052f47, esp = 0xbfbfed7c, ebp = 0xbfbfed98 --- Tracing command swapper pid 0 tid 0 td 0xc0795ba0 sched_switch(c0795ba0,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 scheduler(0,c1ec00,c1e000,0,c0440d25,...) at scheduler+0x199 mi_startup() at mi_startup+0x96 begin() at begin+0x2c db> call doadump Dumping 3070 MB (2 chunks) Aborting dump due to I/O error. status == 0xb, scsi status == 0x0 ** DUMP FAILED (ERROR 5) ** = 0x1d -- Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE Phone: +7 4732 539880 Fax: +7 4732 531415 http://www.vsi.ru CenterTelecom Voronezh ISP http://isp.vsi.ru
On Mon, Oct 22, 2007 at 04:29:32PM +0400, Oleg Derevenetz wrote: > I posted a followup to kern/104406 that includes all information listed in > "Debugging Deadlocks" chapter of FreeBSD Developer's Handbook. Can anyone > take a look on it and say - is this certainly a hardware problem or some > sort of software problem ? The ps output from ddb would simplify the analysis. The buffer space on the machine seems to be contaminated with a critical amount of the dirty buffers that are sent for the write. The pid 844 (that is the only one producing the writes) is blocked in waitrunningbufspace(), waiting for the write queue to be drained. There are other processes, for instance, 937 etc, that wait for the buffer lock. Most likely, these buffers are scheduled for write and cause the processes to be uninterruptibly blocked. If this state of the system is static, then the write queue is not processed. Most plausible cause is the controller (mly0) or the controller driver fault.
OK, here is information grabbed from ddb for the another "hang" case including ps output: db> show pcpu cpuid = 1 curthread = 0xc7ce3900: pid 12 "idle: cpu1" curpcb = 0xe68e7d90 fpcurthread = none idlethread = 0xc7ce3900: pid 12 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: exclusive spin mutex sio r = 0 (0xc07df9c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 db> show allpcpu Current CPU: 1 cpuid = 0 curthread = 0xc7ce3a80: pid 13 "idle: cpu0" curpcb = 0xe68ead90 fpcurthread = none idlethread = 0xc7ce3a80: pid 13 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 curthread = 0xc7ce3900: pid 12 "idle: cpu1" curpcb = 0xe68e7d90 fpcurthread = none idlethread = 0xc7ce3900: pid 12 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: exclusive spin mutex sio r = 0 (0xc07df9c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 cpuid = 2 curthread = 0xc7ce3780: pid 11 "idle: cpu2" curpcb = 0xe68e4d90 fpcurthread = none idlethread = 0xc7ce3780: pid 11 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 curthread = 0xc7ce3600: pid 10 "idle: cpu3" curpcb = 0xe68e1d90 fpcurthread = none idlethread = 0xc7ce3600: pid 10 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: db> show locks exclusive spin mutex sio r = 0 (0xc07df9c0) locked @ /usr/src/sys/dev/sio/sio.c:1390 db> show alllocks db> show lockedvnods Locked vnodes 0xc820f414: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags (VV_ROOT) lock type ufs: EXCL (count 1) by thread 0xc7e05600 (pid 894) with 1 pending#0 0xc054901d at lockmgr+0x4ed #1 0xc0662f0e at ffs_lock+0x76 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87 #3 0xc05b884c at vn_lock+0xac #4 0xc05ac6f2 at vget+0xbe #5 0xc05a43d9 at vfs_hash_get+0x8d #6 0xc0661c77 at ffs_vget+0x27 #7 0xc066b065 at ufs_root+0x19 #8 0xc05a58fc at lookup+0x7c8 #9 0xc05a4e92 at namei+0x39a #10 0xc053c321 at do_execve+0x159 #11 0xc053c168 at kern_execve+0x74 #12 0xc053c0ce at execve+0x32 #13 0xc06c615b at syscall+0x25b #14 0xc06b17bf at Xint0x80_syscall+0x1f ino 2, on dev da0s1d 0xc824815c: tag ufs, type VDIR usecount 1, writecount 0, refcount 4 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xc84e8180 (pid 891) with 1 pending#0 0xc054901d at lockmgr+0x4ed #1 0xc0662f0e at ffs_lock+0x76 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87 #3 0xc05b884c at vn_lock+0xac #4 0xc05ac6f2 at vget+0xbe #5 0xc05a0e6e at cache_lookup+0x34a #6 0xc05a13a2 at vfs_cache_lookup+0x92 #7 0xc06d6c67 at VOP_LOOKUP_APV+0x87 #8 0xc05a55d8 at lookup+0x4a4 #9 0xc05a4e92 at namei+0x39a #10 0xc053c321 at do_execve+0x159 #11 0xc053c168 at kern_execve+0x74 #12 0xc053c0ce at execve+0x32 #13 0xc06c615b at syscall+0x25b #14 0xc06b17bf at Xint0x80_syscall+0x1f ino 518144, on dev da0s1d 0xc98dc414: tag ufs, type VREG usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VV_TEXT) v_object 0xc9752294 ref 1 pages 4 lock type ufs: EXCL (count 1) by thread 0xc8150480 (pid 806)#0 0xc054901d at lockmgr+0x4ed #1 0xc05a44b2 at vfs_hash_insert+0x26 #2 0xc0661dc1 at ffs_vget+0x171 #3 0xc066861a at ufs_lookup+0xab6 #4 0xc06d6d3b at VOP_CACHEDLOOKUP_APV+0x9b #5 0xc05a13c5 at vfs_cache_lookup+0xb5 #6 0xc06d6c67 at VOP_LOOKUP_APV+0x87 #7 0xc05a55d8 at lookup+0x4a4 #8 0xc05a4e92 at namei+0x39a #9 0xc053c321 at do_execve+0x159 #10 0xc053c168 at kern_execve+0x74 #11 0xc053c0ce at execve+0x32 #12 0xc06c615b at syscall+0x25b #13 0xc06b17bf at Xint0x80_syscall+0x1f ino 683213, on dev da0s1d 0xc822e6cc: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type syncer: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc054901d at lockmgr+0x4ed #1 0xc05a3591 at vop_stdlock+0x21 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87 #3 0xc05b884c at vn_lock+0xac #4 0xc05abeb3 at sync_vnode+0xe3 #5 0xc05ac151 at sched_sync+0x1ed #6 0xc05400cc at fork_exit+0xa0 #7 0xc06b17cc at fork_trampoline+0x8 0xc9956570: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () lock type ufs: EXCL (count 1) by thread 0xc8332d80 (pid 851)#0 0xc054901d at lockmgr+0x4ed #1 0xc0662f0e at ffs_lock+0x76 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87 #3 0xc05b884c at vn_lock+0xac #4 0xc05ac6f2 at vget+0xbe #5 0xc05a0e6e at cache_lookup+0x34a #6 0xc05a13a2 at vfs_cache_lookup+0x92 #7 0xc06d6c67 at VOP_LOOKUP_APV+0x87 #8 0xc05a55d8 at lookup+0x4a4 #9 0xc05a4e92 at namei+0x39a #10 0xc05b7637 at vn_open_cred+0x5b #11 0xc05b75da at vn_open+0x1e #12 0xc05b0c81 at kern_open+0x231 #13 0xc05b0a1a at open+0x1a #14 0xc06c615b at syscall+0x25b #15 0xc06b17bf at Xint0x80_syscall+0x1f ino 94845, on dev da1s1d 0xc99a16cc: tag ufs, type VREG usecount 1, writecount 0, refcount 7 mountedhere 0 flags () v_object 0xc99b8a50 ref 0 pages 19 lock type ufs: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc054901d at lockmgr+0x4ed #1 0xc0662f0e at ffs_lock+0x76 #2 0xc06d8ba7 at VOP_LOCK_APV+0x87 #3 0xc05b884c at vn_lock+0xac #4 0xc05ac6f2 at vget+0xbe #5 0xc06619b9 at ffs_sync+0x1c1 #6 0xc05ae264 at sync_fsync+0x164 #7 0xc06d7f37 at VOP_FSYNC_APV+0x9b #8 0xc05abed6 at sync_vnode+0x106 #9 0xc05ac151 at sched_sync+0x1ed #10 0xc05400cc at fork_exit+0xa0 #11 0xc06b17cc at fork_trampoline+0x8 ino 94916, on dev da1s1d db> alltrace Tracing command inetd pid 897 tid 100176 td 0xc8966780 sched_switch(c8966780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c820f46c,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c820f46c,0,c820f4dc,b7,c071716d,...) at sleepq_wait+0x46 msleep(c820f46c,c0780164,50,c0715859,0,...) at msleep+0x27d acquire(eaa608c8,40,60000,c8966780,0,...) at acquire+0x76 lockmgr(c820f46c,2002,c820f4dc,c8966780) at lockmgr+0x44e ffs_lock(eaa60920) at ffs_lock+0x76 VOP_LOCK_APV(c0764180,eaa60920) at VOP_LOCK_APV+0x87 vn_lock(c820f414,2002,c8966780,c820f414) at vn_lock+0xac vget(c820f414,2002,c8966780) at vget+0xbe vfs_hash_get(c822d5c8,2,2,c8966780,eaa60a38,0,0) at vfs_hash_get+0x8d ffs_vget(c822d5c8,2,2,eaa60a38) at ffs_vget+0x27 ufs_root(c822d5c8,2,eaa60a84,c8966780,c7eb0000,...) at ufs_root+0x19 lookup(eaa60c24) at lookup+0x7c8 namei(eaa60c24) at namei+0x39a do_execve(c8966780,eaa60cb4,0) at do_execve+0x159 kern_execve(c8966780,eaa60cb4,0) at kern_execve+0x74 execve(c8966780,eaa60d04) at execve+0x32 syscall(3b,3b,3b,bfbfedb0,805f200,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (59, FreeBSD ELF32, execve), eip = 0x280f14df, esp = 0xbfbfdb6c, ebp = 0xbfbfdb88 --- Tracing command sh pid 894 tid 100041 td 0xc7e05600 sched_switch(c7e05600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c82481b4,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c82481b4,0,c8248224,b7,c071716d,...) at sleepq_wait+0x46 msleep(c82481b4,c07806e0,50,c0715859,0,...) at msleep+0x27d acquire(e83898fc,40,60000,c7e05600,0,...) at acquire+0x76 lockmgr(c82481b4,2002,c8248224,c7e05600) at lockmgr+0x44e ffs_lock(e8389954) at ffs_lock+0x76 VOP_LOCK_APV(c0764180,e8389954) at VOP_LOCK_APV+0x87 vn_lock(c824815c,2002,c7e05600,c824815c) at vn_lock+0xac vget(c824815c,2002,c7e05600) at vget+0xbe cache_lookup(c820f414,e8389c38,e8389c4c) at cache_lookup+0x34a vfs_cache_lookup(e8389a98) at vfs_cache_lookup+0x92 VOP_LOOKUP_APV(c0764180,e8389a98) at VOP_LOOKUP_APV+0x87 lookup(e8389c24) at lookup+0x4a4 namei(e8389c24) at namei+0x39a do_execve(c7e05600,e8389cb4,0) at do_execve+0x159 kern_execve(c7e05600,e8389cb4,0) at kern_execve+0x74 execve(c7e05600,e8389d04) at execve+0x32 syscall(3b,3b,3b,0,8067038,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (59, FreeBSD ELF32, execve), eip = 0x2813d4df, esp = 0xbfbfeb9c, ebp = 0xbfbfebc8 --- Tracing command sh pid 893 tid 100177 td 0xc8966600 sched_switch(c8966600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8964c90) at sleepq_switch+0xc1 sleepq_wait_sig(c8964c90) at sleepq_wait_sig+0x1d msleep(c8964c90,c8964cf8,15c,c0713046,0) at msleep+0x26e kern_wait(c8966600,ffffffff,eaa5dc84,2,0) at kern_wait+0x6bb wait4(c8966600,eaa5dd04) at wait4+0x2a syscall(3b,3b,3b,bfbfebc8,37d,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2813d51f, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 --- Tracing command cron pid 892 tid 100059 td 0xc80e3a80 sched_switch(c80e3a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8107990) at sleepq_switch+0xc1 sleepq_wait_sig(c8107990) at sleepq_wait_sig+0x1d msleep(c8107990,c8107b00,14c,c0712d0e,0) at msleep+0x26e pipe_read(c8859948,ea85acbc,c8052080,0,c80e3a80) at pipe_read+0x377 dofileread(c80e3a80,5,c8859948,ea85acbc,ffffffff,...) at dofileread+0x85 kern_readv(c80e3a80,5,ea85acbc,805a000,1000,...) at kern_readv+0x36 read(c80e3a80,ea85ad04) at read+0x45 syscall(3b,3b,3b,28171040,28171040,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x2814b953, esp = 0xbfbfe65c, ebp = 0xbfbfe678 --- Tracing command inetd pid 891 tid 100159 td 0xc84e8180 sched_switch(c84e8180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(dbdb3fdc,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(dbdb3fdc,0,c0715e2f,e78,c071716d,...) at sleepq_wait+0x46 msleep(dbdb3fdc,c07cf5c0,4c,c07165fb,0) at msleep+0x27d bwait(dbdb3fdc,4c,c07165fb) at bwait+0x47 bufwait(dbdb3fdc,1,0,0,0,...) at bufwait+0x1a breadn(c824815c,0,0,800,0,...) at breadn+0x266 bread(c824815c,0,0,800,0,ea9b18e8) at bread+0x20 ffs_blkatoff(c824815c,0,0,0,ea9b1954) at ffs_blkatoff+0x9e ufs_lookup(ea9b19f8) at ufs_lookup+0x2cd VOP_CACHEDLOOKUP_APV(c0764180,ea9b19f8) at VOP_CACHEDLOOKUP_APV+0x9b vfs_cache_lookup(ea9b1a98) at vfs_cache_lookup+0xb5 VOP_LOOKUP_APV(c0764180,ea9b1a98) at VOP_LOOKUP_APV+0x87 lookup(ea9b1c24) at lookup+0x4a4 namei(ea9b1c24) at namei+0x39a do_execve(c84e8180,ea9b1cb4,0) at do_execve+0x159 kern_execve(c84e8180,ea9b1cb4,0) at kern_execve+0x74 execve(c84e8180,ea9b1d04) at execve+0x32 syscall(3b,3b,3b,bfbfedb0,805f1f0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (59, FreeBSD ELF32, execve), eip = 0x280f14df, esp = 0xbfbfdb6c, ebp = 0xbfbfdb88 --- Tracing command mc pid 851 tid 100161 td 0xc8332d80 sched_switch(c8332d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(dbfdafa4,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(dbfdafa4,0,c9956638,b7,c071716d,...) at sleepq_wait+0x46 msleep(dbfdafa4,c07811b4,50,c0716489,0,...) at msleep+0x27d acquire(ea9ab718,120,60000,c8332d80,0,...) at acquire+0x76 lockmgr(dbfdafa4,202122,c9956638,c8332d80) at lockmgr+0x44e getblk(c9956570,0,0,1000,0,...) at getblk+0x12f breadn(c9956570,0,0,1000,0,...) at breadn+0x31 bread(c9956570,0,0,1000,0,ea9ab7d4) at bread+0x20 ffs_blkatoff(c9956570,800,0,ea9ab810,ea9ab814,c8064600,0,c07243ac,20f) at ffs_blkatoff+0x9e ufsdirhash_findfree(c973b000,18,ea9ab880,c8332d80,1,...) at ufsdirhash_findfree+0x11a ufs_lookup(ea9ab91c) at ufs_lookup+0x126 VOP_CACHEDLOOKUP_APV(c0764180,ea9ab91c) at VOP_CACHEDLOOKUP_APV+0x9b vfs_cache_lookup(ea9ab9bc) at vfs_cache_lookup+0xb5 VOP_LOOKUP_APV(c0764180,ea9ab9bc) at VOP_LOOKUP_APV+0x87 lookup(ea9abbcc) at lookup+0x4a4 namei(ea9abbcc) at namei+0x39a vn_open_cred(ea9abbcc,ea9abccc,180,c8591880,6,...) at vn_open_cred+0x5b vn_open(ea9abbcc,ea9abccc,180,6,c057a284,...) at vn_open+0x1e kern_open(c8332d80,8128900,0,602,180,...) at kern_open+0x231 open(c8332d80,ea9abd04) at open+0x1a syscall(3b,3b,3b,80cfc20,8128900,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x28360ab7, esp = 0xbfbfde7c, ebp = 0xbfbfde98 --- Tracing command csh pid 848 tid 100163 td 0xc8332a80 sched_switch(c8332a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c851dcc4) at sleepq_switch+0xc1 sleepq_wait_sig(c851dcc4) at sleepq_wait_sig+0x1d msleep(c851dcc4,c851dcf8,168,c070e01c,0) at msleep+0x26e kern_sigsuspend(c8332a80,2,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c8332a80,ea9a5d04) at sigsuspend+0x33 syscall(3b,3b,3b,8299008,bfbf1ad0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1abc, ebp = 0xbfbf1ae8 --- Tracing command su pid 847 tid 100162 td 0xc8332c00 sched_switch(c8332c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c851da78) at sleepq_switch+0xc1 sleepq_wait_sig(c851da78) at sleepq_wait_sig+0x1d msleep(c851da78,c851dae0,15c,c0713046,0) at msleep+0x26e kern_wait(c8332c00,350,ea9a8c84,2,0) at kern_wait+0x6bb wait4(c8332c00,ea9a8d04) at wait4+0x2a syscall(3b,3b,3b,bfbfe794,bfbfe810,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x280f451f, esp = 0xbfbfe70c, ebp = 0xbfbfe728 --- Tracing command csh pid 842 tid 100120 td 0xc8157480 sched_switch(c8157480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c835224c) at sleepq_switch+0xc1 sleepq_wait_sig(c835224c) at sleepq_wait_sig+0x1d msleep(c835224c,c8352280,168,c070e01c,0) at msleep+0x26e kern_sigsuspend(c8157480,2,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c8157480,ea8fed04) at sigsuspend+0x33 syscall(3b,3b,3b,81c1308,bfbf1bd0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1bbc, ebp = 0xbfbf1be8 --- Tracing command sshd pid 841 tid 100055 td 0xc8086d80 sched_switch(c8086d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b kern_select(c8086d80,9,807bb70,807bb80,0,...) at kern_select+0x4bf select(c8086d80,e8638d04) at select+0x44 syscall(3b,3b,3b,0,bfbfe11c,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe0ac, ebp = 0xbfbfe0f8 --- Tracing command sshd pid 837 tid 100165 td 0xc8332780 sched_switch(c8332780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c827720c) at sleepq_switch+0xc1 sleepq_wait_sig(c827720c) at sleepq_wait_sig+0x1d msleep(c827720c,c82771d8,158,c0715361,0,...) at msleep+0x26e sbwait(c82771b4,ea99fc0c,c057a284,4,0,...) at sbwait+0x4b soreceive(c8277164,0,ea99fcbc,0,0,0) at soreceive+0x2ee soo_read(c83c2168,ea99fcbc,c849a980,0,c8332780) at soo_read+0x41 dofileread(c8332780,5,c83c2168,ea99fcbc,ffffffff,...) at dofileread+0x85 kern_readv(c8332780,5,ea99fcbc,bfbfe108,4,...) at kern_readv+0x36 read(c8332780,ea99fd04) at read+0x45 syscall(3b,3b,3b,4,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x2834d953, esp = 0xbfbfe0ac, ebp = 0xbfbfe0d8 --- Tracing command httpd pid 836 tid 100124 td 0xc8153d80 sched_switch(c8153d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80bba80) at sleepq_switch+0xc1 sleepq_wait_sig(c80bba80) at sleepq_wait_sig+0x1d msleep(c80bba80,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80bba80,16,ffffffff,ffffffff,ea8f2cb0,...) at lf_setlock+0x181 lf_advlock(ea8f2c90,c83c90b8,0,0,ea8f2c7c,...) at lf_advlock+0x203 ufs_advlock(ea8f2c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea8f2c90) at VOP_ADVLOCK_APV+0xa9 flock(c8153d80,ea8f2d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 835 tid 100125 td 0xc8153c00 sched_switch(c8153c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8113680) at sleepq_switch+0xc1 sleepq_wait_sig(c8113680) at sleepq_wait_sig+0x1d msleep(c8113680,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c8113680,16,ffffffff,ffffffff,ea8efcb0,...) at lf_setlock+0x181 lf_advlock(ea8efc90,c83c90b8,0,0,ea8efc7c,...) at lf_advlock+0x203 ufs_advlock(ea8efc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea8efc90) at VOP_ADVLOCK_APV+0xa9 flock(c8153c00,ea8efd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 834 tid 100136 td 0xc8158d80 sched_switch(c8158d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80c0840) at sleepq_switch+0xc1 sleepq_wait_sig(c80c0840) at sleepq_wait_sig+0x1d msleep(c80c0840,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80c0840,16,ffffffff,ffffffff,ea92ecb0,...) at lf_setlock+0x181 lf_advlock(ea92ec90,c83c90b8,0,0,ea92ec7c,...) at lf_advlock+0x203 ufs_advlock(ea92ec90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea92ec90) at VOP_ADVLOCK_APV+0xa9 flock(c8158d80,ea92ed04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 833 tid 100137 td 0xc8158c00 sched_switch(c8158c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8112880) at sleepq_switch+0xc1 sleepq_wait_sig(c8112880) at sleepq_wait_sig+0x1d msleep(c8112880,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c8112880,16,ffffffff,ffffffff,ea92bcb0,...) at lf_setlock+0x181 lf_advlock(ea92bc90,c83c90b8,0,0,ea92bc7c,...) at lf_advlock+0x203 ufs_advlock(ea92bc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea92bc90) at VOP_ADVLOCK_APV+0xa9 flock(c8158c00,ea92bd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 832 tid 100138 td 0xc8158a80 sched_switch(c8158a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea6080) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea6080) at sleepq_wait_sig+0x1d msleep(c7ea6080,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c7ea6080,16,ffffffff,ffffffff,ea928cb0,...) at lf_setlock+0x181 lf_advlock(ea928c90,c83c90b8,0,0,ea928c7c,...) at lf_advlock+0x203 ufs_advlock(ea928c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea928c90) at VOP_ADVLOCK_APV+0xa9 flock(c8158a80,ea928d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 831 tid 100139 td 0xc8158900 sched_switch(c8158900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80c08c0) at sleepq_switch+0xc1 sleepq_wait_sig(c80c08c0) at sleepq_wait_sig+0x1d msleep(c80c08c0,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80c08c0,16,ffffffff,ffffffff,ea925cb0,...) at lf_setlock+0x181 lf_advlock(ea925c90,c83c90b8,0,0,ea925c7c,...) at lf_advlock+0x203 ufs_advlock(ea925c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea925c90) at VOP_ADVLOCK_APV+0xa9 flock(c8158900,ea925d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 830 tid 100140 td 0xc8158780 sched_switch(c8158780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c81124c0) at sleepq_switch+0xc1 sleepq_wait_sig(c81124c0) at sleepq_wait_sig+0x1d msleep(c81124c0,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c81124c0,16,ffffffff,ffffffff,ea922cb0,...) at lf_setlock+0x181 lf_advlock(ea922c90,c83c90b8,0,0,ea922c7c,...) at lf_advlock+0x203 ufs_advlock(ea922c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea922c90) at VOP_ADVLOCK_APV+0xa9 flock(c8158780,ea922d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 829 tid 100141 td 0xc8086480 sched_switch(c8086480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80c0800) at sleepq_switch+0xc1 sleepq_wait_sig(c80c0800) at sleepq_wait_sig+0x1d msleep(c80c0800,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80c0800,16,ffffffff,ffffffff,e8626cb0,...) at lf_setlock+0x181 lf_advlock(e8626c90,c83c90b8,0,0,e8626c7c,...) at lf_advlock+0x203 ufs_advlock(e8626c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,e8626c90) at VOP_ADVLOCK_APV+0xa9 flock(c8086480,e8626d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 828 tid 100142 td 0xc8087c00 sched_switch(c8087c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80bc840) at sleepq_switch+0xc1 sleepq_wait_sig(c80bc840) at sleepq_wait_sig+0x1d msleep(c80bc840,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80bc840,16,ffffffff,ffffffff,e8653cb0,...) at lf_setlock+0x181 lf_advlock(e8653c90,c83c90b8,0,0,e8653c7c,...) at lf_advlock+0x203 ufs_advlock(e8653c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,e8653c90) at VOP_ADVLOCK_APV+0xa9 flock(c8087c00,e8653d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 827 tid 100143 td 0xc8331480 sched_switch(c8331480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80bc600) at sleepq_switch+0xc1 sleepq_wait_sig(c80bc600) at sleepq_wait_sig+0x1d msleep(c80bc600,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80bc600,16,ffffffff,ffffffff,ea97bcb0,...) at lf_setlock+0x181 lf_advlock(ea97bc90,c83c90b8,0,0,ea97bc7c,...) at lf_advlock+0x203 ufs_advlock(ea97bc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea97bc90) at VOP_ADVLOCK_APV+0xa9 flock(c8331480,ea97bd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 826 tid 100144 td 0xc8087900 sched_switch(c8087900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80c0980) at sleepq_switch+0xc1 sleepq_wait_sig(c80c0980) at sleepq_wait_sig+0x1d msleep(c80c0980,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80c0980,16,ffffffff,ffffffff,e864dcb0,...) at lf_setlock+0x181 lf_advlock(e864dc90,c83c90b8,0,0,e864dc7c,...) at lf_advlock+0x203 ufs_advlock(e864dc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,e864dc90) at VOP_ADVLOCK_APV+0xa9 flock(c8087900,e864dd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 825 tid 100145 td 0xc84e9780 sched_switch(c84e9780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea8cc0) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea8cc0) at sleepq_wait_sig+0x1d msleep(c7ea8cc0,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c7ea8cc0,16,ffffffff,ffffffff,ea9dbcb0,...) at lf_setlock+0x181 lf_advlock(ea9dbc90,c83c90b8,0,0,ea9dbc7c,...) at lf_advlock+0x203 ufs_advlock(ea9dbc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea9dbc90) at VOP_ADVLOCK_APV+0xa9 flock(c84e9780,ea9dbd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 824 tid 100146 td 0xc84e9600 sched_switch(c84e9600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80bbb00) at sleepq_switch+0xc1 sleepq_wait_sig(c80bbb00) at sleepq_wait_sig+0x1d msleep(c80bbb00,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80bbb00,16,ffffffff,ffffffff,ea9d8cb0,...) at lf_setlock+0x181 lf_advlock(ea9d8c90,c83c90b8,0,0,ea9d8c7c,...) at lf_advlock+0x203 ufs_advlock(ea9d8c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea9d8c90) at VOP_ADVLOCK_APV+0xa9 flock(c84e9600,ea9d8d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 823 tid 100147 td 0xc84e9480 sched_switch(c84e9480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80c15c0) at sleepq_switch+0xc1 sleepq_wait_sig(c80c15c0) at sleepq_wait_sig+0x1d msleep(c80c15c0,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80c15c0,16,ffffffff,ffffffff,ea9d5cb0,...) at lf_setlock+0x181 lf_advlock(ea9d5c90,c83c90b8,0,0,ea9d5c7c,...) at lf_advlock+0x203 ufs_advlock(ea9d5c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea9d5c90) at VOP_ADVLOCK_APV+0xa9 flock(c84e9480,ea9d5d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 822 tid 100148 td 0xc84e9300 sched_switch(c84e9300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea8d00) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea8d00) at sleepq_wait_sig+0x1d msleep(c7ea8d00,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c7ea8d00,16,ffffffff,ffffffff,ea9d2cb0,...) at lf_setlock+0x181 lf_advlock(ea9d2c90,c83c90b8,0,0,ea9d2c7c,...) at lf_advlock+0x203 ufs_advlock(ea9d2c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea9d2c90) at VOP_ADVLOCK_APV+0xa9 flock(c84e9300,ea9d2d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 821 tid 100149 td 0xc84e9180 sched_switch(c84e9180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8112180) at sleepq_switch+0xc1 sleepq_wait_sig(c8112180) at sleepq_wait_sig+0x1d msleep(c8112180,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c8112180,16,ffffffff,ffffffff,ea9cfcb0,...) at lf_setlock+0x181 lf_advlock(ea9cfc90,c83c90b8,0,0,ea9cfc7c,...) at lf_advlock+0x203 ufs_advlock(ea9cfc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea9cfc90) at VOP_ADVLOCK_APV+0xa9 flock(c84e9180,ea9cfd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 820 tid 100042 td 0xc7e05480 sched_switch(c7e05480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8113700) at sleepq_switch+0xc1 sleepq_wait_sig(c8113700) at sleepq_wait_sig+0x1d msleep(c8113700,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c8113700,16,ffffffff,ffffffff,e8386cb0,...) at lf_setlock+0x181 lf_advlock(e8386c90,c83c90b8,0,0,e8386c7c,...) at lf_advlock+0x203 ufs_advlock(e8386c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,e8386c90) at VOP_ADVLOCK_APV+0xa9 flock(c7e05480,e8386d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 819 tid 100057 td 0xc80e3d80 sched_switch(c80e3d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c81123c0) at sleepq_switch+0xc1 sleepq_wait_sig(c81123c0) at sleepq_wait_sig+0x1d msleep(c81123c0,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c81123c0,16,ffffffff,ffffffff,ea860cb0,...) at lf_setlock+0x181 lf_advlock(ea860c90,c83c90b8,0,0,ea860c7c,...) at lf_advlock+0x203 ufs_advlock(ea860c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea860c90) at VOP_ADVLOCK_APV+0xa9 flock(c80e3d80,ea860d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 818 tid 100093 td 0xc810fd80 sched_switch(c810fd80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea8dc0) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea8dc0) at sleepq_wait_sig+0x1d msleep(c7ea8dc0,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c7ea8dc0,16,ffffffff,ffffffff,ea8a9cb0,...) at lf_setlock+0x181 lf_advlock(ea8a9c90,c83c90b8,0,0,ea8a9c7c,...) at lf_advlock+0x203 ufs_advlock(ea8a9c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea8a9c90) at VOP_ADVLOCK_APV+0xa9 flock(c810fd80,ea8a9d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 817 tid 100092 td 0xc8150000 sched_switch(c8150000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80c1580) at sleepq_switch+0xc1 sleepq_wait_sig(c80c1580) at sleepq_wait_sig+0x1d msleep(c80c1580,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80c1580,16,ffffffff,ffffffff,ea8b9cb0,...) at lf_setlock+0x181 lf_advlock(ea8b9c90,c83c90b8,0,0,ea8b9c7c,...) at lf_advlock+0x203 ufs_advlock(ea8b9c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea8b9c90) at VOP_ADVLOCK_APV+0xa9 flock(c8150000,ea8b9d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 816 tid 100106 td 0xc8331d80 sched_switch(c8331d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea6b80) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea6b80) at sleepq_wait_sig+0x1d msleep(c7ea6b80,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c7ea6b80,16,ffffffff,ffffffff,ea98dcb0,...) at lf_setlock+0x181 lf_advlock(ea98dc90,c83c90b8,0,0,ea98dc7c,...) at lf_advlock+0x203 ufs_advlock(ea98dc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea98dc90) at VOP_ADVLOCK_APV+0xa9 flock(c8331d80,ea98dd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 815 tid 100054 td 0xc8087000 sched_switch(c8087000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80c0a00) at sleepq_switch+0xc1 sleepq_wait_sig(c80c0a00) at sleepq_wait_sig+0x1d msleep(c80c0a00,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80c0a00,16,ffffffff,ffffffff,e863bcb0,...) at lf_setlock+0x181 lf_advlock(e863bc90,c83c90b8,0,0,e863bc7c,...) at lf_advlock+0x203 ufs_advlock(e863bc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,e863bc90) at VOP_ADVLOCK_APV+0xa9 flock(c8087000,e863bd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 814 tid 100053 td 0xc8087180 sched_switch(c8087180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80c0600) at sleepq_switch+0xc1 sleepq_wait_sig(c80c0600) at sleepq_wait_sig+0x1d msleep(c80c0600,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80c0600,16,ffffffff,ffffffff,e863ecb0,...) at lf_setlock+0x181 lf_advlock(e863ec90,c83c90b8,0,0,e863ec7c,...) at lf_advlock+0x203 ufs_advlock(e863ec90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,e863ec90) at VOP_ADVLOCK_APV+0xa9 flock(c8087180,e863ed04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 813 tid 100115 td 0xc8157c00 sched_switch(c8157c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea8d40) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea8d40) at sleepq_wait_sig+0x1d msleep(c7ea8d40,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c7ea8d40,16,ffffffff,ffffffff,ea90dcb0,...) at lf_setlock+0x181 lf_advlock(ea90dc90,c83c90b8,0,0,ea90dc7c,...) at lf_advlock+0x203 ufs_advlock(ea90dc90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea90dc90) at VOP_ADVLOCK_APV+0xa9 flock(c8157c00,ea90dd04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 812 tid 100066 td 0xc810f600 sched_switch(c810f600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80bba00) at sleepq_switch+0xc1 sleepq_wait_sig(c80bba00) at sleepq_wait_sig+0x1d msleep(c80bba00,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80bba00,16,ffffffff,ffffffff,ea89acb0,...) at lf_setlock+0x181 lf_advlock(ea89ac90,c83c90b8,0,0,ea89ac7c,...) at lf_advlock+0x203 ufs_advlock(ea89ac90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea89ac90) at VOP_ADVLOCK_APV+0xa9 flock(c810f600,ea89ad04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 811 tid 100108 td 0xc8331a80 sched_switch(c8331a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80bc7c0) at sleepq_switch+0xc1 sleepq_wait_sig(c80bc7c0) at sleepq_wait_sig+0x1d msleep(c80bc7c0,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80bc7c0,16,ffffffff,ffffffff,ea987cb0,...) at lf_setlock+0x181 lf_advlock(ea987c90,c83c90b8,0,0,ea987c7c,...) at lf_advlock+0x203 ufs_advlock(ea987c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea987c90) at VOP_ADVLOCK_APV+0xa9 flock(c8331a80,ea987d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 810 tid 100109 td 0xc8331900 sched_switch(c8331900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8112240) at sleepq_switch+0xc1 sleepq_wait_sig(c8112240) at sleepq_wait_sig+0x1d msleep(c8112240,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c8112240,16,ffffffff,ffffffff,ea984cb0,...) at lf_setlock+0x181 lf_advlock(ea984c90,c83c90b8,0,0,ea984c7c,...) at lf_advlock+0x203 ufs_advlock(ea984c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea984c90) at VOP_ADVLOCK_APV+0xa9 flock(c8331900,ea984d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 809 tid 100110 td 0xc8331780 sched_switch(c8331780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80c1e80) at sleepq_switch+0xc1 sleepq_wait_sig(c80c1e80) at sleepq_wait_sig+0x1d msleep(c80c1e80,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c80c1e80,16,ffffffff,ffffffff,ea981cb0,...) at lf_setlock+0x181 lf_advlock(ea981c90,c83c90b8,0,0,ea981c7c,...) at lf_advlock+0x203 ufs_advlock(ea981c90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea981c90) at VOP_ADVLOCK_APV+0xa9 flock(c8331780,ea981d04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 808 tid 100111 td 0xc8331600 sched_switch(c8331600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8112300) at sleepq_switch+0xc1 sleepq_wait_sig(c8112300) at sleepq_wait_sig+0x1d msleep(c8112300,0,168,c074f7c4,0) at msleep+0x26e lf_setlock(c8112300,16,ffffffff,ffffffff,ea97ecb0,...) at lf_setlock+0x181 lf_advlock(ea97ec90,c83c90b8,0,0,ea97ec7c,...) at lf_advlock+0x203 ufs_advlock(ea97ec90) at ufs_advlock+0x1c VOP_ADVLOCK_APV(c0764180,ea97ec90) at VOP_ADVLOCK_APV+0xa9 flock(c8331600,ea97ed04) at flock+0x1a1 syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 --- Tracing command httpd pid 807 tid 100112 td 0xc8158180 sched_switch(c8158180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,3b7,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b poll(c8158180,ea916d04) at poll+0x3b9 syscall(3b,3b,3b,2,28,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (209, FreeBSD ELF32, poll), eip = 0x282c3a87, esp = 0xbfbfebcc, ebp = 0xbfbfec18 --- Tracing command getty pid 806 tid 100089 td 0xc8150480 sched_switch(c8150480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(dbd4b0bc,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(dbd4b0bc,0,c0715e2f,e78,c0727ac1,...) at sleepq_wait+0x46 msleep(dbd4b0bc,c07cf5c0,44,c0727ce4,0) at msleep+0x27d bwait(dbd4b0bc,44,c0727ce4) at bwait+0x47 vnode_pager_generic_getpages(c98dc414,ea8c2acc,5000,0,c9752294,...) at vnode_pager_generic_getpages+0x847 ffs_getpages(ea8c2a88) at ffs_getpages+0x100 VOP_GETPAGES_APV(c0764180,ea8c2a88) at VOP_GETPAGES_APV+0xa9 vnode_pager_getpages(c9752294,ea8c2acc,5,0) at vnode_pager_getpages+0xa5 exec_map_first_page(ea8c2be4) at exec_map_first_page+0x16d do_execve(c8150480,ea8c2cb4,0) at do_execve+0x1e4 kern_execve(c8150480,ea8c2cb4,0) at kern_execve+0x74 execve(c8150480,ea8c2d04) at execve+0x32 syscall(3b,3b,3b,bfbfee18,bfbfedd0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (59, FreeBSD ELF32, execve), eip = 0x280df4df, esp = 0xbfbfedbc, ebp = 0xbfbfee08 --- Tracing command getty pid 805 tid 100084 td 0xc8150c00 sched_switch(c8150c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e9c410) at sleepq_switch+0xc1 sleepq_wait_sig(c7e9c410) at sleepq_wait_sig+0x1d msleep(c7e9c410,0,159,c0714584,0) at msleep+0x26e ttysleep(c7e9c400,c7e9c410,159,c0714584,0,...) at ttysleep+0x21 ttread(c7e9c400,ea8d1cbc,0) at ttread+0x48f ttyread(c7e8f600,ea8d1cbc,0,ea8d1c14,c052f435,...) at ttyread+0x2f scread(c7e8f600,ea8d1cbc,0,c0781300,0,...) at scread+0x22 giant_read(c7e8f600,ea8d1cbc,0) at giant_read+0x2d devfs_read_f(c80f37e0,ea8d1cbc,c7ce0780,0,c8150c00) at devfs_read_f+0x62 dofileread(c8150c00,0,c80f37e0,ea8d1cbc,ffffffff,...) at dofileread+0x85 kern_readv(c8150c00,0,ea8d1cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c8150c00,ea8d1d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 804 tid 100064 td 0xc810f900 sched_switch(c810f900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea4010) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea4010) at sleepq_wait_sig+0x1d msleep(c7ea4010,0,159,c0714584,0) at msleep+0x26e ttysleep(c7ea4000,c7ea4010,159,c0714584,0,...) at ttysleep+0x21 ttread(c7ea4000,ea8a0cbc,0) at ttread+0x48f ttyread(c7e8f700,ea8a0cbc,0,ea8a0c14,c052f435,...) at ttyread+0x2f scread(c7e8f700,ea8a0cbc,0,c0781300,0,...) at scread+0x22 giant_read(c7e8f700,ea8a0cbc,0) at giant_read+0x2d devfs_read_f(c80f6048,ea8a0cbc,c7ce0780,0,c810f900) at devfs_read_f+0x62 dofileread(c810f900,0,c80f6048,ea8a0cbc,ffffffff,...) at dofileread+0x85 kern_readv(c810f900,0,ea8a0cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c810f900,ea8a0d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 803 tid 100117 td 0xc8157900 sched_switch(c8157900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea3c10) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea3c10) at sleepq_wait_sig+0x1d msleep(c7ea3c10,0,159,c0714584,0) at msleep+0x26e ttysleep(c7ea3c00,c7ea3c10,159,c0714584,0,...) at ttysleep+0x21 ttread(c7ea3c00,ea907cbc,0) at ttread+0x48f ttyread(c7e90600,ea907cbc,0,ea907c14,c052f435,...) at ttyread+0x2f scread(c7e90600,ea907cbc,0,c0781300,0,...) at scread+0x22 giant_read(c7e90600,ea907cbc,0) at giant_read+0x2d devfs_read_f(c80f3828,ea907cbc,c7ce0780,0,c8157900) at devfs_read_f+0x62 dofileread(c8157900,0,c80f3828,ea907cbc,ffffffff,...) at dofileread+0x85 kern_readv(c8157900,0,ea907cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c8157900,ea907d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 802 tid 100061 td 0xc80e3780 sched_switch(c80e3780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea4c10) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea4c10) at sleepq_wait_sig+0x1d msleep(c7ea4c10,0,159,c0714584,0) at msleep+0x26e ttysleep(c7ea4c00,c7ea4c10,159,c0714584,0,...) at ttysleep+0x21 ttread(c7ea4c00,ea854cbc,0) at ttread+0x48f ttyread(c7e90500,ea854cbc,0,ea854c14,c052f435,...) at ttyread+0x2f scread(c7e90500,ea854cbc,0,c0781300,0,...) at scread+0x22 giant_read(c7e90500,ea854cbc,0) at giant_read+0x2d devfs_read_f(c82a14c8,ea854cbc,c7ce0780,0,c80e3780) at devfs_read_f+0x62 dofileread(c80e3780,0,c82a14c8,ea854cbc,ffffffff,...) at dofileread+0x85 kern_readv(c80e3780,0,ea854cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c80e3780,ea854d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 801 tid 100078 td 0xc8153600 sched_switch(c8153600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea4810) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea4810) at sleepq_wait_sig+0x1d msleep(c7ea4810,0,159,c0714584,0) at msleep+0x26e ttysleep(c7ea4800,c7ea4810,159,c0714584,0,...) at ttysleep+0x21 ttread(c7ea4800,ea8e3cbc,0) at ttread+0x48f ttyread(c7e90400,ea8e3cbc,0,ea8e3c14,c052f435,...) at ttyread+0x2f scread(c7e90400,ea8e3cbc,0,c0781300,0,...) at scread+0x22 giant_read(c7e90400,ea8e3cbc,0) at giant_read+0x2d devfs_read_f(c82a1678,ea8e3cbc,c7ce0780,0,c8153600) at devfs_read_f+0x62 dofileread(c8153600,0,c82a1678,ea8e3cbc,ffffffff,...) at dofileread+0x85 kern_readv(c8153600,0,ea8e3cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c8153600,ea8e3d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 800 tid 100088 td 0xc8150600 sched_switch(c8150600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea2c10) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea2c10) at sleepq_wait_sig+0x1d msleep(c7ea2c10,0,159,c0714584,0) at msleep+0x26e ttysleep(c7ea2c00,c7ea2c10,159,c0714584,0,...) at ttysleep+0x21 ttread(c7ea2c00,ea8c5cbc,0) at ttread+0x48f ttyread(c7e90200,ea8c5cbc,0,ea8c5c14,c052f435,...) at ttyread+0x2f scread(c7e90200,ea8c5cbc,0,c0781300,0,...) at scread+0x22 giant_read(c7e90200,ea8c5cbc,0) at giant_read+0x2d devfs_read_f(c80f6240,ea8c5cbc,c7ce0780,0,c8150600) at devfs_read_f+0x62 dofileread(c8150600,0,c80f6240,ea8c5cbc,ffffffff,...) at dofileread+0x85 kern_readv(c8150600,0,ea8c5cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c8150600,ea8c5d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 799 tid 100075 td 0xc80e6780 sched_switch(c80e6780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea2810) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea2810) at sleepq_wait_sig+0x1d msleep(c7ea2810,0,159,c0714584,0) at msleep+0x26e ttysleep(c7ea2800,c7ea2810,159,c0714584,0,...) at ttysleep+0x21 ttread(c7ea2800,ea872cbc,0) at ttread+0x48f ttyread(c7e90100,ea872cbc,0,ea872c14,c052f435,...) at ttyread+0x2f scread(c7e90100,ea872cbc,0,c0781300,0,...) at scread+0x22 giant_read(c7e90100,ea872cbc,0) at giant_read+0x2d devfs_read_f(c80f6798,ea872cbc,c7ce0780,0,c80e6780) at devfs_read_f+0x62 dofileread(c80e6780,0,c80f6798,ea872cbc,ffffffff,...) at dofileread+0x85 kern_readv(c80e6780,0,ea872cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c80e6780,ea872d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 798 tid 100040 td 0xc7e05780 sched_switch(c7e05780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ea3810) at sleepq_switch+0xc1 sleepq_wait_sig(c7ea3810) at sleepq_wait_sig+0x1d msleep(c7ea3810,0,159,c0714584,0) at msleep+0x26e ttysleep(c7ea3800,c7ea3810,159,c0714584,0,...) at ttysleep+0x21 ttread(c7ea3800,e838ccbc,0) at ttread+0x48f ttyread(c7e90000,e838ccbc,0,e838cc14,c052f435,...) at ttyread+0x2f scread(c7e90000,e838ccbc,0,c0781300,0,...) at scread+0x22 giant_read(c7e90000,e838ccbc,0) at giant_read+0x2d devfs_read_f(c80f6120,e838ccbc,c7ce0780,0,c7e05780) at devfs_read_f+0x62 dofileread(c7e05780,0,c80f6120,e838ccbc,ffffffff,...) at dofileread+0x85 kern_readv(c7e05780,0,e838ccbc,bfbfee1f,1,...) at kern_readv+0x36 read(c7e05780,e838cd04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command inetd pid 784 tid 100119 td 0xc8157600 sched_switch(c8157600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b kern_select(c8157600,7,bfbfe8a0,0,0,...) at kern_select+0x4bf select(c8157600,ea901d04) at select+0x44 syscall(3b,3b,3b,bfbfe8a0,381,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2815a8d3, esp = 0xbfbfdb7c, ebp = 0xbfbfee68 --- Tracing command httpd pid 764 tid 100116 td 0xc8157a80 sched_switch(c8157a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07cef64,c07cef64,3e9,ea90ab40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07cef64,c07cef40,3e9) at cv_timedwait_sig+0x194 kern_select(c8157a80,0,0,0,0,...) at kern_select+0x4a9 select(c8157a80,ea90ad04) at select+0x44 syscall(3b,3b,3b,0,f4240,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2830c8d3, esp = 0xbfbfec3c, ebp = 0xbfbfec78 --- Tracing command mysqld pid 763 tid 100133 td 0xc8086c00 sched_switch(c8086c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8154034) at sleepq_switch+0xc1 sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e msleep(c8154034,c8156068,168,c070b128,7ba,...) at msleep+0x244 kse_release(c8086c00,e8635d04) at kse_release+0x1df syscall(3b,3b,3b,8530000,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x8534f28, ebp = 0x8534f64 --- Tracing command mysqld pid 763 tid 100135 td 0xc8086900 sched_switch(c8086900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8154034) at sleepq_switch+0xc1 sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e msleep(c8154034,c8156068,168,c070b128,3e9,...) at msleep+0x244 kse_release(c8086900,e862fd04) at kse_release+0x1df syscall(3b,3b,3b,8530b00,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7ef28, ebp = 0x13c7ef64 --- Tracing command mysqld pid 763 tid 100091 td 0xc8086780 sched_switch(c8086780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8154034) at sleepq_switch+0xc1 sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e msleep(c8154034,c8156068,168,c070b128,7ba,...) at msleep+0x244 kse_release(c8086780,e862cd04) at kse_release+0x1df syscall(3b,3b,3b,8530a00,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7af28, ebp = 0x13c7af64 --- Tracing command mysqld pid 763 tid 100127 td 0xc8087780 sched_switch(c8087780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8154034) at sleepq_switch+0xc1 sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e msleep(c8154034,c8156068,168,c070b128,7ba,...) at msleep+0x244 kse_release(c8087780,e864ad04) at kse_release+0x1df syscall(3b,3b,3b,8530900,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c76f28, ebp = 0x13c76f64 --- Tracing command mysqld pid 763 tid 100134 td 0xc8086000 sched_switch(c8086000,c8086c00,1) at sched_switch+0x177 mi_switch(1,c8086c00) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b kern_select(c8086000,f,bfbfebf0,0,0,...) at kern_select+0x4bf select(c8086000,e861dd04) at select+0x44 syscall(3b,3b,3b,8536000,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2861b8d3, esp = 0xbfbfeacc, ebp = 0xbfbfeb08 --- Tracing command mysqld pid 763 tid 100131 td 0xc80e3000 sched_switch(c80e3000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(ea845c14) at sleepq_switch+0xc1 sleepq_wait_sig(ea845c14) at sleepq_wait_sig+0x1d msleep(ea845c14,c8156068,168,c070b120,0,...) at msleep+0x26e kern_sigtimedwait(c80e3000,26005,0,0,0,ea845c80,0) at kern_sigtimedwait+0x30b sigtimedwait(c80e3000,ea845d04) at sigtimedwait+0x6b syscall(3b,3b,3b,13c83e00,bf0f5f37,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (345, FreeBSD ELF32, sigtimedwait), eip = 0x284735b7, esp = 0xbf0f5e2c, ebp = 0xbf0f5ee8 --- Tracing command mysqld pid 763 tid 100126 td 0xc80e3480 sched_switch(c80e3480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8156138) at sleepq_switch+0xc1 sleepq_timedwait_sig(c8156138) at sleepq_timedwait_sig+0x1e msleep(c8156138,c8156068,168,c070b11d,7531,...) at msleep+0x244 kse_release(c80e3480,ea84ed04) at kse_release+0x121 syscall(3b,3b,3b,8530800,81,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0xbf9fef2c, ebp = 0xbf9fef88 --- Tracing command sh pid 733 tid 100096 td 0xc8158480 sched_switch(c8158480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8156a78) at sleepq_switch+0xc1 sleepq_wait_sig(c8156a78) at sleepq_wait_sig+0x1d msleep(c8156a78,c8156ae0,15c,c0713046,0) at msleep+0x26e kern_wait(c8158480,ffffffff,ea91cc84,2,0) at kern_wait+0x6bb wait4(c8158480,ea91cd04) at wait4+0x2a syscall(3b,3b,3b,bfbfe678,2d,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2813d51f, esp = 0xbfbfe62c, ebp = 0xbfbfe648 --- Tracing command cron pid 730 tid 100101 td 0xc8332600 sched_switch(c8332600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c0781bec) at sleepq_switch+0xc1 sleepq_timedwait_sig(c0781bec) at sleepq_timedwait_sig+0x1e msleep(c0781bec,0,15c,c070eeb5,ea61,...) at msleep+0x244 kern_nanosleep(c8332600,ea99cccc,ea99ccc4) at kern_nanosleep+0xab nanosleep(c8332600,ea99cd04) at nanosleep+0x4f syscall(3b,3b,3b,8050e6c,3c,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 --- Tracing command snmpd pid 722 tid 100080 td 0xc8153300 sched_switch(c8153300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07cef64,c07cef64,1389,ea8ddb40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07cef64,c07cef40,1389) at cv_timedwait_sig+0x194 kern_select(c8153300,9,bfbfed50,bfbfecd0,bfbfec50,...) at kern_select+0x4a9 select(c8153300,ea8ddd04) at select+0x44 syscall(3b,3b,3b,bfbfecd0,bfbfec38,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x284598d3, esp = 0xbfbfea4c, ebp = 0xbfbfee68 --- Tracing command cron pid 711 tid 100077 td 0xc8153780 sched_switch(c8153780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c0781bec) at sleepq_switch+0xc1 sleepq_timedwait_sig(c0781bec) at sleepq_timedwait_sig+0x1e msleep(c0781bec,0,15c,c070eeb5,ea61,...) at msleep+0x244 kern_nanosleep(c8153780,ea8e6ccc,ea8e6cc4) at kern_nanosleep+0xab nanosleep(c8153780,ea8e6d04) at nanosleep+0x4f syscall(3b,3b,3b,805008c,3c,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 --- Tracing command sendmail pid 705 tid 100060 td 0xc80e3900 sched_switch(c80e3900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c80e2894) at sleepq_switch+0xc1 sleepq_wait_sig(c80e2894) at sleepq_wait_sig+0x1d msleep(c80e2894,c80e28c8,168,c070e01c,0) at msleep+0x26e kern_sigsuspend(c80e3900,0,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c80e3900,ea857d04) at sigsuspend+0x33 syscall(3b,3b,3b,0,bfbfd000,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2827e3fb, esp = 0xbfbfcfec, ebp = 0xbfbfd018 --- Tracing command sendmail pid 699 tid 100051 td 0xc8087480 sched_switch(c8087480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07cef64,c07cef64,1389,e8644b40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07cef64,c07cef40,1389) at cv_timedwait_sig+0x194 kern_select(c8087480,6,bfbfc530,0,0,...) at kern_select+0x4a9 select(c8087480,e8644d04) at select+0x44 syscall(3b,3b,3b,2,5,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x282fc8d3, esp = 0xbfbfc44c, ebp = 0xbfbfd038 --- Tracing command sshd pid 694 tid 100079 td 0xc8153480 sched_switch(c8153480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b kern_select(c8153480,7,807b920,0,0,...) at kern_select+0x4bf select(c8153480,ea8e0d04) at select+0x44 syscall(3b,3b,3b,1,a,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe16c, ebp = 0xbfbfee78 --- Tracing command ntpd pid 669 tid 100104 td 0xc8332180 sched_switch(c8332180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b kern_select(c8332180,a,bfbfed60,0,0,...) at kern_select+0x4bf select(c8332180,ea993d04) at select+0x44 syscall(3b,3b,3b,bfbfed60,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2828a8d3, esp = 0xbfbfed1c, ebp = 0xbfbfedf8 --- Tracing command usbd pid 638 tid 100056 td 0xc80e6000 sched_switch(c80e6000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07cef64,c07cef64,7531,ea863b40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07cef64,c07cef40,7531) at cv_timedwait_sig+0x194 kern_select(c80e6000,6,bfbfeda0,bfbfed20,0,...) at kern_select+0x4a9 select(c80e6000,ea863d04) at select+0x44 syscall(3b,3b,3b,bfbfed20,1e,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2813c8d3, esp = 0xbfbfeccc, ebp = 0xbfbfee78 --- Tracing command accounting pid 591 tid 100065 td 0xc810f780 sched_switch(c810f780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c077f760,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c077f760) at sleepq_timedwait+0x4a msleep(c077f760,0,7f,c0706bf1,3a98,c077f720,c07093bb,213) at msleep+0x259 acct_thread(0,ea89dd38,0,c052cacc,0,...) at acct_thread+0xd5 fork_exit(c052cacc,0,ea89dd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xea89dd6c, ebp = 0 --- Tracing command named pid 574 tid 100083 td 0xc8150d80 sched_switch(c8150d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07cef64,c07cef64,7531,ea8d4b40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07cef64,c07cef40,7531) at cv_timedwait_sig+0x194 kern_select(c8150d80,1a,bfbfed60,bfbfece0,0,...) at kern_select+0x4a9 select(c8150d80,ea8d4d04) at select+0x44 syscall(3b,3b,3b,0,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2835e8d3, esp = 0xbfbfec9c, ebp = 0xbfbfedf8 --- Tracing command syslogd pid 554 tid 100087 td 0xc8150780 sched_switch(c8150780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b kern_select(c8150780,a,80541c0,0,0,...) at kern_select+0x4bf select(c8150780,ea8c8d04) at select+0x44 syscall(3b,3b,3b,80541c4,80541c0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x2814c8d3, esp = 0xbfbfda0c, ebp = 0xbfbfee58 --- Tracing command ipmon pid 537 tid 100039 td 0xc7e05900 sched_switch(c7e05900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c0781bec) at sleepq_switch+0xc1 sleepq_timedwait_sig(c0781bec) at sleepq_timedwait_sig+0x1e msleep(c0781bec,0,15c,c070eeb5,3e9,...) at msleep+0x244 kern_nanosleep(c7e05900,e838fccc,e838fcc4) at kern_nanosleep+0xab nanosleep(c7e05900,e838fd04) at nanosleep+0x4f syscall(3b,3b,3b,2,1,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2812f513, esp = 0xbfbfcd3c, ebp = 0xbfbfcd68 --- Tracing command devd pid 441 tid 100068 td 0xc810f300 sched_switch(c810f300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cef64) at sleepq_switch+0xc1 sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b kern_select(c810f300,5,bfbfe9d0,0,0,...) at kern_select+0x4bf select(c810f300,ea894d04) at select+0x44 syscall(3b,3b,3b,0,8,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x8075917, esp = 0xbfbfe96c, ebp = 0xbfbfee78 --- Tracing command adjkerntz pid 170 tid 100052 td 0xc8087300 sched_switch(c8087300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c808567c) at sleepq_switch+0xc1 sleepq_wait_sig(c808567c) at sleepq_wait_sig+0x1d msleep(c808567c,c80856b0,168,c070e01c,0) at msleep+0x26e kern_sigsuspend(c8087300,0,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c8087300,e8641d04) at sigsuspend+0x33 syscall(3b,3b,3b,0,ffffc7c0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x280bc3fb, esp = 0xbfbfed9c, ebp = 0xbfbfee78 --- Tracing command schedcpu pid 44 tid 100044 td 0xc7e05180 sched_switch(c7e05180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(e8380d04,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(e8380d04) at sleepq_timedwait+0x4a msleep(e8380d04,0,0,c0706bf1,3e8) at msleep+0x259 schedcpu_thread(0,e8380d38,0,c0565cbc,0,...) at schedcpu_thread+0x26 fork_exit(c0565cbc,0,e8380d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8380d6c, ebp = 0 --- Tracing command softdepflush pid 43 tid 100045 td 0xc7e05000 sched_switch(c7e05000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dca74,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c07dca74) at sleepq_timedwait+0x4a msleep(c07dca74,c07dca40,44,c07229e3,3e8) at msleep+0x259 softdep_flush(0,e837dd38,0,c0655db8,0,...) at softdep_flush+0x227 fork_exit(c0655db8,0,e837dd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe837dd6c, ebp = 0 --- Tracing command vnlru pid 42 tid 100046 td 0xc7dbdd80 sched_switch(c7dbdd80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8084860,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c8084860) at sleepq_timedwait+0x4a msleep(c8084860,c07cf900,250,c0717bd5,3e8,c07cf980) at msleep+0x259 vnlru_proc(0,e836dd38,0,c05aa450,0,...) at vnlru_proc+0xdf fork_exit(c05aa450,0,e836dd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe836dd6c, ebp = 0 --- Tracing command syncer pid 41 tid 100047 td 0xc7dbdc00 sched_switch(c7dbdc00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c0766324,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c0766324,0,c07278c1,144,c0717b07,...) at sleepq_wait+0x46 msleep(c0766324,c07dda00,44,c0727902,0) at msleep+0x27d getpbuf(c0766324,c99a1794,0,0,c99a1794,...) at getpbuf+0x44 cluster_wbuild(c99a16cc,4000,0,0,4,...) at cluster_wbuild+0x16b vfs_bio_awrite(dbe7c9c4) at vfs_bio_awrite+0x1cd ffs_syncvnode(c99a16cc,3) at ffs_syncvnode+0x2c1 ffs_sync(c822d000,3,c7dbdc00,c822d000,2,...) at ffs_sync+0x209 sync_fsync(e836acbc) at sync_fsync+0x164 VOP_FSYNC_APV(c0759ec0,e836acbc) at VOP_FSYNC_APV+0x9b sync_vnode(c822e7d8,c7dbdc00) at sync_vnode+0x106 sched_sync(0,e836ad38,0,c05abf64,0,...) at sched_sync+0x1ed fork_exit(c05abf64,0,e836ad38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe836ad6c, ebp = 0 --- Tracing command bufdaemon pid 40 tid 100048 td 0xc7dbda80 sched_switch(c7dbda80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07cf4e0,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c07cf4e0) at sleepq_timedwait+0x4a msleep(c07cf4e0,c07cf500,44,c07163bf,3e8) at msleep+0x259 buf_daemon(0,e8367d38,0,c059d93c,0,...) at buf_daemon+0x108 fork_exit(c059d93c,0,e8367d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8367d6c, ebp = 0 --- Tracing command pagezero pid 39 tid 100049 td 0xc7dbd900 sched_switch(c7dbd900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dda44,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c07dda44) at sleepq_timedwait+0x4a msleep(c07dda44,c07dd4c0,200,c0727a97,493e0,c07dd4c0,0,c0727a79,9d) at msleep+0x259 vm_pagezero(0,e8364d38,0,c06870c4,0,...) at vm_pagezero+0x6c fork_exit(c06870c4,0,e8364d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8364d6c, ebp = 0 --- Tracing command vmdaemon pid 38 tid 100027 td 0xc7dbd180 sched_switch(c7dbd180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dd54c,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c07dd54c,0,c072771a,63e,0,...) at sleepq_wait+0x46 msleep(c07dd54c,c07dd560,68,c07163bf,0,...) at msleep+0x27d vm_daemon(0,e8355d38,0,c06860d8,0,...) at vm_daemon+0x36 fork_exit(c06860d8,0,e8355d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8355d6c, ebp = 0 --- Tracing command pagedaemon pid 37 tid 100028 td 0xc7dbd000 sched_switch(c7dbd000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dd508,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c07dd508) at sleepq_timedwait+0x4a msleep(c07dd508,c07dd4c0,44,c07163bf,1388) at msleep+0x259 vm_pageout(0,e8352d38,0,c0685d60,0,...) at vm_pageout+0x260 fork_exit(c0685d60,0,e8352d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8352d6c, ebp = 0 --- Tracing command irq7: ppc0 pid 36 tid 100029 td 0xc7ce9d80 fork_trampoline() at fork_trampoline Tracing command fdc0 pid 35 tid 100030 td 0xc7ce9c00 sched_switch(c7ce9c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7e1e63c,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c7e1e63c) at sleepq_timedwait+0x4a msleep(c7e1e63c,c7e1e6f0,4c,c0706bf1,3e8) at msleep+0x259 fdc_worker(c7e1e600,c7e1e6f0,0,c07293da,4a1) at fdc_worker+0x1fe fdc_thread(c7e1e600,e6929d38,c7e1e600,c0694bfc,0,...) at fdc_thread+0x54 fork_exit(c0694bfc,c7e1e600,e6929d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6929d6c, ebp = 0 --- Tracing command swi0: sio pid 34 tid 100031 td 0xc7ce9a80 sched_switch(c7ce9a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7e88c10,e6926d38,c7e88c10,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7e88c10,e6926d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6926d6c, ebp = 0 --- Tracing command irq1: atkbd0 pid 33 tid 100032 td 0xc7ce9900 sched_switch(c7ce9900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7e89030,e6923d38,c7e89030,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7e89030,e6923d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6923d6c, ebp = 0 --- Tracing command irq22: fxp0 pid 32 tid 100033 td 0xc7ce9780 sched_switch(c7ce9780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7e772d0,e6920d38,c7e772d0,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7e772d0,e6920d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6920d6c, ebp = 0 --- Tracing command irq20: mly0 pid 31 tid 100034 td 0xc7ce9600 sched_switch(c7ce9600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7dfd920,e691dd38,c7dfd920,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7dfd920,e691dd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe691dd6c, ebp = 0 --- Tracing command irq15: ata1 pid 30 tid 100035 td 0xc7ce9480 fork_trampoline() at fork_trampoline Tracing command irq14: ata0 pid 29 tid 100036 td 0xc7e05d80 sched_switch(c7e05d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7e07300,e8398d38,c7e07300,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7e07300,e8398d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8398d6c, ebp = 0 --- Tracing command usb1 pid 28 tid 100037 td 0xc7e05c00 sched_switch(c7e05c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7dae210,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c7dae210) at sleepq_timedwait+0x4a msleep(c7dae210,0,5c,c07057e6,ea60,c7debbc0) at msleep+0x259 usb_event_thread(c7debbc0,e8395d38,c7debbc0,c050cf38,0,...) at usb_event_thread+0x70 fork_exit(c050cf38,c7debbc0,e8395d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8395d6c, ebp = 0 --- Tracing command irq17: ehci0 pid 27 tid 100017 td 0xc7ce5900 fork_trampoline() at fork_trampoline Tracing command usbtask pid 26 tid 100018 td 0xc7ce5780 sched_switch(c7ce5780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c077ec04,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c077ec04,0,c054c4d4,c0781300,c07057cb,...) at sleepq_wait+0x46 msleep(c077ec04,0,5c,c07057ed,0,...) at msleep+0x27d usb_task_thread(0,e6902d38,0,c050cfc8,0,...) at usb_task_thread+0x47 fork_exit(c050cfc8,0,e6902d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6902d6c, ebp = 0 --- Tracing command usb0 pid 25 tid 100019 td 0xc7ce5600 sched_switch(c7ce5600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7db4210,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c7db4210) at sleepq_timedwait+0x4a msleep(c7db4210,0,5c,c07057e6,ea60,c7dea580) at msleep+0x259 usb_event_thread(c7dea580,e68ffd38,c7dea580,c050cf38,0,...) at usb_event_thread+0x70 fork_exit(c050cf38,c7dea580,e68ffd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68ffd6c, ebp = 0 --- Tracing command irq16: ohci0 pid 24 tid 100020 td 0xc7ce5480 fork_trampoline() at fork_trampoline Tracing command irq9: acpi0 pid 23 tid 100021 td 0xc7ce5300 fork_trampoline() at fork_trampoline Tracing command swi5: + pid 22 tid 100022 td 0xc7ce5180 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 21 tid 100023 td 0xc7dbd780 sched_switch(c7dbd780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d43d80,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d43d80,0,c07113b9,49,c8053ab0,...) at sleepq_wait+0x46 msleep(c7d43d80,c7d43d9c,0,c0706bf1,0) at msleep+0x27d taskqueue_thread_loop(c07872a8,e8361d38,c07872a8,c05753dc,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05753dc,c07872a8,e8361d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8361d6c, ebp = 0 --- Tracing command swi6: Giant taskq pid 20 tid 100024 td 0xc7dbd600 sched_switch(c7dbd600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8b80,e835ed38,c7cc8b80,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7cc8b80,e835ed38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe835ed6c, ebp = 0 --- Tracing command swi6: task queue pid 19 tid 100025 td 0xc7dbd480 sched_switch(c7dbd480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8b90,e835bd38,c7cc8b90,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7cc8b90,e835bd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe835bd6c, ebp = 0 --- Tracing command kqueue taskq pid 9 tid 100026 td 0xc7dbd300 sched_switch(c7dbd300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d73100,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d73100,0,c07113b9,49,8,...) at sleepq_wait+0x46 msleep(c7d73100,c7d7311c,0,c0706bf1,0) at msleep+0x27d taskqueue_thread_loop(c077f940,e8358d38,c077f940,c05753dc,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05753dc,c077f940,e8358d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe8358d6c, ebp = 0 --- Tracing command acpi_task_2 pid 8 tid 100008 td 0xc7ce3300 sched_switch(c7ce3300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d73180,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d73180,0,c07113b9,49,8,...) at sleepq_wait+0x46 msleep(c7d73180,c7d7319c,0,c0706bf1,0) at msleep+0x27d taskqueue_thread_loop(c077bb08,e68dbd38,c077bb08,c05753dc,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05753dc,c077bb08,e68dbd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68dbd6c, ebp = 0 --- Tracing command acpi_task_1 pid 7 tid 100009 td 0xc7ce3180 sched_switch(c7ce3180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d73180,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d73180,0,c07113b9,49,8,...) at sleepq_wait+0x46 msleep(c7d73180,c7d7319c,0,c0706bf1,0) at msleep+0x27d taskqueue_thread_loop(c077bb08,e68d8d38,c077bb08,c05753dc,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05753dc,c077bb08,e68d8d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68d8d6c, ebp = 0 --- Tracing command acpi_task_0 pid 6 tid 100010 td 0xc7ce3000 sched_switch(c7ce3000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7d73180,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c7d73180,0,c07113b9,49,c7ddf1e0,...) at sleepq_wait+0x46 msleep(c7d73180,c7d7319c,0,c0706bf1,0) at msleep+0x27d taskqueue_thread_loop(c077bb08,e68d5d38,c077bb08,c05753dc,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05753dc,c077bb08,e68d5d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68d5d6c, ebp = 0 --- Tracing command swi2: cambio pid 18 tid 100011 td 0xc7ce9300 sched_switch(c7ce9300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8bc0,e6917d38,c7cc8bc0,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7cc8bc0,e6917d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6917d6c, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100012 td 0xc7ce9180 sched_switch(c7ce9180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c0776004,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1 sleepq_wait(c0776004,0,c054c4d4,c0781300,c06ecd75,...) at sleepq_wait+0x46 msleep(c0776004,0,4c,c06ecd90,0) at msleep+0x27d xpt_scanner_thread(0,e6914d38,0,c0443a54,0,...) at xpt_scanner_thread+0x35 fork_exit(c0443a54,0,e6914d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6914d6c, ebp = 0 --- Tracing command yarrow pid 17 tid 100013 td 0xc7ce9000 sched_switch(c7ce9000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c077cc00,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c077cc00) at sleepq_timedwait+0x4a msleep(c077cc00,0,0,c0706bf1,64) at msleep+0x259 random_kthread(0,e6911d38,0,c04f5f30,0,...) at random_kthread+0x180 fork_exit(c04f5f30,0,e6911d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6911d6c, ebp = 0 --- Tracing command g_down pid 4 tid 100014 td 0xc7ce5d80 sched_switch(c7ce5d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c077f0a8,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c077f0a8) at sleepq_timedwait+0x4a msleep(c077f0a8,c077ef88,24c,c0706bf1,64) at msleep+0x259 g_io_schedule_down(c7ce5d80) at g_io_schedule_down+0x56 g_down_procbody(0,e690ed38,0,c051eeb8,0,...) at g_down_procbody+0x5a fork_exit(c051eeb8,0,e690ed38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe690ed6c, ebp = 0 --- Tracing command g_up pid 3 tid 100015 td 0xc7ce5c00 sched_switch(c7ce5c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c077f0a4,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c077f0a4) at sleepq_timedwait+0x4a msleep(c077f0a4,c077efc8,24c,c0706bf1,64) at msleep+0x259 g_io_schedule_up(c7ce5c00) at g_io_schedule_up+0x127 g_up_procbody(0,e690bd38,0,c051ee58,0,...) at g_up_procbody+0x5a fork_exit(c051ee58,0,e690bd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe690bd6c, ebp = 0 --- Tracing command g_event pid 2 tid 100016 td 0xc7ce5a80 sched_switch(c7ce5a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c077f09c,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1 sleepq_timedwait(c077f09c) at sleepq_timedwait+0x4a msleep(c077f09c,0,4c,c0706bf1,64) at msleep+0x259 g_event_procbody(0,e6908d38,0,c051ef18,0,...) at g_event_procbody+0x9e fork_exit(c051ef18,0,e6908d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6908d6c, ebp = 0 --- Tracing command swi3: vm pid 16 tid 100000 td 0xc7ce5000 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 15 tid 100001 td 0xc7ce3d80 sched_switch(c7ce3d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8870,e68f0d38,c7cc8870,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7cc8870,e68f0d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68f0d6c, ebp = 0 --- Tracing command swi1: net pid 14 tid 100002 td 0xc7ce3c00 sched_switch(c7ce3c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c7cc8880,e68edd38,c7cc8880,c0540f8c,0,...) at ithread_loop+0xda fork_exit(c0540f8c,c7cc8880,e68edd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68edd6c, ebp = 0 --- Tracing command idle: cpu0 pid 13 tid 100003 td 0xc7ce3a80 sched_switch(c07120f0,6a9,c07cee5c,e68eac40,c05798ee,...) at sched_switch+0x177 (null)(c0712111,50000,c7ccb010,c07819d0,0,...) at 0 __func__.0(c054cd38,0,2041,1,3db17,...) at 0xc0712111 (null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa Tracing command idle: cpu1 pid 12 tid 100004 td 0xc7ce3900 kdb_enter(c072a005) at kdb_enter+0x2b siointr1(c7e8c800,c07df9c0,0,c0729e0b,56e,...) at siointr1+0xce siointr(c7e8c800) at siointr+0x21 intr_execute_handlers(c7cd74c8,e68e7c94,4,e68e7cd8,c06b1b23,...) at intr_execute_handlers+0xe1 lapic_handle_intr(38) at lapic_handle_intr+0x2e Xapic_isr1() at Xapic_isr1+0x33 --- interrupt, eip = 0xc06aaa4d, esp = 0xe68e7cd8, ebp = 0xe68e7cd8 --- acpi_cpu_c1(c07812c0,2,e68e7cf8,2,c7ce2860,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(e68e7d0c,c05402e9,c054028c,e68e7d24,c05400cc,...) at acpi_cpu_idle+0x15d cpu_idle(c054028c,e68e7d24,c05400cc,0,e68e7d38,...) at cpu_idle+0x28 idle_proc(0,e68e7d38,0,c054028c,0,...) at idle_proc+0x5d fork_exit(c054028c,0,e68e7d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe68e7d6c, ebp = 0 --- Tracing command idle: cpu2 pid 11 tid 100005 td 0xc7ce3780 sched_switch(c07887c0,0,ffc06034) at sched_switch+0x177 mi_switch(c054c729,c07cee58,e68e4c1c,c057a284,c07887c0,...) at mi_switch+0x270 end(c0712111,50000,c7ccb010,c07819d0,0,...) at 0xe68e4bf0 __func__.0(c054cd38,0,2041,1,3db17,...) at 0xc0712111 (null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa Tracing command idle: cpu3 pid 10 tid 100006 td 0xc7ce3600 sched_switch(28,12,c06b1eb1,c07f95b0,8,...) at sched_switch+0x177 Tracing command init pid 1 tid 100007 td 0xc7ce3480 sched_switch(c7ce3480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c7ce8000) at sleepq_switch+0xc1 sleepq_wait_sig(c7ce8000) at sleepq_wait_sig+0x1d msleep(c7ce8000,c7ce8068,15c,c0713046,0) at msleep+0x26e kern_wait(c7ce3480,ffffffff,e68dec84,0,0) at kern_wait+0x6bb wait4(c7ce3480,e68ded04) at wait4+0x2a syscall(3b,3b,3b,0,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8052f47, esp = 0xbfbfed7c, ebp = 0xbfbfed98 --- Tracing command swapper pid 0 tid 0 td 0xc077f3c0 sched_switch(c077f3c0,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 scheduler(0,c1ec00,c1e000,0,c0440585,...) at scheduler+0x199 mi_startup() at mi_startup+0x96 begin() at begin+0x2c db> ps pid ppid pgrp uid state wmesg wchan cmd 897 784 897 0 Ss ufs 0xc820f46c inetd 894 893 893 0 S ufs 0xc82481b4 sh 893 892 893 0 Ss wait 0xc8964c90 sh 892 711 711 0 S piperd 0xc8107990 cron 891 784 891 0 Ss biord 0xdbdb3fdc inetd 851 848 851 0 S+ getblk 0xdbfdafa4 mc 848 847 848 0 S+ pause 0xc851dcc4 csh 847 842 847 1001 S+ wait 0xc851da78 su 842 841 842 1001 Ss+ pause 0xc835224c csh 841 837 837 1001 S select 0xc07cef64 sshd 837 694 837 0 Ss sbwait 0xc827720c sshd 836 764 764 65534 S lockf 0xc80bba80 httpd 835 764 764 65534 S lockf 0xc8113680 httpd 834 764 764 65534 S lockf 0xc80c0840 httpd 833 764 764 65534 S lockf 0xc8112880 httpd 832 764 764 65534 S lockf 0xc7ea6080 httpd 831 764 764 65534 S lockf 0xc80c08c0 httpd 830 764 764 65534 S lockf 0xc81124c0 httpd 829 764 764 65534 S lockf 0xc80c0800 httpd 828 764 764 65534 S lockf 0xc80bc840 httpd 827 764 764 65534 S lockf 0xc80bc600 httpd 826 764 764 65534 S lockf 0xc80c0980 httpd 825 764 764 65534 S lockf 0xc7ea8cc0 httpd 824 764 764 65534 S lockf 0xc80bbb00 httpd 823 764 764 65534 S lockf 0xc80c15c0 httpd 822 764 764 65534 S lockf 0xc7ea8d00 httpd 821 764 764 65534 S lockf 0xc8112180 httpd 820 764 764 65534 S lockf 0xc8113700 httpd 819 764 764 65534 S lockf 0xc81123c0 httpd 818 764 764 65534 S lockf 0xc7ea8dc0 httpd 817 764 764 65534 S lockf 0xc80c1580 httpd 816 764 764 65534 S lockf 0xc7ea6b80 httpd 815 764 764 65534 S lockf 0xc80c0a00 httpd 814 764 764 65534 S lockf 0xc80c0600 httpd 813 764 764 65534 S lockf 0xc7ea8d40 httpd 812 764 764 65534 S lockf 0xc80bba00 httpd 811 764 764 65534 S lockf 0xc80bc7c0 httpd 810 764 764 65534 S lockf 0xc8112240 httpd 809 764 764 65534 S lockf 0xc80c1e80 httpd 808 764 764 65534 S lockf 0xc8112300 httpd 807 764 764 65534 S select 0xc07cef64 httpd 806 1 806 0 Ss+ vnread 0xdbd4b0bc getty 805 1 805 0 Ss+ ttyin 0xc7e9c410 getty 804 1 804 0 Ss+ ttyin 0xc7ea4010 getty 803 1 803 0 Ss+ ttyin 0xc7ea3c10 getty 802 1 802 0 Ss+ ttyin 0xc7ea4c10 getty 801 1 801 0 Ss+ ttyin 0xc7ea4810 getty 800 1 800 0 Ss+ ttyin 0xc7ea2c10 getty 799 1 799 0 Ss+ ttyin 0xc7ea2810 getty 798 1 798 0 Ss+ ttyin 0xc7ea3810 getty 784 1 784 0 Ss select 0xc07cef64 inetd 764 1 764 0 Ss select 0xc07cef64 httpd 763 733 45 509 S+ (threaded) mysqld 100133 S kserel 0xc8154034 mysqld 100135 S kserel 0xc8154034 mysqld 100091 S kserel 0xc8154034 mysqld 100127 S kserel 0xc8154034 mysqld 100134 S select 0xc07cef64 mysqld 100131 S sigwait 0xea845c14 mysqld 100126 S ksesigwa 0xc8156138 mysqld 733 1 45 0 S+ wait 0xc8156a78 sh 730 1 730 0 Ss nanslp 0xc0781bec cron 722 1 721 0 S select 0xc07cef64 snmpd 711 1 711 0 Ss nanslp 0xc0781bec cron 705 1 705 25 Ss pause 0xc80e2894 sendmail 699 1 699 0 Ss select 0xc07cef64 sendmail 694 1 694 0 Ss select 0xc07cef64 sshd 669 1 669 0 Ss select 0xc07cef64 ntpd 638 1 638 0 Ss select 0xc07cef64 usbd 591 0 0 0 SL - 0xc077f760 [accounting] 574 1 574 53 Ss select 0xc07cef64 named 554 1 554 0 Ss select 0xc07cef64 syslogd 537 1 537 0 Ss nanslp 0xc0781bec ipmon 441 1 441 0 Ss select 0xc07cef64 devd 170 1 170 0 Ss pause 0xc808567c adjkerntz 44 0 0 0 SL - 0xe8380d04 [schedcpu] 43 0 0 0 SL sdflush 0xc07dca74 [softdepflush] 42 0 0 0 SL vlruwt 0xc8084860 [vnlru] 41 0 0 0 SL wswbuf0 0xc0766324 [syncer] 40 0 0 0 SL psleep 0xc07cf4e0 [bufdaemon] 39 0 0 0 SL pgzero 0xc07dda44 [pagezero] 38 0 0 0 SL psleep 0xc07dd54c [vmdaemon] 37 0 0 0 SL psleep 0xc07dd508 [pagedaemon] 36 0 0 0 WL [irq7: ppc0] 35 0 0 0 SL - 0xc7e1e63c [fdc0] 34 0 0 0 WL [swi0: sio] 33 0 0 0 WL [irq1: atkbd0] 32 0 0 0 WL [irq22: fxp0] 31 0 0 0 WL [irq20: mly0] 30 0 0 0 WL [irq15: ata1] 29 0 0 0 WL [irq14: ata0] 28 0 0 0 SL usbevt 0xc7dae210 [usb1] 27 0 0 0 WL [irq17: ehci0] 26 0 0 0 SL usbtsk 0xc077ec04 [usbtask] 25 0 0 0 SL usbevt 0xc7db4210 [usb0] 24 0 0 0 WL [irq16: ohci0] 23 0 0 0 WL [irq9: acpi0] 22 0 0 0 WL [swi5: +] 21 0 0 0 SL - 0xc7d43d80 [thread taskq] 20 0 0 0 WL [swi6: Giant taskq] 19 0 0 0 WL [swi6: task queue] 9 0 0 0 SL - 0xc7d73100 [kqueue taskq] 8 0 0 0 SL - 0xc7d73180 [acpi_task_2] 7 0 0 0 SL - 0xc7d73180 [acpi_task_1] 6 0 0 0 SL - 0xc7d73180 [acpi_task_0] 18 0 0 0 WL [swi2: cambio] 5 0 0 0 SL ccb_scan 0xc0776004 [xpt_thrd] 17 0 0 0 SL - 0xc077cc00 [yarrow] 4 0 0 0 SL - 0xc077f0a8 [g_down] 3 0 0 0 SL - 0xc077f0a4 [g_up] 2 0 0 0 SL - 0xc077f09c [g_event] 16 0 0 0 WL [swi3: vm] 15 0 0 0 WL [swi4: clock sio] 14 0 0 0 WL [swi1: net] 13 0 0 0 RL CPU 0 [idle: cpu0] 12 0 0 0 RL CPU 1 [idle: cpu1] 11 0 0 0 RL CPU 2 [idle: cpu2] 10 0 0 0 RL CPU 3 [idle: cpu3] 1 0 1 0 SLs wait 0xc7ce8000 [init] 0 0 0 0 WLs [swapper] db> -- Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE Phone: +7 4732 539880 Fax: +7 4732 531415 http://www.vsi.ru CenterTelecom Voronezh ISP http://isp.vsi.ru
Here is a kernel dump: ftp://oleg.vsi.ru/private/vmcore.0.zip It was saved for another static hang state. -- Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE Phone: +7 4732 539880 Fax: +7 4732 531415 http://www.vsi.ru CenterTelecom Voronezh ISP http://isp.vsi.ru
On Thu, Nov 01, 2007 at 07:54:39PM +0300, Oleg Derevenetz wrote: > Here is a kernel dump: > > ftp://oleg.vsi.ru/private/vmcore.0.zip > > It was saved for another static hang state. I already analyzed one dump for you.
Responsible Changed From-To: freebsd-bugs->freebsd-fs Over to maintainer(s).
batch change: For bugs that match the following - Status Is In progress AND - Untouched since 2018-01-01. AND - Affects Base System OR Documentation DO: Reset to open status. Note: I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
This particular bug was fixed by the rewrite of the code to deal with low buffer memory situations. Some other low-memory issues have arisen that are discussed in later bugs such as https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224292