Bug 104406 - [ufs] Processes get stuck in "ufs" state under persistent CPU load
Summary: [ufs] Processes get stuck in "ufs" state under persistent CPU load
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 7.0-CURRENT
Hardware: Any Any
: Normal Affects Only Me
Assignee: freebsd-fs (Nobody)
URL:
Keywords:
Depends on: 224292
Blocks:
  Show dependency treegraph
 
Reported: 2006-10-14 14:00 UTC by Sergey Zaharchenko
Modified: 2018-05-29 14:23 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sergey Zaharchenko 2006-10-14 14:00:30 UTC
When a single process loads the CPU for a long(*) time, other processes
which want to access to the filesystem get stuck in the "ufs" state when
trying to do that. Other processes which don't need to access the
filesystem (like top, etc.) proceed normally.

The owner UID, nice- and idprio- status of the offending (or offended)
process do not matter.

It seems essential that a single process is working all the time (e.g.
two hours of compilation don't show up any errors, because there are
many processes).

Example top outputs for this situation:

last pid:  9798;  load averages:  2.00,  1.96,  1.72    up 0+02:29:59  11:02:17
113 processes: 3 running, 109 sleeping, 1 zombie
CPU states:  0.0% user, 96.2% nice,  0.4% system,  3.4% interrupt,  0.0% idle
Mem: 134M Active, 193M Inact, 88M Wired, 316K Cache, 59M Buf, 70M Free
Swap: 4097M Total, 64M Used, 4033M Free, 1% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 2661 df            1 139   20 63532K 58468K RUN    107:21 97.46% generic_slave
 8035 df            1  96    0  7608K  2752K select   0:00  0.05% mc
  702 root          1  96    0  5200K   792K select   0:41  0.00% syslogd
 1245 root          1  -4    0  5152K   572K ufs      0:35  0.00% tail
  912 squid         1  -4    0 13928K  4288K ufs      0:31  0.00% squid
 1163 df            1  -4    0  9716K  2512K ufs      0:27  0.00% fetchmail
 2600 root          1 -32    0  5516K  2052K RUN      0:13  0.00% top
 1042 mysql         6  20    0 59128K  1956K kserel   0:06  0.00% mysqld
 9338 df            1  -4    0 15988K  9812K ufs      0:06  0.00% links
 1179 root          1  96    0  5200K   112K select   0:02  0.00% moused

last pid:  2739;  load averages:  2.00,  1.95,  1.57    up 0+00:31:39  16:46:48
91 processes:  3 running, 87 sleeping, 1 zombie
CPU states: 97.8% user,  0.0% nice,  0.4% system,  1.9% interrupt,  0.0% idle
Mem: 120M Active, 29M Inact, 32M Wired, 17M Cache, 15M Buf, 287M Free
Swap: 4097M Total, 4097M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 1352 df            1 132    0  2248K   896K RUN     25:58 97.41% testcase
 1580 df            1  96    0  7608K  2696K select   0:09  0.00% mc
  970 squid         1  -4    0 13928K  8108K ufs      0:03  0.00% squid
 1316 root          1 -32    0  5516K  1608K RUN      0:03  0.00% top
 1238 df            1  -4    0  7668K  1884K ufs      0:02  0.00% fetchmail
 1110 mysql         6  20    0 59128K 51600K kserel   0:01  0.00% mysqld
  754 root          1  96    0  5200K  1088K select   0:01  0.00% syslogd
 1322 root          1  96    0 22656K 16968K select   0:01  0.00% Xorg
  868 root          1   8    0  9232K  4748K nanslp   0:00  0.00% httpd
 1126 news          1   8    0  5484K  1308K wait     0:00  0.00% sh
 1317 root          1  -4    0  5152K   684K ufs      0:00  0.00% tail
 1144 news          1   4    4  7816K  3396K sbwait   0:00  0.00% perl5.8.8
 1120 news          1  -4    0   150M 12828K ufs      0:00  0.00% innd
 1351 df            1  20    0  4032K  1784K pause    0:00  0.00% csh
  993 squid         1  -4    0  5640K   824K msgwai   0:00  0.00% diskd
 1699 df            1   8    0  5244K  3104K ppwait   0:00  0.00% csh
 1182 root          1   8    0  5200K  1100K nanslp   0:00  0.00% cron

(*) for values of `long' from 10 minutes to 2 hours for me.

Fix: 

I don't know the fix, but an offending process can be stopped with 'kill
-STOP' and continued with 'kill -CONT', which allows other processes to
access the filesystem (until another such failure occurs). Periodic
stopping and starting processes might count as a lousy workaround.
How-To-Repeat: 
A program to generate the necessary load can be quite simple, like

int
main(void)
{
	/* Crunch some numbers (really meaningless) */
	unsigned u=1;
	while (1)
	{
		u*=0x8088405;
	}
}

Compile and run it, run `top', and wait for a long (see above) time.
Browse directories with `ls' from time to time on a different terminal.
See `ls' hang at some time. View the `top' terminal.
Comment 1 Oleg Derevenetz 2007-03-07 02:10:59 UTC
I have the problem with the same symptoms on SMP FreeBSD 6.2-STABLE with 
dual AMD Opteron(tm) Processor 850. Sometimes (apparently when CPU load 
suddenly goes up) all processes that interacts with disk gets stuck in "ufs" 
state, but in my case SIGSTOP/SIGCONT seemingly does not help.

uname -a output:

FreeBSD serv2.vsi.ru 6.2-STABLE FreeBSD 6.2-STABLE #2: Sat Mar  3 01:59:08 
MSK 2007     oleg@serv2.vsi.ru:/usr/obj/usr/src/sys/serv2  i386

dmesg.boot:

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-STABLE #2: Sat Mar  3 01:59:08 MSK 2007
    oleg@serv2.vsi.ru:/usr/obj/usr/src/sys/serv2
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Opteron(tm) Processor 850 (2389.26-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f51  Stepping = 1
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
  AMD Features2=0x1<LAHF>
real memory  = 8589934592 (8192 MB)
avail memory = 8350457856 (7963 MB)
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 24-27 on motherboard
ioapic2 <Version 1.1> irqs 28-31 on motherboard
ioapic3 <Version 1.1> irqs 32-35 on motherboard
ioapic4 <Version 1.1> irqs 36-39 on motherboard
ioapic5 <Version 1.1> irqs 40-43 on motherboard
ioapic6 <Version 1.1> irqs 44-47 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD    XSDT> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
unknown: I/O range not supported
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0xf008-0xf00b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0xf000-0xf07f,0xf080-0xf0ff 
iomem 0xd8000-0xdbfff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci1: <ACPI PCI bus> on pcib1
ohci0: <OHCI (generic) USB controller> mem 0xfc900000-0xfc900fff irq 19 at 
device 0.0 on pci1
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1: <OHCI (generic) USB controller> mem 0xfc901000-0xfc901fff irq 19 at 
device 0.1 on pci1
ohci1: [GIANT-LOCKED]
usb1: OHCI version 1.0, legacy support
usb1: SMM does not respond, resetting
usb1: <OHCI (generic) USB controller> on ohci1
usb1: USB revision 1.0
uhub1: AMD OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
pci1: <display, VGA> at device 5.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <AMD 8111 UDMA133 controller> port 
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1000-0x100f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <bridge> at device 7.3 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> at device 10.0 on pci0
pci2: <ACPI PCI bus> on pcib2
bge0: <Broadcom BCM5704 A3, ASIC rev. 0x2003> mem 
0xfe010000-0xfe01ffff,0xfe000000-0xfe00ffff irq 25 at device 2.0 on pci2
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 
1000baseTX-FDX, auto
bge0: Ethernet address: 00:09:3d:13:fd:00
bge1: <Broadcom BCM5704 A3, ASIC rev. 0x2003> mem 
0xfe030000-0xfe03ffff,0xfe020000-0xfe02ffff irq 26 at device 2.1 on pci2
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 
1000baseTX-FDX, auto
bge1: Ethernet address: 00:09:3d:13:fd:01
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 
0xfe050000-0xfe05ffff,0xfe040000-0xfe04ffff irq 27 at device 4.0 on pci2
mpt0: [GIANT-LOCKED]
mpt0: MPI Version=1.2.15.0
mpt0: Capabilities: ( RAID-1E RAID-1 SAFTE )
mpt0: 0 Active Volumes (1 Max)
mpt0: 0 Hidden Drive Members (6 Max)
pci0: <base peripheral, interrupt controller> at device 10.1 (no driver 
attached)
pcib3: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pci0: <base peripheral, interrupt controller> at device 11.1 (no driver 
attached)
pcib4: <ACPI Host-PCI bridge> iomem 
0xfe301000-0xfe301fff,0xfe303000-0xfe303fff,0xfe305000-0xfe305fff,0xfe307000-0xfe307fff 
on acpi0
pci32: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> mem 0xfe300000-0xfe300fff irq 32 at device 1.0 
on pci32
pci33: <ACPI PCI bus> on pcib5
pci32: <base peripheral, interrupt controller> at device 1.1 (no driver 
attached)
pcib6: <ACPI PCI-PCI bridge> mem 0xfe302000-0xfe302fff irq 36 at device 2.0 
on pci32
pci34: <ACPI PCI bus> on pcib6
pci32: <base peripheral, interrupt controller> at device 2.1 (no driver 
attached)
pcib7: <ACPI PCI-PCI bridge> mem 0xfe304000-0xfe304fff irq 40 at device 3.0 
on pci32
pci35: <ACPI PCI bus> on pcib7
pci32: <base peripheral, interrupt controller> at device 3.1 (no driver 
attached)
pcib8: <ACPI PCI-PCI bridge> mem 0xfe306000-0xfe306fff irq 44 at device 4.0 
on pci32
pci36: <ACPI PCI bus> on pcib8
pci32: <base peripheral, interrupt controller> at device 4.1 (no driver 
attached)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on 
acpi0
sio0: type 16550A
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 
0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcafff,0xcb000-0xcefff on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
IP Filter: v4.1.13 initialized.  Default = block all, Logging = enabled
Waiting 5 seconds for SCSI devices to settle
acd0: DMA limited to UDMA33, controller found non-ATA66 cable
acd0: DVDROM <MATSHITADVD-ROM SR-8178/PZ21> at ata1-master UDMA33
ses0 at mpt0 bus 0 target 6 lun 0
ses0: <SDR GEM318P 1> Fixed Processor SCSI-2 device
ses0: 3.300MB/s transfers
ses0: SAF-TE Compliant Device
SMP: AP CPU #1 Launched!
da0 at mpt0 bus 0 target 0 lun 0
da0: <SEAGATE ST373207LC 0003> Fixed Direct Access SCSI-3 device
da0: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing 
Enabled
da0: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C)
da1 at mpt0 bus 0 target 2 lun 0
da1: <SEAGATE ST336807LC 0C01> Fixed Direct Access SCSI-3 device
da1: 320.000MB/s transfers (160.000MHz, offset 63, 16bit), Tagged Queueing 
Enabled
da1: 35003MB (71687372 512 byte sectors: 255H 63S/T 4462C)
Trying to mount root from ufs:/dev/da0s1a
Accounting enabled

Any ideas on how to debug this ?

--
Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
Phone: +7 4732 539880
Fax:   +7 4732 531415         http://www.vsi.ru
CenterTelecom Voronezh ISP    http://isp.vsi.ru
Comment 2 Jeff Wheelhouse 2007-06-05 21:26:26 UTC
I believe we have also experienced this bug (or a very similar one) on 
our 8-core amd64 systems under 6.2-RELEASE-p4.

In our case, "top" shows that the system is 100% CPU utilized, with the 
vast majority of it as "system" time.  (Ordinarily the system

In the last case, we ended up with about 200 Apache processes that 
looked like this in ps:

   UID   PID  PPID CPU PRI NI   VSZ   RSS MWCHAN STAT  TT       TIME COMMAND
25000 27121 26860 1977  -4  5 146324 33732 ufs    DN    ??    0:03.75 httpd
25000 27147 37257 1994  -4  5 153748 29280 ufs    DN    ??    0:03.72 httpd
25000 27157 36912 1805  -4  5 150756 26592 ufs    DN    ??    0:02.91 httpd
25000 27224 27030 1845  -4  5 137536 24804 ufs    DN    ??    0:01.25 httpd
25000 27274 26794 1829  -4  5 148140 35416 ufs    DN    ??    0:02.90 httpd

Once a process gets "stuck" in WCHAN ufs, it's blocked indefinitely, as 
described here, or at least so slow as to be indistinguishable from 
stuck.  (Typical wait channels for our httpds are accept or kqread, as 
one would expect.)

Each process in this state counts against the load average, so we often 
see load averages north of 200 when this is occurring.  (Typical load 
average is below 2.)

Kill enough processes (or possibly enough to hit the "right" process) 
and everything picks up again right where it left off.

I also have no idea how to debug this.

Thanks,
Jeff

-- 
Jeff Wheelhouse
jdw@wheelhouse.org
Comment 3 Oleg Derevenetz 2007-10-19 14:36:27 UTC
This problem experiences on our another AMD64 machine (also with 6-STABLE). When we copy large amount of small files using mc from 
FTP to local filesystem, after some time mc hangs in "wdrain" state, and all other processes that need to access filesystem are 
hangs in "ufs" state. There are some debug stuff:

uname -a:

FreeBSD serv13.vsi.ru 6.2-STABLE FreeBSD 6.2-STABLE #1: Fri Oct 19 16:28:07 MSD 2007 
oleg@serv13.vsi.ru:/usr/obj/usr/src/sys/serv13  i386

dmesg:

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-STABLE #1: Fri Oct 19 16:28:07 MSD 2007
    oleg@serv13.vsi.ru:/usr/obj/usr/src/sys/serv13
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Dual-Core AMD Opteron(tm) Processor 2212 (2010.31-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x40f12  Stepping = 2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x2001<SSE3,CX16>
  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x1f<LAHF,CMP,SVM,ExtAPIC,CR8>
  Cores per package: 2
real memory  = 3220176896 (3071 MB)
avail memory = 3149598720 (3003 MB)
ACPI APIC Table: <PTLTD          APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
ioapic0 <Version 1.1> irqs 0-23 on motherboard
ioapic1 <Version 1.1> irqs 24-47 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <memory, RAM> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> port 0x1c00-0x1c7f at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xc0040000-0xc0040fff irq 16 at device 2.0 on pci0
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: SMM does not respond, resetting
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 10 ports with 10 removable, self powered
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xc0041000-0xc00410ff irq 17 at device 2.1 on pci0
ehci0: [GIANT-LOCKED]
usb1: EHCI version 1.0
usb1: companion controller, 10 ports each: usb0
usb1: <EHCI (generic) USB 2.0 controller> on ehci0
usb1: USB revision 2.0
uhub1: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub1: 10 ports with 10 removable, self powered
atapci0: <nVidia nForce MCP55 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3480-0x348f at device 4.0 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pcib1: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <PCI-PCI bridge> at device 4.0 on pci1
pci2: <PCI bus> on pcib2
mly0: <Mylex AcceleRAID 170> mem 0xc0600000-0xc0601fff irq 20 at device 4.1 on pci1
mly0: [GIANT-LOCKED]
mly0: AcceleRAID 170  , 1 channel, firmware 6.00-7-00 (20001214), 32MB RAM
pci0: <bridge> at device 8.0 (no driver attached)
pci0: <bridge> at device 9.0 (no driver attached)
pcib3: <ACPI PCI-PCI bridge> at device 13.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
fxp0: <Intel 82559 Pro/100 Ethernet> port 0x4000-0x403f mem 0xc0300000-0xc0300fff,0xc0200000-0xc02fffff irq 22 at device 9.0 on pci4
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:d0:b7:1c:80:7e
pcib5: <ACPI PCI-PCI bridge> mem 0xc0100000-0xc010007f irq 21 at device 0.1 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci128: <ACPI PCI bus> on pcib6
pci128: <memory, RAM> at device 0.0 (no driver attached)
pci128: <memory, RAM> at device 1.0 (no driver attached)
pci128: <serial bus, SMBus> at device 1.1 (no driver attached)
pcib7: <ACPI PCI-PCI bridge> at device 13.0 on pci128
pci129: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 15.0 on pci128
pci130: <ACPI PCI bus> on pcib8
pci130: <display, VGA> at device 0.0 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcefff,0xcf000-0xd07ff on isa0
ppc0: <Parallel port> at port 0x278-0x27f irq 7 on isa0
ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
IP Filter: v4.1.13 initialized.  Default = block all, Logging = enabled
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, default to accept, logging disabled
acd0: CDROM <NEC CD-ROM CD-3002A/C000> at ata0-master UDMA33
da0 at mly0 bus 1 target 0 lun 0
da0: <RAID 7 online > Fixed Direct Access SCSI-3 device
da0: 135.168MB/s transfers
da0: 34712MB (71090176 512 byte sectors: 255H 63S/T 4425C)
da1 at mly0 bus 1 target 1 lun 0
da1: <RAID 7 online > Fixed Direct Access SCSI-3 device
da1: 135.168MB/s transfers
da1: 34712MB (71090176 512 byte sectors: 255H 63S/T 4425C)
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
(da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da1:mly0:1:1:0): Sense Error Code 0x0
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
(da0:mly0:1:0:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da0:mly0:1:0:0): Sense Error Code 0x0
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
(da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da1:mly0:1:1:0): Sense Error Code 0x0
(da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da1:mly0:1:1:0): Sense Error Code 0x0
(da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da1:mly0:1:1:0): Sense Error Code 0x0
(da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da1:mly0:1:1:0): Sense Error Code 0x0
(da1:mly0:1:1:0): SYNCHRONIZE CACHE. CDB: 35 0 0 0 0 0 0 0 0 0
(da1:mly0:1:1:0): Sense Error Code 0x0
acquiring duplicate lock of same type: "vnode interlock"
 1st vnode interlock @ /usr/src/sys/kern/vfs_vnops.c:806
 2nd vnode interlock @ /usr/src/sys/kern/vfs_subr.c:2039
KDB: stack backtrace:
kdb_backtrace(3,c80e6780,c07a89f0,c07a89f0,c07678a4,...) at kdb_backtrace+0x29
witness_checkorder(c7eb0d04,9,c072db32,7f7) at witness_checkorder+0x578
_mtx_lock_flags(c7eb0d04,0,c072db32,7f7,c8055980,...) at _mtx_lock_flags+0x78
vrefcnt(c7eb0c3c) at vrefcnt+0x20
null_checkvp(c81dac3c,c071c808,215) at null_checkvp+0x56
null_lock(ea872a68) at null_lock+0x66
VOP_LOCK_APV(c0760b20,ea872a68) at VOP_LOCK_APV+0x87
vn_lock(c81dac3c,1002,c80e6780,c81dac3c,c81dae60,...) at vn_lock+0xac
nullfs_root(c8209000,2,ea872ae0,c80e6780,0,8,0,c07e5fe0,0,c072d3b0,407) at nullfs_root+0x26
vfs_domount(c80e6780,c8055100,c8055700,d,c8055030,c0797ae0,0,c072d3b0,2bf) at vfs_domount+0x975
vfs_donmount(c80e6780,d,c80ede80,c80ede80,0,...) at vfs_donmount+0x3f9
nmount(c80e6780,ea872d04) at nmount+0x8b
syscall(3b,3b,3b,bfbfe5f5,bfbfeea0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (378, FreeBSD ELF32, nmount), eip = 0x280bc11b, esp = 0xbfbfe5bc, ebp = 0xbfbfee38 ---
Accounting enabled

show pcpu:

cpuid = 0
curthread = 0xc7ce3300: pid 20 "swi6: Giant tasq"
curpcb = 0xe68dbd90
fpcurthread = none
idlethread = 0xc7ce3a80: pid 13 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
spin locks held:

show locks:

exclusive sleep mutex Giant r = 0 (0xc0797ae0) locked @ /usr/src/sys/kern/kern_intr.c: 681

show alllocks:

Process 20 (swi6: Giant tasq) thread 0xc7ce3300 (100008)
exclusive sleep mutex Giant r = 0 (0xc0797ae0) locked @ /usr/src/sys/kern/kern_intr.c: 681

After getting a kernel dump I can obtain more information.

--
Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
Phone: +7 4732 539880
Fax:   +7 4732 531415         http://www.vsi.ru
CenterTelecom Voronezh ISP    http://isp.vsi.ru
Comment 4 Oleg Derevenetz 2007-10-22 13:24:53 UTC
There are all required information from "Debugging Deadlocks" chapter of FreeBSD Developer's Handbook:

db> show pcpu
cpuid        = 1
curthread    = 0xc7ce3900: pid 12 "idle: cpu1"
curpcb       = 0xe68e7d90
fpcurthread  = none
idlethread   = 0xc7ce3900: pid 12 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:
exclusive spin mutex sio r = 0 (0xc07f61a0) locked @ /usr/src/sys/dev/sio/sio.c:1390

db> show allpcpu
Current CPU: 1

cpuid        = 0
curthread    = 0xc7ce3a80: pid 13 "idle: cpu0"
curpcb       = 0xe68ead90
fpcurthread  = none
idlethread   = 0xc7ce3a80: pid 13 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:

cpuid        = 1
curthread    = 0xc7ce3900: pid 12 "idle: cpu1"
curpcb       = 0xe68e7d90
fpcurthread  = none
idlethread   = 0xc7ce3900: pid 12 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:
exclusive spin mutex sio r = 0 (0xc07f61a0) locked @ /usr/src/sys/dev/sio/sio.c:1390

cpuid        = 2
curthread    = 0xc7ce3780: pid 11 "idle: cpu2"
curpcb       = 0xe68e4d90
fpcurthread  = none
idlethread   = 0xc7ce3780: pid 11 "idle: cpu2"
APIC ID      = 2
currentldt   = 0x50
spin locks held:

cpuid        = 3
curthread    = 0xc7ce3600: pid 10 "idle: cpu3"
curpcb       = 0xe68e1d90
fpcurthread  = none
idlethread   = 0xc7ce3600: pid 10 "idle: cpu3"
APIC ID      = 3
currentldt   = 0x50
spin locks held:

db> show locks
exclusive spin mutex sio r = 0 (0xc07f61a0) locked @ /usr/src/sys/dev/sio/sio.c:1390

db> show alllocks
Process 938 (sendmail) thread 0xc82e8300 (100115)
shared sx acct_sx r = 0 (0xc0795f00) locked @ /usr/src/sys/kern/kern_acct.c:308
Process 937 (sendmail) thread 0xc83c4780 (100154)
shared sx acct_sx r = 0 (0xc0795f00) locked @ /usr/src/sys/kern/kern_acct.c:308

db> show lockedvnods
Locked vnodes

0xc7eb1ae0: tag ufs, type VREG
    usecount 3, writecount 0, refcount 4 mountedhere 0
    flags (VV_TEXT)
    v_object 0xc106d318 ref 1 pages 25
     lock type ufs: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc0559c15 at lockmgr+0x4ed
#1 0xc0673b06 at ffs_lock+0x76
#2 0xc06e9797 at VOP_LOCK_APV+0x87
#3 0xc05c9444 at vn_lock+0xac
#4 0xc05bd2ea at vget+0xbe
#5 0xc06725b1 at ffs_sync+0x1c1
#6 0xc05bee5c at sync_fsync+0x164
#7 0xc06e8b27 at VOP_FSYNC_APV+0x9b
#8 0xc05bcace at sync_vnode+0x106
#9 0xc05bcd49 at sched_sync+0x1ed
#10 0xc0550cc4 at fork_exit+0xa0
#11 0xc06c23bc at fork_trampoline+0x8

        ino 70723, on dev da0s1a

0xc817a570: tag syncer, type VNON
    usecount 1, writecount 0, refcount 2 mountedhere 0
    flags ()
     lock type syncer: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc0559c15 at lockmgr+0x4ed
#1 0xc05b4189 at vop_stdlock+0x21
#2 0xc06e9797 at VOP_LOCK_APV+0x87
#3 0xc05c9444 at vn_lock+0xac
#4 0xc05bcaab at sync_vnode+0xe3
#5 0xc05bcd49 at sched_sync+0x1ed
#6 0xc0550cc4 at fork_exit+0xa0
#7 0xc06c23bc at fork_trampoline+0x8


0xc8228000: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 2 mountedhere 0
    flags ()
    v_object 0xc814e4a4 ref 0 pages 0
     lock type ufs: EXCL (count 1) by thread 0xc8105d80 (pid 939)#0 0xc0559c15 at lockmgr+0x4ed
#1 0xc0673b06 at ffs_lock+0x76
#2 0xc06e9797 at VOP_LOCK_APV+0x87
#3 0xc05c9444 at vn_lock+0xac
#4 0xc05bd2ea at vget+0xbe
#5 0xc05b1a66 at cache_lookup+0x34a
#6 0xc05b1f9a at vfs_cache_lookup+0x92
#7 0xc06e7857 at VOP_LOOKUP_APV+0x87
#8 0xc05b61d0 at lookup+0x4a4
#9 0xc05b5a8a at namei+0x39a
#10 0xc05c8443 at vn_open_cred+0x26f
#11 0xc05c81d2 at vn_open+0x1e
#12 0xc05c1879 at kern_open+0x231
#13 0xc05c1612 at open+0x1a
#14 0xc06d6d4b at syscall+0x25b
#15 0xc06c23af at Xint0x80_syscall+0x1f

        ino 282624, on dev da0s1e

0xc8302ae0: tag ufs, type VREG
    usecount 1, writecount 1, refcount 3 mountedhere 0
    flags ()
    v_object 0xc8233840 ref 0 pages 10
     lock type ufs: EXCL (count 1) by thread 0xc8087a80 (pid 548)#0 0xc0559c15 at lockmgr+0x4ed
#1 0xc0673b06 at ffs_lock+0x76
#2 0xc06e9797 at VOP_LOCK_APV+0x87
#3 0xc05c9444 at vn_lock+0xac
#4 0xc05c8de0 at vn_write+0x138
#5 0xc058ba28 at dofilewrite+0x7c
#6 0xc058b8c7 at kern_writev+0x3b
#7 0xc058b870 at writev+0x30
#8 0xc06d6d4b at syscall+0x25b
#9 0xc06c23af at Xint0x80_syscall+0x1f

        ino 259099, on dev da0s1e

0xc830bc3c: tag ufs, type VREG
    usecount 1, writecount 1, refcount 4 mountedhere 0
    flags ()
    v_object 0xc81578c4 ref 0 pages 12
     lock type ufs: EXCL (count 1) by thread 0xc83c4780 (pid 937) with 1 pending#0 0xc0559c15 at lockmgr+0x4ed
#1 0xc0673b06 at ffs_lock+0x76
#2 0xc06e9797 at VOP_LOCK_APV+0x87
#3 0xc05c9444 at vn_lock+0xac
#4 0xc05c87dd at vn_rdwr+0x85
#5 0xc053d3bf at acct_process+0x31f
#6 0xc054eb82 at exit1+0x716
#7 0xc054e46c at exit1+0
#8 0xc06d6d4b at syscall+0x25b
#9 0xc06c23af at Xint0x80_syscall+0x1f

        ino 23555, on dev da0s1e

0xc837915c: tag ufs, type VREG
    usecount 0, writecount 0, refcount 4 mountedhere 0
    flags ()
    v_object 0xc8282210 ref 0 pages 1
     lock type ufs: EXCL (count 1) by thread 0xc83c4480 (pid 936) with 1 pending#0 0xc0559c15 at lockmgr+0x4ed
#1 0xc0673b06 at ffs_lock+0x76
#2 0xc06e9797 at VOP_LOCK_APV+0x87
#3 0xc05c9444 at vn_lock+0xac
#4 0xc05c8708 at vn_close+0x5c
#5 0xc05c9582 at vn_closefile+0xca
#6 0xc05488ac at fdrop_locked+0xb0
#7 0xc05487f4 at fdrop+0x24
#8 0xc05472e3 at closef+0x367
#9 0xc0544dd2 at kern_close+0x1aa
#10 0xc0544c24 at close+0x10
#11 0xc06d6d4b at syscall+0x25b
#12 0xc06c23af at Xint0x80_syscall+0x1f

        ino 282642, on dev da0s1e

0xc9feb984: tag ufs, type VREG
    usecount 1, writecount 1, refcount 19 mountedhere 0
    flags ()
    v_object 0xc9fee210 ref 0 pages 68
     lock type ufs: EXCL (count 1) by thread 0xc8193c00 (pid 844)#0 0xc0559c15 at lockmgr+0x4ed
#1 0xc0673b06 at ffs_lock+0x76
#2 0xc06e9797 at VOP_LOCK_APV+0x87
#3 0xc05c9444 at vn_lock+0xac
#4 0xc05c8de0 at vn_write+0x138
#5 0xc058ba28 at dofilewrite+0x7c
#6 0xc058b8c7 at kern_writev+0x3b
#7 0xc058b7ed at write+0x45
#8 0xc06d6d4b at syscall+0x25b
#9 0xc06c23af at Xint0x80_syscall+0x1f

        ino 4098799, on dev da1s1d

db> alltrace

Tracing command sshd pid 939 tid 100071 td 0xc8105d80
sched_switch(c8105d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c83791b4,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c83791b4,0,c8379224,b7,c072d198,...) at sleepq_wait+0x46
msleep(c83791b4,c07969b0,50,c072b884,0,...) at msleep+0x27d
acquire(ea87e820,40,60000,c8105d80,0,...) at acquire+0x76
lockmgr(c83791b4,2002,c8379224,c8105d80) at lockmgr+0x44e
ffs_lock(ea87e878) at ffs_lock+0x76
VOP_LOCK_APV(c077a640,ea87e878) at VOP_LOCK_APV+0x87
vn_lock(c837915c,2002,c8105d80,c837915c) at vn_lock+0xac
vget(c837915c,2002,c8105d80) at vget+0xbe
cache_lookup(c8228000,ea87ebe0,ea87ebf4) at cache_lookup+0x34a
vfs_cache_lookup(ea87e9bc) at vfs_cache_lookup+0x92
VOP_LOOKUP_APV(c077a640,ea87e9bc) at VOP_LOOKUP_APV+0x87
lookup(ea87ebcc) at lookup+0x4a4
namei(ea87ebcc) at namei+0x39a
vn_open_cred(ea87ebcc,ea87eccc,0,c82fe380,3,...) at vn_open_cred+0x26f
vn_open(ea87ebcc,ea87eccc,0,3,c0797aa0,...) at vn_open+0x1e
kern_open(c8105d80,280915c0,0,1,0,...) at kern_open+0x231
open(c8105d80,ea87ed04) at open+0x1a
syscall(3b,3b,3b,2809d000,804aa41,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x2807ca7f, esp = 0xbfbfec1c, ebp = 0xbfbfecc8 ---

Tracing command sendmail pid 938 tid 100115 td 0xc82e8300
sched_switch(c82e8300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c830bc94,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c830bc94,0,c830bd04,b7,c071f3eb,...) at sleepq_wait+0x46
msleep(c830bc94,c07975ec,50,c072b884,0,...) at msleep+0x27d
acquire(ea951ac0,40,60000,c82e8300,0,...) at acquire+0x76
lockmgr(c830bc94,3002,c830bd04,c82e8300) at lockmgr+0x44e
ffs_lock(ea951b18) at ffs_lock+0x76
VOP_LOCK_APV(c077a640,ea951b18) at VOP_LOCK_APV+0x87
vn_lock(c830bc3c,1002,c82e8300) at vn_lock+0xac
vn_rdwr(1,c830bc3c,ea951c34,30,0,0,1,3,c7ce0780,0,0,c82e8300) at vn_rdwr+0x85
acct_process(c82e8300,c0797c20,c07209a0,163,c8364648,...) at acct_process+0x31f
exit1(c82e8300,0,ea951d30,c06d6d4b,c82e8300,...) at exit1+0x716
exit1(c82e8300,ea951d04) at exit1
syscall(3b,3b,3b,0,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x282fc0fb, esp = 0xbfbfcb1c, ebp = 0xbfbfcb38 ---

Tracing command sendmail pid 937 tid 100154 td 0xc83c4780
sched_switch(c83c4780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(dc011d14,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(dc011d14,0,c830bd04,b7,c071f3eb,...) at sleepq_wait+0x46
msleep(dc011d14,c0796dc4,50,c072c4b4,0,...) at msleep+0x27d
acquire(ea9db860,120,60000,c83c4780,0,...) at acquire+0x76
lockmgr(dc011d14,202122,c830bd04,c83c4780) at lockmgr+0x44e
getblk(c830bc3c,10,0,4000,0,...) at getblk+0x12f
breadn(c830bc3c,10,0,4000,0,...) at breadn+0x31
bread(c830bc3c,10,0,4000,0,ea9db970) at bread+0x20
ffs_balloc_ufs2(c830bc3c,41a60,0,30,c7ce0780,...) at ffs_balloc_ufs2+0x1849
ffs_write(ea9dbb54) at ffs_write+0x2c0
VOP_WRITE_APV(c077a640,ea9dbb54) at VOP_WRITE_APV+0x132
vn_rdwr(1,c830bc3c,ea9dbc34,30,0,0,1,3,c7ce0780,0,0,c83c4780) at vn_rdwr+0x142
acct_process(c83c4780,c0797c20,c07209a0,163,c865b860,...) at acct_process+0x31f
exit1(c83c4780,0,ea9dbd30,c06d6d4b,c83c4780,...) at exit1+0x716
exit1(c83c4780,ea9dbd04) at exit1
syscall(3b,3b,3b,0,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x282fc0fb, esp = 0xbfbfbf4c, ebp = 0xbfbfbf68 ---

Tracing command sh pid 936 tid 100156 td 0xc83c4480
sched_switch(c83c4480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(dbfd0b84,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(dbfd0b84,0,c817a8f0,b7,c072e794,...) at sleepq_wait+0x46
msleep(dbfd0b84,c0796af4,50,c072c4b4,0,...) at msleep+0x27d
acquire(ea9d59bc,120,60000,c83c4480,0,...) at acquire+0x76
lockmgr(dbfd0b84,202122,c817a8f0,c83c4480) at lockmgr+0x44e
getblk(c817a828,44ea60,0,4000,0,...) at getblk+0x12f
breadn(c817a828,44ea60,0,4000,0,...) at breadn+0x31
bread(c817a828,44ea60,0,4000,0,ea9d5aa8) at bread+0x20
ffs_update(c837915c,0) at ffs_update+0x22c
ufs_inactive(ea9d5b04) at ufs_inactive+0x2a6
VOP_INACTIVE_APV(c077a640,ea9d5b04) at VOP_INACTIVE_APV+0x9b
vinactive(c837915c,c83c4480) at vinactive+0x8d
vput(c837915c,c8227000,c078b740,c837915c,1,...) at vput+0x1d3
vn_close(c837915c,1,c8593780,c83c4480) at vn_close+0x97
vn_closefile(c812e2d0,c83c4480) at vn_closefile+0xca
fdrop_locked(c812e2d0,c83c4480,c7ccba0c,0,c072016e,...) at fdrop_locked+0xb0
fdrop(c812e2d0,c83c4480,6ac,c079efa0,0,...) at fdrop+0x24
closef(c812e2d0,c83c4480,0,c83c4480,0,...) at closef+0x367
kern_close(c83c4480,3,ea9d5d30,c06d6d4b,c83c4480,...) at kern_close+0x1aa
close(c83c4480,ea9d5d04) at close+0x10
syscall(3b,3b,3b,2808e000,3,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (6, FreeBSD ELF32, close), eip = 0x2806ba9f, esp = 0xbfbfebcc, ebp = 0xbfbfec78 ---

Tracing command cron pid 935 tid 100160 td 0xc8366d80
sched_switch(c8366d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8121330) at sleepq_switch+0xc1
sleepq_wait_sig(c8121330) at sleepq_wait_sig+0x1d
msleep(c8121330,c81214a0,14c,c0728d39,0) at msleep+0x26e
pipe_read(c8750ca8,ea9c9cbc,c810e400,0,c8366d80) at pipe_read+0x377
dofileread(c8366d80,5,c8750ca8,ea9c9cbc,ffffffff,...) at dofileread+0x85
kern_readv(c8366d80,5,ea9c9cbc,805a000,1000,...) at kern_readv+0x36
read(c8366d80,ea9c9d04) at read+0x45
syscall(3b,3b,3b,28171040,28171040,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x2814b953, esp = 0xbfbfe65c, ebp = 0xbfbfe678 ---

Tracing command mc pid 844 tid 100148 td 0xc8193c00
sched_switch(c8193c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5d04,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c07e5d04,0,c072be5a,180,c072e794,...) at sleepq_wait+0x46
msleep(c07e5d04,c07e5d20,44,c072be76,0) at msleep+0x27d
waitrunningbufspace(dbd8a778,10,0,ea90d9a4,c05acc09,...) at waitrunningbufspace+0x4a
bufwrite(dbd502cc) at bufwrite+0x191
bawrite(dbd502cc,c9feba4c,c1e75650,c9feba4c,c9feba4c,...) at bawrite+0x51
cluster_wbuild(c9feb984,4000,8,0,8) at cluster_wbuild+0x61c
cluster_write(c9feb984,dbd8a778,40000,0,50) at cluster_write+0x503
ffs_write(ea90dbf4) at ffs_write+0x523
VOP_WRITE_APV(c077a640,ea90dbf4) at VOP_WRITE_APV+0x132
vn_write(c8738af8,ea90dcbc,c810b200,0,c8193c00) at vn_write+0x1ea
dofilewrite(c8193c00,7,c8738af8,ea90dcbc,ffffffff,...) at dofilewrite+0x7c
kern_writev(c8193c00,7,ea90dcbc,86c4130,570,...) at kern_writev+0x3b
write(c8193c00,ea90dd04) at write+0x45
syscall(3b,3b,3b,16a0,7,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (4, FreeBSD ELF32, write), eip = 0x2836d933, esp = 0xbfbfde6c, ebp = 0xbfbfde98 ---

Tracing command csh pid 841 tid 100143 td 0xc83c5a80
sched_switch(c83c5a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c852f034) at sleepq_switch+0xc1
sleepq_wait_sig(c852f034) at sleepq_wait_sig+0x1d
msleep(c852f034,c852f068,168,c0724047,0) at msleep+0x26e
kern_sigsuspend(c83c5a80,2,0,0,0,...) at kern_sigsuspend+0xa3
sigsuspend(c83c5a80,ea9ffd04) at sigsuspend+0x33
syscall(3b,3b,3b,8299008,bfbf1ad0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1abc, ebp = 0xbfbf1ae8 ---

Tracing command su pid 840 tid 100146 td 0xc8193d80
sched_switch(c8193d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c852f648) at sleepq_switch+0xc1
sleepq_wait_sig(c852f648) at sleepq_wait_sig+0x1d
msleep(c852f648,c852f6b0,15c,c0729071,0) at msleep+0x26e
kern_wait(c8193d80,349,ea910c84,2,0) at kern_wait+0x6bb
wait4(c8193d80,ea910d04) at wait4+0x2a
syscall(3b,3b,3b,bfbfe794,bfbfe810,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x280f451f, esp = 0xbfbfe70c, ebp = 0xbfbfe728 ---

Tracing command csh pid 835 tid 100098 td 0xc8102480
sched_switch(c8102480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c82e6034) at sleepq_switch+0xc1
sleepq_wait_sig(c82e6034) at sleepq_wait_sig+0x1d
msleep(c82e6034,c82e6068,168,c0724047,0) at msleep+0x26e
kern_sigsuspend(c8102480,2,0,0,0,...) at kern_sigsuspend+0xa3
sigsuspend(c8102480,ea84ed04) at sigsuspend+0x33
syscall(3b,3b,3b,81c1308,bfbf1bd0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1bbc, ebp = 0xbfbf1be8 ---

Tracing command sshd pid 834 tid 100042 td 0xc7e05480
sched_switch(c7e05480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
kern_select(c7e05480,9,807bb60,807bb70,0,...) at kern_select+0x4bf
select(c7e05480,e8386d04) at select+0x44
syscall(3b,3b,3b,0,bfbfe11c,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe0ac, ebp = 0xbfbfe0f8 ---

Tracing command sshd pid 831 tid 100074 td 0xc8105900
sched_switch(c8105900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8275370) at sleepq_switch+0xc1
sleepq_wait_sig(c8275370) at sleepq_wait_sig+0x1d
msleep(c8275370,c827533c,158,c072b38c,0,...) at msleep+0x26e
sbwait(c8275318,ea875c0c,c058ae7c,4,0,...) at sbwait+0x4b
soreceive(c82752c8,0,ea875cbc,0,0,0) at soreceive+0x2ee
soo_read(c8111c18,ea875cbc,c82e1380,0,c8105900) at soo_read+0x41
dofileread(c8105900,5,c8111c18,ea875cbc,ffffffff,...) at dofileread+0x85
kern_readv(c8105900,5,ea875cbc,bfbfe108,4,...) at kern_readv+0x36
read(c8105900,ea875d04) at read+0x45
syscall(3b,3b,3b,4,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x2834d953, esp = 0xbfbfe0ac, ebp = 0xbfbfe0d8 ---

Tracing command httpd pid 830 tid 100087 td 0xc818a780
sched_switch(c818a780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b0900) at sleepq_switch+0xc1
sleepq_wait_sig(c80b0900) at sleepq_wait_sig+0x1d
msleep(c80b0900,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b0900,16,ffffffff,ffffffff,ea8c8cb0,...) at lf_setlock+0x181
lf_advlock(ea8c8c90,c84cfd9c,0,0,ea8c8c7c,...) at lf_advlock+0x203
ufs_advlock(ea8c8c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8c8c90) at VOP_ADVLOCK_APV+0xa9
flock(c818a780,ea8c8d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 829 tid 100084 td 0xc818ac00
sched_switch(c818ac00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b0440) at sleepq_switch+0xc1
sleepq_wait_sig(c80b0440) at sleepq_wait_sig+0x1d
msleep(c80b0440,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b0440,16,ffffffff,ffffffff,ea8d1cb0,...) at lf_setlock+0x181
lf_advlock(ea8d1c90,c84cfd9c,0,0,ea8d1c7c,...) at lf_advlock+0x203
ufs_advlock(ea8d1c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8d1c90) at VOP_ADVLOCK_APV+0xa9
flock(c818ac00,ea8d1d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 828 tid 100063 td 0xc815ea80
sched_switch(c815ea80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c815b3c0) at sleepq_switch+0xc1
sleepq_wait_sig(c815b3c0) at sleepq_wait_sig+0x1d
msleep(c815b3c0,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c815b3c0,16,ffffffff,ffffffff,ea8a3cb0,...) at lf_setlock+0x181
lf_advlock(ea8a3c90,c84cfd9c,0,0,ea8a3c7c,...) at lf_advlock+0x203
ufs_advlock(ea8a3c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8a3c90) at VOP_ADVLOCK_APV+0xa9
flock(c815ea80,ea8a3d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 827 tid 100080 td 0xc818e300
sched_switch(c818e300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8067540) at sleepq_switch+0xc1
sleepq_wait_sig(c8067540) at sleepq_wait_sig+0x1d
msleep(c8067540,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c8067540,16,ffffffff,ffffffff,ea8ddcb0,...) at lf_setlock+0x181
lf_advlock(ea8ddc90,c84cfd9c,0,0,ea8ddc7c,...) at lf_advlock+0x203
ufs_advlock(ea8ddc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8ddc90) at VOP_ADVLOCK_APV+0xa9
flock(c818e300,ea8ddd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 826 tid 100043 td 0xc7e05300
sched_switch(c7e05300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8067c80) at sleepq_switch+0xc1
sleepq_wait_sig(c8067c80) at sleepq_wait_sig+0x1d
msleep(c8067c80,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c8067c80,16,ffffffff,ffffffff,e8383cb0,...) at lf_setlock+0x181
lf_advlock(e8383c90,c84cfd9c,0,0,e8383c7c,...) at lf_advlock+0x203
ufs_advlock(e8383c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,e8383c90) at VOP_ADVLOCK_APV+0xa9
flock(c7e05300,e8383d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 825 tid 100076 td 0xc8105600
sched_switch(c8105600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b0940) at sleepq_switch+0xc1
sleepq_wait_sig(c80b0940) at sleepq_wait_sig+0x1d
msleep(c80b0940,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b0940,16,ffffffff,ffffffff,ea86fcb0,...) at lf_setlock+0x181
lf_advlock(ea86fc90,c84cfd9c,0,0,ea86fc7c,...) at lf_advlock+0x203
ufs_advlock(ea86fc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea86fc90) at VOP_ADVLOCK_APV+0xa9
flock(c8105600,ea86fd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 824 tid 100064 td 0xc815e900
sched_switch(c815e900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b0980) at sleepq_switch+0xc1
sleepq_wait_sig(c80b0980) at sleepq_wait_sig+0x1d
msleep(c80b0980,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b0980,16,ffffffff,ffffffff,ea8a0cb0,...) at lf_setlock+0x181
lf_advlock(ea8a0c90,c84cfd9c,0,0,ea8a0c7c,...) at lf_advlock+0x203
ufs_advlock(ea8a0c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8a0c90) at VOP_ADVLOCK_APV+0xa9
flock(c815e900,ea8a0d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 823 tid 100110 td 0xc82e8a80
sched_switch(c82e8a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c812a540) at sleepq_switch+0xc1
sleepq_wait_sig(c812a540) at sleepq_wait_sig+0x1d
msleep(c812a540,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c812a540,16,ffffffff,ffffffff,ea960cb0,...) at lf_setlock+0x181
lf_advlock(ea960c90,c84cfd9c,0,0,ea960c7c,...) at lf_advlock+0x203
ufs_advlock(ea960c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea960c90) at VOP_ADVLOCK_APV+0xa9
flock(c82e8a80,ea960d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 822 tid 100105 td 0xc8087900
sched_switch(c8087900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c812bd40) at sleepq_switch+0xc1
sleepq_wait_sig(c812bd40) at sleepq_wait_sig+0x1d
msleep(c812bd40,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c812bd40,16,ffffffff,ffffffff,e864dcb0,...) at lf_setlock+0x181
lf_advlock(e864dc90,c84cfd9c,0,0,e864dc7c,...) at lf_advlock+0x203
ufs_advlock(e864dc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,e864dc90) at VOP_ADVLOCK_APV+0xa9
flock(c8087900,e864dd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 821 tid 100039 td 0xc7e05900
sched_switch(c7e05900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b09c0) at sleepq_switch+0xc1
sleepq_wait_sig(c80b09c0) at sleepq_wait_sig+0x1d
msleep(c80b09c0,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b09c0,16,ffffffff,ffffffff,e838fcb0,...) at lf_setlock+0x181
lf_advlock(e838fc90,c84cfd9c,0,0,e838fc7c,...) at lf_advlock+0x203
ufs_advlock(e838fc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,e838fc90) at VOP_ADVLOCK_APV+0xa9
flock(c7e05900,e838fd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 820 tid 100089 td 0xc818a480
sched_switch(c818a480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80af540) at sleepq_switch+0xc1
sleepq_wait_sig(c80af540) at sleepq_wait_sig+0x1d
msleep(c80af540,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80af540,16,ffffffff,ffffffff,ea8c2cb0,...) at lf_setlock+0x181
lf_advlock(ea8c2c90,c84cfd9c,0,0,ea8c2c7c,...) at lf_advlock+0x203
ufs_advlock(ea8c2c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8c2c90) at VOP_ADVLOCK_APV+0xa9
flock(c818a480,ea8c2d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 819 tid 100092 td 0xc818a000
sched_switch(c818a000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8067480) at sleepq_switch+0xc1
sleepq_wait_sig(c8067480) at sleepq_wait_sig+0x1d
msleep(c8067480,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c8067480,16,ffffffff,ffffffff,ea8b9cb0,...) at lf_setlock+0x181
lf_advlock(ea8b9c90,c84cfd9c,0,0,ea8b9c7c,...) at lf_advlock+0x203
ufs_advlock(ea8b9c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8b9c90) at VOP_ADVLOCK_APV+0xa9
flock(c818a000,ea8b9d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 818 tid 100106 td 0xc8087780
sched_switch(c8087780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c815b4c0) at sleepq_switch+0xc1
sleepq_wait_sig(c815b4c0) at sleepq_wait_sig+0x1d
msleep(c815b4c0,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c815b4c0,16,ffffffff,ffffffff,e864acb0,...) at lf_setlock+0x181
lf_advlock(e864ac90,c84cfd9c,0,0,e864ac7c,...) at lf_advlock+0x203
ufs_advlock(e864ac90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,e864ac90) at VOP_ADVLOCK_APV+0xa9
flock(c8087780,e864ad04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 817 tid 100079 td 0xc818e480
sched_switch(c818e480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8067240) at sleepq_switch+0xc1
sleepq_wait_sig(c8067240) at sleepq_wait_sig+0x1d
msleep(c8067240,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c8067240,16,ffffffff,ffffffff,ea8e0cb0,...) at lf_setlock+0x181
lf_advlock(ea8e0c90,c84cfd9c,0,0,ea8e0c7c,...) at lf_advlock+0x203
ufs_advlock(ea8e0c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8e0c90) at VOP_ADVLOCK_APV+0xa9
flock(c818e480,ea8e0d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 816 tid 100041 td 0xc7e05600
sched_switch(c7e05600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b0a00) at sleepq_switch+0xc1
sleepq_wait_sig(c80b0a00) at sleepq_wait_sig+0x1d
msleep(c80b0a00,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b0a00,16,ffffffff,ffffffff,e8389cb0,...) at lf_setlock+0x181
lf_advlock(e8389c90,c84cfd9c,0,0,e8389c7c,...) at lf_advlock+0x203
ufs_advlock(e8389c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,e8389c90) at VOP_ADVLOCK_APV+0xa9
flock(c7e05600,e8389d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 815 tid 100075 td 0xc8105780
sched_switch(c8105780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c815b580) at sleepq_switch+0xc1
sleepq_wait_sig(c815b580) at sleepq_wait_sig+0x1d
msleep(c815b580,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c815b580,16,ffffffff,ffffffff,ea872cb0,...) at lf_setlock+0x181
lf_advlock(ea872c90,c84cfd9c,0,0,ea872c7c,...) at lf_advlock+0x203
ufs_advlock(ea872c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea872c90) at VOP_ADVLOCK_APV+0xa9
flock(c8105780,ea872d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 814 tid 100078 td 0xc818e600
sched_switch(c818e600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8067180) at sleepq_switch+0xc1
sleepq_wait_sig(c8067180) at sleepq_wait_sig+0x1d
msleep(c8067180,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c8067180,16,ffffffff,ffffffff,ea8e3cb0,...) at lf_setlock+0x181
lf_advlock(ea8e3c90,c84cfd9c,0,0,ea8e3c7c,...) at lf_advlock+0x203
ufs_advlock(ea8e3c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8e3c90) at VOP_ADVLOCK_APV+0xa9
flock(c818e600,ea8e3d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 813 tid 100095 td 0xc8195600
sched_switch(c8195600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b0a40) at sleepq_switch+0xc1
sleepq_wait_sig(c80b0a40) at sleepq_wait_sig+0x1d
msleep(c80b0a40,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b0a40,16,ffffffff,ffffffff,ea91fcb0,...) at lf_setlock+0x181
lf_advlock(ea91fc90,c84cfd9c,0,0,ea91fc7c,...) at lf_advlock+0x203
ufs_advlock(ea91fc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea91fc90) at VOP_ADVLOCK_APV+0xa9
flock(c8195600,ea91fd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 812 tid 100120 td 0xc8195a80
sched_switch(c8195a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c815b6c0) at sleepq_switch+0xc1
sleepq_wait_sig(c815b6c0) at sleepq_wait_sig+0x1d
msleep(c815b6c0,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c815b6c0,16,ffffffff,ffffffff,ea928cb0,...) at lf_setlock+0x181
lf_advlock(ea928c90,c84cfd9c,0,0,ea928c7c,...) at lf_advlock+0x203
ufs_advlock(ea928c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea928c90) at VOP_ADVLOCK_APV+0xa9
flock(c8195a80,ea928d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 811 tid 100051 td 0xc8087480
sched_switch(c8087480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8066e00) at sleepq_switch+0xc1
sleepq_wait_sig(c8066e00) at sleepq_wait_sig+0x1d
msleep(c8066e00,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c8066e00,16,ffffffff,ffffffff,e8644cb0,...) at lf_setlock+0x181
lf_advlock(e8644c90,c84cfd9c,0,0,e8644c7c,...) at lf_advlock+0x203
ufs_advlock(e8644c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,e8644c90) at VOP_ADVLOCK_APV+0xa9
flock(c8087480,e8644d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 810 tid 100117 td 0xc82e8000
sched_switch(c82e8000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b0a80) at sleepq_switch+0xc1
sleepq_wait_sig(c80b0a80) at sleepq_wait_sig+0x1d
msleep(c80b0a80,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b0a80,16,ffffffff,ffffffff,ea94bcb0,...) at lf_setlock+0x181
lf_advlock(ea94bc90,c84cfd9c,0,0,ea94bc7c,...) at lf_advlock+0x203
ufs_advlock(ea94bc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea94bc90) at VOP_ADVLOCK_APV+0xa9
flock(c82e8000,ea94bd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 809 tid 100052 td 0xc8087300
sched_switch(c8087300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c815b8c0) at sleepq_switch+0xc1
sleepq_wait_sig(c815b8c0) at sleepq_wait_sig+0x1d
msleep(c815b8c0,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c815b8c0,16,ffffffff,ffffffff,e8641cb0,...) at lf_setlock+0x181
lf_advlock(e8641c90,c84cfd9c,0,0,e8641c7c,...) at lf_advlock+0x203
ufs_advlock(e8641c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,e8641c90) at VOP_ADVLOCK_APV+0xa9
flock(c8087300,e8641d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 808 tid 100038 td 0xc7e05a80
sched_switch(c7e05a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8066c80) at sleepq_switch+0xc1
sleepq_wait_sig(c8066c80) at sleepq_wait_sig+0x1d
msleep(c8066c80,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c8066c80,16,ffffffff,ffffffff,e8392cb0,...) at lf_setlock+0x181
lf_advlock(e8392c90,c84cfd9c,0,0,e8392c7c,...) at lf_advlock+0x203
ufs_advlock(e8392c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,e8392c90) at VOP_ADVLOCK_APV+0xa9
flock(c7e05a80,e8392d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 807 tid 100070 td 0xc815e000
sched_switch(c815e000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b0ac0) at sleepq_switch+0xc1
sleepq_wait_sig(c80b0ac0) at sleepq_wait_sig+0x1d
msleep(c80b0ac0,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b0ac0,16,ffffffff,ffffffff,ea88ecb0,...) at lf_setlock+0x181
lf_advlock(ea88ec90,c84cfd9c,0,0,ea88ec7c,...) at lf_advlock+0x203
ufs_advlock(ea88ec90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea88ec90) at VOP_ADVLOCK_APV+0xa9
flock(c815e000,ea88ed04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 806 tid 100077 td 0xc818e780
sched_switch(c818e780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c815a440) at sleepq_switch+0xc1
sleepq_wait_sig(c815a440) at sleepq_wait_sig+0x1d
msleep(c815a440,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c815a440,16,ffffffff,ffffffff,ea8e6cb0,...) at lf_setlock+0x181
lf_advlock(ea8e6c90,c84cfd9c,0,0,ea8e6c7c,...) at lf_advlock+0x203
ufs_advlock(ea8e6c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8e6c90) at VOP_ADVLOCK_APV+0xa9
flock(c818e780,ea8e6d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 805 tid 100083 td 0xc818ad80
sched_switch(c818ad80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c815a000) at sleepq_switch+0xc1
sleepq_wait_sig(c815a000) at sleepq_wait_sig+0x1d
msleep(c815a000,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c815a000,16,ffffffff,ffffffff,ea8d4cb0,...) at lf_setlock+0x181
lf_advlock(ea8d4c90,c84cfd9c,0,0,ea8d4c7c,...) at lf_advlock+0x203
ufs_advlock(ea8d4c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8d4c90) at VOP_ADVLOCK_APV+0xa9
flock(c818ad80,ea8d4d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 804 tid 100067 td 0xc815e480
sched_switch(c815e480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80b0b00) at sleepq_switch+0xc1
sleepq_wait_sig(c80b0b00) at sleepq_wait_sig+0x1d
msleep(c80b0b00,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c80b0b00,16,ffffffff,ffffffff,ea897cb0,...) at lf_setlock+0x181
lf_advlock(ea897c90,c84cfd9c,0,0,ea897c7c,...) at lf_advlock+0x203
ufs_advlock(ea897c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea897c90) at VOP_ADVLOCK_APV+0xa9
flock(c815e480,ea897d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 803 tid 100112 td 0xc82e8780
sched_switch(c82e8780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8066c40) at sleepq_switch+0xc1
sleepq_wait_sig(c8066c40) at sleepq_wait_sig+0x1d
msleep(c8066c40,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c8066c40,16,ffffffff,ffffffff,ea95acb0,...) at lf_setlock+0x181
lf_advlock(ea95ac90,c84cfd9c,0,0,ea95ac7c,...) at lf_advlock+0x203
ufs_advlock(ea95ac90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea95ac90) at VOP_ADVLOCK_APV+0xa9
flock(c82e8780,ea95ad04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 802 tid 100088 td 0xc818a600
sched_switch(c818a600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c815bbc0) at sleepq_switch+0xc1
sleepq_wait_sig(c815bbc0) at sleepq_wait_sig+0x1d
msleep(c815bbc0,0,168,c0765c84,0) at msleep+0x26e
lf_setlock(c815bbc0,16,ffffffff,ffffffff,ea8c5cb0,...) at lf_setlock+0x181
lf_advlock(ea8c5c90,c84cfd9c,0,0,ea8c5c7c,...) at lf_advlock+0x203
ufs_advlock(ea8c5c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c077a640,ea8c5c90) at VOP_ADVLOCK_APV+0xa9
flock(c818a600,ea8c5d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 801 tid 100082 td 0xc818e000
sched_switch(c818e000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,3b7,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
poll(c818e000,ea8d7d04) at poll+0x3b9
syscall(3b,3b,3b,2,28,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (209, FreeBSD ELF32, poll), eip = 0x282c3a87, esp = 0xbfbfebcc, ebp = 0xbfbfec18 ---

Tracing command getty pid 800 tid 100093 td 0xc815ed80
sched_switch(c815ed80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e95410) at sleepq_switch+0xc1
sleepq_wait_sig(c7e95410) at sleepq_wait_sig+0x1d
msleep(c7e95410,0,159,c072a5af,0) at msleep+0x26e
ttysleep(c7e95400,c7e95410,159,c072a5af,0,...) at ttysleep+0x21
ttread(c7e95400,ea8a9cbc,0) at ttread+0x48f
ttyread(c7e8b800,ea8a9cbc,0,c0797ae0,0,...) at ttyread+0x2f
giant_read(c7e8b800,ea8a9cbc,0) at giant_read+0x2d
devfs_read_f(c83d6048,ea8a9cbc,c7ce0780,0,c815ed80) at devfs_read_f+0x62
dofileread(c815ed80,0,c83d6048,ea8a9cbc,ffffffff,...) at dofileread+0x85
kern_readv(c815ed80,0,ea8a9cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c815ed80,ea8a9d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 799 tid 100123 td 0xc8366900
sched_switch(c8366900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea7c10) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea7c10) at sleepq_wait_sig+0x1d
msleep(c7ea7c10,0,159,c072a5af,0) at msleep+0x26e
ttysleep(c7ea7c00,c7ea7c10,159,c072a5af,0,...) at ttysleep+0x21
ttread(c7ea7c00,ea9c0cbc,0) at ttread+0x48f
ttyread(c7e8c800,ea9c0cbc,0,ea9c0c14,c054002d,...) at ttyread+0x2f
scread(c7e8c800,ea9c0cbc,0,c0797ae0,0,...) at scread+0x22
giant_read(c7e8c800,ea9c0cbc,0) at giant_read+0x2d
devfs_read_f(c811c2d0,ea9c0cbc,c7ce0780,0,c8366900) at devfs_read_f+0x62
dofileread(c8366900,0,c811c2d0,ea9c0cbc,ffffffff,...) at dofileread+0x85
kern_readv(c8366900,0,ea9c0cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c8366900,ea9c0d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 798 tid 100103 td 0xc8087c00
sched_switch(c8087c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea6c10) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea6c10) at sleepq_wait_sig+0x1d
msleep(c7ea6c10,0,159,c072a5af,0) at msleep+0x26e
ttysleep(c7ea6c00,c7ea6c10,159,c072a5af,0,...) at ttysleep+0x21
ttread(c7ea6c00,e8653cbc,0) at ttread+0x48f
ttyread(c7e8c700,e8653cbc,0,e8653c14,c054002d,...) at ttyread+0x2f
scread(c7e8c700,e8653cbc,0,c0797ae0,0,...) at scread+0x22
giant_read(c7e8c700,e8653cbc,0) at giant_read+0x2d
devfs_read_f(c8111e10,e8653cbc,c7ce0780,0,c8087c00) at devfs_read_f+0x62
dofileread(c8087c00,0,c8111e10,e8653cbc,ffffffff,...) at dofileread+0x85
kern_readv(c8087c00,0,e8653cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c8087c00,e8653d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 797 tid 100113 td 0xc82e8600
sched_switch(c82e8600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea5810) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea5810) at sleepq_wait_sig+0x1d
msleep(c7ea5810,0,159,c072a5af,0) at msleep+0x26e
ttysleep(c7ea5800,c7ea5810,159,c072a5af,0,...) at ttysleep+0x21
ttread(c7ea5800,ea957cbc,0) at ttread+0x48f
ttyread(c7e8c600,ea957cbc,0,ea957c14,c054002d,...) at ttyread+0x2f
scread(c7e8c600,ea957cbc,0,c0797ae0,0,...) at scread+0x22
giant_read(c7e8c600,ea957cbc,0) at giant_read+0x2d
devfs_read_f(c811c558,ea957cbc,c7ce0780,0,c82e8600) at devfs_read_f+0x62
dofileread(c82e8600,0,c811c558,ea957cbc,ffffffff,...) at dofileread+0x85
kern_readv(c82e8600,0,ea957cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c82e8600,ea957d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 796 tid 100081 td 0xc818e180
sched_switch(c818e180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea5c10) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea5c10) at sleepq_wait_sig+0x1d
msleep(c7ea5c10,0,159,c072a5af,0) at msleep+0x26e
ttysleep(c7ea5c00,c7ea5c10,159,c072a5af,0,...) at ttysleep+0x21
ttread(c7ea5c00,ea8dacbc,0) at ttread+0x48f
ttyread(c7e8c500,ea8dacbc,0,ea8dac14,c054002d,...) at ttyread+0x2f
scread(c7e8c500,ea8dacbc,0,c0797ae0,0,...) at scread+0x22
giant_read(c7e8c500,ea8dacbc,0) at giant_read+0x2d
devfs_read_f(c8108ca8,ea8dacbc,c7ce0780,0,c818e180) at devfs_read_f+0x62
dofileread(c818e180,0,c8108ca8,ea8dacbc,ffffffff,...) at dofileread+0x85
kern_readv(c818e180,0,ea8dacbc,bfbfee1f,1,...) at kern_readv+0x36
read(c818e180,ea8dad04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 795 tid 100116 td 0xc82e8180
sched_switch(c82e8180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea8410) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea8410) at sleepq_wait_sig+0x1d
msleep(c7ea8410,0,159,c072a5af,0) at msleep+0x26e
ttysleep(c7ea8400,c7ea8410,159,c072a5af,0,...) at ttysleep+0x21
ttread(c7ea8400,ea94ecbc,0) at ttread+0x48f
ttyread(c7e8c400,ea94ecbc,0,ea94ec14,c054002d,...) at ttyread+0x2f
scread(c7e8c400,ea94ecbc,0,c0797ae0,0,...) at scread+0x22
giant_read(c7e8c400,ea94ecbc,0) at giant_read+0x2d
devfs_read_f(c811c480,ea94ecbc,c7ce0780,0,c82e8180) at devfs_read_f+0x62
dofileread(c82e8180,0,c811c480,ea94ecbc,ffffffff,...) at dofileread+0x85
kern_readv(c82e8180,0,ea94ecbc,bfbfee1f,1,...) at kern_readv+0x36
read(c82e8180,ea94ed04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 794 tid 100072 td 0xc8105c00
sched_switch(c8105c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e9f410) at sleepq_switch+0xc1
sleepq_wait_sig(c7e9f410) at sleepq_wait_sig+0x1d
msleep(c7e9f410,0,159,c072a5af,0) at msleep+0x26e
ttysleep(c7e9f400,c7e9f410,159,c072a5af,0,...) at ttysleep+0x21
ttread(c7e9f400,ea87bcbc,0) at ttread+0x48f
ttyread(c7e8c300,ea87bcbc,0,ea87bc14,c054002d,...) at ttyread+0x2f
scread(c7e8c300,ea87bcbc,0,c0797ae0,0,...) at scread+0x22
giant_read(c7e8c300,ea87bcbc,0) at giant_read+0x2d
devfs_read_f(c8111c60,ea87bcbc,c7ce0780,0,c8105c00) at devfs_read_f+0x62
dofileread(c8105c00,0,c8111c60,ea87bcbc,ffffffff,...) at dofileread+0x85
kern_readv(c8105c00,0,ea87bcbc,bfbfee1f,1,...) at kern_readv+0x36
read(c8105c00,ea87bd04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 793 tid 100122 td 0xc8195780
sched_switch(c8195780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea7010) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea7010) at sleepq_wait_sig+0x1d
msleep(c7ea7010,0,159,c072a5af,0) at msleep+0x26e
ttysleep(c7ea7000,c7ea7010,159,c072a5af,0,...) at ttysleep+0x21
ttread(c7ea7000,ea922cbc,0) at ttread+0x48f
ttyread(c7e8c200,ea922cbc,0,ea922c14,c054002d,...) at ttyread+0x2f
scread(c7e8c200,ea922cbc,0,c0797ae0,0,...) at scread+0x22
giant_read(c7e8c200,ea922cbc,0) at giant_read+0x2d
devfs_read_f(c811cd80,ea922cbc,c7ce0780,0,c8195780) at devfs_read_f+0x62
dofileread(c8195780,0,c811cd80,ea922cbc,ffffffff,...) at dofileread+0x85
kern_readv(c8195780,0,ea922cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c8195780,ea922d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 792 tid 100066 td 0xc815e600
sched_switch(c815e600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e9bc10) at sleepq_switch+0xc1
sleepq_wait_sig(c7e9bc10) at sleepq_wait_sig+0x1d
msleep(c7e9bc10,0,159,c072a5af,0) at msleep+0x26e
ttysleep(c7e9bc00,c7e9bc10,159,c072a5af,0,...) at ttysleep+0x21
ttread(c7e9bc00,ea89acbc,0) at ttread+0x48f
ttyread(c7e8c100,ea89acbc,0,ea89ac14,c054002d,...) at ttyread+0x2f
scread(c7e8c100,ea89acbc,0,c0797ae0,0,...) at scread+0x22
giant_read(c7e8c100,ea89acbc,0) at giant_read+0x2d
devfs_read_f(c8108240,ea89acbc,c7ce0780,0,c815e600) at devfs_read_f+0x62
dofileread(c815e600,0,c8108240,ea89acbc,ffffffff,...) at dofileread+0x85
kern_readv(c815e600,0,ea89acbc,bfbfee1f,1,...) at kern_readv+0x36
read(c815e600,ea89ad04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command inetd pid 778 tid 100057 td 0xc8102d80
sched_switch(c8102d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
kern_select(c8102d80,7,bfbfe8a0,0,0,...) at kern_select+0x4bf
select(c8102d80,ea860d04) at select+0x44
syscall(3b,3b,3b,bfbfe8a0,bfbfee30,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815a8d3, esp = 0xbfbfdb7c, ebp = 0xbfbfee68 ---

Tracing command httpd pid 758 tid 100054 td 0xc8087000
sched_switch(c8087000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07e5744,c07e5744,3e9,e863bb40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07e5744,c07e5720,3e9) at cv_timedwait_sig+0x194
kern_select(c8087000,0,0,0,0,...) at kern_select+0x4a9
select(c8087000,e863bd04) at select+0x44
syscall(3b,3b,3b,0,f4240,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2830c8d3, esp = 0xbfbfec3c, ebp = 0xbfbfec78 ---

Tracing command mysqld pid 757 tid 100128 td 0xc82e9480
sched_switch(c82e9480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e066f4) at sleepq_switch+0xc1
sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e
msleep(c7e066f4,c8364cf8,168,c0721153,290,...) at msleep+0x244
kse_release(c82e9480,ea972d04) at kse_release+0x1df
syscall(3b,3b,3b,8530900,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c76f28, ebp = 0x13c76f64 ---

Tracing command mysqld pid 757 tid 100129 td 0xc818ed80
sched_switch(c818ed80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e066f4) at sleepq_switch+0xc1
sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e
msleep(c7e066f4,c8364cf8,168,c0721153,291,...) at msleep+0x244
kse_release(c818ed80,ea8f2d04) at kse_release+0x1df
syscall(3b,3b,3b,8530a00,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7af28, ebp = 0x13c7af64 ---

Tracing command mysqld pid 757 tid 100132 td 0xc82e9a80
sched_switch(c82e9a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e066f4) at sleepq_switch+0xc1
sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e
msleep(c7e066f4,c8364cf8,168,c0721153,290,...) at msleep+0x244
kse_release(c82e9a80,ea97ed04) at kse_release+0x1df
syscall(3b,3b,3b,8530b00,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7ef28, ebp = 0x13c7ef64 ---

Tracing command mysqld pid 757 tid 100131 td 0xc8193a80
sched_switch(c8193a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e066f4) at sleepq_switch+0xc1
sleepq_timedwait_sig(c7e066f4) at sleepq_timedwait_sig+0x1e
msleep(c7e066f4,c8364cf8,168,c0721153,290,...) at msleep+0x244
kse_release(c8193a80,ea90ad04) at kse_release+0x1df
syscall(3b,3b,3b,8530000,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x8534f28, ebp = 0x8534f64 ---

Tracing command mysqld pid 757 tid 100127 td 0xc83c5180
sched_switch(c83c5180,c82e9480,1) at sched_switch+0x177
mi_switch(1,c82e9480) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
kern_select(c83c5180,f,bfbfebf0,0,0,...) at kern_select+0x4bf
select(c83c5180,ea9edd04) at select+0x44
syscall(3b,3b,3b,8536000,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2861b8d3, esp = 0xbfbfeacc, ebp = 0xbfbfeb08 ---

Tracing command mysqld pid 757 tid 100134 td 0xc83c5780
sched_switch(c83c5780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(ea9f9c14) at sleepq_switch+0xc1
sleepq_wait_sig(ea9f9c14) at sleepq_wait_sig+0x1d
msleep(ea9f9c14,c8364cf8,168,c072114b,0,...) at msleep+0x26e
kern_sigtimedwait(c83c5780,26005,0,0,0,ea9f9c80,0) at kern_sigtimedwait+0x30b
sigtimedwait(c83c5780,ea9f9d04) at sigtimedwait+0x6b
syscall(3b,3b,3b,13c83e00,bf0f5f37,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (345, FreeBSD ELF32, sigtimedwait), eip = 0x284735b7, esp = 0xbf0f5e2c, ebp = 0xbf0f5ee8 ---

Tracing command mysqld pid 757 tid 100126 td 0xc8105480
sched_switch(c8105480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8364dc8) at sleepq_switch+0xc1
sleepq_timedwait_sig(c8364dc8) at sleepq_timedwait_sig+0x1e
msleep(c8364dc8,c8364cf8,168,c0721148,7531,...) at msleep+0x244
kse_release(c8105480,ea86cd04) at kse_release+0x121
syscall(3b,3b,3b,8530800,81,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0xbf9fef2c, ebp = 0xbf9fef88 ---

Tracing command sh pid 727 tid 100111 td 0xc82e8900
sched_switch(c82e8900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c82e7c90) at sleepq_switch+0xc1
sleepq_wait_sig(c82e7c90) at sleepq_wait_sig+0x1d
msleep(c82e7c90,c82e7cf8,15c,c0729071,0) at msleep+0x26e
kern_wait(c82e8900,ffffffff,ea95dc84,2,0) at kern_wait+0x6bb
wait4(c82e8900,ea95dd04) at wait4+0x2a
syscall(3b,3b,3b,bfbfe678,2d,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2813d51f, esp = 0xbfbfe62c, ebp = 0xbfbfe648 ---

Tracing command cron pid 724 tid 100108 td 0xc82e8d80
sched_switch(c82e8d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07983cc) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07983cc) at sleepq_timedwait_sig+0x1e
msleep(c07983cc,0,15c,c0724ee0,ea61,...) at msleep+0x244
kern_nanosleep(c82e8d80,ea966ccc,ea966cc4) at kern_nanosleep+0xab
nanosleep(c82e8d80,ea966d04) at nanosleep+0x4f
syscall(3b,3b,3b,8050e6c,3c,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 ---

Tracing command snmpd pid 716 tid 100101 td 0xc8102000
sched_switch(c8102000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07e5744,c07e5744,1389,ea845b40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07e5744,c07e5720,1389) at cv_timedwait_sig+0x194
kern_select(c8102000,9,bfbfed50,bfbfecd0,bfbfec50,...) at kern_select+0x4a9
select(c8102000,ea845d04) at select+0x44
syscall(3b,3b,3b,bfbfecd0,bfbfec38,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x284598d3, esp = 0xbfbfea4c, ebp = 0xbfbfee68 ---

Tracing command cron pid 705 tid 100119 td 0xc8195c00
sched_switch(c8195c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07983cc) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07983cc) at sleepq_timedwait_sig+0x1e
msleep(c07983cc,0,15c,c0724ee0,ea61,...) at msleep+0x244
kern_nanosleep(c8195c00,ea92bccc,ea92bcc4) at kern_nanosleep+0xab
nanosleep(c8195c00,ea92bd04) at nanosleep+0x4f
syscall(3b,3b,3b,805008c,3c,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 ---

Tracing command sendmail pid 699 tid 100121 td 0xc8195900
sched_switch(c8195900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8365464) at sleepq_switch+0xc1
sleepq_wait_sig(c8365464) at sleepq_wait_sig+0x1d
msleep(c8365464,c8365498,168,c0724047,0) at msleep+0x26e
kern_sigsuspend(c8195900,0,0,0,0,...) at kern_sigsuspend+0xa3
sigsuspend(c8195900,ea925d04) at sigsuspend+0x33
syscall(3b,3b,3b,0,bfbfd000,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2827e3fb, esp = 0xbfbfcfec, ebp = 0xbfbfd018 ---

Tracing command sendmail pid 693 tid 100056 td 0xc8105000
sched_switch(c8105000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07e5744,c07e5744,1389,ea863b40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07e5744,c07e5720,1389) at cv_timedwait_sig+0x194
kern_select(c8105000,6,bfbfc530,0,0,...) at kern_select+0x4a9
select(c8105000,ea863d04) at select+0x44
syscall(3b,3b,3b,2,5,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x282fc8d3, esp = 0xbfbfc44c, ebp = 0xbfbfd038 ---

Tracing command sshd pid 688 tid 100059 td 0xc8102a80
sched_switch(c8102a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
kern_select(c8102a80,7,807b920,0,0,...) at kern_select+0x4bf
select(c8102a80,ea85ad04) at select+0x44
syscall(3b,3b,3b,1,a,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe16c, ebp = 0xbfbfee78 ---

Tracing command ntpd pid 663 tid 100069 td 0xc815e180
sched_switch(c815e180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
kern_select(c815e180,a,bfbfed60,0,0,...) at kern_select+0x4bf
select(c815e180,ea891d04) at select+0x44
syscall(3b,3b,3b,bfbfed60,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2828a8d3, esp = 0xbfbfed1c, ebp = 0xbfbfedf8 ---

Tracing command usbd pid 632 tid 100085 td 0xc818aa80
sched_switch(c818aa80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07e5744,c07e5744,7531,ea8ceb40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07e5744,c07e5720,7531) at cv_timedwait_sig+0x194
kern_select(c818aa80,6,bfbfeda0,bfbfed20,0,...) at kern_select+0x4a9
select(c818aa80,ea8ced04) at select+0x44
syscall(3b,3b,3b,bfbfed20,1e,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2813c8d3, esp = 0xbfbfeccc, ebp = 0xbfbfee78 ---

Tracing command accounting pid 585 tid 100053 td 0xc8087180
sched_switch(c8087180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c0795f00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c0795f00) at sleepq_wait+0x46
_sx_xlock_hard(c0795f00,c8087180,0,c071f3eb,215) at _sx_xlock_hard+0x195
_sx_xlock(c0795f00,0,c071f3eb,215) at _sx_xlock+0x85
acct_thread(0,e863ed38,0,c053d6c4,0,...) at acct_thread+0xee
fork_exit(c053d6c4,0,e863ed38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe863ed6c, ebp = 0 ---

Tracing command named pid 568 tid 100086 td 0xc818a900
sched_switch(c818a900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07e5744,c07e5744,7531,ea8cbb40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07e5744,c07e5720,7531) at cv_timedwait_sig+0x194
kern_select(c818a900,1a,bfbfed60,bfbfece0,0,...) at kern_select+0x4a9
select(c818a900,ea8cbd04) at select+0x44
syscall(3b,3b,3b,0,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2835e8d3, esp = 0xbfbfec9c, ebp = 0xbfbfedf8 ---

Tracing command syslogd pid 548 tid 100104 td 0xc8087a80
sched_switch(c8087a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(dbfc2388,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(dbfc2388,0,c8302ba8,b7,c072e794,...) at sleepq_wait+0x46
msleep(dbfc2388,c0797a48,50,c072c4b4,0,...) at msleep+0x27d
acquire(e865090c,120,60000,c8087a80,0,...) at acquire+0x76
lockmgr(dbfc2388,202122,c8302ba8,c8087a80) at lockmgr+0x44e
getblk(c8302ae0,4,0,1800,0,...) at getblk+0x12f
breadn(c8302ae0,4,0,1800,0,...) at breadn+0x31
bread(c8302ae0,4,0,1800,0,e8650a2c) at bread+0x20
ffs_balloc_ufs2(c8302ae0,1173b,0,7d,c7ce0780,...) at ffs_balloc_ufs2+0xcd3
ffs_write(e8650c10) at ffs_write+0x2c0
VOP_WRITE_APV(c077a640,e8650c10) at VOP_WRITE_APV+0x132
vn_write(c8111ab0,c8593980,c7ce0780,0,c8087a80) at vn_write+0x1ea
dofilewrite(c8087a80,c,c8111ab0,c8593980,ffffffff,...) at dofilewrite+0x7c
kern_writev(c8087a80,c,c8593980,c8593980,0,...) at kern_writev+0x3b
writev(c8087a80,e8650d04) at writev+0x30
syscall(3b,3b,3b,0,bfbfd050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (121, FreeBSD ELF32, writev), eip = 0x28137f1f, esp = 0xbfbfc96c, ebp = 0xbfbfd078 ---

Tracing command ipmon pid 531 tid 100050 td 0xc8087600
sched_switch(c8087600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07983cc) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07983cc) at sleepq_timedwait_sig+0x1e
msleep(c07983cc,0,15c,c0724ee0,3e9,...) at msleep+0x244
kern_nanosleep(c8087600,e8647ccc,e8647cc4) at kern_nanosleep+0xab
nanosleep(c8087600,e8647d04) at nanosleep+0x4f
syscall(3b,3b,3b,2,1,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2812f513, esp = 0xbfbfcd3c, ebp = 0xbfbfcd68 ---

Tracing command devd pid 435 tid 100107 td 0xc82e9000
sched_switch(c82e9000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5744) at sleepq_switch+0xc1
sleepq_wait_sig(c07e5744,246,c0797aa0,c0728ac4,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07e5744,c07e5720) at cv_wait_sig+0x18b
kern_select(c82e9000,5,bfbfe9d0,0,0,...) at kern_select+0x4bf
select(c82e9000,ea969d04) at select+0x44
syscall(3b,3b,3b,0,8,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x8075917, esp = 0xbfbfe96c, ebp = 0xbfbfee78 ---

Tracing command adjkerntz pid 164 tid 100060 td 0xc8102900
sched_switch(c8102900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8101894) at sleepq_switch+0xc1
sleepq_wait_sig(c8101894) at sleepq_wait_sig+0x1d
msleep(c8101894,c81018c8,168,c0724047,0) at msleep+0x26e
kern_sigsuspend(c8102900,0,0,0,0,...) at kern_sigsuspend+0xa3
sigsuspend(c8102900,ea857d04) at sigsuspend+0x33
syscall(3b,3b,3b,0,ffffc7c0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x280bc3fb, esp = 0xbfbfed9c, ebp = 0xbfbfee78 ---

Tracing command schedcpu pid 44 tid 100044 td 0xc7e05180
sched_switch(c7e05180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(e8380d04,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(e8380d04) at sleepq_timedwait+0x4a
msleep(e8380d04,0,0,c071cc2c,3e8) at msleep+0x259
schedcpu_thread(0,e8380d38,0,c05768b4,0,...) at schedcpu_thread+0x26
fork_exit(c05768b4,0,e8380d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8380d6c, ebp = 0 ---

Tracing command softdepflush pid 43 tid 100045 td 0xc7e05000
sched_switch(c7e05000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07f3254,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c07f3254) at sleepq_timedwait+0x4a
msleep(c07f3254,c07f3220,44,c0738a08,3e8) at msleep+0x259
softdep_flush(0,e837dd38,0,c06669b0,0,...) at softdep_flush+0x227
fork_exit(c06669b0,0,e837dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe837dd6c, ebp = 0 ---

Tracing command vnlru pid 42 tid 100046 td 0xc7dbdd80
sched_switch(c7dbdd80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8084860,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c8084860) at sleepq_timedwait+0x4a
msleep(c8084860,c07e60e0,250,c072dc00,3e8,c07e6160) at msleep+0x259
vnlru_proc(0,e836dd38,0,c05bb048,0,...) at vnlru_proc+0xdf
fork_exit(c05bb048,0,e836dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe836dd6c, ebp = 0 ---

Tracing command syncer pid 41 tid 100047 td 0xc7dbdc00
sched_switch(c7dbdc00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(dbf1b09c,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(dbf1b09c,0,c7eb0ba8,b7,c072db32,...) at sleepq_wait+0x46
msleep(dbf1b09c,c07975ec,50,c072c4b4,0,...) at msleep+0x27d
acquire(e836aacc,120,60000,c7dbdc00,0,...) at acquire+0x76
lockmgr(dbf1b09c,202122,c7eb0ba8,c7dbdc00) at lockmgr+0x44e
getblk(c7eb0ae0,113b60,0,4000,0,...) at getblk+0x12f
breadn(c7eb0ae0,113b60,0,4000,0,...) at breadn+0x31
bread(c7eb0ae0,113b60,0,4000,0,e836abb8) at bread+0x20
ffs_update(c7eb1ae0,0,7,0,0,...) at ffs_update+0x22c
ffs_syncvnode(c7eb1ae0,3) at ffs_syncvnode+0x3be
ffs_sync(c80cbb90,3,c7dbdc00,c80cbb90,2,...) at ffs_sync+0x209
sync_fsync(e836acbc) at sync_fsync+0x164
VOP_FSYNC_APV(c0770380,e836acbc) at VOP_FSYNC_APV+0x9b
sync_vnode(c817a67c,c7dbdc00) at sync_vnode+0x106
sched_sync(0,e836ad38,0,c05bcb5c,0,...) at sched_sync+0x1ed
fork_exit(c05bcb5c,0,e836ad38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe836ad6c, ebp = 0 ---

Tracing command bufdaemon pid 40 tid 100048 td 0xc7dbda80
sched_switch(c7dbda80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07e5cc0,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c07e5cc0) at sleepq_timedwait+0x4a
msleep(c07e5cc0,c07e5ce0,44,c072c3ea,3e8) at msleep+0x259
buf_daemon(0,e8367d38,0,c05ae534,0,...) at buf_daemon+0x108
fork_exit(c05ae534,0,e8367d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8367d6c, ebp = 0 ---

Tracing command pagezero pid 39 tid 100049 td 0xc7dbd900
sched_switch(c7dbd900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07f4224,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c07f4224) at sleepq_timedwait+0x4a
msleep(c07f4224,c07f3ca0,200,c073dabc,493e0,c07f3ca0,0,c073da9e,9d) at msleep+0x259
vm_pagezero(0,e8364d38,0,c0697cbc,0,...) at vm_pagezero+0x6c
fork_exit(c0697cbc,0,e8364d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8364d6c, ebp = 0 ---

Tracing command vmdaemon pid 38 tid 100027 td 0xc7dbd180
sched_switch(c7dbd180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07f3d2c,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c07f3d2c,0,c073d73f,63e,0,...) at sleepq_wait+0x46
msleep(c07f3d2c,c07f3d40,68,c072c3ea,0,...) at msleep+0x27d
vm_daemon(0,e8355d38,0,c0696cd0,0,...) at vm_daemon+0x36
fork_exit(c0696cd0,0,e8355d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8355d6c, ebp = 0 ---

Tracing command pagedaemon pid 37 tid 100028 td 0xc7dbd000
sched_switch(c7dbd000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07f3ce8,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c07f3ce8) at sleepq_timedwait+0x4a
msleep(c07f3ce8,c07f3ca0,44,c072c3ea,1388) at msleep+0x259
vm_pageout(0,e8352d38,0,c0696958,0,...) at vm_pageout+0x260
fork_exit(c0696958,0,e8352d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8352d6c, ebp = 0 ---

Tracing command irq7: ppc0 pid 36 tid 100029 td 0xc7ce9d80
fork_trampoline() at fork_trampoline

Tracing command fdc0 pid 35 tid 100030 td 0xc7ce9c00
sched_switch(c7ce9c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e1f03c,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c7e1f03c) at sleepq_timedwait+0x4a
msleep(c7e1f03c,c7e1f0f0,4c,c071cc2c,3e8) at msleep+0x259
fdc_worker(c7e1f000,c7e1f0f0,0,c073f3ff,4a1) at fdc_worker+0x1fe
fdc_thread(c7e1f000,e6929d38,c7e1f000,c06a57f4,0,...) at fdc_thread+0x54
fork_exit(c06a57f4,c7e1f000,e6929d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6929d6c, ebp = 0 ---

Tracing command swi0: sio pid 34 tid 100031 td 0xc7ce9a80
sched_switch(c7ce9a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7e89540,e6926d38,c7e89540,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7e89540,e6926d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6926d6c, ebp = 0 ---

Tracing command irq1: atkbd0 pid 33 tid 100032 td 0xc7ce9900
sched_switch(c7ce9900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7e846c0,e6923d38,c7e846c0,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7e846c0,e6923d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6923d6c, ebp = 0 ---

Tracing command irq22: fxp0 pid 32 tid 100033 td 0xc7ce9780
sched_switch(c7ce9780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7e25010,e6920d38,c7e25010,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7e25010,e6920d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6920d6c, ebp = 0 ---

Tracing command irq20: mly0 pid 31 tid 100034 td 0xc7ce9600
sched_switch(c7ce9600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7dfea40,e691dd38,c7dfea40,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7dfea40,e691dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe691dd6c, ebp = 0 ---

Tracing command irq15: ata1 pid 30 tid 100035 td 0xc7ce9480
fork_trampoline() at fork_trampoline

Tracing command irq14: ata0 pid 29 tid 100036 td 0xc7e05d80
sched_switch(c7e05d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7e072e0,e8398d38,c7e072e0,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7e072e0,e8398d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8398d6c, ebp = 0 ---

Tracing command usb1 pid 28 tid 100037 td 0xc7e05c00
sched_switch(c7e05c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7dada10,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c7dada10) at sleepq_timedwait+0x4a
msleep(c7dada10,0,5c,c071b821,ea60,c7debd00) at msleep+0x259
usb_event_thread(c7debd00,e8395d38,c7debd00,c051db30,0,...) at usb_event_thread+0x70
fork_exit(c051db30,c7debd00,e8395d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8395d6c, ebp = 0 ---

Tracing command irq17: ehci0 pid 27 tid 100017 td 0xc7ce5900
fork_trampoline() at fork_trampoline

Tracing command usbtask pid 26 tid 100018 td 0xc7ce5780
sched_switch(c7ce5780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07953e4,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c07953e4,0,c055d0cc,c0797ae0,c071b806,...) at sleepq_wait+0x46
msleep(c07953e4,0,5c,c071b828,0,...) at msleep+0x27d
usb_task_thread(0,e6902d38,0,c051dbc0,0,...) at usb_task_thread+0x47
fork_exit(c051dbc0,0,e6902d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6902d6c, ebp = 0 ---

Tracing command usb0 pid 25 tid 100019 td 0xc7ce5600
sched_switch(c7ce5600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7db4210,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c7db4210) at sleepq_timedwait+0x4a
msleep(c7db4210,0,5c,c071b821,ea60,c7dea640) at msleep+0x259
usb_event_thread(c7dea640,e68ffd38,c7dea640,c051db30,0,...) at usb_event_thread+0x70
fork_exit(c051db30,c7dea640,e68ffd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68ffd6c, ebp = 0 ---

Tracing command irq16: ohci0 pid 24 tid 100020 td 0xc7ce5480
fork_trampoline() at fork_trampoline

Tracing command irq9: acpi0 pid 23 tid 100021 td 0xc7ce5300
fork_trampoline() at fork_trampoline

Tracing command acpi_task_2 pid 22 tid 100022 td 0xc7ce5180
sched_switch(c7ce5180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d43c00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d43c00,0,c07273e4,49,8,...) at sleepq_wait+0x46
msleep(c7d43c00,c7d43c1c,0,c071cc2c,0) at msleep+0x27d
taskqueue_thread_loop(c0791fc8,e68f6d38,c0791fc8,c0585fd4,0,...) at taskqueue_thread_loop+0x78
fork_exit(c0585fd4,c0791fc8,e68f6d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68f6d6c, ebp = 0 ---

Tracing command acpi_task_1 pid 9 tid 100023 td 0xc7dbd780
sched_switch(c7dbd780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d43c00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d43c00,0,c07273e4,49,8,...) at sleepq_wait+0x46
msleep(c7d43c00,c7d43c1c,0,c071cc2c,0) at msleep+0x27d
taskqueue_thread_loop(c0791fc8,e8361d38,c0791fc8,c0585fd4,0,...) at taskqueue_thread_loop+0x78
fork_exit(c0585fd4,c0791fc8,e8361d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8361d6c, ebp = 0 ---

Tracing command acpi_task_0 pid 8 tid 100024 td 0xc7dbd600
sched_switch(c7dbd600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d43c00,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d43c00,0,c07273e4,49,c7ddf260,...) at sleepq_wait+0x46
msleep(c7d43c00,c7d43c1c,0,c071cc2c,0) at msleep+0x27d
taskqueue_thread_loop(c0791fc8,e835ed38,c0791fc8,c0585fd4,0,...) at taskqueue_thread_loop+0x78
fork_exit(c0585fd4,c0791fc8,e835ed38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe835ed6c, ebp = 0 ---

Tracing command swi5: + pid 21 tid 100025 td 0xc7dbd480
fork_trampoline() at fork_trampoline

Tracing command thread taskq pid 7 tid 100026 td 0xc7dbd300
sched_switch(c7dbd300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d43d80,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d43d80,0,c07273e4,49,c806aeb0,...) at sleepq_wait+0x46
msleep(c7d43d80,c7d43d9c,0,c071cc2c,0) at msleep+0x27d
taskqueue_thread_loop(c079da88,e8358d38,c079da88,c0585fd4,0,...) at taskqueue_thread_loop+0x78
fork_exit(c0585fd4,c079da88,e8358d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8358d6c, ebp = 0 ---

Tracing command swi6: Giant taskq pid 20 tid 100008 td 0xc7ce3300
sched_switch(c7ce3300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8b70,e68dbd38,c7cc8b70,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7cc8b70,e68dbd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68dbd6c, ebp = 0 ---

Tracing command swi6: task queue pid 19 tid 100009 td 0xc7ce3180
sched_switch(c7ce3180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8b80,e68d8d38,c7cc8b80,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7cc8b80,e68d8d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68d8d6c, ebp = 0 ---

Tracing command kqueue taskq pid 6 tid 100010 td 0xc7ce3000
sched_switch(c7ce3000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d72100,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d72100,0,c07273e4,49,8,...) at sleepq_wait+0x46
msleep(c7d72100,c7d7211c,0,c071cc2c,0) at msleep+0x27d
taskqueue_thread_loop(c0796120,e68d5d38,c0796120,c0585fd4,0,...) at taskqueue_thread_loop+0x78
fork_exit(c0585fd4,c0796120,e68d5d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68d5d6c, ebp = 0 ---

Tracing command swi2: cambio pid 18 tid 100011 td 0xc7ce9300
sched_switch(c7ce9300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8ba0,e6917d38,c7cc8ba0,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7cc8ba0,e6917d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6917d6c, ebp = 0 ---

Tracing command xpt_thrd pid 5 tid 100012 td 0xc7ce9180
sched_switch(c7ce9180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c078c4c4,c0797aa0,0,c0726d95,220,...) at sleepq_switch+0xc1
sleepq_wait(c078c4c4,0,c055d0cc,c0797ae0,c06fdf9f,...) at sleepq_wait+0x46
msleep(c078c4c4,0,4c,c06fdfba,0) at msleep+0x27d
xpt_scanner_thread(0,e6914d38,0,c04441f4,0,...) at xpt_scanner_thread+0x35
fork_exit(c04441f4,0,e6914d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6914d6c, ebp = 0 ---

Tracing command yarrow pid 17 tid 100013 td 0xc7ce9000
sched_switch(c7ce9000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07933e0,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c07933e0) at sleepq_timedwait+0x4a
msleep(c07933e0,0,0,c071cc2c,64) at msleep+0x259
random_kthread(0,e6911d38,0,c0506b28,0,...) at random_kthread+0x180
fork_exit(c0506b28,0,e6911d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6911d6c, ebp = 0 ---

Tracing command g_down pid 4 tid 100014 td 0xc7ce5d80
sched_switch(c7ce5d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c0795888,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c0795888) at sleepq_timedwait+0x4a
msleep(c0795888,c0795768,24c,c071cc2c,64) at msleep+0x259
g_io_schedule_down(c7ce5d80) at g_io_schedule_down+0x56
g_down_procbody(0,e690ed38,0,c052fab0,0,...) at g_down_procbody+0x5a
fork_exit(c052fab0,0,e690ed38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe690ed6c, ebp = 0 ---

Tracing command g_up pid 3 tid 100015 td 0xc7ce5c00
sched_switch(c7ce5c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c0795884,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c0795884) at sleepq_timedwait+0x4a
msleep(c0795884,c07957a8,24c,c071cc2c,64) at msleep+0x259
g_io_schedule_up(c7ce5c00) at g_io_schedule_up+0x127
g_up_procbody(0,e690bd38,0,c052fa50,0,...) at g_up_procbody+0x5a
fork_exit(c052fa50,0,e690bd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe690bd6c, ebp = 0 ---

Tracing command g_event pid 2 tid 100016 td 0xc7ce5a80
sched_switch(c7ce5a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c079587c,c0797aa0,0,c0726d95,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c079587c) at sleepq_timedwait+0x4a
msleep(c079587c,0,4c,c071cc2c,64) at msleep+0x259
g_event_procbody(0,e6908d38,0,c052fb10,0,...) at g_event_procbody+0x9e
fork_exit(c052fb10,0,e6908d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6908d6c, ebp = 0 ---

Tracing command swi3: vm pid 16 tid 100000 td 0xc7ce5000
fork_trampoline() at fork_trampoline

Tracing command swi4: clock sio pid 15 tid 100001 td 0xc7ce3d80
sched_switch(c7ce3d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8870,e68f0d38,c7cc8870,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7cc8870,e68f0d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68f0d6c, ebp = 0 ---

Tracing command swi1: net pid 14 tid 100002 td 0xc7ce3c00
sched_switch(c7ce3c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8880,e68edd38,c7cc8880,c0551b84,0,...) at ithread_loop+0xda
fork_exit(c0551b84,c7cc8880,e68edd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68edd6c, ebp = 0 ---

Tracing command idle: cpu0 pid 13 tid 100003 td 0xc7ce3a80
sched_switch(c072811b,6a9,c07e5594,e68eac40,c058a4e6,...) at sched_switch+0x177
(null)(c072813c,50000,c7ccb010,c07981b0,0,...) at 0
__func__.0(c055d930,0,2001,1,94378,...) at 0xc072813c
(null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa

Tracing command idle: cpu1 pid 12 tid 100004 td 0xc7ce3900
kdb_enter(c074002a) at kdb_enter+0x2b
siointr1(c7e95800,c07f61a0,0,c073fe30,56e,...) at siointr1+0xce
siointr(c7e95800) at siointr+0x21
intr_execute_handlers(c7cd74c8,e68e7c94,4,e68e7cd8,c06c2713,...) at intr_execute_handlers+0xe1
lapic_handle_intr(38) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc06bb645, esp = 0xe68e7cd8, ebp = 0xe68e7cd8 ---
acpi_cpu_c1(c0797aa0,2,e68e7cf8,2,c7ce2860,...) at acpi_cpu_c1+0x5
acpi_cpu_idle(e68e7d0c,c0550ee1,c0550e84,e68e7d24,c0550cc4,...) at acpi_cpu_idle+0x15d
cpu_idle(c0550e84,e68e7d24,c0550cc4,0,e68e7d38,...) at cpu_idle+0x28
idle_proc(0,e68e7d38,0,c0550e84,0,...) at idle_proc+0x5d
fork_exit(c0550e84,0,e68e7d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68e7d6c, ebp = 0 ---

Tracing command idle: cpu2 pid 11 tid 100005 td 0xc7ce3780
sched_switch(c079efa0,0,ffc06034) at sched_switch+0x177
mi_switch(c055d321,c07e5590,e68e4c1c,c058ae7c,c079efa0,...) at mi_switch+0x270
end(c072813c,50000,c7ccb010,c07981b0,0,...) at 0xe68e4bf0
__func__.0(c055d930,0,2001,1,94378,...) at 0xc072813c
(null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa

Tracing command idle: cpu3 pid 10 tid 100006 td 0xc7ce3600
sched_switch(28,12,c06c2aa1,c080fd90,8,...) at sched_switch+0x177

Tracing command init pid 1 tid 100007 td 0xc7ce3480
sched_switch(c7ce3480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ce8000) at sleepq_switch+0xc1
sleepq_wait_sig(c7ce8000) at sleepq_wait_sig+0x1d
msleep(c7ce8000,c7ce8068,15c,c0729071,0) at msleep+0x26e
kern_wait(c7ce3480,ffffffff,e68dec84,0,0) at kern_wait+0x6bb
wait4(c7ce3480,e68ded04) at wait4+0x2a
syscall(3b,3b,3b,0,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x8052f47, esp = 0xbfbfed7c, ebp = 0xbfbfed98 ---

Tracing command swapper pid 0 tid 0 td 0xc0795ba0
sched_switch(c0795ba0,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
scheduler(0,c1ec00,c1e000,0,c0440d25,...) at scheduler+0x199
mi_startup() at mi_startup+0x96
begin() at begin+0x2c

db> call doadump
Dumping 3070 MB (2 chunks)
Aborting dump due to I/O error.
status == 0xb, scsi status == 0x0

** DUMP FAILED (ERROR 5) **
= 0x1d

--
Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
Phone: +7 4732 539880
Fax:   +7 4732 531415         http://www.vsi.ru
CenterTelecom Voronezh ISP    http://isp.vsi.ru
Comment 5 Kostik Belousov 2007-10-23 10:54:15 UTC
On Mon, Oct 22, 2007 at 04:29:32PM +0400, Oleg Derevenetz wrote:
> I posted a followup to kern/104406 that includes all information listed in 
> "Debugging Deadlocks" chapter of FreeBSD Developer's Handbook. Can anyone 
> take a look on it and say - is this certainly a hardware problem or some 
> sort of software problem ?


The ps output from ddb would simplify the analysis.

The buffer space on the machine seems to be contaminated with a critical
amount of the dirty buffers that are sent for the write. The pid
844 (that is the only one producing the writes) is blocked in
waitrunningbufspace(), waiting for the write queue to be drained. There
are other processes, for instance, 937 etc, that wait for the buffer
lock. Most likely, these buffers are scheduled for write and cause the
processes to be uninterruptibly blocked.

If this state of the system is static, then the write queue is not
processed. Most plausible cause is the controller (mly0) or the controller
driver fault.
Comment 6 Oleg Derevenetz 2007-10-23 13:19:57 UTC
OK, here is information grabbed from ddb for the another "hang" case including ps output:

db> show pcpu
cpuid        = 1
curthread    = 0xc7ce3900: pid 12 "idle: cpu1"
curpcb       = 0xe68e7d90
fpcurthread  = none
idlethread   = 0xc7ce3900: pid 12 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:
exclusive spin mutex sio r = 0 (0xc07df9c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
db> show allpcpu
Current CPU: 1

cpuid        = 0
curthread    = 0xc7ce3a80: pid 13 "idle: cpu0"
curpcb       = 0xe68ead90
fpcurthread  = none
idlethread   = 0xc7ce3a80: pid 13 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:

cpuid        = 1
curthread    = 0xc7ce3900: pid 12 "idle: cpu1"
curpcb       = 0xe68e7d90
fpcurthread  = none
idlethread   = 0xc7ce3900: pid 12 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:
exclusive spin mutex sio r = 0 (0xc07df9c0) locked @ /usr/src/sys/dev/sio/sio.c:1390

cpuid        = 2
curthread    = 0xc7ce3780: pid 11 "idle: cpu2"
curpcb       = 0xe68e4d90
fpcurthread  = none
idlethread   = 0xc7ce3780: pid 11 "idle: cpu2"
APIC ID      = 2
currentldt   = 0x50
spin locks held:

cpuid        = 3
curthread    = 0xc7ce3600: pid 10 "idle: cpu3"
curpcb       = 0xe68e1d90
fpcurthread  = none
idlethread   = 0xc7ce3600: pid 10 "idle: cpu3"
APIC ID      = 3
currentldt   = 0x50
spin locks held:

db> show locks
exclusive spin mutex sio r = 0 (0xc07df9c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
db> show alllocks
db> show lockedvnods
Locked vnodes

0xc820f414: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 3 mountedhere 0
    flags (VV_ROOT)
     lock type ufs: EXCL (count 1) by thread 0xc7e05600 (pid 894) with 1 pending#0 0xc054901d at lockmgr+0x4ed
#1 0xc0662f0e at ffs_lock+0x76
#2 0xc06d8ba7 at VOP_LOCK_APV+0x87
#3 0xc05b884c at vn_lock+0xac
#4 0xc05ac6f2 at vget+0xbe
#5 0xc05a43d9 at vfs_hash_get+0x8d
#6 0xc0661c77 at ffs_vget+0x27
#7 0xc066b065 at ufs_root+0x19
#8 0xc05a58fc at lookup+0x7c8
#9 0xc05a4e92 at namei+0x39a
#10 0xc053c321 at do_execve+0x159
#11 0xc053c168 at kern_execve+0x74
#12 0xc053c0ce at execve+0x32
#13 0xc06c615b at syscall+0x25b
#14 0xc06b17bf at Xint0x80_syscall+0x1f

        ino 2, on dev da0s1d

0xc824815c: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 4 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc84e8180 (pid 891) with 1 pending#0 0xc054901d at lockmgr+0x4ed
#1 0xc0662f0e at ffs_lock+0x76
#2 0xc06d8ba7 at VOP_LOCK_APV+0x87
#3 0xc05b884c at vn_lock+0xac
#4 0xc05ac6f2 at vget+0xbe
#5 0xc05a0e6e at cache_lookup+0x34a
#6 0xc05a13a2 at vfs_cache_lookup+0x92
#7 0xc06d6c67 at VOP_LOOKUP_APV+0x87
#8 0xc05a55d8 at lookup+0x4a4
#9 0xc05a4e92 at namei+0x39a
#10 0xc053c321 at do_execve+0x159
#11 0xc053c168 at kern_execve+0x74
#12 0xc053c0ce at execve+0x32
#13 0xc06c615b at syscall+0x25b
#14 0xc06b17bf at Xint0x80_syscall+0x1f

        ino 518144, on dev da0s1d

0xc98dc414: tag ufs, type VREG
    usecount 2, writecount 0, refcount 3 mountedhere 0
    flags (VV_TEXT)
    v_object 0xc9752294 ref 1 pages 4
     lock type ufs: EXCL (count 1) by thread 0xc8150480 (pid 806)#0 0xc054901d at lockmgr+0x4ed
#1 0xc05a44b2 at vfs_hash_insert+0x26
#2 0xc0661dc1 at ffs_vget+0x171
#3 0xc066861a at ufs_lookup+0xab6
#4 0xc06d6d3b at VOP_CACHEDLOOKUP_APV+0x9b
#5 0xc05a13c5 at vfs_cache_lookup+0xb5
#6 0xc06d6c67 at VOP_LOOKUP_APV+0x87
#7 0xc05a55d8 at lookup+0x4a4
#8 0xc05a4e92 at namei+0x39a
#9 0xc053c321 at do_execve+0x159
#10 0xc053c168 at kern_execve+0x74
#11 0xc053c0ce at execve+0x32
#12 0xc06c615b at syscall+0x25b
#13 0xc06b17bf at Xint0x80_syscall+0x1f

        ino 683213, on dev da0s1d

0xc822e6cc: tag syncer, type VNON
    usecount 1, writecount 0, refcount 2 mountedhere 0
    flags ()
     lock type syncer: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc054901d at lockmgr+0x4ed
#1 0xc05a3591 at vop_stdlock+0x21
#2 0xc06d8ba7 at VOP_LOCK_APV+0x87
#3 0xc05b884c at vn_lock+0xac
#4 0xc05abeb3 at sync_vnode+0xe3
#5 0xc05ac151 at sched_sync+0x1ed
#6 0xc05400cc at fork_exit+0xa0
#7 0xc06b17cc at fork_trampoline+0x8


0xc9956570: tag ufs, type VDIR
    usecount 1, writecount 0, refcount 3 mountedhere 0
    flags ()
     lock type ufs: EXCL (count 1) by thread 0xc8332d80 (pid 851)#0 0xc054901d at lockmgr+0x4ed
#1 0xc0662f0e at ffs_lock+0x76
#2 0xc06d8ba7 at VOP_LOCK_APV+0x87
#3 0xc05b884c at vn_lock+0xac
#4 0xc05ac6f2 at vget+0xbe
#5 0xc05a0e6e at cache_lookup+0x34a
#6 0xc05a13a2 at vfs_cache_lookup+0x92
#7 0xc06d6c67 at VOP_LOOKUP_APV+0x87
#8 0xc05a55d8 at lookup+0x4a4
#9 0xc05a4e92 at namei+0x39a
#10 0xc05b7637 at vn_open_cred+0x5b
#11 0xc05b75da at vn_open+0x1e
#12 0xc05b0c81 at kern_open+0x231
#13 0xc05b0a1a at open+0x1a
#14 0xc06c615b at syscall+0x25b
#15 0xc06b17bf at Xint0x80_syscall+0x1f

        ino 94845, on dev da1s1d

0xc99a16cc: tag ufs, type VREG
    usecount 1, writecount 0, refcount 7 mountedhere 0
    flags ()
    v_object 0xc99b8a50 ref 0 pages 19
     lock type ufs: EXCL (count 1) by thread 0xc7dbdc00 (pid 41)#0 0xc054901d at lockmgr+0x4ed
#1 0xc0662f0e at ffs_lock+0x76
#2 0xc06d8ba7 at VOP_LOCK_APV+0x87
#3 0xc05b884c at vn_lock+0xac
#4 0xc05ac6f2 at vget+0xbe
#5 0xc06619b9 at ffs_sync+0x1c1
#6 0xc05ae264 at sync_fsync+0x164
#7 0xc06d7f37 at VOP_FSYNC_APV+0x9b
#8 0xc05abed6 at sync_vnode+0x106
#9 0xc05ac151 at sched_sync+0x1ed
#10 0xc05400cc at fork_exit+0xa0
#11 0xc06b17cc at fork_trampoline+0x8

        ino 94916, on dev da1s1d
db> alltrace

Tracing command inetd pid 897 tid 100176 td 0xc8966780
sched_switch(c8966780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c820f46c,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c820f46c,0,c820f4dc,b7,c071716d,...) at sleepq_wait+0x46
msleep(c820f46c,c0780164,50,c0715859,0,...) at msleep+0x27d
acquire(eaa608c8,40,60000,c8966780,0,...) at acquire+0x76
lockmgr(c820f46c,2002,c820f4dc,c8966780) at lockmgr+0x44e
ffs_lock(eaa60920) at ffs_lock+0x76
VOP_LOCK_APV(c0764180,eaa60920) at VOP_LOCK_APV+0x87
vn_lock(c820f414,2002,c8966780,c820f414) at vn_lock+0xac
vget(c820f414,2002,c8966780) at vget+0xbe
vfs_hash_get(c822d5c8,2,2,c8966780,eaa60a38,0,0) at vfs_hash_get+0x8d
ffs_vget(c822d5c8,2,2,eaa60a38) at ffs_vget+0x27
ufs_root(c822d5c8,2,eaa60a84,c8966780,c7eb0000,...) at ufs_root+0x19
lookup(eaa60c24) at lookup+0x7c8
namei(eaa60c24) at namei+0x39a
do_execve(c8966780,eaa60cb4,0) at do_execve+0x159
kern_execve(c8966780,eaa60cb4,0) at kern_execve+0x74
execve(c8966780,eaa60d04) at execve+0x32
syscall(3b,3b,3b,bfbfedb0,805f200,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (59, FreeBSD ELF32, execve), eip = 0x280f14df, esp = 0xbfbfdb6c, ebp = 0xbfbfdb88 ---

Tracing command sh pid 894 tid 100041 td 0xc7e05600
sched_switch(c7e05600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c82481b4,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c82481b4,0,c8248224,b7,c071716d,...) at sleepq_wait+0x46
msleep(c82481b4,c07806e0,50,c0715859,0,...) at msleep+0x27d
acquire(e83898fc,40,60000,c7e05600,0,...) at acquire+0x76
lockmgr(c82481b4,2002,c8248224,c7e05600) at lockmgr+0x44e
ffs_lock(e8389954) at ffs_lock+0x76
VOP_LOCK_APV(c0764180,e8389954) at VOP_LOCK_APV+0x87
vn_lock(c824815c,2002,c7e05600,c824815c) at vn_lock+0xac
vget(c824815c,2002,c7e05600) at vget+0xbe
cache_lookup(c820f414,e8389c38,e8389c4c) at cache_lookup+0x34a
vfs_cache_lookup(e8389a98) at vfs_cache_lookup+0x92
VOP_LOOKUP_APV(c0764180,e8389a98) at VOP_LOOKUP_APV+0x87
lookup(e8389c24) at lookup+0x4a4
namei(e8389c24) at namei+0x39a
do_execve(c7e05600,e8389cb4,0) at do_execve+0x159
kern_execve(c7e05600,e8389cb4,0) at kern_execve+0x74
execve(c7e05600,e8389d04) at execve+0x32
syscall(3b,3b,3b,0,8067038,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (59, FreeBSD ELF32, execve), eip = 0x2813d4df, esp = 0xbfbfeb9c, ebp = 0xbfbfebc8 ---

Tracing command sh pid 893 tid 100177 td 0xc8966600
sched_switch(c8966600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8964c90) at sleepq_switch+0xc1
sleepq_wait_sig(c8964c90) at sleepq_wait_sig+0x1d
msleep(c8964c90,c8964cf8,15c,c0713046,0) at msleep+0x26e
kern_wait(c8966600,ffffffff,eaa5dc84,2,0) at kern_wait+0x6bb
wait4(c8966600,eaa5dd04) at wait4+0x2a
syscall(3b,3b,3b,bfbfebc8,37d,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2813d51f, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 ---

Tracing command cron pid 892 tid 100059 td 0xc80e3a80
sched_switch(c80e3a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8107990) at sleepq_switch+0xc1
sleepq_wait_sig(c8107990) at sleepq_wait_sig+0x1d
msleep(c8107990,c8107b00,14c,c0712d0e,0) at msleep+0x26e
pipe_read(c8859948,ea85acbc,c8052080,0,c80e3a80) at pipe_read+0x377
dofileread(c80e3a80,5,c8859948,ea85acbc,ffffffff,...) at dofileread+0x85
kern_readv(c80e3a80,5,ea85acbc,805a000,1000,...) at kern_readv+0x36
read(c80e3a80,ea85ad04) at read+0x45
syscall(3b,3b,3b,28171040,28171040,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x2814b953, esp = 0xbfbfe65c, ebp = 0xbfbfe678 ---

Tracing command inetd pid 891 tid 100159 td 0xc84e8180
sched_switch(c84e8180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(dbdb3fdc,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(dbdb3fdc,0,c0715e2f,e78,c071716d,...) at sleepq_wait+0x46
msleep(dbdb3fdc,c07cf5c0,4c,c07165fb,0) at msleep+0x27d
bwait(dbdb3fdc,4c,c07165fb) at bwait+0x47
bufwait(dbdb3fdc,1,0,0,0,...) at bufwait+0x1a
breadn(c824815c,0,0,800,0,...) at breadn+0x266
bread(c824815c,0,0,800,0,ea9b18e8) at bread+0x20
ffs_blkatoff(c824815c,0,0,0,ea9b1954) at ffs_blkatoff+0x9e
ufs_lookup(ea9b19f8) at ufs_lookup+0x2cd
VOP_CACHEDLOOKUP_APV(c0764180,ea9b19f8) at VOP_CACHEDLOOKUP_APV+0x9b
vfs_cache_lookup(ea9b1a98) at vfs_cache_lookup+0xb5
VOP_LOOKUP_APV(c0764180,ea9b1a98) at VOP_LOOKUP_APV+0x87
lookup(ea9b1c24) at lookup+0x4a4
namei(ea9b1c24) at namei+0x39a
do_execve(c84e8180,ea9b1cb4,0) at do_execve+0x159
kern_execve(c84e8180,ea9b1cb4,0) at kern_execve+0x74
execve(c84e8180,ea9b1d04) at execve+0x32
syscall(3b,3b,3b,bfbfedb0,805f1f0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (59, FreeBSD ELF32, execve), eip = 0x280f14df, esp = 0xbfbfdb6c, ebp = 0xbfbfdb88 ---

Tracing command mc pid 851 tid 100161 td 0xc8332d80
sched_switch(c8332d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(dbfdafa4,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(dbfdafa4,0,c9956638,b7,c071716d,...) at sleepq_wait+0x46
msleep(dbfdafa4,c07811b4,50,c0716489,0,...) at msleep+0x27d
acquire(ea9ab718,120,60000,c8332d80,0,...) at acquire+0x76
lockmgr(dbfdafa4,202122,c9956638,c8332d80) at lockmgr+0x44e
getblk(c9956570,0,0,1000,0,...) at getblk+0x12f
breadn(c9956570,0,0,1000,0,...) at breadn+0x31
bread(c9956570,0,0,1000,0,ea9ab7d4) at bread+0x20
ffs_blkatoff(c9956570,800,0,ea9ab810,ea9ab814,c8064600,0,c07243ac,20f) at ffs_blkatoff+0x9e
ufsdirhash_findfree(c973b000,18,ea9ab880,c8332d80,1,...) at ufsdirhash_findfree+0x11a
ufs_lookup(ea9ab91c) at ufs_lookup+0x126
VOP_CACHEDLOOKUP_APV(c0764180,ea9ab91c) at VOP_CACHEDLOOKUP_APV+0x9b
vfs_cache_lookup(ea9ab9bc) at vfs_cache_lookup+0xb5
VOP_LOOKUP_APV(c0764180,ea9ab9bc) at VOP_LOOKUP_APV+0x87
lookup(ea9abbcc) at lookup+0x4a4
namei(ea9abbcc) at namei+0x39a
vn_open_cred(ea9abbcc,ea9abccc,180,c8591880,6,...) at vn_open_cred+0x5b
vn_open(ea9abbcc,ea9abccc,180,6,c057a284,...) at vn_open+0x1e
kern_open(c8332d80,8128900,0,602,180,...) at kern_open+0x231
open(c8332d80,ea9abd04) at open+0x1a
syscall(3b,3b,3b,80cfc20,8128900,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x28360ab7, esp = 0xbfbfde7c, ebp = 0xbfbfde98 ---

Tracing command csh pid 848 tid 100163 td 0xc8332a80
sched_switch(c8332a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c851dcc4) at sleepq_switch+0xc1
sleepq_wait_sig(c851dcc4) at sleepq_wait_sig+0x1d
msleep(c851dcc4,c851dcf8,168,c070e01c,0) at msleep+0x26e
kern_sigsuspend(c8332a80,2,0,0,0,...) at kern_sigsuspend+0xa3
sigsuspend(c8332a80,ea9a5d04) at sigsuspend+0x33
syscall(3b,3b,3b,8299008,bfbf1ad0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1abc, ebp = 0xbfbf1ae8 ---

Tracing command su pid 847 tid 100162 td 0xc8332c00
sched_switch(c8332c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c851da78) at sleepq_switch+0xc1
sleepq_wait_sig(c851da78) at sleepq_wait_sig+0x1d
msleep(c851da78,c851dae0,15c,c0713046,0) at msleep+0x26e
kern_wait(c8332c00,350,ea9a8c84,2,0) at kern_wait+0x6bb
wait4(c8332c00,ea9a8d04) at wait4+0x2a
syscall(3b,3b,3b,bfbfe794,bfbfe810,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x280f451f, esp = 0xbfbfe70c, ebp = 0xbfbfe728 ---

Tracing command csh pid 842 tid 100120 td 0xc8157480
sched_switch(c8157480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c835224c) at sleepq_switch+0xc1
sleepq_wait_sig(c835224c) at sleepq_wait_sig+0x1d
msleep(c835224c,c8352280,168,c070e01c,0) at msleep+0x26e
kern_sigsuspend(c8157480,2,0,0,0,...) at kern_sigsuspend+0xa3
sigsuspend(c8157480,ea8fed04) at sigsuspend+0x33
syscall(3b,3b,3b,81c1308,bfbf1bd0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x281573fb, esp = 0xbfbf1bbc, ebp = 0xbfbf1be8 ---

Tracing command sshd pid 841 tid 100055 td 0xc8086d80
sched_switch(c8086d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
kern_select(c8086d80,9,807bb70,807bb80,0,...) at kern_select+0x4bf
select(c8086d80,e8638d04) at select+0x44
syscall(3b,3b,3b,0,bfbfe11c,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe0ac, ebp = 0xbfbfe0f8 ---

Tracing command sshd pid 837 tid 100165 td 0xc8332780
sched_switch(c8332780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c827720c) at sleepq_switch+0xc1
sleepq_wait_sig(c827720c) at sleepq_wait_sig+0x1d
msleep(c827720c,c82771d8,158,c0715361,0,...) at msleep+0x26e
sbwait(c82771b4,ea99fc0c,c057a284,4,0,...) at sbwait+0x4b
soreceive(c8277164,0,ea99fcbc,0,0,0) at soreceive+0x2ee
soo_read(c83c2168,ea99fcbc,c849a980,0,c8332780) at soo_read+0x41
dofileread(c8332780,5,c83c2168,ea99fcbc,ffffffff,...) at dofileread+0x85
kern_readv(c8332780,5,ea99fcbc,bfbfe108,4,...) at kern_readv+0x36
read(c8332780,ea99fd04) at read+0x45
syscall(3b,3b,3b,4,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x2834d953, esp = 0xbfbfe0ac, ebp = 0xbfbfe0d8 ---

Tracing command httpd pid 836 tid 100124 td 0xc8153d80
sched_switch(c8153d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80bba80) at sleepq_switch+0xc1
sleepq_wait_sig(c80bba80) at sleepq_wait_sig+0x1d
msleep(c80bba80,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80bba80,16,ffffffff,ffffffff,ea8f2cb0,...) at lf_setlock+0x181
lf_advlock(ea8f2c90,c83c90b8,0,0,ea8f2c7c,...) at lf_advlock+0x203
ufs_advlock(ea8f2c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea8f2c90) at VOP_ADVLOCK_APV+0xa9
flock(c8153d80,ea8f2d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 835 tid 100125 td 0xc8153c00
sched_switch(c8153c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8113680) at sleepq_switch+0xc1
sleepq_wait_sig(c8113680) at sleepq_wait_sig+0x1d
msleep(c8113680,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c8113680,16,ffffffff,ffffffff,ea8efcb0,...) at lf_setlock+0x181
lf_advlock(ea8efc90,c83c90b8,0,0,ea8efc7c,...) at lf_advlock+0x203
ufs_advlock(ea8efc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea8efc90) at VOP_ADVLOCK_APV+0xa9
flock(c8153c00,ea8efd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 834 tid 100136 td 0xc8158d80
sched_switch(c8158d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80c0840) at sleepq_switch+0xc1
sleepq_wait_sig(c80c0840) at sleepq_wait_sig+0x1d
msleep(c80c0840,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80c0840,16,ffffffff,ffffffff,ea92ecb0,...) at lf_setlock+0x181
lf_advlock(ea92ec90,c83c90b8,0,0,ea92ec7c,...) at lf_advlock+0x203
ufs_advlock(ea92ec90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea92ec90) at VOP_ADVLOCK_APV+0xa9
flock(c8158d80,ea92ed04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 833 tid 100137 td 0xc8158c00
sched_switch(c8158c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8112880) at sleepq_switch+0xc1
sleepq_wait_sig(c8112880) at sleepq_wait_sig+0x1d
msleep(c8112880,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c8112880,16,ffffffff,ffffffff,ea92bcb0,...) at lf_setlock+0x181
lf_advlock(ea92bc90,c83c90b8,0,0,ea92bc7c,...) at lf_advlock+0x203
ufs_advlock(ea92bc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea92bc90) at VOP_ADVLOCK_APV+0xa9
flock(c8158c00,ea92bd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 832 tid 100138 td 0xc8158a80
sched_switch(c8158a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea6080) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea6080) at sleepq_wait_sig+0x1d
msleep(c7ea6080,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c7ea6080,16,ffffffff,ffffffff,ea928cb0,...) at lf_setlock+0x181
lf_advlock(ea928c90,c83c90b8,0,0,ea928c7c,...) at lf_advlock+0x203
ufs_advlock(ea928c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea928c90) at VOP_ADVLOCK_APV+0xa9
flock(c8158a80,ea928d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 831 tid 100139 td 0xc8158900
sched_switch(c8158900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80c08c0) at sleepq_switch+0xc1
sleepq_wait_sig(c80c08c0) at sleepq_wait_sig+0x1d
msleep(c80c08c0,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80c08c0,16,ffffffff,ffffffff,ea925cb0,...) at lf_setlock+0x181
lf_advlock(ea925c90,c83c90b8,0,0,ea925c7c,...) at lf_advlock+0x203
ufs_advlock(ea925c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea925c90) at VOP_ADVLOCK_APV+0xa9
flock(c8158900,ea925d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 830 tid 100140 td 0xc8158780
sched_switch(c8158780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c81124c0) at sleepq_switch+0xc1
sleepq_wait_sig(c81124c0) at sleepq_wait_sig+0x1d
msleep(c81124c0,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c81124c0,16,ffffffff,ffffffff,ea922cb0,...) at lf_setlock+0x181
lf_advlock(ea922c90,c83c90b8,0,0,ea922c7c,...) at lf_advlock+0x203
ufs_advlock(ea922c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea922c90) at VOP_ADVLOCK_APV+0xa9
flock(c8158780,ea922d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 829 tid 100141 td 0xc8086480
sched_switch(c8086480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80c0800) at sleepq_switch+0xc1
sleepq_wait_sig(c80c0800) at sleepq_wait_sig+0x1d
msleep(c80c0800,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80c0800,16,ffffffff,ffffffff,e8626cb0,...) at lf_setlock+0x181
lf_advlock(e8626c90,c83c90b8,0,0,e8626c7c,...) at lf_advlock+0x203
ufs_advlock(e8626c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,e8626c90) at VOP_ADVLOCK_APV+0xa9
flock(c8086480,e8626d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 828 tid 100142 td 0xc8087c00
sched_switch(c8087c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80bc840) at sleepq_switch+0xc1
sleepq_wait_sig(c80bc840) at sleepq_wait_sig+0x1d
msleep(c80bc840,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80bc840,16,ffffffff,ffffffff,e8653cb0,...) at lf_setlock+0x181
lf_advlock(e8653c90,c83c90b8,0,0,e8653c7c,...) at lf_advlock+0x203
ufs_advlock(e8653c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,e8653c90) at VOP_ADVLOCK_APV+0xa9
flock(c8087c00,e8653d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 827 tid 100143 td 0xc8331480
sched_switch(c8331480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80bc600) at sleepq_switch+0xc1
sleepq_wait_sig(c80bc600) at sleepq_wait_sig+0x1d
msleep(c80bc600,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80bc600,16,ffffffff,ffffffff,ea97bcb0,...) at lf_setlock+0x181
lf_advlock(ea97bc90,c83c90b8,0,0,ea97bc7c,...) at lf_advlock+0x203
ufs_advlock(ea97bc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea97bc90) at VOP_ADVLOCK_APV+0xa9
flock(c8331480,ea97bd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 826 tid 100144 td 0xc8087900
sched_switch(c8087900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80c0980) at sleepq_switch+0xc1
sleepq_wait_sig(c80c0980) at sleepq_wait_sig+0x1d
msleep(c80c0980,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80c0980,16,ffffffff,ffffffff,e864dcb0,...) at lf_setlock+0x181
lf_advlock(e864dc90,c83c90b8,0,0,e864dc7c,...) at lf_advlock+0x203
ufs_advlock(e864dc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,e864dc90) at VOP_ADVLOCK_APV+0xa9
flock(c8087900,e864dd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 825 tid 100145 td 0xc84e9780
sched_switch(c84e9780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea8cc0) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea8cc0) at sleepq_wait_sig+0x1d
msleep(c7ea8cc0,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c7ea8cc0,16,ffffffff,ffffffff,ea9dbcb0,...) at lf_setlock+0x181
lf_advlock(ea9dbc90,c83c90b8,0,0,ea9dbc7c,...) at lf_advlock+0x203
ufs_advlock(ea9dbc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea9dbc90) at VOP_ADVLOCK_APV+0xa9
flock(c84e9780,ea9dbd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 824 tid 100146 td 0xc84e9600
sched_switch(c84e9600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80bbb00) at sleepq_switch+0xc1
sleepq_wait_sig(c80bbb00) at sleepq_wait_sig+0x1d
msleep(c80bbb00,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80bbb00,16,ffffffff,ffffffff,ea9d8cb0,...) at lf_setlock+0x181
lf_advlock(ea9d8c90,c83c90b8,0,0,ea9d8c7c,...) at lf_advlock+0x203
ufs_advlock(ea9d8c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea9d8c90) at VOP_ADVLOCK_APV+0xa9
flock(c84e9600,ea9d8d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 823 tid 100147 td 0xc84e9480
sched_switch(c84e9480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80c15c0) at sleepq_switch+0xc1
sleepq_wait_sig(c80c15c0) at sleepq_wait_sig+0x1d
msleep(c80c15c0,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80c15c0,16,ffffffff,ffffffff,ea9d5cb0,...) at lf_setlock+0x181
lf_advlock(ea9d5c90,c83c90b8,0,0,ea9d5c7c,...) at lf_advlock+0x203
ufs_advlock(ea9d5c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea9d5c90) at VOP_ADVLOCK_APV+0xa9
flock(c84e9480,ea9d5d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 822 tid 100148 td 0xc84e9300
sched_switch(c84e9300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea8d00) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea8d00) at sleepq_wait_sig+0x1d
msleep(c7ea8d00,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c7ea8d00,16,ffffffff,ffffffff,ea9d2cb0,...) at lf_setlock+0x181
lf_advlock(ea9d2c90,c83c90b8,0,0,ea9d2c7c,...) at lf_advlock+0x203
ufs_advlock(ea9d2c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea9d2c90) at VOP_ADVLOCK_APV+0xa9
flock(c84e9300,ea9d2d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 821 tid 100149 td 0xc84e9180
sched_switch(c84e9180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8112180) at sleepq_switch+0xc1
sleepq_wait_sig(c8112180) at sleepq_wait_sig+0x1d
msleep(c8112180,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c8112180,16,ffffffff,ffffffff,ea9cfcb0,...) at lf_setlock+0x181
lf_advlock(ea9cfc90,c83c90b8,0,0,ea9cfc7c,...) at lf_advlock+0x203
ufs_advlock(ea9cfc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea9cfc90) at VOP_ADVLOCK_APV+0xa9
flock(c84e9180,ea9cfd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 820 tid 100042 td 0xc7e05480
sched_switch(c7e05480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8113700) at sleepq_switch+0xc1
sleepq_wait_sig(c8113700) at sleepq_wait_sig+0x1d
msleep(c8113700,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c8113700,16,ffffffff,ffffffff,e8386cb0,...) at lf_setlock+0x181
lf_advlock(e8386c90,c83c90b8,0,0,e8386c7c,...) at lf_advlock+0x203
ufs_advlock(e8386c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,e8386c90) at VOP_ADVLOCK_APV+0xa9
flock(c7e05480,e8386d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 819 tid 100057 td 0xc80e3d80
sched_switch(c80e3d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c81123c0) at sleepq_switch+0xc1
sleepq_wait_sig(c81123c0) at sleepq_wait_sig+0x1d
msleep(c81123c0,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c81123c0,16,ffffffff,ffffffff,ea860cb0,...) at lf_setlock+0x181
lf_advlock(ea860c90,c83c90b8,0,0,ea860c7c,...) at lf_advlock+0x203
ufs_advlock(ea860c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea860c90) at VOP_ADVLOCK_APV+0xa9
flock(c80e3d80,ea860d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 818 tid 100093 td 0xc810fd80
sched_switch(c810fd80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea8dc0) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea8dc0) at sleepq_wait_sig+0x1d
msleep(c7ea8dc0,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c7ea8dc0,16,ffffffff,ffffffff,ea8a9cb0,...) at lf_setlock+0x181
lf_advlock(ea8a9c90,c83c90b8,0,0,ea8a9c7c,...) at lf_advlock+0x203
ufs_advlock(ea8a9c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea8a9c90) at VOP_ADVLOCK_APV+0xa9
flock(c810fd80,ea8a9d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 817 tid 100092 td 0xc8150000
sched_switch(c8150000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80c1580) at sleepq_switch+0xc1
sleepq_wait_sig(c80c1580) at sleepq_wait_sig+0x1d
msleep(c80c1580,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80c1580,16,ffffffff,ffffffff,ea8b9cb0,...) at lf_setlock+0x181
lf_advlock(ea8b9c90,c83c90b8,0,0,ea8b9c7c,...) at lf_advlock+0x203
ufs_advlock(ea8b9c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea8b9c90) at VOP_ADVLOCK_APV+0xa9
flock(c8150000,ea8b9d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 816 tid 100106 td 0xc8331d80
sched_switch(c8331d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea6b80) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea6b80) at sleepq_wait_sig+0x1d
msleep(c7ea6b80,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c7ea6b80,16,ffffffff,ffffffff,ea98dcb0,...) at lf_setlock+0x181
lf_advlock(ea98dc90,c83c90b8,0,0,ea98dc7c,...) at lf_advlock+0x203
ufs_advlock(ea98dc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea98dc90) at VOP_ADVLOCK_APV+0xa9
flock(c8331d80,ea98dd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 815 tid 100054 td 0xc8087000
sched_switch(c8087000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80c0a00) at sleepq_switch+0xc1
sleepq_wait_sig(c80c0a00) at sleepq_wait_sig+0x1d
msleep(c80c0a00,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80c0a00,16,ffffffff,ffffffff,e863bcb0,...) at lf_setlock+0x181
lf_advlock(e863bc90,c83c90b8,0,0,e863bc7c,...) at lf_advlock+0x203
ufs_advlock(e863bc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,e863bc90) at VOP_ADVLOCK_APV+0xa9
flock(c8087000,e863bd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 814 tid 100053 td 0xc8087180
sched_switch(c8087180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80c0600) at sleepq_switch+0xc1
sleepq_wait_sig(c80c0600) at sleepq_wait_sig+0x1d
msleep(c80c0600,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80c0600,16,ffffffff,ffffffff,e863ecb0,...) at lf_setlock+0x181
lf_advlock(e863ec90,c83c90b8,0,0,e863ec7c,...) at lf_advlock+0x203
ufs_advlock(e863ec90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,e863ec90) at VOP_ADVLOCK_APV+0xa9
flock(c8087180,e863ed04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 813 tid 100115 td 0xc8157c00
sched_switch(c8157c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea8d40) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea8d40) at sleepq_wait_sig+0x1d
msleep(c7ea8d40,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c7ea8d40,16,ffffffff,ffffffff,ea90dcb0,...) at lf_setlock+0x181
lf_advlock(ea90dc90,c83c90b8,0,0,ea90dc7c,...) at lf_advlock+0x203
ufs_advlock(ea90dc90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea90dc90) at VOP_ADVLOCK_APV+0xa9
flock(c8157c00,ea90dd04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 812 tid 100066 td 0xc810f600
sched_switch(c810f600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80bba00) at sleepq_switch+0xc1
sleepq_wait_sig(c80bba00) at sleepq_wait_sig+0x1d
msleep(c80bba00,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80bba00,16,ffffffff,ffffffff,ea89acb0,...) at lf_setlock+0x181
lf_advlock(ea89ac90,c83c90b8,0,0,ea89ac7c,...) at lf_advlock+0x203
ufs_advlock(ea89ac90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea89ac90) at VOP_ADVLOCK_APV+0xa9
flock(c810f600,ea89ad04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 811 tid 100108 td 0xc8331a80
sched_switch(c8331a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80bc7c0) at sleepq_switch+0xc1
sleepq_wait_sig(c80bc7c0) at sleepq_wait_sig+0x1d
msleep(c80bc7c0,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80bc7c0,16,ffffffff,ffffffff,ea987cb0,...) at lf_setlock+0x181
lf_advlock(ea987c90,c83c90b8,0,0,ea987c7c,...) at lf_advlock+0x203
ufs_advlock(ea987c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea987c90) at VOP_ADVLOCK_APV+0xa9
flock(c8331a80,ea987d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 810 tid 100109 td 0xc8331900
sched_switch(c8331900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8112240) at sleepq_switch+0xc1
sleepq_wait_sig(c8112240) at sleepq_wait_sig+0x1d
msleep(c8112240,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c8112240,16,ffffffff,ffffffff,ea984cb0,...) at lf_setlock+0x181
lf_advlock(ea984c90,c83c90b8,0,0,ea984c7c,...) at lf_advlock+0x203
ufs_advlock(ea984c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea984c90) at VOP_ADVLOCK_APV+0xa9
flock(c8331900,ea984d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 809 tid 100110 td 0xc8331780
sched_switch(c8331780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80c1e80) at sleepq_switch+0xc1
sleepq_wait_sig(c80c1e80) at sleepq_wait_sig+0x1d
msleep(c80c1e80,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c80c1e80,16,ffffffff,ffffffff,ea981cb0,...) at lf_setlock+0x181
lf_advlock(ea981c90,c83c90b8,0,0,ea981c7c,...) at lf_advlock+0x203
ufs_advlock(ea981c90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea981c90) at VOP_ADVLOCK_APV+0xa9
flock(c8331780,ea981d04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 808 tid 100111 td 0xc8331600
sched_switch(c8331600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8112300) at sleepq_switch+0xc1
sleepq_wait_sig(c8112300) at sleepq_wait_sig+0x1d
msleep(c8112300,0,168,c074f7c4,0) at msleep+0x26e
lf_setlock(c8112300,16,ffffffff,ffffffff,ea97ecb0,...) at lf_setlock+0x181
lf_advlock(ea97ec90,c83c90b8,0,0,ea97ec7c,...) at lf_advlock+0x203
ufs_advlock(ea97ec90) at ufs_advlock+0x1c
VOP_ADVLOCK_APV(c0764180,ea97ec90) at VOP_ADVLOCK_APV+0xa9
flock(c8331600,ea97ed04) at flock+0x1a1
syscall(3b,3b,3b,8635018,862e050,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (131, FreeBSD ELF32, flock), eip = 0x282ab5cf, esp = 0xbfbfebec, ebp = 0xbfbfec08 ---

Tracing command httpd pid 807 tid 100112 td 0xc8158180
sched_switch(c8158180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,3b7,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
poll(c8158180,ea916d04) at poll+0x3b9
syscall(3b,3b,3b,2,28,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (209, FreeBSD ELF32, poll), eip = 0x282c3a87, esp = 0xbfbfebcc, ebp = 0xbfbfec18 ---

Tracing command getty pid 806 tid 100089 td 0xc8150480
sched_switch(c8150480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(dbd4b0bc,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(dbd4b0bc,0,c0715e2f,e78,c0727ac1,...) at sleepq_wait+0x46
msleep(dbd4b0bc,c07cf5c0,44,c0727ce4,0) at msleep+0x27d
bwait(dbd4b0bc,44,c0727ce4) at bwait+0x47
vnode_pager_generic_getpages(c98dc414,ea8c2acc,5000,0,c9752294,...) at vnode_pager_generic_getpages+0x847
ffs_getpages(ea8c2a88) at ffs_getpages+0x100
VOP_GETPAGES_APV(c0764180,ea8c2a88) at VOP_GETPAGES_APV+0xa9
vnode_pager_getpages(c9752294,ea8c2acc,5,0) at vnode_pager_getpages+0xa5
exec_map_first_page(ea8c2be4) at exec_map_first_page+0x16d
do_execve(c8150480,ea8c2cb4,0) at do_execve+0x1e4
kern_execve(c8150480,ea8c2cb4,0) at kern_execve+0x74
execve(c8150480,ea8c2d04) at execve+0x32
syscall(3b,3b,3b,bfbfee18,bfbfedd0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (59, FreeBSD ELF32, execve), eip = 0x280df4df, esp = 0xbfbfedbc, ebp = 0xbfbfee08 ---

Tracing command getty pid 805 tid 100084 td 0xc8150c00
sched_switch(c8150c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e9c410) at sleepq_switch+0xc1
sleepq_wait_sig(c7e9c410) at sleepq_wait_sig+0x1d
msleep(c7e9c410,0,159,c0714584,0) at msleep+0x26e
ttysleep(c7e9c400,c7e9c410,159,c0714584,0,...) at ttysleep+0x21
ttread(c7e9c400,ea8d1cbc,0) at ttread+0x48f
ttyread(c7e8f600,ea8d1cbc,0,ea8d1c14,c052f435,...) at ttyread+0x2f
scread(c7e8f600,ea8d1cbc,0,c0781300,0,...) at scread+0x22
giant_read(c7e8f600,ea8d1cbc,0) at giant_read+0x2d
devfs_read_f(c80f37e0,ea8d1cbc,c7ce0780,0,c8150c00) at devfs_read_f+0x62
dofileread(c8150c00,0,c80f37e0,ea8d1cbc,ffffffff,...) at dofileread+0x85
kern_readv(c8150c00,0,ea8d1cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c8150c00,ea8d1d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 804 tid 100064 td 0xc810f900
sched_switch(c810f900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea4010) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea4010) at sleepq_wait_sig+0x1d
msleep(c7ea4010,0,159,c0714584,0) at msleep+0x26e
ttysleep(c7ea4000,c7ea4010,159,c0714584,0,...) at ttysleep+0x21
ttread(c7ea4000,ea8a0cbc,0) at ttread+0x48f
ttyread(c7e8f700,ea8a0cbc,0,ea8a0c14,c052f435,...) at ttyread+0x2f
scread(c7e8f700,ea8a0cbc,0,c0781300,0,...) at scread+0x22
giant_read(c7e8f700,ea8a0cbc,0) at giant_read+0x2d
devfs_read_f(c80f6048,ea8a0cbc,c7ce0780,0,c810f900) at devfs_read_f+0x62
dofileread(c810f900,0,c80f6048,ea8a0cbc,ffffffff,...) at dofileread+0x85
kern_readv(c810f900,0,ea8a0cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c810f900,ea8a0d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 803 tid 100117 td 0xc8157900
sched_switch(c8157900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea3c10) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea3c10) at sleepq_wait_sig+0x1d
msleep(c7ea3c10,0,159,c0714584,0) at msleep+0x26e
ttysleep(c7ea3c00,c7ea3c10,159,c0714584,0,...) at ttysleep+0x21
ttread(c7ea3c00,ea907cbc,0) at ttread+0x48f
ttyread(c7e90600,ea907cbc,0,ea907c14,c052f435,...) at ttyread+0x2f
scread(c7e90600,ea907cbc,0,c0781300,0,...) at scread+0x22
giant_read(c7e90600,ea907cbc,0) at giant_read+0x2d
devfs_read_f(c80f3828,ea907cbc,c7ce0780,0,c8157900) at devfs_read_f+0x62
dofileread(c8157900,0,c80f3828,ea907cbc,ffffffff,...) at dofileread+0x85
kern_readv(c8157900,0,ea907cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c8157900,ea907d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 802 tid 100061 td 0xc80e3780
sched_switch(c80e3780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea4c10) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea4c10) at sleepq_wait_sig+0x1d
msleep(c7ea4c10,0,159,c0714584,0) at msleep+0x26e
ttysleep(c7ea4c00,c7ea4c10,159,c0714584,0,...) at ttysleep+0x21
ttread(c7ea4c00,ea854cbc,0) at ttread+0x48f
ttyread(c7e90500,ea854cbc,0,ea854c14,c052f435,...) at ttyread+0x2f
scread(c7e90500,ea854cbc,0,c0781300,0,...) at scread+0x22
giant_read(c7e90500,ea854cbc,0) at giant_read+0x2d
devfs_read_f(c82a14c8,ea854cbc,c7ce0780,0,c80e3780) at devfs_read_f+0x62
dofileread(c80e3780,0,c82a14c8,ea854cbc,ffffffff,...) at dofileread+0x85
kern_readv(c80e3780,0,ea854cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c80e3780,ea854d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 801 tid 100078 td 0xc8153600
sched_switch(c8153600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea4810) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea4810) at sleepq_wait_sig+0x1d
msleep(c7ea4810,0,159,c0714584,0) at msleep+0x26e
ttysleep(c7ea4800,c7ea4810,159,c0714584,0,...) at ttysleep+0x21
ttread(c7ea4800,ea8e3cbc,0) at ttread+0x48f
ttyread(c7e90400,ea8e3cbc,0,ea8e3c14,c052f435,...) at ttyread+0x2f
scread(c7e90400,ea8e3cbc,0,c0781300,0,...) at scread+0x22
giant_read(c7e90400,ea8e3cbc,0) at giant_read+0x2d
devfs_read_f(c82a1678,ea8e3cbc,c7ce0780,0,c8153600) at devfs_read_f+0x62
dofileread(c8153600,0,c82a1678,ea8e3cbc,ffffffff,...) at dofileread+0x85
kern_readv(c8153600,0,ea8e3cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c8153600,ea8e3d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 800 tid 100088 td 0xc8150600
sched_switch(c8150600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea2c10) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea2c10) at sleepq_wait_sig+0x1d
msleep(c7ea2c10,0,159,c0714584,0) at msleep+0x26e
ttysleep(c7ea2c00,c7ea2c10,159,c0714584,0,...) at ttysleep+0x21
ttread(c7ea2c00,ea8c5cbc,0) at ttread+0x48f
ttyread(c7e90200,ea8c5cbc,0,ea8c5c14,c052f435,...) at ttyread+0x2f
scread(c7e90200,ea8c5cbc,0,c0781300,0,...) at scread+0x22
giant_read(c7e90200,ea8c5cbc,0) at giant_read+0x2d
devfs_read_f(c80f6240,ea8c5cbc,c7ce0780,0,c8150600) at devfs_read_f+0x62
dofileread(c8150600,0,c80f6240,ea8c5cbc,ffffffff,...) at dofileread+0x85
kern_readv(c8150600,0,ea8c5cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c8150600,ea8c5d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 799 tid 100075 td 0xc80e6780
sched_switch(c80e6780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea2810) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea2810) at sleepq_wait_sig+0x1d
msleep(c7ea2810,0,159,c0714584,0) at msleep+0x26e
ttysleep(c7ea2800,c7ea2810,159,c0714584,0,...) at ttysleep+0x21
ttread(c7ea2800,ea872cbc,0) at ttread+0x48f
ttyread(c7e90100,ea872cbc,0,ea872c14,c052f435,...) at ttyread+0x2f
scread(c7e90100,ea872cbc,0,c0781300,0,...) at scread+0x22
giant_read(c7e90100,ea872cbc,0) at giant_read+0x2d
devfs_read_f(c80f6798,ea872cbc,c7ce0780,0,c80e6780) at devfs_read_f+0x62
dofileread(c80e6780,0,c80f6798,ea872cbc,ffffffff,...) at dofileread+0x85
kern_readv(c80e6780,0,ea872cbc,bfbfee1f,1,...) at kern_readv+0x36
read(c80e6780,ea872d04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command getty pid 798 tid 100040 td 0xc7e05780
sched_switch(c7e05780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ea3810) at sleepq_switch+0xc1
sleepq_wait_sig(c7ea3810) at sleepq_wait_sig+0x1d
msleep(c7ea3810,0,159,c0714584,0) at msleep+0x26e
ttysleep(c7ea3800,c7ea3810,159,c0714584,0,...) at ttysleep+0x21
ttread(c7ea3800,e838ccbc,0) at ttread+0x48f
ttyread(c7e90000,e838ccbc,0,e838cc14,c052f435,...) at ttyread+0x2f
scread(c7e90000,e838ccbc,0,c0781300,0,...) at scread+0x22
giant_read(c7e90000,e838ccbc,0) at giant_read+0x2d
devfs_read_f(c80f6120,e838ccbc,c7ce0780,0,c7e05780) at devfs_read_f+0x62
dofileread(c7e05780,0,c80f6120,e838ccbc,ffffffff,...) at dofileread+0x85
kern_readv(c7e05780,0,e838ccbc,bfbfee1f,1,...) at kern_readv+0x36
read(c7e05780,e838cd04) at read+0x45
syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x28148953, esp = 0xbfbfedfc, ebp = 0xbfbfee28 ---

Tracing command inetd pid 784 tid 100119 td 0xc8157600
sched_switch(c8157600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
kern_select(c8157600,7,bfbfe8a0,0,0,...) at kern_select+0x4bf
select(c8157600,ea901d04) at select+0x44
syscall(3b,3b,3b,bfbfe8a0,381,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2815a8d3, esp = 0xbfbfdb7c, ebp = 0xbfbfee68 ---

Tracing command httpd pid 764 tid 100116 td 0xc8157a80
sched_switch(c8157a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07cef64,c07cef64,3e9,ea90ab40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07cef64,c07cef40,3e9) at cv_timedwait_sig+0x194
kern_select(c8157a80,0,0,0,0,...) at kern_select+0x4a9
select(c8157a80,ea90ad04) at select+0x44
syscall(3b,3b,3b,0,f4240,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2830c8d3, esp = 0xbfbfec3c, ebp = 0xbfbfec78 ---

Tracing command mysqld pid 763 tid 100133 td 0xc8086c00
sched_switch(c8086c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8154034) at sleepq_switch+0xc1
sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e
msleep(c8154034,c8156068,168,c070b128,7ba,...) at msleep+0x244
kse_release(c8086c00,e8635d04) at kse_release+0x1df
syscall(3b,3b,3b,8530000,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x8534f28, ebp = 0x8534f64 ---

Tracing command mysqld pid 763 tid 100135 td 0xc8086900
sched_switch(c8086900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8154034) at sleepq_switch+0xc1
sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e
msleep(c8154034,c8156068,168,c070b128,3e9,...) at msleep+0x244
kse_release(c8086900,e862fd04) at kse_release+0x1df
syscall(3b,3b,3b,8530b00,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7ef28, ebp = 0x13c7ef64 ---

Tracing command mysqld pid 763 tid 100091 td 0xc8086780
sched_switch(c8086780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8154034) at sleepq_switch+0xc1
sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e
msleep(c8154034,c8156068,168,c070b128,7ba,...) at msleep+0x244
kse_release(c8086780,e862cd04) at kse_release+0x1df
syscall(3b,3b,3b,8530a00,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c7af28, ebp = 0x13c7af64 ---

Tracing command mysqld pid 763 tid 100127 td 0xc8087780
sched_switch(c8087780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8154034) at sleepq_switch+0xc1
sleepq_timedwait_sig(c8154034) at sleepq_timedwait_sig+0x1e
msleep(c8154034,c8156068,168,c070b128,7ba,...) at msleep+0x244
kse_release(c8087780,e864ad04) at kse_release+0x1df
syscall(3b,3b,3b,8530900,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0x13c76f28, ebp = 0x13c76f64 ---

Tracing command mysqld pid 763 tid 100134 td 0xc8086000
sched_switch(c8086000,c8086c00,1) at sched_switch+0x177
mi_switch(1,c8086c00) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
kern_select(c8086000,f,bfbfebf0,0,0,...) at kern_select+0x4bf
select(c8086000,e861dd04) at select+0x44
syscall(3b,3b,3b,8536000,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2861b8d3, esp = 0xbfbfeacc, ebp = 0xbfbfeb08 ---

Tracing command mysqld pid 763 tid 100131 td 0xc80e3000
sched_switch(c80e3000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(ea845c14) at sleepq_switch+0xc1
sleepq_wait_sig(ea845c14) at sleepq_wait_sig+0x1d
msleep(ea845c14,c8156068,168,c070b120,0,...) at msleep+0x26e
kern_sigtimedwait(c80e3000,26005,0,0,0,ea845c80,0) at kern_sigtimedwait+0x30b
sigtimedwait(c80e3000,ea845d04) at sigtimedwait+0x6b
syscall(3b,3b,3b,13c83e00,bf0f5f37,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (345, FreeBSD ELF32, sigtimedwait), eip = 0x284735b7, esp = 0xbf0f5e2c, ebp = 0xbf0f5ee8 ---

Tracing command mysqld pid 763 tid 100126 td 0xc80e3480
sched_switch(c80e3480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8156138) at sleepq_switch+0xc1
sleepq_timedwait_sig(c8156138) at sleepq_timedwait_sig+0x1e
msleep(c8156138,c8156068,168,c070b11d,7531,...) at msleep+0x244
kse_release(c80e3480,ea84ed04) at kse_release+0x121
syscall(3b,3b,3b,8530800,81,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (383, FreeBSD ELF32, kse_release), eip = 0x284734f7, esp = 0xbf9fef2c, ebp = 0xbf9fef88 ---

Tracing command sh pid 733 tid 100096 td 0xc8158480
sched_switch(c8158480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8156a78) at sleepq_switch+0xc1
sleepq_wait_sig(c8156a78) at sleepq_wait_sig+0x1d
msleep(c8156a78,c8156ae0,15c,c0713046,0) at msleep+0x26e
kern_wait(c8158480,ffffffff,ea91cc84,2,0) at kern_wait+0x6bb
wait4(c8158480,ea91cd04) at wait4+0x2a
syscall(3b,3b,3b,bfbfe678,2d,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2813d51f, esp = 0xbfbfe62c, ebp = 0xbfbfe648 ---

Tracing command cron pid 730 tid 100101 td 0xc8332600
sched_switch(c8332600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c0781bec) at sleepq_switch+0xc1
sleepq_timedwait_sig(c0781bec) at sleepq_timedwait_sig+0x1e
msleep(c0781bec,0,15c,c070eeb5,ea61,...) at msleep+0x244
kern_nanosleep(c8332600,ea99cccc,ea99ccc4) at kern_nanosleep+0xab
nanosleep(c8332600,ea99cd04) at nanosleep+0x4f
syscall(3b,3b,3b,8050e6c,3c,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 ---

Tracing command snmpd pid 722 tid 100080 td 0xc8153300
sched_switch(c8153300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07cef64,c07cef64,1389,ea8ddb40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07cef64,c07cef40,1389) at cv_timedwait_sig+0x194
kern_select(c8153300,9,bfbfed50,bfbfecd0,bfbfec50,...) at kern_select+0x4a9
select(c8153300,ea8ddd04) at select+0x44
syscall(3b,3b,3b,bfbfecd0,bfbfec38,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x284598d3, esp = 0xbfbfea4c, ebp = 0xbfbfee68 ---

Tracing command cron pid 711 tid 100077 td 0xc8153780
sched_switch(c8153780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c0781bec) at sleepq_switch+0xc1
sleepq_timedwait_sig(c0781bec) at sleepq_timedwait_sig+0x1e
msleep(c0781bec,0,15c,c070eeb5,ea61,...) at msleep+0x244
kern_nanosleep(c8153780,ea8e6ccc,ea8e6cc4) at kern_nanosleep+0xab
nanosleep(c8153780,ea8e6d04) at nanosleep+0x4f
syscall(3b,3b,3b,805008c,3c,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28132513, esp = 0xbfbfecdc, ebp = 0xbfbfed08 ---

Tracing command sendmail pid 705 tid 100060 td 0xc80e3900
sched_switch(c80e3900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c80e2894) at sleepq_switch+0xc1
sleepq_wait_sig(c80e2894) at sleepq_wait_sig+0x1d
msleep(c80e2894,c80e28c8,168,c070e01c,0) at msleep+0x26e
kern_sigsuspend(c80e3900,0,0,0,0,...) at kern_sigsuspend+0xa3
sigsuspend(c80e3900,ea857d04) at sigsuspend+0x33
syscall(3b,3b,3b,0,bfbfd000,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2827e3fb, esp = 0xbfbfcfec, ebp = 0xbfbfd018 ---

Tracing command sendmail pid 699 tid 100051 td 0xc8087480
sched_switch(c8087480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07cef64,c07cef64,1389,e8644b40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07cef64,c07cef40,1389) at cv_timedwait_sig+0x194
kern_select(c8087480,6,bfbfc530,0,0,...) at kern_select+0x4a9
select(c8087480,e8644d04) at select+0x44
syscall(3b,3b,3b,2,5,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x282fc8d3, esp = 0xbfbfc44c, ebp = 0xbfbfd038 ---

Tracing command sshd pid 694 tid 100079 td 0xc8153480
sched_switch(c8153480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
kern_select(c8153480,7,807b920,0,0,...) at kern_select+0x4bf
select(c8153480,ea8e0d04) at select+0x44
syscall(3b,3b,3b,1,a,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2834d8d3, esp = 0xbfbfe16c, ebp = 0xbfbfee78 ---

Tracing command ntpd pid 669 tid 100104 td 0xc8332180
sched_switch(c8332180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
kern_select(c8332180,a,bfbfed60,0,0,...) at kern_select+0x4bf
select(c8332180,ea993d04) at select+0x44
syscall(3b,3b,3b,bfbfed60,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2828a8d3, esp = 0xbfbfed1c, ebp = 0xbfbfedf8 ---

Tracing command usbd pid 638 tid 100056 td 0xc80e6000
sched_switch(c80e6000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07cef64,c07cef64,7531,ea863b40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07cef64,c07cef40,7531) at cv_timedwait_sig+0x194
kern_select(c80e6000,6,bfbfeda0,bfbfed20,0,...) at kern_select+0x4a9
select(c80e6000,ea863d04) at select+0x44
syscall(3b,3b,3b,bfbfed20,1e,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2813c8d3, esp = 0xbfbfeccc, ebp = 0xbfbfee78 ---

Tracing command accounting pid 591 tid 100065 td 0xc810f780
sched_switch(c810f780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c077f760,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c077f760) at sleepq_timedwait+0x4a
msleep(c077f760,0,7f,c0706bf1,3a98,c077f720,c07093bb,213) at msleep+0x259
acct_thread(0,ea89dd38,0,c052cacc,0,...) at acct_thread+0xd5
fork_exit(c052cacc,0,ea89dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xea89dd6c, ebp = 0 ---

Tracing command named pid 574 tid 100083 td 0xc8150d80
sched_switch(c8150d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_timedwait_sig(c07cef64,c07cef64,7531,ea8d4b40,246,...) at sleepq_timedwait_sig+0x1e
cv_timedwait_sig(c07cef64,c07cef40,7531) at cv_timedwait_sig+0x194
kern_select(c8150d80,1a,bfbfed60,bfbfece0,0,...) at kern_select+0x4a9
select(c8150d80,ea8d4d04) at select+0x44
syscall(3b,3b,3b,0,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2835e8d3, esp = 0xbfbfec9c, ebp = 0xbfbfedf8 ---

Tracing command syslogd pid 554 tid 100087 td 0xc8150780
sched_switch(c8150780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
kern_select(c8150780,a,80541c0,0,0,...) at kern_select+0x4bf
select(c8150780,ea8c8d04) at select+0x44
syscall(3b,3b,3b,80541c4,80541c0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x2814c8d3, esp = 0xbfbfda0c, ebp = 0xbfbfee58 ---

Tracing command ipmon pid 537 tid 100039 td 0xc7e05900
sched_switch(c7e05900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c0781bec) at sleepq_switch+0xc1
sleepq_timedwait_sig(c0781bec) at sleepq_timedwait_sig+0x1e
msleep(c0781bec,0,15c,c070eeb5,3e9,...) at msleep+0x244
kern_nanosleep(c7e05900,e838fccc,e838fcc4) at kern_nanosleep+0xab
nanosleep(c7e05900,e838fd04) at nanosleep+0x4f
syscall(3b,3b,3b,2,1,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x2812f513, esp = 0xbfbfcd3c, ebp = 0xbfbfcd68 ---

Tracing command devd pid 441 tid 100068 td 0xc810f300
sched_switch(c810f300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cef64) at sleepq_switch+0xc1
sleepq_wait_sig(c07cef64,246,c07812c0,c0712a99,303,...) at sleepq_wait_sig+0x1d
cv_wait_sig(c07cef64,c07cef40) at cv_wait_sig+0x18b
kern_select(c810f300,5,bfbfe9d0,0,0,...) at kern_select+0x4bf
select(c810f300,ea894d04) at select+0x44
syscall(3b,3b,3b,0,8,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x8075917, esp = 0xbfbfe96c, ebp = 0xbfbfee78 ---

Tracing command adjkerntz pid 170 tid 100052 td 0xc8087300
sched_switch(c8087300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c808567c) at sleepq_switch+0xc1
sleepq_wait_sig(c808567c) at sleepq_wait_sig+0x1d
msleep(c808567c,c80856b0,168,c070e01c,0) at msleep+0x26e
kern_sigsuspend(c8087300,0,0,0,0,...) at kern_sigsuspend+0xa3
sigsuspend(c8087300,e8641d04) at sigsuspend+0x33
syscall(3b,3b,3b,0,ffffc7c0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x280bc3fb, esp = 0xbfbfed9c, ebp = 0xbfbfee78 ---

Tracing command schedcpu pid 44 tid 100044 td 0xc7e05180
sched_switch(c7e05180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(e8380d04,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(e8380d04) at sleepq_timedwait+0x4a
msleep(e8380d04,0,0,c0706bf1,3e8) at msleep+0x259
schedcpu_thread(0,e8380d38,0,c0565cbc,0,...) at schedcpu_thread+0x26
fork_exit(c0565cbc,0,e8380d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8380d6c, ebp = 0 ---

Tracing command softdepflush pid 43 tid 100045 td 0xc7e05000
sched_switch(c7e05000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07dca74,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c07dca74) at sleepq_timedwait+0x4a
msleep(c07dca74,c07dca40,44,c07229e3,3e8) at msleep+0x259
softdep_flush(0,e837dd38,0,c0655db8,0,...) at softdep_flush+0x227
fork_exit(c0655db8,0,e837dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe837dd6c, ebp = 0 ---

Tracing command vnlru pid 42 tid 100046 td 0xc7dbdd80
sched_switch(c7dbdd80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8084860,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c8084860) at sleepq_timedwait+0x4a
msleep(c8084860,c07cf900,250,c0717bd5,3e8,c07cf980) at msleep+0x259
vnlru_proc(0,e836dd38,0,c05aa450,0,...) at vnlru_proc+0xdf
fork_exit(c05aa450,0,e836dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe836dd6c, ebp = 0 ---

Tracing command syncer pid 41 tid 100047 td 0xc7dbdc00
sched_switch(c7dbdc00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c0766324,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c0766324,0,c07278c1,144,c0717b07,...) at sleepq_wait+0x46
msleep(c0766324,c07dda00,44,c0727902,0) at msleep+0x27d
getpbuf(c0766324,c99a1794,0,0,c99a1794,...) at getpbuf+0x44
cluster_wbuild(c99a16cc,4000,0,0,4,...) at cluster_wbuild+0x16b
vfs_bio_awrite(dbe7c9c4) at vfs_bio_awrite+0x1cd
ffs_syncvnode(c99a16cc,3) at ffs_syncvnode+0x2c1
ffs_sync(c822d000,3,c7dbdc00,c822d000,2,...) at ffs_sync+0x209
sync_fsync(e836acbc) at sync_fsync+0x164
VOP_FSYNC_APV(c0759ec0,e836acbc) at VOP_FSYNC_APV+0x9b
sync_vnode(c822e7d8,c7dbdc00) at sync_vnode+0x106
sched_sync(0,e836ad38,0,c05abf64,0,...) at sched_sync+0x1ed
fork_exit(c05abf64,0,e836ad38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe836ad6c, ebp = 0 ---

Tracing command bufdaemon pid 40 tid 100048 td 0xc7dbda80
sched_switch(c7dbda80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07cf4e0,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c07cf4e0) at sleepq_timedwait+0x4a
msleep(c07cf4e0,c07cf500,44,c07163bf,3e8) at msleep+0x259
buf_daemon(0,e8367d38,0,c059d93c,0,...) at buf_daemon+0x108
fork_exit(c059d93c,0,e8367d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8367d6c, ebp = 0 ---

Tracing command pagezero pid 39 tid 100049 td 0xc7dbd900
sched_switch(c7dbd900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07dda44,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c07dda44) at sleepq_timedwait+0x4a
msleep(c07dda44,c07dd4c0,200,c0727a97,493e0,c07dd4c0,0,c0727a79,9d) at msleep+0x259
vm_pagezero(0,e8364d38,0,c06870c4,0,...) at vm_pagezero+0x6c
fork_exit(c06870c4,0,e8364d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8364d6c, ebp = 0 ---

Tracing command vmdaemon pid 38 tid 100027 td 0xc7dbd180
sched_switch(c7dbd180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07dd54c,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c07dd54c,0,c072771a,63e,0,...) at sleepq_wait+0x46
msleep(c07dd54c,c07dd560,68,c07163bf,0,...) at msleep+0x27d
vm_daemon(0,e8355d38,0,c06860d8,0,...) at vm_daemon+0x36
fork_exit(c06860d8,0,e8355d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8355d6c, ebp = 0 ---

Tracing command pagedaemon pid 37 tid 100028 td 0xc7dbd000
sched_switch(c7dbd000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c07dd508,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c07dd508) at sleepq_timedwait+0x4a
msleep(c07dd508,c07dd4c0,44,c07163bf,1388) at msleep+0x259
vm_pageout(0,e8352d38,0,c0685d60,0,...) at vm_pageout+0x260
fork_exit(c0685d60,0,e8352d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8352d6c, ebp = 0 ---

Tracing command irq7: ppc0 pid 36 tid 100029 td 0xc7ce9d80
fork_trampoline() at fork_trampoline

Tracing command fdc0 pid 35 tid 100030 td 0xc7ce9c00
sched_switch(c7ce9c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7e1e63c,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c7e1e63c) at sleepq_timedwait+0x4a
msleep(c7e1e63c,c7e1e6f0,4c,c0706bf1,3e8) at msleep+0x259
fdc_worker(c7e1e600,c7e1e6f0,0,c07293da,4a1) at fdc_worker+0x1fe
fdc_thread(c7e1e600,e6929d38,c7e1e600,c0694bfc,0,...) at fdc_thread+0x54
fork_exit(c0694bfc,c7e1e600,e6929d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6929d6c, ebp = 0 ---

Tracing command swi0: sio pid 34 tid 100031 td 0xc7ce9a80
sched_switch(c7ce9a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7e88c10,e6926d38,c7e88c10,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7e88c10,e6926d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6926d6c, ebp = 0 ---

Tracing command irq1: atkbd0 pid 33 tid 100032 td 0xc7ce9900
sched_switch(c7ce9900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7e89030,e6923d38,c7e89030,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7e89030,e6923d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6923d6c, ebp = 0 ---

Tracing command irq22: fxp0 pid 32 tid 100033 td 0xc7ce9780
sched_switch(c7ce9780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7e772d0,e6920d38,c7e772d0,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7e772d0,e6920d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6920d6c, ebp = 0 ---

Tracing command irq20: mly0 pid 31 tid 100034 td 0xc7ce9600
sched_switch(c7ce9600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7dfd920,e691dd38,c7dfd920,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7dfd920,e691dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe691dd6c, ebp = 0 ---

Tracing command irq15: ata1 pid 30 tid 100035 td 0xc7ce9480
fork_trampoline() at fork_trampoline

Tracing command irq14: ata0 pid 29 tid 100036 td 0xc7e05d80
sched_switch(c7e05d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7e07300,e8398d38,c7e07300,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7e07300,e8398d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8398d6c, ebp = 0 ---

Tracing command usb1 pid 28 tid 100037 td 0xc7e05c00
sched_switch(c7e05c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7dae210,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c7dae210) at sleepq_timedwait+0x4a
msleep(c7dae210,0,5c,c07057e6,ea60,c7debbc0) at msleep+0x259
usb_event_thread(c7debbc0,e8395d38,c7debbc0,c050cf38,0,...) at usb_event_thread+0x70
fork_exit(c050cf38,c7debbc0,e8395d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8395d6c, ebp = 0 ---

Tracing command irq17: ehci0 pid 27 tid 100017 td 0xc7ce5900
fork_trampoline() at fork_trampoline

Tracing command usbtask pid 26 tid 100018 td 0xc7ce5780
sched_switch(c7ce5780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c077ec04,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c077ec04,0,c054c4d4,c0781300,c07057cb,...) at sleepq_wait+0x46
msleep(c077ec04,0,5c,c07057ed,0,...) at msleep+0x27d
usb_task_thread(0,e6902d38,0,c050cfc8,0,...) at usb_task_thread+0x47
fork_exit(c050cfc8,0,e6902d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6902d6c, ebp = 0 ---

Tracing command usb0 pid 25 tid 100019 td 0xc7ce5600
sched_switch(c7ce5600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7db4210,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c7db4210) at sleepq_timedwait+0x4a
msleep(c7db4210,0,5c,c07057e6,ea60,c7dea580) at msleep+0x259
usb_event_thread(c7dea580,e68ffd38,c7dea580,c050cf38,0,...) at usb_event_thread+0x70
fork_exit(c050cf38,c7dea580,e68ffd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68ffd6c, ebp = 0 ---

Tracing command irq16: ohci0 pid 24 tid 100020 td 0xc7ce5480
fork_trampoline() at fork_trampoline

Tracing command irq9: acpi0 pid 23 tid 100021 td 0xc7ce5300
fork_trampoline() at fork_trampoline

Tracing command swi5: + pid 22 tid 100022 td 0xc7ce5180
fork_trampoline() at fork_trampoline

Tracing command thread taskq pid 21 tid 100023 td 0xc7dbd780
sched_switch(c7dbd780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d43d80,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d43d80,0,c07113b9,49,c8053ab0,...) at sleepq_wait+0x46
msleep(c7d43d80,c7d43d9c,0,c0706bf1,0) at msleep+0x27d
taskqueue_thread_loop(c07872a8,e8361d38,c07872a8,c05753dc,0,...) at taskqueue_thread_loop+0x78
fork_exit(c05753dc,c07872a8,e8361d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8361d6c, ebp = 0 ---

Tracing command swi6: Giant taskq pid 20 tid 100024 td 0xc7dbd600
sched_switch(c7dbd600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8b80,e835ed38,c7cc8b80,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7cc8b80,e835ed38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe835ed6c, ebp = 0 ---

Tracing command swi6: task queue pid 19 tid 100025 td 0xc7dbd480
sched_switch(c7dbd480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8b90,e835bd38,c7cc8b90,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7cc8b90,e835bd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe835bd6c, ebp = 0 ---

Tracing command kqueue taskq pid 9 tid 100026 td 0xc7dbd300
sched_switch(c7dbd300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d73100,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d73100,0,c07113b9,49,8,...) at sleepq_wait+0x46
msleep(c7d73100,c7d7311c,0,c0706bf1,0) at msleep+0x27d
taskqueue_thread_loop(c077f940,e8358d38,c077f940,c05753dc,0,...) at taskqueue_thread_loop+0x78
fork_exit(c05753dc,c077f940,e8358d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8358d6c, ebp = 0 ---

Tracing command acpi_task_2 pid 8 tid 100008 td 0xc7ce3300
sched_switch(c7ce3300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d73180,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d73180,0,c07113b9,49,8,...) at sleepq_wait+0x46
msleep(c7d73180,c7d7319c,0,c0706bf1,0) at msleep+0x27d
taskqueue_thread_loop(c077bb08,e68dbd38,c077bb08,c05753dc,0,...) at taskqueue_thread_loop+0x78
fork_exit(c05753dc,c077bb08,e68dbd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68dbd6c, ebp = 0 ---

Tracing command acpi_task_1 pid 7 tid 100009 td 0xc7ce3180
sched_switch(c7ce3180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d73180,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d73180,0,c07113b9,49,8,...) at sleepq_wait+0x46
msleep(c7d73180,c7d7319c,0,c0706bf1,0) at msleep+0x27d
taskqueue_thread_loop(c077bb08,e68d8d38,c077bb08,c05753dc,0,...) at taskqueue_thread_loop+0x78
fork_exit(c05753dc,c077bb08,e68d8d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68d8d6c, ebp = 0 ---

Tracing command acpi_task_0 pid 6 tid 100010 td 0xc7ce3000
sched_switch(c7ce3000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7d73180,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c7d73180,0,c07113b9,49,c7ddf1e0,...) at sleepq_wait+0x46
msleep(c7d73180,c7d7319c,0,c0706bf1,0) at msleep+0x27d
taskqueue_thread_loop(c077bb08,e68d5d38,c077bb08,c05753dc,0,...) at taskqueue_thread_loop+0x78
fork_exit(c05753dc,c077bb08,e68d5d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68d5d6c, ebp = 0 ---

Tracing command swi2: cambio pid 18 tid 100011 td 0xc7ce9300
sched_switch(c7ce9300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8bc0,e6917d38,c7cc8bc0,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7cc8bc0,e6917d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6917d6c, ebp = 0 ---

Tracing command xpt_thrd pid 5 tid 100012 td 0xc7ce9180
sched_switch(c7ce9180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c0776004,c07812c0,0,c0710d6a,220,...) at sleepq_switch+0xc1
sleepq_wait(c0776004,0,c054c4d4,c0781300,c06ecd75,...) at sleepq_wait+0x46
msleep(c0776004,0,4c,c06ecd90,0) at msleep+0x27d
xpt_scanner_thread(0,e6914d38,0,c0443a54,0,...) at xpt_scanner_thread+0x35
fork_exit(c0443a54,0,e6914d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6914d6c, ebp = 0 ---

Tracing command yarrow pid 17 tid 100013 td 0xc7ce9000
sched_switch(c7ce9000,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c077cc00,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c077cc00) at sleepq_timedwait+0x4a
msleep(c077cc00,0,0,c0706bf1,64) at msleep+0x259
random_kthread(0,e6911d38,0,c04f5f30,0,...) at random_kthread+0x180
fork_exit(c04f5f30,0,e6911d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6911d6c, ebp = 0 ---

Tracing command g_down pid 4 tid 100014 td 0xc7ce5d80
sched_switch(c7ce5d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c077f0a8,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c077f0a8) at sleepq_timedwait+0x4a
msleep(c077f0a8,c077ef88,24c,c0706bf1,64) at msleep+0x259
g_io_schedule_down(c7ce5d80) at g_io_schedule_down+0x56
g_down_procbody(0,e690ed38,0,c051eeb8,0,...) at g_down_procbody+0x5a
fork_exit(c051eeb8,0,e690ed38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe690ed6c, ebp = 0 ---

Tracing command g_up pid 3 tid 100015 td 0xc7ce5c00
sched_switch(c7ce5c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c077f0a4,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c077f0a4) at sleepq_timedwait+0x4a
msleep(c077f0a4,c077efc8,24c,c0706bf1,64) at msleep+0x259
g_io_schedule_up(c7ce5c00) at g_io_schedule_up+0x127
g_up_procbody(0,e690bd38,0,c051ee58,0,...) at g_up_procbody+0x5a
fork_exit(c051ee58,0,e690bd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe690bd6c, ebp = 0 ---

Tracing command g_event pid 2 tid 100016 td 0xc7ce5a80
sched_switch(c7ce5a80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c077f09c,c07812c0,0,c0710d6a,245,...) at sleepq_switch+0xc1
sleepq_timedwait(c077f09c) at sleepq_timedwait+0x4a
msleep(c077f09c,0,4c,c0706bf1,64) at msleep+0x259
g_event_procbody(0,e6908d38,0,c051ef18,0,...) at g_event_procbody+0x9e
fork_exit(c051ef18,0,e6908d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe6908d6c, ebp = 0 ---

Tracing command swi3: vm pid 16 tid 100000 td 0xc7ce5000
fork_trampoline() at fork_trampoline

Tracing command swi4: clock sio pid 15 tid 100001 td 0xc7ce3d80
sched_switch(c7ce3d80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8870,e68f0d38,c7cc8870,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7cc8870,e68f0d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68f0d6c, ebp = 0 ---

Tracing command swi1: net pid 14 tid 100002 td 0xc7ce3c00
sched_switch(c7ce3c00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
ithread_loop(c7cc8880,e68edd38,c7cc8880,c0540f8c,0,...) at ithread_loop+0xda
fork_exit(c0540f8c,c7cc8880,e68edd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68edd6c, ebp = 0 ---

Tracing command idle: cpu0 pid 13 tid 100003 td 0xc7ce3a80
sched_switch(c07120f0,6a9,c07cee5c,e68eac40,c05798ee,...) at sched_switch+0x177
(null)(c0712111,50000,c7ccb010,c07819d0,0,...) at 0
__func__.0(c054cd38,0,2041,1,3db17,...) at 0xc0712111
(null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa

Tracing command idle: cpu1 pid 12 tid 100004 td 0xc7ce3900
kdb_enter(c072a005) at kdb_enter+0x2b
siointr1(c7e8c800,c07df9c0,0,c0729e0b,56e,...) at siointr1+0xce
siointr(c7e8c800) at siointr+0x21
intr_execute_handlers(c7cd74c8,e68e7c94,4,e68e7cd8,c06b1b23,...) at intr_execute_handlers+0xe1
lapic_handle_intr(38) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc06aaa4d, esp = 0xe68e7cd8, ebp = 0xe68e7cd8 ---
acpi_cpu_c1(c07812c0,2,e68e7cf8,2,c7ce2860,...) at acpi_cpu_c1+0x5
acpi_cpu_idle(e68e7d0c,c05402e9,c054028c,e68e7d24,c05400cc,...) at acpi_cpu_idle+0x15d
cpu_idle(c054028c,e68e7d24,c05400cc,0,e68e7d38,...) at cpu_idle+0x28
idle_proc(0,e68e7d38,0,c054028c,0,...) at idle_proc+0x5d
fork_exit(c054028c,0,e68e7d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe68e7d6c, ebp = 0 ---

Tracing command idle: cpu2 pid 11 tid 100005 td 0xc7ce3780
sched_switch(c07887c0,0,ffc06034) at sched_switch+0x177
mi_switch(c054c729,c07cee58,e68e4c1c,c057a284,c07887c0,...) at mi_switch+0x270
end(c0712111,50000,c7ccb010,c07819d0,0,...) at 0xe68e4bf0
__func__.0(c054cd38,0,2041,1,3db17,...) at 0xc0712111
(null)(2f007865,2f727375,2f637273,2f737973,6e72656b,...) at 0xa

Tracing command idle: cpu3 pid 10 tid 100006 td 0xc7ce3600
sched_switch(28,12,c06b1eb1,c07f95b0,8,...) at sched_switch+0x177

Tracing command init pid 1 tid 100007 td 0xc7ce3480
sched_switch(c7ce3480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c7ce8000) at sleepq_switch+0xc1
sleepq_wait_sig(c7ce8000) at sleepq_wait_sig+0x1d
msleep(c7ce8000,c7ce8068,15c,c0713046,0) at msleep+0x26e
kern_wait(c7ce3480,ffffffff,e68dec84,0,0) at kern_wait+0x6bb
wait4(c7ce3480,e68ded04) at wait4+0x2a
syscall(3b,3b,3b,0,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x8052f47, esp = 0xbfbfed7c, ebp = 0xbfbfed98 ---

Tracing command swapper pid 0 tid 0 td 0xc077f3c0
sched_switch(c077f3c0,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
scheduler(0,c1ec00,c1e000,0,c0440585,...) at scheduler+0x199
mi_startup() at mi_startup+0x96
begin() at begin+0x2c
db> ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
  897   784   897     0  Ss      ufs      0xc820f46c inetd
  894   893   893     0  S       ufs      0xc82481b4 sh
  893   892   893     0  Ss      wait     0xc8964c90 sh
  892   711   711     0  S       piperd   0xc8107990 cron
  891   784   891     0  Ss      biord    0xdbdb3fdc inetd
  851   848   851     0  S+      getblk   0xdbfdafa4 mc
  848   847   848     0  S+      pause    0xc851dcc4 csh
  847   842   847  1001  S+      wait     0xc851da78 su
  842   841   842  1001  Ss+     pause    0xc835224c csh
  841   837   837  1001  S       select   0xc07cef64 sshd
  837   694   837     0  Ss      sbwait   0xc827720c sshd
  836   764   764 65534  S       lockf    0xc80bba80 httpd
  835   764   764 65534  S       lockf    0xc8113680 httpd
  834   764   764 65534  S       lockf    0xc80c0840 httpd
  833   764   764 65534  S       lockf    0xc8112880 httpd
  832   764   764 65534  S       lockf    0xc7ea6080 httpd
  831   764   764 65534  S       lockf    0xc80c08c0 httpd
  830   764   764 65534  S       lockf    0xc81124c0 httpd
  829   764   764 65534  S       lockf    0xc80c0800 httpd
  828   764   764 65534  S       lockf    0xc80bc840 httpd
  827   764   764 65534  S       lockf    0xc80bc600 httpd
  826   764   764 65534  S       lockf    0xc80c0980 httpd
  825   764   764 65534  S       lockf    0xc7ea8cc0 httpd
  824   764   764 65534  S       lockf    0xc80bbb00 httpd
  823   764   764 65534  S       lockf    0xc80c15c0 httpd
  822   764   764 65534  S       lockf    0xc7ea8d00 httpd
  821   764   764 65534  S       lockf    0xc8112180 httpd
  820   764   764 65534  S       lockf    0xc8113700 httpd
  819   764   764 65534  S       lockf    0xc81123c0 httpd
  818   764   764 65534  S       lockf    0xc7ea8dc0 httpd
  817   764   764 65534  S       lockf    0xc80c1580 httpd
  816   764   764 65534  S       lockf    0xc7ea6b80 httpd
  815   764   764 65534  S       lockf    0xc80c0a00 httpd
  814   764   764 65534  S       lockf    0xc80c0600 httpd
  813   764   764 65534  S       lockf    0xc7ea8d40 httpd
  812   764   764 65534  S       lockf    0xc80bba00 httpd
  811   764   764 65534  S       lockf    0xc80bc7c0 httpd
  810   764   764 65534  S       lockf    0xc8112240 httpd
  809   764   764 65534  S       lockf    0xc80c1e80 httpd
  808   764   764 65534  S       lockf    0xc8112300 httpd
  807   764   764 65534  S       select   0xc07cef64 httpd
  806     1   806     0  Ss+     vnread   0xdbd4b0bc getty
  805     1   805     0  Ss+     ttyin    0xc7e9c410 getty
  804     1   804     0  Ss+     ttyin    0xc7ea4010 getty
  803     1   803     0  Ss+     ttyin    0xc7ea3c10 getty
  802     1   802     0  Ss+     ttyin    0xc7ea4c10 getty
  801     1   801     0  Ss+     ttyin    0xc7ea4810 getty
  800     1   800     0  Ss+     ttyin    0xc7ea2c10 getty
  799     1   799     0  Ss+     ttyin    0xc7ea2810 getty
  798     1   798     0  Ss+     ttyin    0xc7ea3810 getty
  784     1   784     0  Ss      select   0xc07cef64 inetd
  764     1   764     0  Ss      select   0xc07cef64 httpd
  763   733    45   509  S+      (threaded)          mysqld
100133                   S       kserel   0xc8154034 mysqld
100135                   S       kserel   0xc8154034 mysqld
100091                   S       kserel   0xc8154034 mysqld
100127                   S       kserel   0xc8154034 mysqld
100134                   S       select   0xc07cef64 mysqld
100131                   S       sigwait  0xea845c14 mysqld
100126                   S       ksesigwa 0xc8156138 mysqld
  733     1    45     0  S+      wait     0xc8156a78 sh
  730     1   730     0  Ss      nanslp   0xc0781bec cron
  722     1   721     0  S       select   0xc07cef64 snmpd
  711     1   711     0  Ss      nanslp   0xc0781bec cron
  705     1   705    25  Ss      pause    0xc80e2894 sendmail
  699     1   699     0  Ss      select   0xc07cef64 sendmail
  694     1   694     0  Ss      select   0xc07cef64 sshd
  669     1   669     0  Ss      select   0xc07cef64 ntpd
  638     1   638     0  Ss      select   0xc07cef64 usbd
  591     0     0     0  SL      -        0xc077f760 [accounting]
  574     1   574    53  Ss      select   0xc07cef64 named
  554     1   554     0  Ss      select   0xc07cef64 syslogd
  537     1   537     0  Ss      nanslp   0xc0781bec ipmon
  441     1   441     0  Ss      select   0xc07cef64 devd
  170     1   170     0  Ss      pause    0xc808567c adjkerntz
   44     0     0     0  SL      -        0xe8380d04 [schedcpu]
   43     0     0     0  SL      sdflush  0xc07dca74 [softdepflush]
   42     0     0     0  SL      vlruwt   0xc8084860 [vnlru]
   41     0     0     0  SL      wswbuf0  0xc0766324 [syncer]
   40     0     0     0  SL      psleep   0xc07cf4e0 [bufdaemon]
   39     0     0     0  SL      pgzero   0xc07dda44 [pagezero]
   38     0     0     0  SL      psleep   0xc07dd54c [vmdaemon]
   37     0     0     0  SL      psleep   0xc07dd508 [pagedaemon]
   36     0     0     0  WL                          [irq7: ppc0]
   35     0     0     0  SL      -        0xc7e1e63c [fdc0]
   34     0     0     0  WL                          [swi0: sio]
   33     0     0     0  WL                          [irq1: atkbd0]
   32     0     0     0  WL                          [irq22: fxp0]
   31     0     0     0  WL                          [irq20: mly0]
   30     0     0     0  WL                          [irq15: ata1]
   29     0     0     0  WL                          [irq14: ata0]
   28     0     0     0  SL      usbevt   0xc7dae210 [usb1]
   27     0     0     0  WL                          [irq17: ehci0]
   26     0     0     0  SL      usbtsk   0xc077ec04 [usbtask]
   25     0     0     0  SL      usbevt   0xc7db4210 [usb0]
   24     0     0     0  WL                          [irq16: ohci0]
   23     0     0     0  WL                          [irq9: acpi0]
   22     0     0     0  WL                          [swi5: +]
   21     0     0     0  SL      -        0xc7d43d80 [thread taskq]
   20     0     0     0  WL                          [swi6: Giant taskq]
   19     0     0     0  WL                          [swi6: task queue]
    9     0     0     0  SL      -        0xc7d73100 [kqueue taskq]
    8     0     0     0  SL      -        0xc7d73180 [acpi_task_2]
    7     0     0     0  SL      -        0xc7d73180 [acpi_task_1]
    6     0     0     0  SL      -        0xc7d73180 [acpi_task_0]
   18     0     0     0  WL                          [swi2: cambio]
    5     0     0     0  SL      ccb_scan 0xc0776004 [xpt_thrd]
   17     0     0     0  SL      -        0xc077cc00 [yarrow]
    4     0     0     0  SL      -        0xc077f0a8 [g_down]
    3     0     0     0  SL      -        0xc077f0a4 [g_up]
    2     0     0     0  SL      -        0xc077f09c [g_event]
   16     0     0     0  WL                          [swi3: vm]
   15     0     0     0  WL                          [swi4: clock sio]
   14     0     0     0  WL                          [swi1: net]
   13     0     0     0  RL      CPU 0               [idle: cpu0]
   12     0     0     0  RL      CPU 1               [idle: cpu1]
   11     0     0     0  RL      CPU 2               [idle: cpu2]
   10     0     0     0  RL      CPU 3               [idle: cpu3]
    1     0     1     0  SLs     wait     0xc7ce8000 [init]
    0     0     0     0  WLs                         [swapper]
db>

--
Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
Phone: +7 4732 539880
Fax:   +7 4732 531415         http://www.vsi.ru
CenterTelecom Voronezh ISP    http://isp.vsi.ru
Comment 7 Oleg Derevenetz 2007-11-01 16:54:39 UTC
Here is a kernel dump:

ftp://oleg.vsi.ru/private/vmcore.0.zip

It was saved for another static hang state.

--
Oleg Derevenetz <oleg@vsi.ru> OOD3-RIPE
Phone: +7 4732 539880
Fax:   +7 4732 531415         http://www.vsi.ru
CenterTelecom Voronezh ISP    http://isp.vsi.ru
Comment 8 Kostik Belousov 2007-11-01 17:00:34 UTC
On Thu, Nov 01, 2007 at 07:54:39PM +0300, Oleg Derevenetz wrote:
> Here is a kernel dump:
> 
> ftp://oleg.vsi.ru/private/vmcore.0.zip
> 
> It was saved for another static hang state.


I already analyzed one dump for you.
Comment 9 Mark Linimon freebsd_committer freebsd_triage 2009-05-18 04:02:12 UTC
Responsible Changed
From-To: freebsd-bugs->freebsd-fs

Over to maintainer(s).
Comment 10 Eitan Adler freebsd_committer freebsd_triage 2018-05-28 19:49:26 UTC
batch change:

For bugs that match the following
-  Status Is In progress 
AND
- Untouched since 2018-01-01.
AND
- Affects Base System OR Documentation

DO:

Reset to open status.


Note:
I did a quick pass but if you are getting this email it might be worthwhile to double check to see if this bug ought to be closed.
Comment 11 Kirk McKusick freebsd_committer freebsd_triage 2018-05-29 14:23:14 UTC
This particular bug was fixed by the rewrite of the code to deal with low buffer memory situations. Some other low-memory issues have arisen that are discussed in later bugs such as https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224292