Bug 107109
| Summary: | [nfs] Netbeans/Eclipse/Java provoke deadlocks when used with NFS | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | Joerg Lehners <Joerg.Lehners> | ||||||||
| Component: | kern | Assignee: | Alfred Perlstein <alfred> | ||||||||
| Status: | Closed FIXED | ||||||||||
| Severity: | Affects Only Me | ||||||||||
| Priority: | Normal | ||||||||||
| Version: | 6.2-PRERELEASE | ||||||||||
| Hardware: | Any | ||||||||||
| OS: | Any | ||||||||||
| Attachments: |
|
||||||||||
|
Description
Joerg Lehners
2006-12-22 21:10:13 UTC
I have experimented further and narrowed down the problem somewhat.
In the meantime I upgraded to FreeBSD 6.2-STABLE as of 8. Jan 2007
and Java 1.5.0 p4 (patchkit 4 from a few days ago).
I got someone to wrote a short Java program for me (thanks to Felix
Kronlage!) that is able to provoke the problem more easily than with
Netbeans and 'do this and than that and ...'.
With the program running the Java process locks up after a few seconds
of running it, sometime it takes longer. As I suspected initially it is the
intermixed fork()ing and stat()ing of NFS files/directories in different
threads and some other magic.
I tried the Java program with Java 1.5.0 for FreeBSD (patchkit 4), which
locks up after some seconds. With Java 1.5.0 for Linux the program crashes
after some seconds with some HotSpot error (which is not uncommon for
Java 1.5.0 for Linux on FreeBSD 6). With Java 1.6.0 for Linux the
program ran for many minutes without any crashes or lockups jet.
I also tried Java 1.4.2 for FreeBSD (built from Java SCSL source code with
patchkit 8): no crashes or lockups even after many minutes of running.
So there seems to be some magic in Java 1.5.0 for FreeBSD (built from
Java SCSL source code with at least patchkit p3 and p4) that provoke
some deadlock in the kernel when threads do execute fork() and stat()
on NFS files/directories intermixed.
In the program NFSStressTest.java the variable 'directories' must list
some directories that are accessible via NFS.
--- NFSStressTest.java begins here ---
import java.io.*;
public class NFSStressTest {
public static void main(String[] args) {
int intervall= 1000;
boolean debug= true;
String[] directories = { "/user/admin/joerg/Src/%Archiv",
"/user/admin/joerg/Src/%DONE",
"/user/admin/joerg/Bin" };
String[] commands = { "/bin/echo 'hallo'",
"/bin/ls /tmp" ,
"/sbin/md5 /bin/ls" };
IOMaker[] ioMakers= new IOMaker[directories.length];
CMDMaker[] cmdMakers= new CMDMaker[commands.length];
for(int i=0;i<directories.length;i++) {
ioMakers[i]= new IOMaker(directories[i], intervall);
ioMakers[i].setDebug(false);
ioMakers[i].start();
}
for(int i=0;i<commands.length;i++) {
cmdMakers[i]= new CMDMaker(commands[i], intervall);
cmdMakers[i].setDebug(true);
cmdMakers[i].start();
}
}
}
class IOMaker extends Thread {
private String directory;
private int intervall;
private boolean debug= false;
IOMaker(String directory) {
this.directory= directory;
}
IOMaker(String directory, int intervall) {
this.directory= directory;
this.intervall= intervall;
}
void setDebug(boolean debug) {
this.debug= debug;
}
public void run() {
File myDirectory= new File(this.directory);
while(true) {
File[] myFiles= myDirectory.listFiles();
if(myFiles == null)
break;
for(int i=0;i<myFiles.length;i++) {
String absolutePath= myFiles[i].getAbsolutePath();
long lastModified= myFiles[i].lastModified();
if(this.debug) {
System.out.println(this +": " + absolutePath +" : " + lastModified);
}
}
Thread.yield();
try {
Thread.sleep(this.intervall);
} catch (Exception e) {
System.out.println("Caught exception: " + e.getMessage());
e.printStackTrace();
}
}
}
}
class CMDMaker extends Thread {
private String command;
private int intervall;
private boolean debug= false;
CMDMaker(String command, int intervall) {
this.command= command;
this.intervall= intervall;
}
void setDebug(boolean debug) {
this.debug= debug;
}
public void run() {
while(true) {
try {
if(this.debug) {
System.out.println("command: " + this.command);
}
Runtime.getRuntime().exec(this.command);
} catch (IOException e) {
System.out.println("Caught exception in CMDMaker ("+ this.command +
": "+ e.getMessage());
e.printStackTrace();
}
Thread.yield();
try {
Thread.sleep(this.intervall);
} catch (Exception e) {
System.out.println("Caught exception: " + e.getMessage());
e.printStackTrace();
}
}
}
}
--- NFSStressTest.java ends here ---
--
Mail: Joerg.Lehners@Informatik.Uni-Oldenburg.DE Tel: 2198
Real: Joerg Lehners, Informatik ARBI, Uni Oldenburg, D-26111 Oldenburg
Unwoerter: Kostensenkung - Gewinnmaximierung - billig, billig, billig
Responsible Changed From-To: freebsd-bugs->glewis I'll take a look at this from the Java perspective. Finally, I was able to reproduce the problem with Java 1.4.2 for FreeBSD
(patchkit 8).
Traces of the offending process:
db> ps
...
16569 319 16569 0 Ss select 0xc07e30e4 xdm
14940 352 14940 1002 T+ (threaded) java
100119 s java
100087 D nfs 0xc69e81b4 java
100126 D nfs 0xc69e81b4 java
100081 s java
100068 s java
731 704 731 1002 S+ select 0xc07e30e4 top
...
db> trace 14940
Tracing pid 14940 tid 100119 td 0xc747fd80
sched_switch(c747fd80,0,1) at sched_switch+0x177
mi_switch(1,0,c6b60498,0,c071e095,335,c747fd80,c6b60430) at mi_switch+0x270
thread_single(0,c6b60498,0,c071a546,111) at thread_single+0x24f
fork1(c747fd80,14,0,e8ec3cd4) at fork1+0x16b
fork(c747fd80,e8ec3d04) at fork+0x18
syscall(3b,8ca6003b,bf6f003b,8051830,8818fa24,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (2, FreeBSD ELF32, fork), eip = 0x881026cf, esp = 0xbf6320c8, ebp = 0xbf632114 ---
db> trace 100087
Tracing pid 14940 tid 100087 td 0xc7480180
sched_switch(c7480180,c747fd80,1) at sched_switch+0x177
mi_switch(1,c747fd80) at mi_switch+0x270
sleepq_switch(c69e81b4,c0797c60,0,c071ff50,211,...) at sleepq_switch+0xc1
sleepq_wait(c69e81b4,0,c69e8224,b7,c0725ece,...) at sleepq_wait+0x46
msleep(c69e81b4,c0796a74,50,c072c707,0,...) at msleep+0x279
acquire(e8ec9998,40,60000,c7480180,0,...) at acquire+0x76
lockmgr(c69e81b4,2002,c69e8224,c7480180,e8ec99bc,...) at lockmgr+0x44e
vop_stdlock(e8ec99dc) at vop_stdlock+0x21
VOP_LOCK_APV(c0778160,e8ec99dc) at VOP_LOCK_APV+0x87
vn_lock(c69e815c,2002,c7480180,c69e815c) at vn_lock+0xac
vget(c69e815c,2002,c7480180) at vget+0xc2
vfs_hash_get(c67f75c8,4c8a5263,2,c7480180,e8ec9a84,c0601488,e8ec9a88) at vfs_hash_get+0x8d
nfs_nget(c67f75c8,c69e3790,20,e8ec9ab4) at nfs_nget+0x93
nfs_root(c67f75c8,2,e8ec9b00,c7480180,1,...) at nfs_root+0x25
lookup(e8ec9c00) at lookup+0x743
namei(e8ec9c00) at namei+0x39a
kern_stat(c7480180,8239e80,0,e8ec9c74) at kern_stat+0x35
stat(c7480180,e8ec9d04) at stat+0x1b
syscall(3b,8ca6003b,bf6f003b,818,8c4c94f0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x8815e447, esp = 0xbf6f5b1c, ebp = 0xbf6f5bb8 ---
db> trace 100126
Tracing pid 14940 tid 100126 td 0xc7480300
sched_switch(c7480300,c7480180,1) at sched_switch+0x177
mi_switch(1,c7480180) at mi_switch+0x270
sleepq_switch(c69e81b4,c0797c60,0,c071ff50,211,...) at sleepq_switch+0xc1
sleepq_wait(c69e81b4,0,c69e8224,b7,c0725ece,...) at sleepq_wait+0x46
msleep(c69e81b4,c0796a74,50,c072c707,0,...) at msleep+0x279
acquire(e8ecc998,40,60000,c7480300,0,...) at acquire+0x76
lockmgr(c69e81b4,2002,c69e8224,c7480300,e8ecc9bc,...) at lockmgr+0x44e
vop_stdlock(e8ecc9dc) at vop_stdlock+0x21
VOP_LOCK_APV(c0778160,e8ecc9dc) at VOP_LOCK_APV+0x87
vn_lock(c69e815c,2002,c7480300,c69e815c) at vn_lock+0xac
vget(c69e815c,2002,c7480300) at vget+0xc2
vfs_hash_get(c67f75c8,4c8a5263,2,c7480300,e8ecca84,c0601488,e8ecca88) at vfs_hash_get+0x8d
nfs_nget(c67f75c8,c69e3790,20,e8eccab4) at nfs_nget+0x93
nfs_root(c67f75c8,2,e8eccb00,c7480300,1,...) at nfs_root+0x25
lookup(e8eccc00) at lookup+0x743
namei(e8eccc00) at namei+0x39a
kern_stat(c7480300,820fb60,0,e8eccc74) at kern_stat+0x35
stat(c7480300,e8eccd04) at stat+0x1b
syscall(3b,8ca6003b,bf6f003b,30,8c4cf7a0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x8815e447, esp = 0xbf6b4b1c, ebp = 0xbf6b4bb8 ---
db>
Looking around in /sys/kern/* one can see that a process/thread doing a
fork() needs to suspend all other threads first (thread_single()) in
/sys/kern/kern_thread.c called from fork1() in /sys/kern/kern_fork.c.
The comment at the top of thread_single() says amongst other things:
Threads in the kernel must be
* allowed to continue until they get to the user boundary.
Somehow this cannot happen due to some lock contention/deadlock
when there are outstanding NFS request. So the whole thing locks up!
Someone with experience in the locking model of the kernel
might want to look, where the deadlock results from.
--
Mail: Joerg.Lehners@Informatik.Uni-Oldenburg.DE Tel: 2198
Real: Joerg Lehners, Informatik ARBI, Uni Oldenburg, D-26111 Oldenburg
Unwoerter: Kostensenkung - Gewinnmaximierung - billig, billig, billig
On Tue, 30 Jan 2007, Joerg Lehners wrote: > Looking around in /sys/kern/* one can see that a process/thread doing a > fork() needs to suspend all other threads first (thread_single()) in > /sys/kern/kern_thread.c called from fork1() in /sys/kern/kern_fork.c. > The comment at the top of thread_single() says amongst other things: > > Threads in the kernel must be > * allowed to continue until they get to the user boundary. Reading more: * allowed to continue until they get to the user boundary. They may even * copy out their return values and data before suspending. They may however be * accellerated in reaching the user boundary as we will wake up * any sleeping threads that are interruptable. (PCATCH). 'PCATCH'! I mounted the NFS volumes with 'intr' (see above in the original PR) to be able to CTRL-C/SIGINT out of dead NFS mounts. I mounted the offending NFS Volume without 'intr' and was not yet able to reproduce the deadlock. So I think: the issue is not specific to Java, but specific to processes that are threaded where at least one thread is doing fork() regulary and there are other threads doing NFS I/O on a NFS volume mounted with 'intr' at the same time. As an immediate measure against locked up process making the machines unusable I'll mount NFS volumes without 'intr' for now. But I won't miss 'intr' for the long term, because being able to CTRL-C/SIGINT processes out of dead NFS mounts is important! I'll try to write a threaded C program to try to show that this issue is not specific to Java. Perhaps someone ele is faster? :) -- Mail: Joerg.Lehners@Informatik.Uni-Oldenburg.DE Tel: 2198 Real: Joerg Lehners, Informatik ARBI, Uni Oldenburg, D-26111 Oldenburg Unwoerter: Kostensenkung - Gewinnmaximierung - billig, billig, billig Responsible Changed From-To: glewis->freebsd-bugs Putting this back in the pool. This turns out to be not specific to Java (its just easy to aggravate the problem with a multi-threaded Java process). Submitter has added some very useful traces and analysis which should help someone who is familiar with the kernel resolve the problem. On Tue, 30 Jan 2007, Joerg Lehners wrote: > [...] > I'll try to write a threaded C program to try to show that > this issue is not specific to Java. Perhaps someone ele is faster? :) Done, see attached file. While writing this short program and trying to reproduce the problem more easily I discovered that it is NOT neccessary to do stat() on files/directories that are on a NFS volume. It is neccessary for the test program to have its CURRENT WORKING DIRECTORY on a NFS volume that is mounted with 'intr'. For the I/O part I decided have the program do a opendir()/readdir()/stat() loop over the root-directory. So to provoke to lockup: compile and run the program with a working directory on a NFS volume mounted with 'intr'. I got lockups within milliseconds of runtime. So the subject of this PR is now wrong: the issue is not Netbeans/Eclipse/ Java specific at all. Someone might want to change the subject of this PR. Suggestion: NFS volumes with 'intr' may cause deadlocks when used with threaded programs --- c.c begins here --- #include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <pthread.h> #include <errno.h> #include <time.h> #include <sys/types.h> #include <sys/wait.h> #include <sys/stat.h> #include <dirent.h> void *forker(void *); void *looper(void *); int main (int argc, char **argv) { pthread_t forker_thrid1; pthread_t forker_thrid2; pthread_t looper_thrid1; pthread_t looper_thrid2; pthread_t looper_thrid3; fprintf (stderr, "started!\n"); pthread_create (&forker_thrid1, NULL, forker, NULL); fprintf (stderr, "forker1 created!\n"); pthread_create (&forker_thrid2, NULL, forker, NULL); fprintf (stderr, "forker2 created!\n"); pthread_create (&looper_thrid1, NULL, looper, NULL); fprintf (stderr, "looper1 created!\n"); pthread_create (&looper_thrid2, NULL, looper, NULL); fprintf (stderr, "looper2 created!\n"); pthread_create (&looper_thrid3, NULL, looper, NULL); fprintf (stderr, "looper3 created!\n"); pause(); exit (0); } void *forker(void *arg) { pid_t child_pid; int child_status; struct timespec sleep_time; fprintf (stderr, "forker started\n"); while (1) { child_pid = fork(); if (child_pid < 0) { fprintf (stderr, "fork failed, errno: %d\n", errno); } else if (child_pid == 0) { /* child */ exit (0); } else { /* parent */ wait (&child_status); /* fprintf (stderr, "forked and waited for the child\n"); */ fprintf (stderr, "f"); } sleep_time.tv_sec = 0; sleep_time.tv_nsec = 10 * 1000 * 1000; nanosleep (&sleep_time, NULL); } return (NULL); } void *looper(void *arg) { char *dirname; DIR* dir; struct dirent dir_entry; struct dirent *dir_entry_p; struct stat stat_buf; int count; struct timespec sleep_time; fprintf (stderr, "looper started\n"); dirname = "/"; dir = opendir (dirname); if (dir == NULL) { fprintf (stderr, "opendir of %s failed\n", dirname); return (NULL); } fprintf (stderr, "opened directory %s\n", dirname); while (1) { count = 0; rewinddir (dir); while (1) { if (readdir_r (dir, &dir_entry, &dir_entry_p) < 0) { fprintf (stderr, "error reading directory\n"); break; } if (dir_entry_p == NULL) break; /* fprintf (stderr, "filename: %s\n", dir_entry.d_name); */ count++; if (stat (dir_entry.d_name, &stat_buf) < 0) { /* fprintf (stderr, "error stat()ing %s\n", dir_entry.d_name); */ } } /* fprintf (stderr, "looper looped, %d entries\n", count); */ fprintf (stderr, "l"); sleep_time.tv_sec = 0; sleep_time.tv_nsec = 100 * 1000 * 1000; nanosleep (&sleep_time, NULL); } return (NULL); } --- c.c ends here --- --- Makefile begins here --- CFLAGS=-Wall LDFLAGS=-lpthread c: c.o cc -o c c.o ${LDFLAGS} c.o: c.c --- Makefile ends here --- -- Mail: Joerg.Lehners@Informatik.Uni-Oldenburg.DE Tel: 2198 Real: Joerg Lehners, Informatik ARBI, Uni Oldenburg, D-26111 Oldenburg Unwoerter: Kostensenkung - Gewinnmaximierung - billig, billig, billig On Fri, 2 Feb 2007, Joerg Lehners wrote: > On Tue, 30 Jan 2007, Joerg Lehners wrote: >> [...] >> I'll try to write a threaded C program to try to show that >> this issue is not specific to Java. Perhaps someone ele is faster? :) > > Done, see attached file. > [...] On request of Craig Rodrigues I put the following ddb ouput here. Note: this output is from another machine (testido). The ddb output from 22 Dec. 2006 was from the machine cenis, which has slightly different hardware and its kernel config did not include all debugging options listed in kerneldebug-deadlocks.html. --- ddb.output begins here --- FreeBSD/i386 (testido) (ttyd0) login: [thread pid 10 tid 100006 ] Stopped at kdb_enter+0x2b: nop db> ps pid ppid pgrp uid state wmesg wchan cmd 371 355 371 1002 T+ (threaded) c 100078 s c 100081 D nfs 0xc36dc5c8 c 100080 s c 100077 Ss ksesigwa 0xc3515dc8 c 355 354 355 1002 S+ pause 0xc370c67c ksh 354 353 354 0 Ss+ wait 0xc36c1a78 login 353 152 353 0 Ss select 0xc07dbf24 rlogind 352 1 337 0 S select 0xc07dbf24 xconsole 331 301 331 0 Ss select 0xc07dbf24 xdm 325 304 325 0 S select 0xc07dbf24 Xorg 304 301 304 0 Ss pause 0xc370567c ksh 301 1 301 0 Ss select 0xc07dbf24 xdm 300 1 300 0 Ss+ ttyin 0xc33bb410 getty 299 1 299 0 Ss+ ttyin 0xc33d2410 getty 298 1 298 0 Ss+ ttyin 0xc33bd810 getty 297 1 297 0 Ss+ ttyin 0xc33bd410 getty 296 1 296 0 Ss+ ttyin 0xc33d2c10 getty 290 1 290 0 Ss select 0xc07dbf24 usbd 287 1 40 0 S+ fifoor 0xc34ee6c8 ksh 259 1 259 0 Ss select 0xc07dbf24 nwhod 248 1 248 0 Ss select 0xc07dbf24 lpd 245 1 245 0 Ss nanslp 0xc07913cc cron 243 1 243 0 Ss select 0xc07dbf24 ntpd 238 1 238 501 Ss select 0xc07dbf24 exim-4.66-1 193 188 188 0 S - 0xc34f0a00 nfsd 191 188 188 0 S - 0xc34f0400 nfsd 190 188 188 0 S - 0xc34f0c00 nfsd 189 188 188 0 S - 0xc34ef200 nfsd 188 1 188 0 Ss accept 0xc3600892 nfsd 183 1 183 0 Ss select 0xc07dbf24 mountd 174 1 174 0 Ss select 0xc07dbf24 ypbind 165 1 165 0 Ss select 0xc07dbf24 rpcbind 157 1 157 0 Ss select 0xc07dbf24 sshd 152 1 152 0 Ss select 0xc07dbf24 inetd 147 1 147 0 Ss select 0xc07dbf24 syslogd 135 1 135 0 Ss select 0xc07dbf24 devd 39 0 0 0 SL - 0xd56d9d04 [schedcpu] 38 0 0 0 SL sdflush 0xc07e4f94 [softdepflush] 37 0 0 0 SL vlruwt 0xc33be000 [vnlru] 36 0 0 0 SL syncer 0xc079113c [syncer] 35 0 0 0 SL psleep 0xc07dc48c [bufdaemon] 34 0 0 0 SL pgzero 0xc07e5f04 [pagezero] 33 0 0 0 SL psleep 0xc07e5a54 [vmdaemon] 32 0 0 0 SL psleep 0xc07e5a10 [pagedaemon] 31 0 0 0 WL [irq12: psm0] 30 0 0 0 WL [irq1: atkbd0] 29 0 0 0 WL [swi0: sio] 28 0 0 0 WL [irq7: ppc0] 27 0 0 0 SL - 0xc33a623c [fdc0] 26 0 0 0 WL [irq10: atapci1] 25 0 0 0 SL usbevt 0xc339c210 [usb1] 24 0 0 0 SL usbtsk 0xc078e204 [usbtask] 23 0 0 0 SL usbevt 0xc3390210 [usb0] 22 0 0 0 WL [irq5: xl0 uhci0++] 21 0 0 0 WL [irq15: ata1] 20 0 0 0 WL [irq14: ata0] 19 0 0 0 WL [irq9: acpi0] 18 0 0 0 WL [swi6: Giant taskq] 17 0 0 0 WL [swi6: task queue] 16 0 0 0 WL [swi2: cambio] 9 0 0 0 SL - 0xc3351300 [acpi_task_2] 8 0 0 0 SL - 0xc3351300 [acpi_task_1] 7 0 0 0 SL - 0xc3351300 [acpi_task_0] 6 0 0 0 SL - 0xc3351380 [kqueue taskq] 15 0 0 0 WL [swi5: +] 5 0 0 0 SL - 0xc329f380 [thread taskq] 14 0 0 0 SL - 0xc078b120 [yarrow] 4 0 0 0 SL - 0xc078e888 [g_down] 3 0 0 0 SL - 0xc078e884 [g_up] 2 0 0 0 SL - 0xc078e87c [g_event] 13 0 0 0 WL [swi3: vm] 12 0 0 0 WL [swi4: clock sio] 11 0 0 0 WL [swi1: net] 10 0 0 0 RL CPU 0 [idle] 1 0 1 0 SLs wait 0xc32a5000 [init] 0 0 0 0 WLs [swapper] db> show pcpu cpuid = 0 curthread = 0xc32a1600: pid 10 "idle" curpcb = 0xd44afd90 fpcurthread = none idlethread = 0xc32a1600: pid 10 "idle" APIC ID = 0 currentldt = 0x50 spin locks held: exclusive spin mutex sio r = 0 (0xc07e7f80) locked @ dev/sio/sio.c:1390 db> show allpcpu Current CPU: 0 cpuid = 0 curthread = 0xc32a1600: pid 10 "idle" curpcb = 0xd44afd90 fpcurthread = none idlethread = 0xc32a1600: pid 10 "idle" APIC ID = 0 currentldt = 0x50 spin locks held: exclusive spin mutex sio r = 0 (0xc07e7f80) locked @ dev/sio/sio.c:1390 db> show locks exclusive spin mutex sio r = 0 (0xc07e7f80) locked @ dev/sio/sio.c:1390 db> show alllocks db> show lockedvnods Locked vnodes 0xc36dc570: tag nfs, type VDIR usecount 5, writecount 0, refcount 5 mountedhere 0 flags (VV_ROOT) lock type nfs: EXCL (count 1) by thread 0xc3709900 (pid 371) with 1 pending#0 0xc054ec81 at lockmgr+0x4ed #1 0xc05a6c39 at vop_stdlock+0x21 #2 0xc06da3ff at VOP_LOCK_APV+0x87 #3 0xc05bbc2c at vn_lock+0xac #4 0xc05a885e at lookup+0xd2 #5 0xc05a850e at namei+0x39a #6 0xc05b5f65 at kern_stat+0x35 #7 0xc05b5f13 at stat+0x1b #8 0xc06c49fb at syscall+0x25b #9 0xc06b2b3f at Xint0x80_syscall+0x1f fileid 2 fsid 0x100ff05 db> alltrace Tracing command c pid 371 tid 100078 td 0xc3709600 sched_switch(c3709600,0,1) at sched_switch+0x177 mi_switch(1,0,c3515cf8,0,c0717139,335,c3709600,c3515c90) at mi_switch+0x270 thread_single(0,c3515cf8,0,c07135ea,111) at thread_single+0x24f fork1(c3709600,14,0,da9dacd4) at fork1+0x16b fork(c3709600,da9dad04) at fork+0x18 syscall(3b,3b,3b,8807c050,8818aa24,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (2, FreeBSD ELF32, fork), eip = 0x880fd6cf, esp = 0xbf7fcf4c, ebp = 0xbf7fcf98 --- Tracing command c pid 371 tid 100081 td 0xc3709780 sched_switch(c3709780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c36dc5c8,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c36dc5c8,0,c36dc638,b7,c071ef72,...) at sleepq_wait+0x46 msleep(c36dc5c8,c0790268,50,c07257ab,0,...) at msleep+0x279 acquire(da9dda64,40,60000,c3709780,0,...) at acquire+0x76 lockmgr(c36dc5c8,3002,c36dc638,c3709780,da9dda88,...) at lockmgr+0x44e vop_stdlock(da9ddaa8) at vop_stdlock+0x21 VOP_LOCK_APV(c0770fa0,da9ddaa8) at VOP_LOCK_APV+0x87 vn_lock(c36dc570,1002,c3709780) at vn_lock+0xac lookup(da9ddc00) at lookup+0xd2 namei(da9ddc00) at namei+0x39a kern_stat(c3709780,bf6fbe98,0,da9ddc74) at kern_stat+0x35 stat(c3709780,da9ddd04) at stat+0x1b syscall(3b,3b,3b,8807c050,bf6fbfec,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x88159447, esp = 0xbf6fbdec, ebp = 0xbf6fbfb8 --- Tracing command c pid 371 tid 100080 td 0xc3709900 sched_switch(c3709900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c37efe00) at sleepq_switch+0xc1 sleepq_wait_sig(c37efe00) at sleepq_wait_sig+0x1d msleep(c37efe00,c07df2e0,153,c072514f,0) at msleep+0x26a nfs_reply(c37efe00,0,29d,0,c34e0300,...) at nfs_reply+0x1ec nfs_request(c36dc570,c34e2100,3,c3709900,c37efc00,da9e0a04,da9e0a08,da9e0a0c,c3709900,da9e0a18) at nfs_request+0x371 nfs_lookup(da9e0b14) at nfs_lookup+0x2d4 VOP_LOOKUP_APV(c0770fa0,da9e0b14) at VOP_LOOKUP_APV+0x87 lookup(da9e0c00) at lookup+0x46e namei(da9e0c00) at namei+0x39a kern_stat(c3709900,bf4f9e98,0,da9e0c74) at kern_stat+0x35 stat(c3709900,da9e0d04) at stat+0x1b syscall(3b,3b,3b,8807c050,bf4f9fec,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x88159447, esp = 0xbf4f9dec, ebp = 0xbf4f9fb8 --- Tracing command c pid 371 tid 100077 td 0xc351bc00 sched_switch(c351bc00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c3515dc8) at sleepq_switch+0xc1 sleepq_timedwait_sig(c3515dc8) at sleepq_timedwait_sig+0x1e msleep(c3515dc8,c3515cf8,168,c0713c77,7531,...) at msleep+0x240 kse_release(c351bc00,d97f7d04) at kse_release+0x121 syscall(3b,3b,3b,804d100,81,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (383, FreeBSD ELF32, kse_release), eip = 0x880a04f7, esp = 0xbf9fef2c, ebp = 0xbf9fef88 --- Tracing command ksh pid 355 tid 100073 td 0xc3706300 sched_switch(c3706300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c370c67c) at sleepq_switch+0xc1 sleepq_wait_sig(c370c67c) at sleepq_wait_sig+0x1d msleep(c370c67c,c370c6b0,168,c07166be,0) at msleep+0x26a kern_sigsuspend(c3706300,0,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c3706300,da9b6d04) at sigsuspend+0x33 syscall(3b,3b,3b,0,8079be8,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x880eb797, esp = 0xbfbfea8c, ebp = 0xbfbfeab8 --- Tracing command login pid 354 tid 100061 td 0xc3519480 sched_switch(c3519480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c36c1a78) at sleepq_switch+0xc1 sleepq_wait_sig(c36c1a78) at sleepq_wait_sig+0x1d msleep(c36c1a78,c36c1ae0,15c,c071af18,0) at msleep+0x26a kern_wait(c3519480,163,d97cac84,0,0) at kern_wait+0x6b7 wait4(c3519480,d97cad04) at wait4+0x2a syscall(3b,3b,3b,246,1,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x880fe6af, esp = 0xbfbfed2c, ebp = 0xbfbfed48 --- Tracing command rlogind pid 353 tid 100055 td 0xc3519d80 sched_switch(c3519d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c3519d80,4,bfbfe0d8,0,bfbfe0dc,...) at kern_select+0x4bf select(c3519d80,d97dcd04) at select+0x44 syscall(3b,3b,3b,0,3,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x8814e2c3, esp = 0xbfbfe06c, ebp = 0xbfbfe908 --- Tracing command xconsole pid 352 tid 100076 td 0xc351bd80 sched_switch(c351bd80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,3b7,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 poll(c351bd80,d97fad04) at poll+0x3b9 syscall(3b,3b,3b,805a9e0,6,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (209, FreeBSD ELF32, poll), eip = 0x882b0dc3, esp = 0xbfbfea5c, ebp = 0xbfbfea78 --- Tracing command xdm pid 331 tid 100065 td 0xc3709000 sched_switch(c3709000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,3b7,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 poll(c3709000,da9ced04) at poll+0x3b9 syscall(3b,3b,3b,0,1,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (209, FreeBSD ELF32, poll), eip = 0x8829fdc3, esp = 0xbfbfe5ec, ebp = 0xbfbfe608 --- Tracing command Xorg pid 325 tid 100058 td 0xc3519900 sched_switch(c3519900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07dbf24,c07dbf24,ea4c,d97d3b40,203246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07dbf24,c07dbf00,ea4c) at cv_timedwait_sig+0x192 kern_select(c3519900,14,81efe60,0,0,...) at kern_select+0x4a9 select(c3519900,d97d3d04) at select+0x44 syscall(3b,3b,3b,0,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x882d52c3, esp = 0xbfbfe6ec, ebp = 0xbfbfe938 --- Tracing command ksh pid 304 tid 100066 td 0xc3706d80 sched_switch(c3706d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c370567c) at sleepq_switch+0xc1 sleepq_wait_sig(c370567c) at sleepq_wait_sig+0x1d msleep(c370567c,c37056b0,168,c07166be,0) at msleep+0x26a kern_sigsuspend(c3706d80,0,0,0,0,...) at kern_sigsuspend+0xa3 sigsuspend(c3706d80,da9cbd04) at sigsuspend+0x33 syscall(3b,3b,3b,5,8079360,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x880eb797, esp = 0xbfbfe9bc, ebp = 0xbfbfe9e8 --- Tracing command xdm pid 301 tid 100069 td 0xc3706900 sched_switch(c3706900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c3706900,6,bfbfe960,0,0,...) at kern_select+0x4bf select(c3706900,da9c2d04) at select+0x44 syscall(3b,3b,3b,8050528,bfbfe960,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x882e82c3, esp = 0xbfbfe93c, ebp = 0xbfbfe9e8 --- Tracing command getty pid 300 tid 100052 td 0xc351b300 sched_switch(c351b300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c33bb410) at sleepq_switch+0xc1 sleepq_wait_sig(c33bb410) at sleepq_wait_sig+0x1d msleep(c33bb410,0,159,c071c433,0) at msleep+0x26a ttysleep(c33bb400,c33bb410,159,c071c433,0,...) at ttysleep+0x21 ttread(c33bb400,d97e5cbc,0) at ttread+0x48f ttyread(c33c1000,d97e5cbc,0,c0790ae0,0,...) at ttyread+0x2f giant_read(c33c1000,d97e5cbc,0) at giant_read+0x2d devfs_read_f(c353dbd0,d97e5cbc,c329ed80,0,c351b300) at devfs_read_f+0x62 dofileread(c351b300,0,c353dbd0,d97e5cbc,ffffffff,...) at dofileread+0x85 kern_readv(c351b300,0,d97e5cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c351b300,d97e5d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x88150343, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 299 tid 100070 td 0xc3706780 sched_switch(c3706780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c33d2410) at sleepq_switch+0xc1 sleepq_wait_sig(c33d2410) at sleepq_wait_sig+0x1d msleep(c33d2410,0,159,c071c433,0) at msleep+0x26a ttysleep(c33d2400,c33d2410,159,c071c433,0,...) at ttysleep+0x21 ttread(c33d2400,da9bfcbc,0) at ttread+0x48f ttyread(c33c1900,da9bfcbc,0,da9bfc14,c05350bd,...) at ttyread+0x2f scread(c33c1900,da9bfcbc,0,c0790ae0,0,...) at scread+0x22 giant_read(c33c1900,da9bfcbc,0) at giant_read+0x2d devfs_read_f(c353d438,da9bfcbc,c329ed80,0,c3706780) at devfs_read_f+0x62 dofileread(c3706780,0,c353d438,da9bfcbc,ffffffff,...) at dofileread+0x85 kern_readv(c3706780,0,da9bfcbc,bfbfee1f,1,...) at kern_readv+0x36 read(c3706780,da9bfd04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x88150343, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 298 tid 100072 td 0xc3706480 sched_switch(c3706480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c33bd810) at sleepq_switch+0xc1 sleepq_wait_sig(c33bd810) at sleepq_wait_sig+0x1d msleep(c33bd810,0,159,c071c433,0) at msleep+0x26a ttysleep(c33bd800,c33bd810,159,c071c433,0,...) at ttysleep+0x21 ttread(c33bd800,da9b9cbc,0) at ttread+0x48f ttyread(c33c1a00,da9b9cbc,0,da9b9c14,c05350bd,...) at ttyread+0x2f scread(c33c1a00,da9b9cbc,0,c0790ae0,0,...) at scread+0x22 giant_read(c33c1a00,da9b9cbc,0) at giant_read+0x2d devfs_read_f(c353d5e8,da9b9cbc,c329ed80,0,c3706480) at devfs_read_f+0x62 dofileread(c3706480,0,c353d5e8,da9b9cbc,ffffffff,...) at dofileread+0x85 kern_readv(c3706480,0,da9b9cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c3706480,da9b9d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x88150343, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 297 tid 100064 td 0xc3519000 sched_switch(c3519000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c33bd410) at sleepq_switch+0xc1 sleepq_wait_sig(c33bd410) at sleepq_wait_sig+0x1d msleep(c33bd410,0,159,c071c433,0) at msleep+0x26a ttysleep(c33bd400,c33bd410,159,c071c433,0,...) at ttysleep+0x21 ttread(c33bd400,d97c1cbc,0) at ttread+0x48f ttyread(c33c1b00,d97c1cbc,0,d97c1c14,c05350bd,...) at ttyread+0x2f scread(c33c1b00,d97c1cbc,0,c0790ae0,0,...) at scread+0x22 giant_read(c33c1b00,d97c1cbc,0) at giant_read+0x2d devfs_read_f(c353dcf0,d97c1cbc,c329ed80,0,c3519000) at devfs_read_f+0x62 dofileread(c3519000,0,c353dcf0,d97c1cbc,ffffffff,...) at dofileread+0x85 kern_readv(c3519000,0,d97c1cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c3519000,d97c1d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x88150343, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command getty pid 296 tid 100051 td 0xc351b480 sched_switch(c351b480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c33d2c10) at sleepq_switch+0xc1 sleepq_wait_sig(c33d2c10) at sleepq_wait_sig+0x1d msleep(c33d2c10,0,159,c071c433,0) at msleep+0x26a ttysleep(c33d2c00,c33d2c10,159,c071c433,0,...) at ttysleep+0x21 ttread(c33d2c00,d97e8cbc,0) at ttread+0x48f ttyread(c33c1c00,d97e8cbc,0,d97e8c14,c05350bd,...) at ttyread+0x2f scread(c33c1c00,d97e8cbc,0,c0790ae0,0,...) at scread+0x22 giant_read(c33c1c00,d97e8cbc,0) at giant_read+0x2d devfs_read_f(c353da68,d97e8cbc,c329ed80,0,c351b480) at devfs_read_f+0x62 dofileread(c351b480,0,c353da68,d97e8cbc,ffffffff,...) at dofileread+0x85 kern_readv(c351b480,0,d97e8cbc,bfbfee1f,1,...) at kern_readv+0x36 read(c351b480,d97e8d04) at read+0x45 syscall(3b,3b,3b,0,804e580,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x88150343, esp = 0xbfbfedfc, ebp = 0xbfbfee28 --- Tracing command usbd pid 290 tid 100074 td 0xc3706180 sched_switch(c3706180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07dbf24,c07dbf24,7531,da9b3b40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07dbf24,c07dbf00,7531) at cv_timedwait_sig+0x192 kern_select(c3706180,6,bfbfed80,bfbfed00,0,...) at kern_select+0x4a9 select(c3706180,da9b3d04) at select+0x44 syscall(3b,3b,3b,bfbfed00,1e,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x881442c3, esp = 0xbfbfecac, ebp = 0xbfbfee58 --- Tracing command ksh pid 287 tid 100071 td 0xc3706600 sched_switch(c3706600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c34ee6c8) at sleepq_switch+0xc1 sleepq_wait_sig(c34ee6c8) at sleepq_wait_sig+0x1d msleep(c34ee6c8,c078e400,358,c070e041,0,...) at msleep+0x26a fifo_open(da9bca64) at fifo_open+0x3b3 VOP_OPEN_APV(c0775200,da9bca64) at VOP_OPEN_APV+0x9b vn_open_cred(da9bcbcc,da9bcccc,1a4,c329ed80,3,...) at vn_open_cred+0x3f6 vn_open(da9bcbcc,da9bcccc,1a4,3) at vn_open+0x1e kern_open(c3706600,80818e0,0,1,1b6,...) at kern_open+0xb6 open(c3706600,da9bcd04) at open+0x1a syscall(3b,3b,3b,bfbfeac4,80799ec,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x8815c4a7, esp = 0xbfbfea1c, ebp = 0xbfbfeae8 --- Tracing command nwhod pid 259 tid 100054 td 0xc351b000 sched_switch(c351b000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07dbf24,c07dbf24,1d4c1,d97dfb40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07dbf24,c07dbf00,1d4c1) at cv_timedwait_sig+0x192 kern_select(c351b000,14,bfbfe8b0,0,0,...) at kern_select+0x4a9 select(c351b000,d97dfd04) at select+0x44 syscall(3b,3b,3b,18,20,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x881442c3, esp = 0xbfbfe82c, ebp = 0xbfbfee58 --- Tracing command lpd pid 248 tid 100059 td 0xc3519780 sched_switch(c3519780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c3519780,14,bfbfeab0,0,0,...) at kern_select+0x4bf select(c3519780,d97d0d04) at select+0x44 syscall(3b,3b,3b,805d030,805d030,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x8814f2c3, esp = 0xbfbfe0cc, ebp = 0xbfbfee78 --- Tracing command cron pid 245 tid 100050 td 0xc351b600 sched_switch(c351b600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07913cc) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07913cc) at sleepq_timedwait_sig+0x1e msleep(c07913cc,0,15c,c0717308,ea61,...) at msleep+0x240 kern_nanosleep(c351b600,d97ebccc,d97ebcc4) at kern_nanosleep+0xab nanosleep(c351b600,d97ebd04) at nanosleep+0x4f syscall(3b,3b,3b,805008c,3c,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x88139f03, esp = 0xbfbfecdc, ebp = 0xbfbfed08 --- Tracing command ntpd pid 243 tid 100053 td 0xc351b180 sched_switch(c351b180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c351b180,7,bfbfedb0,0,0,...) at kern_select+0x4bf select(c351b180,d97e2d04) at select+0x44 syscall(3b,3b,3b,bfbfedb0,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x882922c3, esp = 0xbfbfed6c, ebp = 0xbfbfee48 --- Tracing command exim-4.66-1 pid 238 tid 100057 td 0xc3519a80 sched_switch(c3519a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c3519a80,4,bfbfea00,0,0,...) at kern_select+0x4bf select(c3519a80,d97d6d04) at select+0x44 syscall(3b,3b,3b,8,3,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x8844b2c3, esp = 0xbfbfe94c, ebp = 0xbfbfeab8 --- Tracing command nfsd pid 193 tid 100060 td 0xc3519600 sched_switch(c3519600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c34f0a00) at sleepq_switch+0xc1 sleepq_wait_sig(c34f0a00) at sleepq_wait_sig+0x1d msleep(c34f0a00,c07e4ca0,158,c070f1de,0,...) at msleep+0x26a nfssvc_nfsd(c3519600) at nfssvc_nfsd+0xe5 nfssvc(c3519600,d97cdd04) at nfssvc+0x18c syscall(3b,3b,3b,1,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x880bd197, esp = 0xbfbfeb1c, ebp = 0xbfbfeb38 --- Tracing command nfsd pid 191 tid 100062 td 0xc3519300 sched_switch(c3519300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c34f0400) at sleepq_switch+0xc1 sleepq_wait_sig(c34f0400) at sleepq_wait_sig+0x1d msleep(c34f0400,c07e4ca0,158,c070f1de,0,...) at msleep+0x26a nfssvc_nfsd(c3519300) at nfssvc_nfsd+0xe5 nfssvc(c3519300,d97c7d04) at nfssvc+0x18c syscall(3b,3b,3b,1,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x880bd197, esp = 0xbfbfeb1c, ebp = 0xbfbfeb38 --- Tracing command nfsd pid 190 tid 100038 td 0xc33c0a80 sched_switch(c33c0a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c34f0c00) at sleepq_switch+0xc1 sleepq_wait_sig(c34f0c00) at sleepq_wait_sig+0x1d msleep(c34f0c00,c07e4ca0,158,c070f1de,0,...) at msleep+0x26a nfssvc_nfsd(c33c0a80) at nfssvc_nfsd+0xe5 nfssvc(c33c0a80,d97b8d04) at nfssvc+0x18c syscall(3b,3b,3b,1,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x880bd197, esp = 0xbfbfeb1c, ebp = 0xbfbfeb38 --- Tracing command nfsd pid 189 tid 100042 td 0xc33c0480 sched_switch(c33c0480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c34ef200) at sleepq_switch+0xc1 sleepq_wait_sig(c34ef200) at sleepq_wait_sig+0x1d msleep(c34ef200,c07e4ca0,158,c070f1de,0,...) at msleep+0x26a nfssvc_nfsd(c33c0480) at nfssvc_nfsd+0xe5 nfssvc(c33c0480,d97acd04) at nfssvc+0x18c syscall(3b,3b,3b,1,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x880bd197, esp = 0xbfbfeb1c, ebp = 0xbfbfeb38 --- Tracing command nfsd pid 188 tid 100044 td 0xc33c0180 sched_switch(c33c0180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c3600892) at sleepq_switch+0xc1 sleepq_wait_sig(c3600892) at sleepq_wait_sig+0x1d msleep(c3600892,c07dc2e0,158,c071af86,0) at msleep+0x26a accept1(c33c0180,d97a6d04,0,d97a6d30,c06c49fb,...) at accept1+0x142 accept(c33c0180,d97a6d04) at accept+0x10 syscall(3b,3b,3b,bfbfee40,3,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (30, FreeBSD ELF32, accept), eip = 0x880e475f, esp = 0xbfbfeb3c, ebp = 0xbfbfee68 --- Tracing command mountd pid 183 tid 100043 td 0xc33c0300 sched_switch(c33c0300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c33c0300,6,bfbfede0,0,0,...) at kern_select+0x4bf select(c33c0300,d97a9d04) at select+0x44 syscall(3b,3b,3b,4,1,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x8814a2c3, esp = 0xbfbfed3c, ebp = 0xbfbfee78 --- Tracing command ypbind pid 174 tid 100045 td 0xc33c0000 sched_switch(c33c0000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07dbf24,c07dbf24,ea61,d97a3b40,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07dbf24,c07dbf00,ea61) at cv_timedwait_sig+0x192 kern_select(c33c0000,14,804c380,0,0,...) at kern_select+0x4a9 select(c33c0000,d97a3d04) at select+0x44 syscall(3b,3b,3b,bfbfea18,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x8813b2c3, esp = 0xbfbfe9cc, ebp = 0xbfbfee38 --- Tracing command rpcbind pid 165 tid 100039 td 0xc33c0900 sched_switch(c33c0900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_timedwait_sig(c07dbf24,c07dbf24,7531,d97b5b84,246,...) at sleepq_timedwait_sig+0x1e cv_timedwait_sig(c07dbf24,c07dbf00,7531) at cv_timedwait_sig+0x192 poll(c33c0900,d97b5d04) at poll+0x3a3 syscall(3b,3b,3b,1,4,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (209, FreeBSD ELF32, poll), eip = 0x8810adc3, esp = 0xbfbfcc5c, ebp = 0xbfbfee28 --- Tracing command sshd pid 157 tid 100041 td 0xc33c0600 sched_switch(c33c0600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c33c0600,4,8140b20,0,0,...) at kern_select+0x4bf select(c33c0600,d97afd04) at select+0x44 syscall(3b,3b,3b,813f040,a,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x882522c3, esp = 0xbfbfe15c, ebp = 0xbfbfee78 --- Tracing command inetd pid 152 tid 100040 td 0xc33c0780 sched_switch(c33c0780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c33c0780,b,bfbfe890,0,0,...) at kern_select+0x4bf select(c33c0780,d97b2d04) at select+0x44 syscall(3b,3b,3b,bfbfe890,161,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x881592c3, esp = 0xbfbfdb6c, ebp = 0xbfbfee58 --- Tracing command syslogd pid 147 tid 100046 td 0xc335bd80 sched_switch(c335bd80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c335bd80,8,8056040,0,0,...) at kern_select+0x4bf select(c335bd80,d56e2d04) at select+0x44 syscall(3b,3b,3b,8056044,8056040,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x8814b2c3, esp = 0xbfbfda2c, ebp = 0xbfbfee78 --- Tracing command devd pid 135 tid 100047 td 0xc335bc00 sched_switch(c335bc00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dbf24) at sleepq_switch+0xc1 sleepq_wait_sig(c07dbf24,246,c0790aa0,c071a984,303,...) at sleepq_wait_sig+0x1d cv_wait_sig(c07dbf24,c07dbf00) at cv_wait_sig+0x189 kern_select(c335bc00,5,bfbfe9d0,0,0,...) at kern_select+0x4bf select(c335bc00,d56dfd04) at select+0x44 syscall(3b,3b,3b,0,8,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (93, FreeBSD ELF32, select), eip = 0x80757f7, esp = 0xbfbfe96c, ebp = 0xbfbfee78 --- Tracing command schedcpu pid 39 tid 100049 td 0xc335b900 sched_switch(c335b900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(d56d9d04,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(d56d9d04) at sleepq_timedwait+0x4a msleep(d56d9d04,0,0,c070f1de,3e8) at msleep+0x255 schedcpu_thread(0,d56d9d38,0,c056a360,0,...) at schedcpu_thread+0x26 fork_exit(c056a360,0,d56d9d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd56d9d6c, ebp = 0 --- Tracing command softdepflush pid 38 tid 100027 td 0xc335b180 sched_switch(c335b180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e4f94,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c07e4f94) at sleepq_timedwait+0x4a msleep(c07e4f94,c07e4f60,44,c0729939,3e8) at msleep+0x255 softdep_flush(0,d56cad38,0,c06429dc,0,...) at softdep_flush+0x227 fork_exit(c06429dc,0,d56cad38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd56cad6c, ebp = 0 --- Tracing command vnlru pid 37 tid 100028 td 0xc335b000 sched_switch(c335b000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c33be000,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c33be000) at sleepq_timedwait+0x4a msleep(c33be000,c07dc8a0,250,c071f95a,3e8,c07dc920) at msleep+0x255 vnlru_proc(0,d56c7d38,0,c05ad800,0,...) at vnlru_proc+0xdf fork_exit(c05ad800,0,d56c7d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd56c7d6c, ebp = 0 --- Tracing command syncer pid 36 tid 100029 td 0xc32a6d80 sched_switch(c32a6d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c079113c,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c079113c,0,c055217e,c07dc8e0,c071f899,...) at sleepq_wait+0x46 msleep(c079113c,0,68,c071fc68,0) at msleep+0x279 sched_sync(0,d44fad38,0,c05af30c,0,...) at sched_sync+0x309 fork_exit(c05af30c,0,d44fad38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44fad6c, ebp = 0 --- Tracing command bufdaemon pid 35 tid 100030 td 0xc32a6c00 sched_switch(c32a6c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07dc48c,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c07dc48c) at sleepq_timedwait+0x4a msleep(c07dc48c,c07dc4a0,44,c071e1d7,3e8) at msleep+0x255 buf_daemon(0,d44f7d38,0,c05a0fec,0,...) at buf_daemon+0x108 fork_exit(c05a0fec,0,d44f7d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44f7d6c, ebp = 0 --- Tracing command pagezero pid 34 tid 100031 td 0xc32a6a80 sched_switch(c32a6a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5f04,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c07e5f04) at sleepq_timedwait+0x4a msleep(c07e5f04,c07e59e0,200,c072ea6b,493e0,c07e59e0,0,c072ea51,9d) at msleep+0x255 vm_pagezero(0,d44f4d38,0,c0674508,0,...) at vm_pagezero+0x6c fork_exit(c0674508,0,d44f4d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44f4d6c, ebp = 0 --- Tracing command vmdaemon pid 33 tid 100032 td 0xc32a6900 sched_switch(c32a6900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5a54,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c07e5a54,0,c05520d8,c0790ae0,c072e727,...) at sleepq_wait+0x46 msleep(c07e5a54,0,68,c071e1d7,0) at msleep+0x279 vm_daemon(0,d44f1d38,0,c0673534,0,...) at vm_daemon+0x36 fork_exit(c0673534,0,d44f1d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44f1d6c, ebp = 0 --- Tracing command pagedaemon pid 32 tid 100033 td 0xc32a6780 sched_switch(c32a6780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c07e5a10,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c07e5a10) at sleepq_timedwait+0x4a msleep(c07e5a10,c07e59e0,44,c071e1d7,1388) at msleep+0x255 vm_pageout(0,d44eed38,0,c06731f4,0,...) at vm_pageout+0x260 fork_exit(c06731f4,0,d44eed38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44eed6c, ebp = 0 --- Tracing command irq12: psm0 pid 31 tid 100034 td 0xc32a6600 fork_trampoline() at fork_trampoline Tracing command irq1: atkbd0 pid 30 tid 100035 td 0xc32a6480 sched_switch(c32a6480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c33cd0b0,d44e8d38,c33cd0b0,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c33cd0b0,d44e8d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44e8d6c, ebp = 0 --- Tracing command swi0: sio pid 29 tid 100036 td 0xc33c0d80 sched_switch(c33c0d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c33cd0d0,d97bed38,c33cd0d0,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c33cd0d0,d97bed38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd97bed6c, ebp = 0 --- Tracing command irq7: ppc0 pid 28 tid 100037 td 0xc33c0c00 sched_switch(c33c0c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c33b3bc0,d97bbd38,c33b3bc0,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c33b3bc0,d97bbd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd97bbd6c, ebp = 0 --- Tracing command fdc0 pid 27 tid 100017 td 0xc32a2900 sched_switch(c32a2900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c33a623c,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c33a623c) at sleepq_timedwait+0x4a msleep(c33a623c,c33a62f0,4c,c070f1de,3e8) at msleep+0x255 fdc_worker(c33a6200,c33a62f0,0,c0730830,48e) at fdc_worker+0x1ca fdc_thread(c33a6200,d44d3d38,c33a6200,c068c828,0,...) at fdc_thread+0x54 fork_exit(c068c828,c33a6200,d44d3d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44d3d6c, ebp = 0 --- Tracing command irq10: atapci1 pid 26 tid 100018 td 0xc32a2780 sched_switch(c32a2780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c33ab310,d44d0d38,c33ab310,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c33ab310,d44d0d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44d0d6c, ebp = 0 --- Tracing command usb1 pid 25 tid 100019 td 0xc32a2600 sched_switch(c32a2600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c339c210,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c339c210) at sleepq_timedwait+0x4a msleep(c339c210,0,5c,c070d0d9,ea60,c3398c40) at msleep+0x255 usb_event_thread(c3398c40,d44cdd38,c3398c40,c05004bc,0,...) at usb_event_thread+0x70 fork_exit(c05004bc,c3398c40,d44cdd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44cdd6c, ebp = 0 --- Tracing command usbtask pid 24 tid 100020 td 0xc32a2480 sched_switch(c32a2480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c078e204,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c078e204,0,c05520d8,c0790ae0,c070d0cb,...) at sleepq_wait+0x46 msleep(c078e204,0,5c,c070d0e0,0,...) at msleep+0x279 usb_task_thread(0,d44cad38,0,c050054c,0,...) at usb_task_thread+0x47 fork_exit(c050054c,0,d44cad38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44cad6c, ebp = 0 --- Tracing command usb0 pid 23 tid 100021 td 0xc32a2300 sched_switch(c32a2300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c3390210,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c3390210) at sleepq_timedwait+0x4a msleep(c3390210,0,5c,c070d0d9,ea60,c338a900) at msleep+0x255 usb_event_thread(c338a900,d44c7d38,c338a900,c05004bc,0,...) at usb_event_thread+0x70 fork_exit(c05004bc,c338a900,d44c7d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44c7d6c, ebp = 0 --- Tracing command irq5: xl0 uhci0++ pid 22 tid 100022 td 0xc32a2180 sched_switch(c32a2180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c3389140,d44c4d38,c3389140,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c3389140,d44c4d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44c4d6c, ebp = 0 --- Tracing command irq15: ata1 pid 21 tid 100023 td 0xc335b780 sched_switch(c335b780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c33892e0,d56d6d38,c33892e0,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c33892e0,d56d6d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd56d6d6c, ebp = 0 --- Tracing command irq14: ata0 pid 20 tid 100024 td 0xc335b600 sched_switch(c335b600,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c3389430,d56d3d38,c3389430,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c3389430,d56d3d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd56d3d6c, ebp = 0 --- Tracing command irq9: acpi0 pid 19 tid 100025 td 0xc335b480 fork_trampoline() at fork_trampoline Tracing command swi6: Giant taskq pid 18 tid 100026 td 0xc335b300 fork_trampoline() at fork_trampoline Tracing command swi6: task queue pid 17 tid 100008 td 0xc32a1300 sched_switch(c32a1300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c3350520,d44a9d38,c3350520,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c3350520,d44a9d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44a9d6c, ebp = 0 --- Tracing command swi2: cambio pid 16 tid 100009 td 0xc32a1180 sched_switch(c32a1180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c3350530,d44a6d38,c3350530,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c3350530,d44a6d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44a6d6c, ebp = 0 --- Tracing command acpi_task_2 pid 9 tid 100010 td 0xc32a1000 sched_switch(c32a1000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c3351300,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c3351300,0,c07194d2,49,8,...) at sleepq_wait+0x46 msleep(c3351300,c335131c,0,c070f1de,0) at msleep+0x279 taskqueue_thread_loop(c08ff200,d44a3d38,c08ff200,c05790d4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05790d4,c08ff200,d44a3d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44a3d6c, ebp = 0 --- Tracing command acpi_task_1 pid 8 tid 100011 td 0xc32a6300 sched_switch(c32a6300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c3351300,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c3351300,0,c07194d2,49,8,...) at sleepq_wait+0x46 msleep(c3351300,c335131c,0,c070f1de,0) at msleep+0x279 taskqueue_thread_loop(c08ff200,d44e5d38,c08ff200,c05790d4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05790d4,c08ff200,d44e5d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44e5d6c, ebp = 0 --- Tracing command acpi_task_0 pid 7 tid 100012 td 0xc32a6180 sched_switch(c32a6180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c3351300,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c3351300,0,c07194d2,49,c336a1c0,...) at sleepq_wait+0x46 msleep(c3351300,c335131c,0,c070f1de,0) at msleep+0x279 taskqueue_thread_loop(c08ff200,d44e2d38,c08ff200,c05790d4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05790d4,c08ff200,d44e2d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44e2d6c, ebp = 0 --- Tracing command kqueue taskq pid 6 tid 100013 td 0xc32a6000 sched_switch(c32a6000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c3351380,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c3351380,0,c07194d2,49,8,...) at sleepq_wait+0x46 msleep(c3351380,c335139c,0,c070f1de,0) at msleep+0x279 taskqueue_thread_loop(c078f120,d44dfd38,c078f120,c05790d4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05790d4,c078f120,d44dfd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44dfd6c, ebp = 0 --- Tracing command swi5: + pid 15 tid 100014 td 0xc32a2d80 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 5 tid 100015 td 0xc32a2c00 sched_switch(c32a2c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c329f380,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c329f380,0,c07194d2,49,c33852b0,...) at sleepq_wait+0x46 msleep(c329f380,c329f39c,0,c070f1de,0) at msleep+0x279 taskqueue_thread_loop(c0794908,d44d9d38,c0794908,c05790d4,0,...) at taskqueue_thread_loop+0x78 fork_exit(c05790d4,c0794908,d44d9d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44d9d6c, ebp = 0 --- Tracing command yarrow pid 14 tid 100016 td 0xc32a2a80 sched_switch(c32a2a80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c078b120,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c078b120) at sleepq_timedwait+0x4a msleep(c078b120,0,0,c070f1de,64) at msleep+0x255 random_kthread(0,d44d6d38,0,c04ca7a8,0,...) at random_kthread+0x180 fork_exit(c04ca7a8,0,d44d6d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44d6d6c, ebp = 0 --- Tracing command g_down pid 4 tid 100000 td 0xc32a2000 sched_switch(c32a2000,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c078e888,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c078e888) at sleepq_timedwait+0x4a msleep(c078e888,c078e768,24c,c070f1de,64) at msleep+0x255 g_io_schedule_down(c32a2000) at g_io_schedule_down+0x56 g_down_procbody(0,d44c1d38,0,c0524c78,0,...) at g_down_procbody+0x5a fork_exit(c0524c78,0,d44c1d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44c1d6c, ebp = 0 --- Tracing command g_up pid 3 tid 100001 td 0xc32a1d80 sched_switch(c32a1d80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c078e884,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c078e884) at sleepq_timedwait+0x4a msleep(c078e884,c078e7a8,24c,c070f1de,64) at msleep+0x255 g_io_schedule_up(c32a1d80) at g_io_schedule_up+0x127 g_up_procbody(0,d44bed38,0,c0524c18,0,...) at g_up_procbody+0x5a fork_exit(c0524c18,0,d44bed38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44bed6c, ebp = 0 --- Tracing command g_event pid 2 tid 100002 td 0xc32a1c00 sched_switch(c32a1c00,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c078e87c,c0790aa0,0,c0718ff4,236,...) at sleepq_switch+0xc1 sleepq_timedwait(c078e87c) at sleepq_timedwait+0x4a msleep(c078e87c,0,4c,c070f1de,64) at msleep+0x255 g_event_procbody(0,d44bbd38,0,c0524cd8,0,...) at g_event_procbody+0x9e fork_exit(c0524cd8,0,d44bbd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44bbd6c, ebp = 0 --- Tracing command swi3: vm pid 13 tid 100003 td 0xc32a1a80 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 12 tid 100004 td 0xc32a1900 sched_switch(c32a1900,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c3285710,d44b5d38,c3285710,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c3285710,d44b5d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44b5d6c, ebp = 0 --- Tracing command swi1: net pid 11 tid 100005 td 0xc32a1780 sched_switch(c32a1780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 ithread_loop(c3285720,d44b2d38,c3285720,c0546cfc,0,...) at ithread_loop+0xd9 fork_exit(c0546cfc,c3285720,d44b2d38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44b2d6c, ebp = 0 --- Tracing command idle pid 10 tid 100006 td 0xc32a1600 kdb_enter(c0732314) at kdb_enter+0x2b siointr1(c33bb800,c07e7f80,0,c0732124,56e,...) at siointr1+0xce siointr(c33bb800) at siointr+0x21 intr_execute_handlers(c0785310,d44afc98,0,c3352818,c3352800,...) at intr_execute_handlers+0xe1 atpic_handle_intr(4) at atpic_handle_intr+0x96 Xatpic_intr4() at Xatpic_intr4+0x20 --- interrupt, eip = 0xc08f0b49, esp = 0xd44afcdc, ebp = 0xd44afcdc --- acpi_cpu_c1(0,0,1,c054604c,c32a0c90,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(d44afd0c,c0546075,d44afd24,c0545eac,0,...) at acpi_cpu_idle+0x152 cpu_idle(d44afd24,c0545eac,0,d44afd38,0,...) at cpu_idle+0x1f idle_proc(0,d44afd38,0,c054604c,0,...) at idle_proc+0x29 fork_exit(c054604c,0,d44afd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44afd6c, ebp = 0 --- Tracing command init pid 1 tid 100007 td 0xc32a1480 sched_switch(c32a1480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c32a5000) at sleepq_switch+0xc1 sleepq_wait_sig(c32a5000) at sleepq_wait_sig+0x1d msleep(c32a5000,c32a5068,15c,c071af18,0) at msleep+0x26a kern_wait(c32a1480,ffffffff,d44acc84,0,0) at kern_wait+0x6b7 wait4(c32a1480,d44acd04) at wait4+0x2a syscall(3b,3b,3b,0,0,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8052f47, esp = 0xbfbfed7c, ebp = 0xbfbfed98 --- Tracing command swapper pid 0 tid 0 td 0xc078eba0 sched_switch(c078eba0,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 scheduler(0,c1ec00,c1e000,0,c0438ed5,...) at scheduler+0x195 mi_startup() at mi_startup+0x96 begin() at begin+0x2c db> ps pid ppid pgrp uid state wmesg wchan cmd 371 355 371 1002 T+ (threaded) c 100078 s c 100081 D nfs 0xc36dc5c8 c 100080 s c 100077 Ss ksesigwa 0xc3515dc8 c 355 354 355 1002 S+ pause 0xc370c67c ksh 354 353 354 0 Ss+ wait 0xc36c1a78 login 353 152 353 0 Ss select 0xc07dbf24 rlogind 352 1 337 0 S select 0xc07dbf24 xconsole 331 301 331 0 Ss select 0xc07dbf24 xdm 325 304 325 0 S select 0xc07dbf24 Xorg 304 301 304 0 Ss pause 0xc370567c ksh 301 1 301 0 Ss select 0xc07dbf24 xdm 300 1 300 0 Ss+ ttyin 0xc33bb410 getty 299 1 299 0 Ss+ ttyin 0xc33d2410 getty 298 1 298 0 Ss+ ttyin 0xc33bd810 getty 297 1 297 0 Ss+ ttyin 0xc33bd410 getty 296 1 296 0 Ss+ ttyin 0xc33d2c10 getty 290 1 290 0 Ss select 0xc07dbf24 usbd db> where Tracing pid 10 tid 100006 td 0xc32a1600 kdb_enter(c0732314) at kdb_enter+0x2b siointr1(c33bb800,c07e7f80,0,c0732124,56e,...) at siointr1+0xce siointr(c33bb800) at siointr+0x21 intr_execute_handlers(c0785310,d44afc98,0,c3352818,c3352800,...) at intr_execute_handlers+0xe1 atpic_handle_intr(4) at atpic_handle_intr+0x96 Xatpic_intr4() at Xatpic_intr4+0x20 --- interrupt, eip = 0xc08f0b49, esp = 0xd44afcdc, ebp = 0xd44afcdc --- acpi_cpu_c1(0,0,1,c054604c,c32a0c90,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(d44afd0c,c0546075,d44afd24,c0545eac,0,...) at acpi_cpu_idle+0x152 cpu_idle(d44afd24,c0545eac,0,d44afd38,0,...) at cpu_idle+0x1f idle_proc(0,d44afd38,0,c054604c,0,...) at idle_proc+0x29 fork_exit(c054604c,0,d44afd38) at fork_exit+0xa0 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd44afd6c, ebp = 0 --- db> thread 100081 [thread pid 371 tid 100081 ] sched_switch+0x177: addl $0x8,%esp db> where Tracing pid 371 tid 100081 td 0xc3709780 sched_switch(c3709780,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c36dc5c8,c0790aa0,0,c0718ff4,211,...) at sleepq_switch+0xc1 sleepq_wait(c36dc5c8,0,c36dc638,b7,c071ef72,...) at sleepq_wait+0x46 msleep(c36dc5c8,c0790268,50,c07257ab,0,...) at msleep+0x279 acquire(da9dda64,40,60000,c3709780,0,...) at acquire+0x76 lockmgr(c36dc5c8,3002,c36dc638,c3709780,da9dda88,...) at lockmgr+0x44e vop_stdlock(da9ddaa8) at vop_stdlock+0x21 VOP_LOCK_APV(c0770fa0,da9ddaa8) at VOP_LOCK_APV+0x87 vn_lock(c36dc570,1002,c3709780) at vn_lock+0xac lookup(da9ddc00) at lookup+0xd2 namei(da9ddc00) at namei+0x39a kern_stat(c3709780,bf6fbe98,0,da9ddc74) at kern_stat+0x35 stat(c3709780,da9ddd04) at stat+0x1b syscall(3b,3b,3b,8807c050,bf6fbfec,...) at syscall+0x25b Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x88159447, esp = 0xbf6fbdec, ebp = 0xbf6fbfb8 --- db> reset --- ddb.output ends here --- -- Mail: Joerg.Lehners@Informatik.Uni-Oldenburg.DE Tel: 2198 Real: Joerg Lehners, Informatik ARBI, Uni Oldenburg, D-26111 Oldenburg Unwoerter: Kostensenkung - Gewinnmaximierung - billig, billig, billig State Changed From-To: open->feedback Need confirmation that this isn't a recently fixed issue. Responsible Changed From-To: freebsd-bugs->alfred Need confirmation that this isn't a recently fixed issue. State Changed From-To: feedback->closed Graciously fixed by Julian and Alfred in: Revision Changes Path 1.284 +15 -5 src/sys/kern/kern_fork.c |
