Bug 135692 - [mutex] [witness] blockable sleep lock - kernel panic on 8.0-CURRENT
Summary: [mutex] [witness] blockable sleep lock - kernel panic on 8.0-CURRENT
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 8.0-CURRENT
Hardware: Any Any
: Normal Affects Only Me
Assignee: Jaakko Heinonen
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-06-18 11:30 UTC by Adam K Kirchhoff
Modified: 2010-10-30 09:09 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Adam K Kirchhoff 2009-06-18 11:30:01 UTC
I recently updated my workstation at home to -CURRENT, and then noticed (thanks to some discussion on ##freebsd) that VirtualBox had made it into the ports tree.  I installed it via ports, but upon trying to boot a VDI image created under Linux (where it works just fine), I ran into a kernel panic.  When starting the VM, VirtualBox informs me of the key combination to grab/release the mouse and keyboard.  I click "OK" and then the entire machine locks up.
The VM is Windows XP.  256 megs of RAM, 1 processor, VT-x/AMD-V is enabled, with Nesting Paging disabled.  The only option I changed was the memory.  All the others were the defaults.

I reproduced the panic quite easily by rebooting into single user mode, fsck'ing the filesystems manually, booting into X, and then starting up VirtualBox.  Same thing happened when I launched the VM.

The panic:

This GDB was configured as "i386-marcel-freebsd"...                        
Unread portion of the kernel message buffer:
panic: blockable sleep lock (sleep mutex) 32 @ /usr/src/sys/vm/uma_core.c:1990
cpuid = 2                                                                    KDB: enter: panic                                                            Uptime: 1h57m3s                                                              Physical memory: 2029 MB                                                     Dumping 252 MB: 237 221 205 189 173 157 141 125 109 93 77 61 45 29 13       
bReading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done.                                                                                                         
done.                                                                                                                                                                                                            
Loaded symbols for /boot/kernel/linux.ko                                                                                                                                                                         
Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from /boot/kernel/snd_hda.ko.symbols...tdone.                                                                                                     
done.                                                                                                                                                                                                            
Loaded symbols for /boot/kernel/snd_hda.ko                                                                                                                                                                       
Reading symbols from /boot/kernel/sound.ko...Reading symbols from /boot/kernel/sound.ko.symbols...done.                                                                                                          
done.                                                                                                                                                                                                            
Loaded symbols for /boot/kernel/sound.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/aio.ko...Reading symbols from /boot/kernel/aio.ko.symbols...done.
done.
^R
Loaded symbols for /boot/kernel/aio.ko
Reading symbols from /boot/modules/kqemu.ko...done.
Loaded symbols for /boot/modules/kqemu.ko
Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linprocfs.ko
Reading symbols from /boot/kernel/smbfs.ko...Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/smbfs.ko
Reading symbols from /boot/kernel/libiconv.ko...Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/libiconv.ko
Reading symbols from /boot/kernel/libmchain.ko...Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/libmchain.ko
Reading symbols from /boot/kernel/radeon.ko...Reading symbols from /boot/kernel/radeon.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/radeon.ko
Reading symbols from /boot/kernel/drm.ko...Reading symbols from /boot/kernel/drm.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/drm.ko
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
#0  doadump () at pcpu.h:246
246     pcpu.h: No such file or directory.
       in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:246
#1  0xc0870a8e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
#2  0xc0870d62 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:575
#3  0xc08b1eb0 in witness_checkorder (lock=0xc1894788, flags=9, file=0xc0c81d4e "/usr/src/sys/vm/uma_core.c", line=1990, interlock=0x0)
   at /usr/src/sys/kern/subr_witness.c:1063
#4  0xc0861b74 in _mtx_lock_flags (m=0xc1894788, opts=0, file=0xc0c81d4e "/usr/src/sys/vm/uma_core.c", line=1990)
   at /usr/src/sys/kern/kern_mutex.c:200
#5  0xc0ab0ae3 in uma_zalloc_arg (zone=0xc188c700, udata=0x0, flags=1) at /usr/src/sys/vm/uma_core.c:1990
#6  0xc085e1e4 in malloc (size=28, mtp=0xc97c09d0, flags=1) at uma.h:305
#7  0xc97b54b6 in rtMemAlloc () from /boot/modules/vboxdrv.ko
#8  0xc97b3519 in RTMemAlloc () from /boot/modules/vboxdrv.ko
#9  0xc97a6e62 in SUPR0ObjAddRefEx () from /boot/modules/vboxdrv.ko
#10 0xc97a6f00 in SUPR0ObjAddRef () from /boot/modules/vboxdrv.ko
#11 0xc97aee2b in RTHandleTableLookupWithCtx () from /boot/modules/vboxdrv.ko
#12 0xc97a6b26 in SUPSemEventWaitNoResume () from /boot/modules/vboxdrv.ko
#13 0xc97a9305 in supdrvIOCtl () from /boot/modules/vboxdrv.ko
#14 0xc97a4e3a in VBoxDrvFreeBSDIOCtl () from /boot/modules/vboxdrv.ko
#15 0xc07f48d8 in devfs_ioctl_f (fp=0xc6dd4738, com=3364738944, data=0xc88dd780, cred=0xc7adbe00, td=0xc99c9480)
   at /usr/src/sys/fs/devfs/devfs_vnops.c:658
#16 0xc08b4a7d in kern_ioctl (td=0xc99c9480, fd=22, com=3223868955, data=0xc88dd780 "birddrib(") at file.h:262
#17 0xc08b4c04 in ioctl (td=0xc99c9480, uap=0xeab19cf8) at /usr/src/sys/kern/sys_generic.c:677
#18 0xc0b8ed93 in syscall (frame=0xeab19d38) at /usr/src/sys/i386/i386/trap.c:1073
#19 0xc0b71fd0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:261
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)

The host:

FreeBSD 8.0-CURRENT #8: Tue Jun 16 14:10:29 EDT 2009
   root@sorrow.ashke.com:/usr/obj/usr/src/sys/GENERIC
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.20GHz (3191.87-MHz 686-class CPU)
 Origin = "GenuineIntel"  Id = 0xf64  Stepping = 4
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 Features2=0xe43d<SSE3,DTES64,MON,DS_CPL,VMX,CNXT-ID,CX16,xTPR,PDCM>
 AMD Features=0x20100000<NX,LM>
 AMD Features2=0x1<LAHF>
 TSC: P-state invariant
real memory  = 2147483648 (2048 MB)
avail memory = 2086457344 (1989 MB)

And I am running virtualbox-2.2.51.r20457_2. 

I have tested this, and experienced the crash, on two systems.  The first is that dualcore xeon (P4 based) with Virtualization enabled in the BIOS.  The second is a core 2 duo, which does not have such an option in the BIOS.  Both are running 8.0-CURRENT as of Tuesday, June 16th, 2009.

How-To-Repeat: 
Install FreeBSD.  Upgrade to 8.0-CURRENT.  Install virtualbox from the ports tree.  Load the vboxdrv kernel module and then try to start a VM.
Comment 1 Volker Werth freebsd_committer freebsd_triage 2009-06-18 13:06:21 UTC
Responsible Changed
From-To: freebsd-ports-bugs->freebsd-bugs

not a ports bug
Comment 2 Volker Werth freebsd_committer freebsd_triage 2009-06-18 21:54:21 UTC
State Changed
From-To: open->feedback

Adam, 
while checking your issue for a moment, it seems like your issue 
should not be seen when using a kernel w/o INVARIANTS support. 
That does not mean, it will work properly and is the suggested way. 
To get the real source of trouble, I would like to see a full backtrace. 
Please do the following (while vboxdrv.ko is loaded): 
``asf -ks `sysctl -n kern.module_path` kernel.asf '' 
kgdb /boot/kernel/kernel /path/to/dump/dir 
and inside kgdb: 
add-symbol-file kernel.asf 
bt full
Comment 3 Adam K Kirchhoff 2009-06-19 14:30:46 UTC
asf -ks `sysctl -n kern.module_path` kernel.asf

worked without any errors.  It created /root/kernel.asf:

[ root@sorrow - ~ ]: ls -l /root/kernel.asf
-rw-r--r--  1 root  wheel  1067 Jun 18 17:13 /root/kernel.asf

But when I try to add it in kgdb, this is what I get:

(kgdb) add-symbol-file /root/kernel.asf
add symbol table from file "/root/kernel.asf" at
(y or n) y
"/root/kernel.asf": can't read symbols: File format not recognized.

However, I got the gist of things and simply copied and pasted each line 
of the kernel.asf file into kgdb.  It adds the symbols for each module 
without any errors.  'bt full' however, only returns a few lines:

Here's the full output from kgdb:

[ root@sorrow - ~ ]: kgdb /boot/kernel/kernel /var/crash/vmcore.7
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you 
are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: blockable sleep lock (sleep mutex) 32 @ 
/usr/src/sys/vm/uma_core.c:1990
cpuid = 2
KDB: enter: panic
Uptime: 1h57m3s
Physical memory: 2029 MB
Dumping 252 MB: 237 221 205 189 173 157 141 125 109 93 77 61 45 29 13

Reading symbols from /boot/kernel/linux.ko...Reading symbols from 
/boot/kernel/linux.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from 
/boot/kernel/snd_hda.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/snd_hda.ko
Reading symbols from /boot/kernel/sound.ko...Reading symbols from 
/boot/kernel/sound.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/sound.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from 
/boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/aio.ko...Reading symbols from 
/boot/kernel/aio.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/aio.ko
Reading symbols from /boot/modules/kqemu.ko...done.
Loaded symbols for /boot/modules/kqemu.ko
Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from 
/boot/kernel/linprocfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linprocfs.ko
Reading symbols from /boot/kernel/smbfs.ko...Reading symbols from 
/boot/kernel/smbfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/smbfs.ko
Reading symbols from /boot/kernel/libiconv.ko...Reading symbols from 
/boot/kernel/libiconv.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/libiconv.ko
Reading symbols from /boot/kernel/libmchain.ko...Reading symbols from 
/boot/kernel/libmchain.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/libmchain.ko
Reading symbols from /boot/kernel/radeon.ko...Reading symbols from 
/boot/kernel/radeon.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/radeon.ko
Reading symbols from /boot/kernel/drm.ko...Reading symbols from 
/boot/kernel/drm.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/drm.ko
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
#0  doadump () at pcpu.h:246
246    pcpu.h: No such file or directory.
   in pcpu.h
(kgdb) add-symbol-file /boot/kernel/linux.ko.symbols 0xc10f2a90 -s .data 
0xc110c000 -s .bss 0xc110ed54
add symbol table from file "/boot/kernel/linux.ko.symbols" at
   .text_addr = 0xc10f2a90
   .data_addr = 0xc110c000
   .bss_addr = 0xc110ed54
(y or n) y
Reading symbols from /boot/kernel/linux.ko.symbols...done.
(kgdb) add-symbol-file /boot/kernel/snd_hda.ko.symbols 0xc111c740 -s 
.data 0xc112fb20 -s .bss 0xc112ff3c
add symbol table from file "/boot/kernel/snd_hda.ko.symbols" at
   .text_addr = 0xc111c740
   .data_addr = 0xc112fb20
   .bss_addr = 0xc112ff3c
(y or n) y
Reading symbols from /boot/kernel/snd_hda.ko.symbols...done.
(kgdb) add-symbol-file /boot/kernel/sound.ko.symbols 0xc1148fb0 -s .data 
0xc117f000 -s .bss 0xc1189a20
add symbol table from file "/boot/kernel/sound.ko.symbols" at
   .text_addr = 0xc1148fb0
   .data_addr = 0xc117f000
   .bss_addr = 0xc1189a20
(y or n) y
Reading symbols from /boot/kernel/sound.ko.symbols...done.
(kgdb) add-symbol-file /boot/kernel/accf_http.ko.symbols 0xc11956b0 -s 
.data 0xc1196ca0 -s .bss 0xc1196e24
add symbol table from file "/boot/kernel/accf_http.ko.symbols" at
   .text_addr = 0xc11956b0
   .data_addr = 0xc1196ca0
   .bss_addr = 0xc1196e24
(y or n) y
Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
(kgdb) add-symbol-file /boot/kernel/linprocfs.ko.symbols 0xc5c9d9f0 -s 
.data 0xc5ca2480 -s .bss 0xc5ca2740
add symbol table from file "/boot/kernel/linprocfs.ko.symbols" at
   .text_addr = 0xc5c9d9f0
   .data_addr = 0xc5ca2480
   .bss_addr = 0xc5ca2740
(y or n) y
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
(kgdb) add-symbol-file /boot/kernel/smbfs.ko.symbols 0xc5e9d170 -s .data 
0xc5eb43c0 -s .bss 0xc5eb4de0
add symbol table from file "/boot/kernel/smbfs.ko.symbols" at
   .text_addr = 0xc5e9d170
   .data_addr = 0xc5eb43c0
   .bss_addr = 0xc5eb4de0
(y or n) y
Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
(kgdb) add-symbol-file /boot/kernel/libiconv.ko.symbols 0xc5eb6250 -s 
.data 0xc5eb87e0 -s .bss 0xc5eb8c4c
add symbol table from file "/boot/kernel/libiconv.ko.symbols" at
   .text_addr = 0xc5eb6250
   .data_addr = 0xc5eb87e0
   .bss_addr = 0xc5eb8c4c
(y or n) y
Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
(kgdb) add-symbol-file /boot/kernel/libmchain.ko.symbols 0xc5f3c640 -s 
.data 0xc5f3e410 -s .bss 0xc5f3e4a8
add symbol table from file "/boot/kernel/libmchain.ko.symbols" at
   .text_addr = 0xc5f3c640
   .data_addr = 0xc5f3e410
   .bss_addr = 0xc5f3e4a8
(y or n) y
Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
(kgdb) add-symbol-file /boot/modules/vboxdrv.ko 0xc6072b30 -s .data 
0xc608d000 -s .bss 0xc608eb00
add symbol table from file "/boot/modules/vboxdrv.ko" at
   .text_addr = 0xc6072b30
   .data_addr = 0xc608d000
   .bss_addr = 0xc608eb00
(y or n) y
Reading symbols from /boot/modules/vboxdrv.ko...done.
(kgdb) add-symbol-file /boot/kernel/radeon.ko.symbols 0xc6563b60 -s 
.data 0xc65b6440 -s .bss 0xc65b7d20
add symbol table from file "/boot/kernel/radeon.ko.symbols" at
   .text_addr = 0xc6563b60
   .data_addr = 0xc65b6440
   .bss_addr = 0xc65b7d20
(y or n) y
Reading symbols from /boot/kernel/radeon.ko.symbols...done.
(kgdb) add-symbol-file /boot/kernel/drm.ko.symbols 0xc65c08b0 -s .data 
0xc65ce580 -s .bss 0xc65cf6a4
add symbol table from file "/boot/kernel/drm.ko.symbols" at
   .text_addr = 0xc65c08b0
   .data_addr = 0xc65ce580
   .bss_addr = 0xc65cf6a4
(y or n) y
Reading symbols from /boot/kernel/drm.ko.symbols...done.
(kgdb) bt full
#0  doadump () at pcpu.h:246
No locals.
#1  0xc0870a8e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
   _giantcnt = Variable "_giantcnt" is not available.


So 'bt full' only gives me the giantcnt error.  Just a regular 'bt' 
gives me the same output I posted in my pr.

Am I doing something wrong?

I can possibly try a -CURRENT kernel without invariants in the next day 
or two, if you think that will make a difference, or just want to 
confirm your thought.
Comment 4 Adam K Kirchhoff 2009-06-21 13:51:37 UTC
You were correct that this problem does not happen with invariants 
disabled.  However, I am willing to test anything you'd like with 
invariants enabled again to help fix the source of the problem rather 
than work around it.

Adam
Comment 5 Jaakko Heinonen freebsd_committer freebsd_triage 2010-09-19 16:16:39 UTC
Responsible Changed
From-To: freebsd-bugs->jh

Do you still see this on 8.1?
Comment 6 Jaakko Heinonen freebsd_committer freebsd_triage 2010-10-30 09:09:23 UTC
State Changed
From-To: feedback->closed

Feedback timeout.