Bug 205469
| Summary: | [panic] lock order reversal, userret: returning with the following locks held | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Shawn Debnath <sd> |
| Component: | kern | Assignee: | Konstantin Belousov <kib> |
| Status: | Closed FIXED | ||
| Severity: | Affects Only Me | CC: | alc, glebius, grahamperrin, kib, mjg, op, pho, shawn.webb |
| Priority: | --- | Keywords: | crash |
| Version: | CURRENT | ||
| Hardware: | Any | ||
| OS: | Any | ||
Just a headsup, I am seeing this issue with FreeBSD-11.0-CURRENT-amd64-20151130-r291495 as well. I will test more if I can get access to an ISO to install a base system with. Confirmed, I can reproduce this error in bhyve: WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from cd9660:/dev/iso9660/11_0__HBSD_AMD64_CD [ro]... lock order reversal: 1st 0xfffffe001d822e40 bufwait (bufwait) @ /usr/src/sys/vm/vm_pager.c:380 2nd 0xfffff8000512d240 isofs (isofs) @ /usr/src/sys/kern/imgact_elf.c:883 stack backtrace: #0 0xffffffff80a7d0a0 at witness_debugger+0x70 #1 0xffffffff80a7cfa1 at witness_checkorder+0xe71 #2 0xffffffff80a0056b at __lockmgr_args+0xd3b #3 0xffffffff80ac320c at vop_stdlock+0x3c #4 0xffffffff80fc0fa0 at VOP_LOCK1_APV+0x100 #5 0xffffffff80ae3baa at _vn_lock+0x9a #6 0xffffffff809c4d81 at exec_elf64_imgact+0xa91 #7 0xffffffff809e3809 at kern_execve+0x459 #8 0xffffffff809e305c at sys_execve+0x4c #9 0xffffffff809c788a at start_init+0x26a #10 0xffffffff809eafe4 at fork_exit+0x84 #11 0xffffffff80e4ec6e at fork_trampoline+0xe userret: returning with the following locks held: exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe001d822c10) locked @ /usr/src/sys/vm/vm_pager.c:380 exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe001d822e40) locked @ /usr/src/sys/vm/vm_pager.c:380 panic: witness_warn cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe001d7906b0 vpanic() at vpanic+0x182/frame 0xfffffe001d790730 kassert_panic() at kassert_panic+0x126/frame 0xfffffe001d7907a0 witness_warn() at witness_warn+0x3c6/frame 0xfffffe001d790870 userret() at userret+0x98/frame 0xfffffe001d7908a0 trap() at trap+0x3f4/frame 0xfffffe001d790ab0 calltrap() at calltrap+0x8/frame 0xfffffe001d790ab0 --- trap 0xc, rip = 0x4001c4, rsp = 0x7231f647c380, rbp = 0x7231f647c3b0 --- KDB: enter: panic [ thread pid 1 tid 100002 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why Exec of date(1) copied to a isofs file system triggered this: https://people.freebsd.org/~pho/stress/log/isofs2.txt This mail contains a possible (not yet tested) fix from kib@: https://lists.freebsd.org/pipermail/freebsd-current/2015-December/059069.html And the fix in HEAD: Author: kib Date: Sun Dec 27 14:42:39 2015 New Revision: 292772 URL: https://svnweb.freebsd.org/changeset/base/292772 Log: Add missed relpbuf() for a smallfs page-in. Reported by: Shawn Webb Tested by: pho Sponsored by: The FreeBSD Foundation Modified: head/sys/vm/vnode_pager.c Tested - issue has been resolved. Issue seems to have been resolved with: Author: kib Date: Sun Dec 27 14:42:39 2015 New Revision: 292772 URL: https://svnweb.freebsd.org/changeset/base/292772 Can't find the bug for changeset 292772. Marking this as fixed. Please change as necessary. |
Ran into panic while trying to get the latest 11-current ISO to load in bhyve. ISO === FreeBSD-11.0-CURRENT-amd64-20151217-r292413-disc1.iso BHYVE CMDS ========== sudo bhyveload -m 8G -d ~sd/FreeBSD-11.0-CURRENT-amd64-20151217-r292413-disc1.iso -h /dev/zvol/zroot/albert -c /dev/nmdm0A albert sudo bhyve -c 8 -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/dev/zvol/zroot/albert -s 3:0,virtio-net,tap0 -s 4:0,ahci-cd,FreeBSD-11.0-CURRENT-amd64-20151217-r292413-disc1.iso -l com1,/dev/nmdm0A -A -H -P -m 8G albert DETAILS ======= Trying to mount root from cd9660:/dev/iso9660/11_0_CURRENT_AMD64_CD [ro]... lock order reversal: 1st 0xfffffe01efa22e40 bufwait (bufwait) @ /usr/src/sys/vm/vm_pager.c:380 2nd 0xfffff800098405f0 isofs (isofs) @ /usr/src/sys/kern/imgact_elf.c:877 stack backtrace: #0 0xffffffff80a80560 at witness_debugger+0x70 #1 0xffffffff80a80461 at witness_checkorder+0xe71 #2 0xffffffff80a01e6b at __lockmgr_args+0xd3b #3 0xffffffff80ac845c at vop_stdlock+0x3c #4 0xffffffff80fbb260 at VOP_LOCK1_APV+0x100 #5 0xffffffff80ae93aa at _vn_lock+0x9a #6 0xffffffff809c1f20 at exec_elf64_imgact+0xa50 #7 0xffffffff809e526d at kern_execve+0x42d #8 0xffffffff809e4aec at sys_execve+0x4c #9 0xffffffff809c891a at start_init+0x26a #10 0xffffffff809ec8a4 at fork_exit+0x84 #11 0xffffffff80e4fbae at fork_trampoline+0xe userret: returning with the following locks held: exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe01efa22c10) locked @ /usr/src/sys/vm/vm_pager.c:380 exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe01efa22e40) locked @ /usr/src/sys/vm/vm_pager.c:380 panic: witness_warn BOOT LOG ======== /boot/kernel/kernel text=0x13a80d8 data=0x1343e0+0x4d8ab8 syms=[0x8+0x15c180+0x8+0x173e0c] Booting... GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2015 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #0 r292413: Fri Dec 18 01:11:55 UTC 2015 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906 WARNING: WITNESS option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU X5670 @ 2.93GHz (2933.07-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2 Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE> Features2=0x829e6217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI,HV> AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant Hypervisor: Origin = "bhyve bhyve " real memory = 9663676416 (9216 MB) avail memory = 8249942016 (7867 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <BHYVE BVMADT > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 8 package(s) x 1 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0 <Version 1.1> irqs 0-23 on motherboard random: entropy device external interface kbd1 at kbdmux0 netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff80ee1f10, 0) error 19 vtvga0: <VT VGA driver> on motherboard cryptosoft0: <software crypto> on motherboard acpi0: <BHYVE BVXSDT> on motherboard acpi0: Power Button (fixed) atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 10000000 Hz quality 950 Event timer "HPET" frequency 10000000 Hz quality 550 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 isab0: <PCI-ISA bridge> at device 1.0 on pci0 isa0: <ISA bus> on isab0 ahci0: <Intel ICH8 AHCI SATA controller> mem 0xc0000000-0xc00003ff irq 16 at device 2.0 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: <AHCI channel> at channel 0 on ahci0 virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem 0xc0002000-0xc0003fff irq 17 at device 3.0 on pci0 vtnet0: <VirtIO Networking Adapter> on virtio_pci0 vtnet0: Ethernet address: 00:a0:98:95:a2:c5 001.000048 [ 421] vtnet_netmap_attach max rings 1 vtnet0: netmap queues/slots: TX 1/1024, RX 1/1024 001.000049 [ 426] vtnet_netmap_attach virtio attached txq=1, txd=1024 rxq=1, rxd=1024 ahci1: <Intel ICH8 AHCI SATA controller> mem 0xc0004000-0xc00043ff irq 18 at device 4.0 on pci0 ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich6: <AHCI channel> at channel 0 on ahci1 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (9600,n,8,1) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 qpi0: <QPI system bus> on motherboard vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. cd0 at ahcich6 bus 0 scbus1 target 0 lun 0 cd0: <BHYVE BHYVE DVD-ROM 001> Removable CD-ROM SCSI device cd0: Serial Number BHYVE-E5BA-7EDF-596F cd0: 600.000MB/s transfers (SATA 3.x, UDMA6, ATAPI 12bytes, PIO 8192bytes) cd0: 808MB (413788 2048 byte sectors) ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <BHYVE SATA DISK 001> ACS-2 ATA SATA 3.x device ada0: Serial Number BHYVE-0DD8-8ED8-A4CE ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 32768MB (67108864 512 byte sectors) SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #6 Launched! WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from cd9660:/dev/iso9660/11_0_CURRENT_AMD64_CD [ro]... lock order reversal: 1st 0xfffffe01efa22e40 bufwait (bufwait) @ /usr/src/sys/vm/vm_pager.c:380 2nd 0xfffff800098405f0 isofs (isofs) @ /usr/src/sys/kern/imgact_elf.c:877 stack backtrace: #0 0xffffffff80a80560 at witness_debugger+0x70 #1 0xffffffff80a80461 at witness_checkorder+0xe71 #2 0xffffffff80a01e6b at __lockmgr_args+0xd3b #3 0xffffffff80ac845c at vop_stdlock+0x3c #4 0xffffffff80fbb260 at VOP_LOCK1_APV+0x100 #5 0xffffffff80ae93aa at _vn_lock+0x9a #6 0xffffffff809c1f20 at exec_elf64_imgact+0xa50 #7 0xffffffff809e526d at kern_execve+0x42d #8 0xffffffff809e4aec at sys_execve+0x4c #9 0xffffffff809c891a at start_init+0x26a #10 0xffffffff809ec8a4 at fork_exit+0x84 #11 0xffffffff80e4fbae at fork_trampoline+0xe userret: returning with the following locks held: exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe01efa22c10) locked @ /usr/src/sys/vm/vm_pager.c:380 exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe01efa22e40) locked @ /usr/src/sys/vm/vm_pager.c:380 panic: witness_warn cpuid = 7 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe022f27d7f0 vpanic() at vpanic+0x182/frame 0xfffffe022f27d870 kassert_panic() at kassert_panic+0x126/frame 0xfffffe022f27d8e0 witness_warn() at witness_warn+0x3c6/frame 0xfffffe022f27d9b0 userret() at userret+0x98/frame 0xfffffe022f27d9e0 trap() at trap+0x3f4/frame 0xfffffe022f27dbf0 calltrap() at calltrap+0x8/frame 0xfffffe022f27dbf0 --- trap 0xc, rip = 0x4001c4, rsp = 0x7fffffffed80, rbp = 0x7fffffffedb0 --- KDB: enter: panic [ thread pid 1 tid 100002 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db>