Bug 205469 - [panic] lock order reversal, userret: returning with the following locks held
Summary: [panic] lock order reversal, userret: returning with the following locks held
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Konstantin Belousov
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2015-12-20 21:33 UTC by Shawn Debnath
Modified: 2016-03-24 23:53 UTC (History)
8 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Shawn Debnath 2015-12-20 21:33:17 UTC
Ran into panic while trying to get the latest 11-current ISO to load in bhyve.


ISO
===
FreeBSD-11.0-CURRENT-amd64-20151217-r292413-disc1.iso

BHYVE CMDS
==========
sudo bhyveload -m 8G -d ~sd/FreeBSD-11.0-CURRENT-amd64-20151217-r292413-disc1.iso -h /dev/zvol/zroot/albert -c /dev/nmdm0A albert

sudo bhyve -c 8 -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/dev/zvol/zroot/albert -s 3:0,virtio-net,tap0  -s 4:0,ahci-cd,FreeBSD-11.0-CURRENT-amd64-20151217-r292413-disc1.iso -l com1,/dev/nmdm0A -A -H -P -m 8G albert  

DETAILS
=======
Trying to mount root from cd9660:/dev/iso9660/11_0_CURRENT_AMD64_CD [ro]...
lock order reversal:
 1st 0xfffffe01efa22e40 bufwait (bufwait) @ /usr/src/sys/vm/vm_pager.c:380
 2nd 0xfffff800098405f0 isofs (isofs) @ /usr/src/sys/kern/imgact_elf.c:877
stack backtrace:
#0 0xffffffff80a80560 at witness_debugger+0x70
#1 0xffffffff80a80461 at witness_checkorder+0xe71
#2 0xffffffff80a01e6b at __lockmgr_args+0xd3b
#3 0xffffffff80ac845c at vop_stdlock+0x3c
#4 0xffffffff80fbb260 at VOP_LOCK1_APV+0x100
#5 0xffffffff80ae93aa at _vn_lock+0x9a
#6 0xffffffff809c1f20 at exec_elf64_imgact+0xa50
#7 0xffffffff809e526d at kern_execve+0x42d
#8 0xffffffff809e4aec at sys_execve+0x4c
#9 0xffffffff809c891a at start_init+0x26a
#10 0xffffffff809ec8a4 at fork_exit+0x84
#11 0xffffffff80e4fbae at fork_trampoline+0xe
userret: returning with the following locks held:
exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe01efa22c10) locked @ /usr/src/sys/vm/vm_pager.c:380
exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe01efa22e40) locked @ /usr/src/sys/vm/vm_pager.c:380
panic: witness_warn


BOOT LOG
========
/boot/kernel/kernel text=0x13a80d8 data=0x1343e0+0x4d8ab8 syms=[0x8+0x15c180+0x8+0x173e0c]
Booting...
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2015 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r292413: Fri Dec 18 01:11:55 UTC 2015
    root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz (2933.07-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
  Features2=0x829e6217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI,HV>
  AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
Hypervisor: Origin = "bhyve bhyve "
real memory  = 9663676416 (9216 MB)
avail memory = 8249942016 (7867 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BHYVE  BVMADT  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 8 package(s) x 1 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 1.1> irqs 0-23 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff80ee1f10, 0) error 19
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <BHYVE BVXSDT> on motherboard
acpi0: Power Button (fixed)
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 10000000 Hz quality 950
Event timer "HPET" frequency 10000000 Hz quality 550
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH8 AHCI SATA controller> mem 0xc0000000-0xc00003ff irq 16 at device 2.0 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
virtio_pci0: <VirtIO PCI Network adapter> port 0x2000-0x201f mem 0xc0002000-0xc0003fff irq 17 at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 00:a0:98:95:a2:c5
001.000048 [ 421] vtnet_netmap_attach       max rings 1
vtnet0: netmap queues/slots: TX 1/1024, RX 1/1024
001.000049 [ 426] vtnet_netmap_attach       virtio attached txq=1, txd=1024 rxq=1, rxd=1024
ahci1: <Intel ICH8 AHCI SATA controller> mem 0xc0004000-0xc00043ff irq 18 at device 4.0 on pci0
ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich6: <AHCI channel> at channel 0 on ahci1
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
qpi0: <QPI system bus> on motherboard
vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
cd0 at ahcich6 bus 0 scbus1 target 0 lun 0
cd0: <BHYVE BHYVE DVD-ROM 001> Removable CD-ROM SCSI device
cd0: Serial Number BHYVE-E5BA-7EDF-596F
cd0: 600.000MB/s transfers (SATA 3.x, UDMA6, ATAPI 12bytes, PIO 8192bytes)
cd0: 808MB (413788 2048 byte sectors)
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <BHYVE SATA DISK 001> ACS-2 ATA SATA 3.x device
ada0: Serial Number BHYVE-0DD8-8ED8-A4CE
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 32768MB (67108864 512 byte sectors)
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from cd9660:/dev/iso9660/11_0_CURRENT_AMD64_CD [ro]...
lock order reversal:
 1st 0xfffffe01efa22e40 bufwait (bufwait) @ /usr/src/sys/vm/vm_pager.c:380
 2nd 0xfffff800098405f0 isofs (isofs) @ /usr/src/sys/kern/imgact_elf.c:877
stack backtrace:
#0 0xffffffff80a80560 at witness_debugger+0x70
#1 0xffffffff80a80461 at witness_checkorder+0xe71
#2 0xffffffff80a01e6b at __lockmgr_args+0xd3b
#3 0xffffffff80ac845c at vop_stdlock+0x3c
#4 0xffffffff80fbb260 at VOP_LOCK1_APV+0x100
#5 0xffffffff80ae93aa at _vn_lock+0x9a
#6 0xffffffff809c1f20 at exec_elf64_imgact+0xa50
#7 0xffffffff809e526d at kern_execve+0x42d
#8 0xffffffff809e4aec at sys_execve+0x4c
#9 0xffffffff809c891a at start_init+0x26a
#10 0xffffffff809ec8a4 at fork_exit+0x84
#11 0xffffffff80e4fbae at fork_trampoline+0xe
userret: returning with the following locks held:
exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe01efa22c10) locked @ /usr/src/sys/vm/vm_pager.c:380
exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe01efa22e40) locked @ /usr/src/sys/vm/vm_pager.c:380
panic: witness_warn
cpuid = 7
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe022f27d7f0
vpanic() at vpanic+0x182/frame 0xfffffe022f27d870
kassert_panic() at kassert_panic+0x126/frame 0xfffffe022f27d8e0
witness_warn() at witness_warn+0x3c6/frame 0xfffffe022f27d9b0
userret() at userret+0x98/frame 0xfffffe022f27d9e0
trap() at trap+0x3f4/frame 0xfffffe022f27dbf0
calltrap() at calltrap+0x8/frame 0xfffffe022f27dbf0
--- trap 0xc, rip = 0x4001c4, rsp = 0x7fffffffed80, rbp = 0x7fffffffedb0 ---
KDB: enter: panic
[ thread pid 1 tid 100002 ]
Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
db>
Comment 1 Shawn Debnath 2015-12-21 18:31:48 UTC
Just a headsup, I am seeing this issue with FreeBSD-11.0-CURRENT-amd64-20151130-r291495 as well. I will test more if I can get access to an ISO to install a base system with.
Comment 2 Oliver Pinter freebsd_committer freebsd_triage 2015-12-26 19:06:34 UTC
Confirmed, I can reproduce this error in bhyve: 

WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from cd9660:/dev/iso9660/11_0__HBSD_AMD64_CD [ro]...
lock order reversal:
 1st 0xfffffe001d822e40 bufwait (bufwait) @ /usr/src/sys/vm/vm_pager.c:380
 2nd 0xfffff8000512d240 isofs (isofs) @ /usr/src/sys/kern/imgact_elf.c:883
stack backtrace:
#0 0xffffffff80a7d0a0 at witness_debugger+0x70
#1 0xffffffff80a7cfa1 at witness_checkorder+0xe71
#2 0xffffffff80a0056b at __lockmgr_args+0xd3b
#3 0xffffffff80ac320c at vop_stdlock+0x3c
#4 0xffffffff80fc0fa0 at VOP_LOCK1_APV+0x100
#5 0xffffffff80ae3baa at _vn_lock+0x9a
#6 0xffffffff809c4d81 at exec_elf64_imgact+0xa91
#7 0xffffffff809e3809 at kern_execve+0x459
#8 0xffffffff809e305c at sys_execve+0x4c
#9 0xffffffff809c788a at start_init+0x26a
#10 0xffffffff809eafe4 at fork_exit+0x84
#11 0xffffffff80e4ec6e at fork_trampoline+0xe
userret: returning with the following locks held:
exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe001d822c10) locked @ /usr/src/sys/vm/vm_pager.c:380
exclusive lockmgr bufwait (bufwait) r = 0 (0xfffffe001d822e40) locked @ /usr/src/sys/vm/vm_pager.c:380
panic: witness_warn
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe001d7906b0
vpanic() at vpanic+0x182/frame 0xfffffe001d790730
kassert_panic() at kassert_panic+0x126/frame 0xfffffe001d7907a0
witness_warn() at witness_warn+0x3c6/frame 0xfffffe001d790870
userret() at userret+0x98/frame 0xfffffe001d7908a0
trap() at trap+0x3f4/frame 0xfffffe001d790ab0
calltrap() at calltrap+0x8/frame 0xfffffe001d790ab0
--- trap 0xc, rip = 0x4001c4, rsp = 0x7231f647c380, rbp = 0x7231f647c3b0 ---
KDB: enter: panic
[ thread pid 1 tid 100002 ]
Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
Comment 3 Peter Holm freebsd_committer freebsd_triage 2015-12-26 22:12:31 UTC
Exec of date(1) copied to a isofs file system triggered this:
https://people.freebsd.org/~pho/stress/log/isofs2.txt
Comment 4 Oliver Pinter freebsd_committer freebsd_triage 2015-12-27 12:00:11 UTC
This mail contains a possible (not yet tested) fix from kib@: https://lists.freebsd.org/pipermail/freebsd-current/2015-December/059069.html
Comment 5 Oliver Pinter freebsd_committer freebsd_triage 2015-12-27 14:53:37 UTC
And the fix in HEAD: 

Author: kib
Date: Sun Dec 27 14:42:39 2015
New Revision: 292772
URL: https://svnweb.freebsd.org/changeset/base/292772

Log:
  Add missed relpbuf() for a smallfs page-in.

  Reported by:  Shawn Webb
  Tested by:    pho
  Sponsored by: The FreeBSD Foundation

Modified:
  head/sys/vm/vnode_pager.c
Comment 6 Shawn Debnath 2015-12-30 04:08:08 UTC
Tested - issue has been resolved.
Comment 7 Shawn Debnath 2016-03-24 23:50:25 UTC
Issue seems to have been resolved with:

Author: kib
Date: Sun Dec 27 14:42:39 2015
New Revision: 292772
URL: https://svnweb.freebsd.org/changeset/base/292772
Comment 8 Shawn Debnath 2016-03-24 23:53:10 UTC
Can't find the bug for changeset 292772. Marking this as fixed. Please change as necessary.