A function _mtx_lock_sleep() (sys/kern/kern_mutex.c) has a code: #ifdef ADAPTIVE_MUTEXES /* * If the owner is running on another CPU, spin until the * owner stops running or the state of the lock changes. */ v = m->mtx_lock; if (v != MTX_UNOWNED) { owner = (struct thread *)(v & ~MTX_FLAGMASK); if (TD_IS_RUNNING(owner)) { A macros TD_IS_RUNNING(owner) dereferences "owner" pointer and there is no check for NULL. There is 100% repeatable test case in which "owner" is always null, so this code panices kernel. Kernel config file and crashdump info follow. Fix: Unknown. How-To-Repeat: 1. Use kernel with options MAC (default in GENERIC) and options ADAPTIVE_MUTEXES (not present in GENERIC). 2. Add mac_portacl_load="YES" to /boot/loader.conf 3. Run shell command security.mac.portacl.rules=uid:88:tcp:80 4. Observe a panic. Here comes kgdb backtrace. Script started on Sun Feb 2 12:04:59 2014 command: kgdb kernel.debug /var/crash/vmcore.7 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.2-STABLE #10 r256953M: Sat Feb 1 17:35:38 NOVT 2014 root@grosbein.net:/usr/obj/usr/local/src/sys/DADV amd64 gcc version 4.2.1 20070831 patched [FreeBSD] can't re-use a leaf (geom_cache)! can't re-use a leaf (hwpstate_verbose)! module_register: module cpu/ichss already exists! Module cpu/ichss failed to register: 17 module_register: module g_cache already exists! Module g_cache failed to register: 17 module_register: module mac_portacl already exists! Module mac_portacl failed to register: 17 module_register: module cpu/coretemp already exists! Module cpu/coretemp failed to register: 17 module_register: module cpu/powernow already exists! Module cpu/powernow failed to register: 17 module_register: module cpu/est already exists! Module cpu/est failed to register: 17 module_register: module cpu/hwpstate already exists! Module cpu/hwpstate failed to register: 17 module_register: module cpu/p4tcc already exists! Module cpu/p4tcc failed to register: 17 CPU: Intel(R) Pentium(R) D CPU 2.80GHz (2800.16-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0xf62 Family = 0xf Model = 0x6 Stepping = 2 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0xe43d<SSE3,DTES64,MON,DS_CPL,VMX,CNXT-ID,CX16,xTPR,PDCM> AMD Features=0x20100800<SYSCALL,NX,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant real memory = 6442450944 (6144 MB) avail memory = 6152839168 (5867 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <INTEL D975XBX > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 Security policy loaded: TrustedBSD MAC/portacl (mac_portacl) Security policy unload: TrustedBSD MAC/portacl (mac_portacl) module_register_init: MOD_LOAD (mac_portacl, 0xffffffff8079c889, 0xffffffff80c074c0) error 17 WARNING: VIMAGE (virtualized network stack) is a highly experimental feature. ioapic0: Changing APIC ID to 2 ioapic0 <Version 2.0> irqs 0-23 on motherboard kbd1 at kbdmux0 cryptosoft0: <software crypto> on motherboard acpi0: <INTEL D975XBX> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_button0: <Sleep Button> on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pcib0: Length mismatch for 3 range: 6c00000 vs 6bfffff pcib0: Length mismatch for 3 range: 20000001 vs 20000000 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 vgapci0: <VGA-compatible display> port 0x3000-0x307f mem 0xe1000000-0xe1ffffff,0xd0000000-0xdfffffff,0xe0000000-0xe0ffffff irq 16 at device 0.0 on pci1 nvidia0: <GeForce 7600 GS> on vgapci0 vgapci0: child nvidia0 requested pci_enable_io vgapci0: child nvidia0 requested pci_enable_io hdac0: <Intel 82801G HDA Controller> mem 0xe2300000-0xe2303fff irq 22 at device 27.0 on pci0 pcib2: <ACPI PCI-PCI bridge> at device 28.0 on pci0 pci2: <ACPI PCI bus> on pcib2 ath0: <Atheros 9285> mem 0xe2200000-0xe220ffff irq 16 at device 0.0 on pci2 ath0: AR9285 mac 192.2 RF5133 phy 14.0 pcib3: <ACPI PCI-PCI bridge> at device 28.4 on pci0 pci3: <ACPI PCI bus> on pcib3 pcib4: <ACPI PCI-PCI bridge> at device 28.5 on pci0 pci4: <ACPI PCI bus> on pcib4 em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2000-0x201f mem 0xe2100000-0xe211ffff irq 17 at device 0.0 on pci4 em0: Using an MSI interrupt em0: Ethernet address: 00:16:76:6b:04:d7 uhci0: <Intel 82801G (ICH7) USB controller USB-A> port 0x4080-0x409f irq 23 at device 29.0 on pci0 usbus0 on uhci0 uhci1: <Intel 82801G (ICH7) USB controller USB-B> port 0x4060-0x407f irq 19 at device 29.1 on pci0 usbus1 on uhci1 uhci2: <Intel 82801G (ICH7) USB controller USB-C> port 0x4040-0x405f irq 18 at device 29.2 on pci0 usbus2 on uhci2 uhci3: <Intel 82801G (ICH7) USB controller USB-D> port 0x4020-0x403f irq 16 at device 29.3 on pci0 usbus3 on uhci3 ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xe2304400-0xe23047ff irq 23 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4 on ehci0 pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci5: <ACPI PCI bus> on pcib5 fwohci0: <Texas Instruments TSB43AB23> mem 0xe2004000-0xe20047ff,0xe2000000-0xe2003fff irq 18 at device 4.0 on pci5 fwohci0: OHCI version 1.10 (ROM=0) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:90:27:00:01:a6:c5:f9 fwohci0: Phy 1394a available S400, 3 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: <IEEE1394(FireWire) bus> on fwohci0 dcons_crom0: <dcons configuration ROM> on firewire0 dcons_crom0: bus_addr 0x2de4000 fwe0: <Ethernet over FireWire> on firewire0 if_fwe0: Fake Ethernet address: 02:90:27:a6:c5:f9 fwe0: Ethernet address: 02:90:27:a6:c5:f9 fwip0: <IP over FireWire> on firewire0 fwip0: Firewire address: 00:90:27:00:01:a6:c5:f9 @ 0xfffe00000000, S400, maxrec 2048 sbp0: <SBP-2/SCSI over FireWire> on firewire0 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode atapci0: <SiI 3114 SATA150 controller> port 0x1018-0x101f,0x1024-0x1027,0x1010-0x1017,0x1020-0x1023,0x1000-0x100f mem 0xe2004800-0xe2004bff irq 17 at device 5.0 on pci5 ata2: <ATA channel> at channel 0 on atapci0 ata3: <ATA channel> at channel 1 on atapci0 ata4: <ATA channel> at channel 2 on atapci0 ata5: <ATA channel> at channel 3 on atapci0 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci1: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x40b0-0x40bf irq 18 at device 31.1 on pci0 ata0: <ATA channel> at channel 0 on atapci1 atapci2: <Intel ICH7 SATA300 controller> port 0x40c8-0x40cf,0x40e4-0x40e7,0x40c0-0x40c7,0x40e0-0x40e3,0x40a0-0x40af mem 0xe2304000-0xe23043ff irq 19 at device 31.2 on pci0 atapci2: AHCI called from vendor specific driver atapci2: AHCI v1.10 controller with 4 3Gbps ports, PM not supported ata6: <ATA channel> at channel 0 on atapci2 ata7: <ATA channel> at channel 1 on atapci2 ata8: <ATA channel> at channel 2 on atapci2 ata9: <ATA channel> at channel 3 on atapci2 ichsmb0: <Intel 82801GB (ICH7) SMBus controller> port 0x4000-0x401f irq 19 at device 31.3 on pci0 smbus0: <System Management Bus> on ichsmb0 smb0: <SMBus generic I/O> on smbus0 ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/8 bytes threshold ppbus0: <Parallel port bus> on ppc0 lpt0: <Printer> on ppbus0 lpt0: Interrupt-driven port atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 ichwd0 on isa0 orm0: <ISA Option ROMs> at iomem 0xcf000-0xd3fff,0xd4000-0xd87ff on isa0 sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 p4tcc0: <CPU Frequency Thermal Control> on cpu0 p4tcc1: <CPU Frequency Thermal Control> on cpu1 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 vboxdrv: fAsync=0 offMin=0x2d8 offMax=0x69e IPsec: Initialized Security Association Processing. ipfw2 initialized, divert enabled, nat enabled, default to deny, logging disabled DUMMYNET 0xfffffe0002bad580 with IPv6 initialized (100409) load_dn_sched dn_sched FIFO loaded load_dn_sched dn_sched PRIO loaded load_dn_sched dn_sched QFQ loaded load_dn_sched dn_sched RR loaded load_dn_sched dn_sched WF2Q+ loaded hdacc0: <Sigmatel STAC9221D HDA CODEC> at cad 2 on hdac0 hdaa0: <Sigmatel STAC9221D Audio Function Group> at nid 1 on hdacc0 pcm0: <Sigmatel STAC9221D (Analog 5.1+HP/2.0)> at nid 13,15,11,10 and 14,21,12 on hdaa0 pcm1: <Sigmatel STAC9221D (Rear Digital)> at nid 16 on hdaa0 The GEOM class CACHE is already loaded. usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 480Mbps High Speed USB v2.0 ugen0.1: <Intel> at usbus0 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ugen1.1: <Intel> at usbus1 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 ugen2.1: <Intel> at usbus2 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 ugen3.1: <Intel> at usbus3 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 ugen4.1: <Intel> at usbus4 uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered ada0 at ata6 bus 0 scbus6 target 0 lun 0 ada0: <ST3500410AS CC31> ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada0: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad12 ada1 at ata7 bus 0 scbus7 target 0 lun 0 ada1: <WDC WD20EARX-00PASB0 51.0AB51> ATA-8 SATA 3.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada1: quirks=0x1<4K> ada1: Previously was known as ad14 ada2 at ata9 bus 0 scbus9 target 0 lun 0 ada2: <WDC WD5001AALS-00L3B2 01.03B01> ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada2: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C) ada2: Previously was known as ad18 cd0 at ata2 bus 0 scbus1 target 0 lun 0 SMP: AP CPU #1 Launched! cd0: <Optiarc DVD RW AD-7241S 1.02> Removable CD-ROM SCSI-0 device cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed Timecounter "TSC-low" frequency 1400079352 Hz quality 1000 GEOM_RAID: Intel-3b6bf030: Array Intel-3b6bf030 created. GEOM_RAID: Intel-3b6bf030: Disk ada0 state changed from NONE to FAILED. GEOM_RAID: Intel-3b6bf030: Subdisk DUAL:0-ada0 state changed from NONE to FAILED. GEOM_RAID: Intel-3b6bf030: Disk ada2 state changed from NONE to ACTIVE. GEOM_RAID: Intel-3b6bf030: Subdisk DUAL:1-ada2 state changed from NONE to ACTIVE. GEOM_RAID: Intel-3b6bf030: Array started. GEOM_RAID: Intel-3b6bf030: Volume DUAL state changed from STARTING to DEGRADED. GEOM_RAID: Intel-3b6bf030: Provider raid/r0 for volume DUAL created. GEOM_CACHE: Device ca0 created. GEOM_JOURNAL: Journal 1867989504: ada1s2a contains data. GEOM_JOURNAL: Journal 1867989504: ada1s2a contains journal. GEOM_JOURNAL: Journal ada1s2a clean. GEOM_JOURNAL: Journal 2090598348: cache/ca0s4g contains data. GEOM_JOURNAL: Journal 2090598348: cache/ca0s4g contains journal. GEOM_JOURNAL: Journal cache/ca0s4g clean. Root mount waiting for: GJOURNAL usbus4 uhub4: 8 ports with 8 removable, self powered Root mount waiting for: usbus4 ugen4.2: <NEC> at usbus4 uhub5: <NEC USB 2.0 4-Port Hub, class 9/0, rev 2.00/1.00, addr 2> on usbus4 uhub5: 4 ports with 4 removable, self powered Root mount waiting for: usbus4 ugen0.2: <American Power Conversion> at usbus0 Root mount waiting for: usbus4 Trying to mount root from ufs:/dev/cache/ca0s4a [rw]... <118>Enter full pathname of shell or RETURN for /bin/sh: ugen3.2: <Prolific Technology Inc.> at usbus3 ugen1.2: <Razer> at usbus1 ums0: <Razer Razer Abyssus, class 0/0, rev 2.00/1.00, addr 2> on usbus1 ums0: 7 buttons and [XYZ] coordinates ID=0 <118>Cannot read termcap database; <118>using dumb terminal settings. <118># dumpon /dev/label/swap <118># ecurity.mac.portacl.rules=uid:88:tcp:80 <118>security.mac.portacl.rules: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x368 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8056134e stack pointer = 0x28:0xffffff81be785850 frame pointer = 0x28:0xffffff81be7858a0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 40 (sysctl) trap number = 12 panic: page fault cpuid = 0 Uptime: 1m34s Dumping 347 out of 6112 MB:..5%..14%..24%..33%..42%..51%..65%..74%..83%..93% Reading symbols from /boot/modules/coretemp.ko...done. Loaded symbols for /boot/modules/coretemp.ko Reading symbols from /boot/modules/mac_portacl.ko...done. Loaded symbols for /boot/modules/mac_portacl.ko Reading symbols from /boot/modules/nvidia.ko...done. Loaded symbols for /boot/modules/nvidia.ko Reading symbols from /boot/modules/vboxdrv.ko...done. Loaded symbols for /boot/modules/vboxdrv.ko Reading symbols from /boot/modules/geom_cache.ko...done. Loaded symbols for /boot/modules/geom_cache.ko Reading symbols from /boot/modules/geom_raid.ko...done. Loaded symbols for /boot/modules/geom_raid.ko Reading symbols from /boot/modules/cpufreq.ko...done. Loaded symbols for /boot/modules/cpufreq.ko #0 doadump (textdump=1) at pcpu.h:234 234 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) bt full #0 doadump (textdump=1) at pcpu.h:234 No locals. #1 0xffffffff805729b8 in kern_reboot (howto=260) at /usr/local/src/sys/kern/kern_shutdown.c:449 _ep = (struct eventhandler_entry *) 0x0 _el = <value optimized out> first_buf_printf = 1 #2 0xffffffff80572ee7 in panic (fmt=0x0) at /usr/local/src/sys/kern/kern_shutdown.c:637 td = (struct thread *) 0x1 bootopt = <value optimized out> newpanic = <value optimized out> ap = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0xffffff81be785560, reg_save_area = 0xffffff81be785480}} panic_cpu = 0 buf = "page fault", '\0' <repeats 245 times> #3 0xffffffff80833556 in trap_fatal (frame=0xc, eva=<value optimized out>) at /usr/local/src/sys/amd64/amd64/trap.c:879 code = 40 ss = 40 type = 12 esp = <value optimized out> ---Type <return> to continue, or q <return> to quit--- softseg = {ssd_base = 0, ssd_limit = 1048575, ssd_type = 27, ssd_dpl = 0, ssd_p = 1, ssd_long = 1, ssd_def32 = 0, ssd_gran = 1} msg = <value optimized out> #4 0xffffffff80833848 in trap_pfault (frame=0xffffff81be7857a0, usermode=0) at /usr/local/src/sys/amd64/amd64/trap.c:795 va = 0 vm = <value optimized out> map = 0xfffffe00068b6000 rv = 0 ftype = 255 'ÿ' td = (struct thread *) 0xfffffe0006801000 p = (struct proc *) 0xfffffe00067fb000 eva = 872 #5 0xffffffff80833e3e in trap (frame=0xffffff81be7857a0) at /usr/local/src/sys/amd64/amd64/trap.c:463 td = (struct thread *) 0xfffffe0006801000 p = <value optimized out> i = <value optimized out> ucode = <value optimized out> code = 0 type = 12 addr = <value optimized out> ksi = {ksi_link = {tqe_next = 0xffffff81be78585c, tqe_prev = 0x0}, ---Type <return> to continue, or q <return> to quit--- ksi_info = {si_signo = -1099409648, si_errno = -127, si_code = -2141412496, si_pid = -1, si_uid = 107410432, si_status = -512, si_addr = 0xffffff81be785750, si_value = {sival_int = -1099409600, sival_ptr = 0xffffff81be785740, sigval_int = -1099409600, sigval_ptr = 0xffffff81be785740}, _reason = {_fault = { _trapno = -2141384299}, _timer = {_timerid = -2141384299, _overrun = -1}, _mesgq = {_mqd = -2141384299}, _poll = { _band = -2141384299}, __spare__ = {__spare1__ = -2141384299, __spare2__ = {-1099409584, -127, -2134554144, -1, -1099409584, -127, 107410432}}}}, ksi_flags = -1099409408, ksi_sigq = 0xffffffff8043df1c} #6 0xffffffff8081da2f in calltrap () at /usr/local/src/sys/amd64/amd64/exception.S:232 No locals. #7 0xffffffff8056134e in _mtx_lock_sleep (m=0xffffffff80cc8720, tid=18446741874795352064, opts=<value optimized out>, file=<value optimized out>, line=0) at /usr/local/src/sys/kern/kern_mutex.c:394 ts = <value optimized out> v = 0 owner = (volatile struct thread *) 0x0 #8 0xffffffff807ab3ea in sysctl_rules (oidp=0xffffffff80c07160, arg1=<value optimized out>, arg2=<value optimized out>, ---Type <return> to continue, or q <return> to quit--- req=0xffffff81be7859b0) at /usr/local/src/sys/security/mac_portacl/mac_portacl.c:347 string = <value optimized out> copy_string = <value optimized out> new_string = 0xfffffe0006730c00 "" head = {tqh_first = 0xffffff81be785910, tqh_last = 0xffffffff807ffd22} save_head = {tqh_first = 0xfffffe0002d4f500, tqh_last = 0x7fffffffd000} error = <value optimized out> #9 0xffffffff8057c504 in sysctl_root (oidp=<value optimized out>, arg1=0x0, arg2=0, req=0xffffff81be7859b0) at /usr/local/src/sys/kern/kern_sysctl.c:1493 oid = (struct sysctl_oid *) 0xffffffff80c07160 error = 0 indx = 4 sysctl___debug_fail_point_sysctl_running = { oid_parent = 0xffffffff80c660c0, oid_link = {sle_next = 0xffffffff80beae60}, oid_number = 756, oid_kind = 3221487619, oid_arg1 = 0xffffffff80bde2e0, oid_arg2 = 0, oid_name = 0xffffffff8092eb75 "sysctl_running", oid_handler = 0xffffffff805442b8 <fail_point_sysctl>, oid_fmt = 0xffffffff808fb93f "A", oid_refcnt = 0, oid_running = 0, oid_descr = 0xffffffff808f8f74 ""} __set_sysctl_set_sym_sysctl___debug_fail_point_sysctl_running = ( const void * const) 0xffffffff80bde5e0 ---Type <return> to continue, or q <return> to quit--- _fail_point_sysctl_running = { fp_name = 0xffffffff8092eb75 "sysctl_running", fp_location = 0xffffffff8092eaf0 "(/usr/local/src/sys/kern/kern_sysctl.c:1497)", fp_entries = {tqh_first = 0x0, tqh_last = 0xffffffff80bde2f0}, fp_flags = 0, fp_sleep_fn = 0, fp_sleep_arg = 0x0} #10 0xffffffff8057c834 in userland_sysctl (td=0xfffffe0006801000, name=0xffffff81be785a70, namelen=4, old=<value optimized out>, oldlenp=<value optimized out>, inkernel=0, new=0x7fffffffd7eb, newlen=13, retval=0xffffff81be785ad8, flags=0) at /usr/local/src/sys/kern/kern_sysctl.c:1603 error = 0 memlocked = 0 req = {td = 0xfffffe0006801000, lock = 1, oldptr = 0x0, oldlen = 0, oldidx = 0, oldfunc = 0xffffffff8057cd82 <sysctl_old_user>, newptr = 0x7fffffffd7eb, newlen = 13, newidx = 0, newfunc = 0xffffffff8057c378 <sysctl_new_user>, validlen = 0, flags = 0} saved_vnet = (struct vnet *) 0x0 #11 0xffffffff8057cd4c in sys___sysctl (td=0xfffffe0006801000, uap=0xffffff81be785bb0) at /usr/local/src/sys/kern/kern_sysctl.c:1529 error = 0 i = <value optimized out> name = {716, 1384, 1393, 1388, -1099408704, -127, -2141473826, -1, 1, 5, 12615680, 8, 12619776, 8, -2134826080, -1, 47514880, -512, 109056000, ---Type <return> to continue, or q <return> to quit--- -512, -2134826080, -1, 0, 0} j = 18446744071570665925 #12 0xffffffff80832e72 in amd64_syscall (td=0xfffffe0006801000, traced=0) at subr_syscall.c:135 sa = {code = 202, callp = 0xffffffff80bcfe40, args = {140737488343824, 4, 0, 0, 140737488345067, 13, -542265287632, -2141958881}, narg = 6} error = 0 ksi = {ksi_link = {tqe_next = 0x31, tqe_prev = 0xdddd15a414c3ff7b}, ksi_info = {si_signo = -1099408512, si_errno = -127, si_code = 19409280, si_pid = 0, si_uid = 3195558768, si_status = 1, si_addr = 0xddd3e89d69d5e07e, si_value = {sival_int = -2115297920, sival_ptr = 0xffffffff81eb1d80, sigval_int = -2115297920, sigval_ptr = 0xffffffff81eb1d80}, _reason = {_fault = { _trapno = -2133633504}, _timer = {_timerid = -2133633504, _overrun = -1}, _mesgq = {_mqd = -2133633504}, _poll = { _band = -2133633504}, __spare__ = {__spare1__ = -2133633504, __spare2__ = {0, 0, 12615680, 8, -1099408400, -127, -2138782070}}}}, ksi_flags = -11344, ksi_sigq = 0x0} #13 0xffffffff8081dd17 in Xfast_syscall () at /usr/local/src/sys/amd64/amd64/exception.S:391 No locals. #14 0x000000080091b41c in ?? () No symbol table info available. Previous frame inner to this frame (corrupt stack?) (kgdb) frame 7 #7 0xffffffff8056134e in _mtx_lock_sleep (m=0xffffffff80cc8720, tid=18446741874795352064, opts=<value optimized out>, file=<value optimized out>, line=0) at /usr/local/src/sys/kern/kern_mutex.c:394 394 owner = (struct thread *)(v & ~MTX_FLAGMASK); (kgdb) l 389 * If the owner is running on another CPU, spin until the 390 * owner stops running or the state of the lock changes. 391 */ 392 v = m->mtx_lock; 393 if (v != MTX_UNOWNED) { 394 owner = (struct thread *)(v & ~MTX_FLAGMASK); 395 if (TD_IS_RUNNING(owner)) { 396 if (LOCK_LOG_TEST(&m->lock_object, 0)) 397 CTR3(KTR_LOCK, 398 "%s: spinning on %p held by %p", (kgdb) p v $1 = 0 (kgdb) p owner $2 = (volatile struct thread *) 0x0 (kgdb) quit Script done on Sun Feb 2 12:05:14 2014 The kernel config file follows. cpu HAMMER ident DADV options COMPAT_FREEBSD32 options INCLUDE_CONFIG_FILE # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. # Use the following to compile in values accessible to the kernel # through getenv() (or kenv(1) in userland). The format of the file # is 'variable=value', see kenv(1) # # env "GENERIC.env" makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols #options KVA_PAGES=512 options DEADLKRES options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking #options INET6 # IPv6 communications protocols options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options GEOM_JOURNAL options GEOM_CACHE options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFSLOCKD # Network Lock Manager options NFS_ROOT # NFS usable as /, requires NFSCLIENT options NFSCL options NFSD options MSDOSFS # MSDOS Filesystem options NTFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options FDESCFS options LINPROCFS options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty) options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 #options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=512 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework options MAC_PORTACL #options FLOWTABLE # per-cpu routing cache #options KDTRACE_HOOKS # Kernel DTrace hooks options UDF options LIBICONV options CD9660_ICONV options MSDOSFS_ICONV options NTFS_ICONV options UDF_ICONV # Debugging for use in -current options BREAK_TO_DEBUGGER options KDB # Enable kernel debugger support. options KDB_UNATTENDED # Enable kernel debugger support. options DDB # Support DDB. options DDB_NUMSYM # Support DDB. options GDB # Support remote GDB. #options INVARIANTS # Enable calls of extra sanity checking #options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS #options WITNESS # Enable checks to detect deadlocks and cycles #options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed # To make an SMP kernel, the next two lines are needed options SMP # Symmetric MultiProcessor Kernel #device apic # I/O APIC # CPU frequency control #device cpufreq # Bus support. device acpi #device eisa device pci # Floppy drives #device fdc # ATA and ATAPI devices device ata #device atadisk # ATA disk drives #device ataraid # ATA RAID drives #device atapicd # ATAPI CDROM drives #device atapifd # ATAPI floppy drives #device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering options ATA_CAM # SCSI peripherals device scbus # SCSI bus (required for SCSI) #device ch # SCSI media changers device da # Direct Access (disks) #device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) #device ses # SCSI Environmental Services (and SAF-TE) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc #device agp # support several AGP chipsets # Serial (COM) ports device uart # Generic UART driver device sound device snd_hda # Parallel port device ppc device ppbus # Parallel port bus (required) device lpt # Printer #device plip # TCP/IP over parallel #device ppi # Parallel port interface device #device vpo # Requires scbus and da # If you've got a "dumb" serial or parallel PCI card that is # supported by the puc(4) glue driver, uncomment the following # line to enable it (connects to sio, uart and/or ppc drivers): #device puc device cpuctl device smbus device smb device ichsmb device iicbus device iicbb device ic device iic device iicsmb device ichwd # PCI Ethernet NICs. #device de # DEC/Intel DC21x4x (``Tulip'') #device em # Intel PRO/1000 Gigabit Ethernet Family #device igb # Intel PRO/1000 PCIE Server Gigabit Family #device ixgb # Intel PRO/10GbE Ethernet Card #device le # AMD Am7900 LANCE and Am79C9xx PCnet #device ti # Alteon Networks Tigon I/II gigabit Ethernet #device txp # 3Com 3cR990 (``Typhoon'') #device vx # 3Com 3c590, 3c595 (``Vortex'') # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device wlan device ath device ath_pci device ath_hal device ath_rate_sample options AH_SUPPORT_AR5416 # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device tun # Packet tunnel. device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module device tap device if_bridge device coretemp device cpufreq # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support options USB_VERBOSE device uhci # UHCI PCI->USB interface #device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices device uhid # "Human Interface Devices" device ukbd # Keyboard device ulpt # Printer device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse #device rum # Ralink Technology RT2501USB wireless NICs #device ural # Ralink Technology RT2500USB wireless NICs #device uath # Atheros AR5523 wireless NICs #device zyd # ZyDAS zb1211/zb1211b wireless NICs #device urio # Diamond Rio 500 MP3 player # USB Serial devices #device u3g # USB-based 3G modems (Option, Huawei, Sierra) #device uark # Technologies ARK3116 based serial adapters #device ubsa # Belkin F5U103 and compatible serial adapters #device uftdi # For FTDI usb serial adapters #device uipaq # Some WinCE based devices #device uplcom # Prolific PL-2303 serial adapters #device uslcom # SI Labs CP2101/CP2102 serial adapters #device uvisor # Visor and Palm devices #device uvscom # USB serial support for DDI pocket's PHS # USB Ethernet, requires miibus #device aue # ADMtek USB Ethernet #device axe # ASIX Electronics USB Ethernet #device cdce # Generic USB over Ethernet #device cue # CATC USB Ethernet #device kue # Kawasaki LSI USB Ethernet #device rue # RealTek RTL8150 USB Ethernet #device udav # Davicom DM9601E USB # FireWire support device firewire # FireWire bus code device sbp # SCSI over FireWire (Requires scbus and da) device fwe # Ethernet over FireWire (non-standard!) device fwip # IP over FireWire (RFC 2734,3146) device dcons # Dumb console driver device dcons_crom # Configuration ROM for dcons options LIBALIAS options IPFIREWALL options IPFIREWALL_NAT options IPDIVERT options DUMMYNET options IPSEC_FILTERTUNNEL options IPSEC device crypto #options NETGRAPH #options NETGRAPH_ETHER options COMPAT_LINUX32 #options DEVICE_POLLING #options DEADLKRES device em options VIMAGE
"How-To-Repeat" sequence, step 3 should be: 3. Run shell command "sysctl security.mac.portacl.rules=uid:88:tcp:80"
NULL check would be redundant there as it is already established that the lock is owned and thus must have an owner. What seems to be happening that is that the thread is trying to acquire a lock that has been corrupted somehow. E.g. never initialized or its memory location overwritten. Better diagnostics for that case when INVARIANTS are enabled is warranted though. Returning to the main issue, I wonder if the following message are related to the problem: module_register: module mac_portacl already exists! Module mac_portacl failed to register: 17 I seem to recall that in some cases our kernel module loading infrastructure may refuse to load a duplicate module, bu nevertheless re-resolve a symbol name to point to a symbol in the duplicate module rather than in the original. Which is a bug, of course. -- Andriy Gapon
On 02.02.2014 19:55, Andriy Gapon wrote: > > NULL check would be redundant there as it is already established that the lock > is owned and thus must have an owner. > What seems to be happening that is that the thread is trying to acquire a lock > that has been corrupted somehow. E.g. never initialized or its memory location > overwritten. > > Better diagnostics for that case when INVARIANTS are enabled is warranted though. > > Returning to the main issue, I wonder if the following message are related to > the problem: > > module_register: module mac_portacl already exists! > Module mac_portacl failed to register: 17 > > I seem to recall that in some cases our kernel module loading infrastructure may > refuse to load a duplicate module, bu nevertheless re-resolve a symbol name to > point to a symbol in the duplicate module rather than in the original. Which is > a bug, of course. I've disabled loading of mac_portacl in loader.conf keeping options MAC_PORTACL in my kernel and panic is no more. Do you still need additional INVARIANTS-enabled debug info? Eugene Grosbein
*** This bug has been marked as a duplicate of bug 183817 ***