Bug 213903
| Summary: | Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837) | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Ben Woods <woodsb02> |
| Component: | kern | Assignee: | Mateusz Guzik <mjg> |
| Status: | Closed FIXED | ||
| Severity: | Affects Some People | CC: | bugs, chrysalis, franco, jhb, kostikbel, marklmi26-fbsd, mjg, mjguzik, muxx.dev, peixoto.cassiano, shawn.webb, stl |
| Priority: | --- | Keywords: | crash, needs-qa, patch |
| Version: | CURRENT | Flags: | koobs:
mfc-stable11+
koobs: mfc-stable10+ |
| Hardware: | Any | ||
| OS: | Any | ||
|
Description
Ben Woods
2016-10-30 03:49:30 UTC
We at OPNsense have a user that might be affected by this as well. Here's the bug report on OPNsense's side: https://forum.opnsense.org/index.php?topic=4315.0 Any updates on this issue? I still experience the issue about 2-3 times a month. Not sure how to investigate further... We have over a dozen user reports on this collected in two weeks, some with daily crashes. Trying to bring in a developer now... Please reproduce with: https://people.freebsd.org/~mjg/patches/rwlock-debug.diff the patch is against 11.0 Thank you. A public call for testing will be out today based on your diff. :) Cheers, Franco In kgdb, can you go up to the frame that faulted (13) and print out 'ts' and 'queue'? So far we haven't heard back from users regarding the debug info. But we have a suspect: https://lists.freebsd.org/pipermail/freebsd-current/2015-November/058122.html Some CPU info of previous user reports: CPU: Intel(R) Atom(TM) CPU C2758 @ 2.40GHz (2400.06-MHz K8-class CPU) Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch> Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 16515608576 (15750 MB) CPU: Intel(R) Celeron(R) CPU N2930 @ 1.83GHz (1833.38-MHz K8-class CPU) Origin="GenuineIntel" Id=0x30678 Family=0x6 Model=0x37 Stepping=8 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch> Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 8589934592 (8192 MB) avail memory = 8072794112 (7698 MB) CPU: Intel(R) Celeron(R) CPU J1900 @ 1.99GHz (2000.05-MHz K8-class CPU) Origin="GenuineIntel" Id=0x30678 Family=0x6 Model=0x37 Stepping=8 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch> Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 4294967296 (4096 MB) avail memory = 3961688064 (3778 MB) (In reply to John Baldwin from comment #6) > In kgdb, can you go up to the frame that faulted (13) and print out 'ts' and 'queue'? Apologies for the delay in replying, I had deleted this core, and moved poudriere to another box to avoid these crashes. I had to reinstall poudriere to recreate the high load on the server in order to reproduce this kernel crash. I have printed the values you requested below. Please let me know if you would like any more info. I'll keep the core handy this time :) # kgdb /usr/lib/debug/boot/kernel.GENERIC-NODEBUG-VIMAGE/kernel.debug /var/crash/vmcore.last GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 04 fault virtual address = 0x30 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80b2db5c stack pointer = 0x28:0xfffffe04687ad8c0 frame pointer = 0x28:0xfffffe04687ad8e0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 98303 (sh) Reading symbols from /usr/lib/debug/boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done. Loaded symbols for /usr/lib/debug/boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/cpuctl.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/cpuctl.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/cpuctl.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linprocfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linprocfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linprocfs.ko #0 doadump (textdump=0) at pcpu.h:222 222 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump (textdump=0) at pcpu.h:222 #1 0xffffffff803a571c in db_fncall (dummy1=<value optimized out>, dummy2=<value optimized out>, dummy3=<value optimized out>, dummy4=<value optimized out>) at /usr/src/sys/ddb/db_command.c:581 #2 0xffffffff803a529f in db_command (cmd_table=<value optimized out>) at /usr/src/sys/ddb/db_command.c:453 #3 0xffffffff803a5014 in db_command_loop () at /usr/src/sys/ddb/db_command.c:506 #4 0xffffffff803a806f in db_trap (type=<value optimized out>, code=<value optimized out>) at /usr/src/sys/ddb/db_main.c:248 #5 0xffffffff80b168b3 in kdb_trap (type=<value optimized out>, code=<value optimized out>, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654 #6 0xffffffff80fbb322 in trap_fatal (frame=0xfffffe04687ad800, eva=48) at /usr/src/sys/amd64/amd64/trap.c:796 #7 0xffffffff80fbb52c in trap_pfault (frame=0xfffffe04687ad800, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:658 #8 0xffffffff80fbabe0 in trap (frame=0xfffffe04687ad800) at /usr/src/sys/amd64/amd64/trap.c:421 #9 0xffffffff80f9d9f1 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 #10 0xffffffff80b2db5c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:840 #11 0xffffffff80ac693f in __rw_wunlock_hard (c=0xfffff803574082e8, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1051 #12 0xffffffff80e317ec in vm_map_delete (map=<value optimized out>, start=<value optimized out>, end=<value optimized out>) at /usr/src/sys/vm/vm_map.c:2956 #13 0xffffffff80e2f40e in vmspace_exit (td=<value optimized out>) at /usr/src/sys/vm/vm_map.c:3073 #14 0xffffffff80a7bd19 in exit1 (td=<value optimized out>, rval=<value optimized out>, signo=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:399 #15 0xffffffff80a7b4cd in sys_sys_exit (td=<value optimized out>, uap=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:178 #16 0xffffffff80fbbcee in amd64_syscall (td=0xfffff801ec058a00, traced=0) at subr_syscall.c:135 #17 0xffffffff80f9dcdb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #18 0x0000000800b661fa in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) up 10 #10 0xffffffff80b2db5c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:840 840 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); (kgdb) print ts $1 = (struct turnstile *) 0x0 (kgdb) print queue $2 = 1 (kgdb) % (In reply to Franco Fichtner from comment #7) Indeed, my CPU is one of these. I am experiencing this on a FreeNAS mini bought from IxSystems. Some details from my dmesg: CPU: Intel(R) Atom(TM) CPU C2750 @ 2.40GHz (2400.14-MHz K8-class CPU) Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch> Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 16534667264 (15768 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <INTEL TIANO > WARNING: L1 data cache covers less APIC IDs than a core 0 < 1 FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 8 core(s) random: unblocking device. WARNING: VIMAGE (virtualized network stack) is a highly experimental feature. ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20170119/tbfadt-742) ioapic0 <Version 2.0> irqs 0-23 on motherboard SMP: AP CPU #6 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! turnstile of NULL value and the queue of 1 are consistent with the previous core showing the lock value being that of the curthread. Can you please reproduce with this https://people.freebsd.org/~mjg/patches/rwlock-debug-head.diff Guys, I'm having the same issue here on FreeBSD 10.3-STABLE. I'm using Atom C2758 as well. It has began after 10.3 update. It's very serious issue because many production servers are crashing. Can someone take a look please? Thanks. r301157 was backported to 10-STABLE, but 10.3 is unaffected. There is no 10.3-STABLE. Which one did you mean? From our experience r301157 is the bad commit as the panics have disappeared in our latest OPNsense version which reverted the rwlock bits of this particular patch. (In reply to Franco Fichtner from comment #12) Hi Franco, I don't know exactly which svn version i'm using, because when i run uname -a it doesn't show me. But anyway i updated my FreeBSD 10.3 on February 6th. Is it makes sense to you? How can i revert this commit? Thanks. Hi Cassiano, What's your output of uname -v? Can you make sure to include a backtrace here from ddb? type "bt" at the prompt when the panic happens. It may be related but not the same code path. Cheers, Franco (In reply to Franco Fichtner from comment #14) Hi Franco, Here it is: FreeBSD 10.3-STABLE #4: Mon Feb 6 09:29:52 BRST 2017 root@bgp.server.us:/usr/obj/usr/src/sys/GENERIC My debug bellow: # kgdb kernel.debug /var/crash/vmcore.last GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 7; apic id = 0e fault virtual address = 0x30 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80b2b4fa stack pointer = 0x28:0xfffffe0237a4f450 frame pointer = 0x28:0xfffffe0237a4f480 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 79530 (sh) trap number = 12 panic: page fault cpuid = 7 KDB: stack backtrace: #0 0xffffffff80b16230 at kdb_backtrace+0x60 #1 0xffffffff80ad7036 at vpanic+0x126 #2 0xffffffff80ad6f03 at panic+0x43 #3 0xffffffff80f810cd at trap_fatal+0x35d #4 0xffffffff80f813e8 at trap_pfault+0x308 #5 0xffffffff80f80a2a at trap+0x47a #6 0xffffffff80f661dc at calltrap+0x8 #7 0xffffffff80ad4d80 at __rw_wunlock_hard+0x90 #8 0xffffffff80dffd9a at vm_map_delete+0x33a #9 0xffffffff80e01b47 at vm_map_remove+0x47 #10 0xffffffff80a96759 at exec_new_vmspace+0x1e9 #11 0xffffffff80a73284 at exec_elf64_imgact+0xa44 #12 0xffffffff80a94ec4 at kern_execve+0x7d4 #13 0xffffffff80a9438c at sys_execve+0x4c #14 0xffffffff80f81b00 at amd64_syscall+0x450 #15 0xffffffff80f664cb at Xfast_syscall+0xfb Uptime: 19h0m34s Dumping 1063 out of 8149 MB: (CTRL-C to abort) ..2%..11%..22%..31%..41%..52%..61%..71%..82%..91% Reading symbols from /boot/kernel.off/coretemp.ko.symbols...done. Loaded symbols for /boot/kernel.off/coretemp.ko.symbols Reading symbols from /boot/modules/plcm.ko...done. Loaded symbols for /boot/modules/plcm.ko #0 doadump (textdump=<value optimized out>) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) list *0xffffffff80b2b4fa 0xffffffff80b2b4fa is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:838). 833 834 /* 835 * Transfer the blocked list to the pending list. 836 */ 837 mtx_lock_spin(&td_contested_lock); 838 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); 839 mtx_unlock_spin(&td_contested_lock); 840 841 /* 842 * Give a turnstile to each thread. The last thread gets Current language: auto; currently minimal (kgdb) bt #0 doadump (textdump=<value optimized out>) at pcpu.h:219 #1 0xffffffff80ad6c53 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486 #2 0xffffffff80ad7075 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:889 #3 0xffffffff80ad6f03 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:818 #4 0xffffffff80f810cd in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:858 #5 0xffffffff80f813e8 in trap_pfault (frame=0xfffffe0237a4f3a0, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:681 #6 0xffffffff80f80a2a in trap (frame=0xfffffe0237a4f3a0) at /usr/src/sys/amd64/amd64/trap.c:447 #7 0xffffffff80f661dc in calltrap () at /usr/src/sys/amd64/amd64/exception.S:238 #8 0xffffffff80b2b4fa in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:838 #9 0xffffffff80ad4d80 in __rw_wunlock_hard (c=0xfffff8013a3b5318, tid=1, file=0xfffff80009947001 "8?\201????", line=1) at /usr/src/sys/kern/kern_rwlock.c:1027 #10 0xffffffff80dffd9a in vm_map_delete (map=0xfffff8000c22b8c0, start=<value optimized out>, end=140737488355328) at /usr/src/sys/vm/vm_map.c:2911 #11 0xffffffff80e01b47 in vm_map_remove (map=0xfffff8000c22b8c0, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3028 #12 0xffffffff80a96759 in exec_new_vmspace (imgp=0xfffffe0237a4f868, sv=0xffffffff819858e8) at /usr/src/sys/kern/kern_exec.c:1084 #13 0xffffffff80a73284 in exec_elf64_imgact (imgp=0xfffffe0237a4f868) at /usr/src/sys/kern/imgact_elf.c:881 #14 0xffffffff80a94ec4 in kern_execve (td=0xfffff80009947000, args=0xfffffe0237a4fa78, mac_p=<value optimized out>) at /usr/src/sys/kern/kern_exec.c:606 #15 0xffffffff80a9438c in sys_execve (td=0xfffff80009947000, uap=<value optimized out>) at /usr/src/sys/kern/kern_exec.c:222 #16 0xffffffff80f81b00 in amd64_syscall (td=0xfffff80009947000, traced=0) at subr_syscall.c:141 #17 0xffffffff80f664cb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:398 #18 0x0000000800d7a97a in ?? () Previous frame inner to this frame (corrupt stack?) Thanks for your help. Can you please reproduce with https://people.freebsd.org/~mjg/patches/rwlock-debug-10.diff appliled on top. E.g. like this: cd /usr/src fetch https://people.freebsd.org/~mjg/patches/rwlock-debug-10.diff patch -p1 < rwlock-debug-10.diff (In reply to Mateusz Guzik from comment #16) Hi Mateusz, Sorry but i can't try this patch, i had to rollback the old kernel to avoid crashes. It's a production server and i can't let it down. :( (In reply to Cassiano Peixoto from comment #17) Hi Mateusz, I got another customer to try. Can i apply this patch and only replace the kernel or do i need to replace userland as well? After that do you need only kgdb trace output? Thanks. (In reply to Cassiano Peixoto from comment #15) Tossing in that I've been seeing this panic on stock kernel 11.0-p8 machines running on Atom C2750s. The failure shown by kgdb output looks identical to what Cassiano posted for 10.3. In my case it doesn't require heavy traffic however, most often it's happening at periods of off-peak load. Hi guys, After i applied Mateusz's patch it stopped crashing (2 days with no crash). So it seems worked. Mateusz are you planning to commit it? Thank you for your help. This was only supposed to print further debug output, but something in here is probably responsible for fixing it as it touches the same lines as r301157 that's been known to cause the panic in the first place...
- else if ((rw)->rw_lock != _tid || !_rw_write_unlock((rw), _tid))\
- _rw_wunlock_hard((rw), _tid, (file), (line)); \
+ else { \
+ _v = (rw)->rw_lock; \
+ if (_v != _tid || !_rw_write_unlock_fetch((rw), &_v)) \
+ _rw_wunlock_hard((rw), _v, _tid, (file), (line));\
+ }
(In reply to Franco Fichtner from comment #21) Hi Franco, i agree with you. Now it's 6 days with no crashing anymore. Mateusz, can you take a look please? I have also been running a week with this patch, with no more crashes. It appears to have solved it - thank you! So guys, any committer available to commit this fix? It's really important. Thanks. I'm working on an update on this, likely will be ready tomorrow. So, can I please get confirmation that the problem is still present in current and still present in stable/11 past r315395? There were several changes which could have make the apparent cpu problem stop manifesting itself. If so, I'll simply merge them to stable/10 as well. (In reply to Mateusz Guzik from comment #26) Hi Mateuz, i can confirm only on stable/10. Maybe someone else can confirm it to you. After the patch is running 13 days without crash. Thanks. (In reply to Cassiano Peixoto from comment #27) Hi Mateusz, any news about this issue? (In reply to Mateusz Guzik from comment #26) I have been running with the patch with no crashes. If you would like me to confirm whether the issue is still in CURRENT, I will have to upgrade to the latest CURRENT without the patch, and let you know if I get crashes again. Let me know if this would be helpful for you. Hi Mateusz, Any update about this issue? Thanks. Hi Mateusz,
After some time i had a server with FreeBSD 11-p1 crashed with your patch applied.
See bellow:
(kgdb) list *0xffffffff80b317ac
0xffffffff80b317ac is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837).
832
833 /*
834 * Transfer the blocked list to the pending list.
835 */
836 mtx_lock_spin(&td_contested_lock);
837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq);
838 mtx_unlock_spin(&td_contested_lock);
839
840 /*
841 * Give a turnstile to each thread. The last thread gets
Current language: auto; currently minimal
(kgdb) bt
#0 doadump (textdump=<value optimized out>) at pcpu.h:221
#1 0xffffffff80acfd69 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366
#2 0xffffffff80ad031b in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3 0xffffffff80ad0153 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:690
#4 0xffffffff80ee2d21 in trap_fatal (frame=0xfffffe02378e0850, eva=32) at /usr/src/sys/amd64/amd64/trap.c:841
#5 0xffffffff80ee2f13 in trap_pfault (frame=0xfffffe02378e0850, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:691
#6 0xffffffff80ee24bc in trap (frame=0xfffffe02378e0850) at /usr/src/sys/amd64/amd64/trap.c:442
#7 0xffffffff80ec5951 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
#8 0xffffffff80b317ac in turnstile_broadcast (ts=0x0, queue=0) at /usr/src/sys/kern/subr_turnstile.c:837
#9 0xffffffff80aabb70 in __mtx_unlock_sleep (c=0xffffffff81cab130, opts=<value optimized out>, file=0x1 <Address 0x1 out of bounds>, line=1) at /usr/src/sys/kern/kern_mutex.c:865
#10 0xffffffff80d42dd1 in swap_reserve_by_cred (incr=<value optimized out>, cred=<value optimized out>) at /usr/src/sys/vm/swap_pager.c:224
#11 0xffffffff80d58f49 in kmap_alloc_wait (map=0xfffff800061d7c30, size=266240) at /usr/src/sys/vm/vm_kern.c:437
#12 0xffffffff80a7cf5c in exec_copyin_args (args=0xfffffe02378e0a70, fname=0x801436b58 <Address 0x801436b58 out of bounds>, segflg=UIO_USERSPACE, argv=0x801436a98, envv=0x801436b20)
at /usr/src/sys/kern/kern_exec.c:1326
#13 0xffffffff80a7cdb8 in sys_execve (td=0xfffff8013d03ba00, uap=0xfffffe02378e0b80) at /usr/src/sys/kern/kern_exec.c:215
#14 0xffffffff80ee367e in amd64_syscall (td=<value optimized out>, traced=0) at subr_syscall.c:135
#15 0xffffffff80ec5c3b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
#16 0x0000000800b45daa in ?? ()
I am getting turnstile_broadcast() at turnstile_broadcast panic's on a PFSense unit which is based on FreeBSD 11.0-p8 The hardware is a intel celeron N3150. Is it possible to provide a patch which I can pass onto the PFSense developers to see if the proposed fix works on that? Thanks I have applied your patch now Mateusz on 11-p10 so will report back if it works. (In reply to Mateusz Guzik from comment #26) Dear Mateusz, Please, Can you confirm if it has been fixed in last STABLE? I've seen some commits but i'm no sure. Thanks. Hi there, sorry for late reply. This somehow fell through the cracks. First of all there is no kernel bug per se that I can see, rather a bug in the atom cpu which started manifesting itself. There were several changes to the affected code path on stable/11 and in particular the condition which triggered the bug is gone. I don't have any reports, but I suspect stable/11 is now cleared. stable/10 will require a hack, but I'll need someone to test it as don't have the hardware to reproduce. You can use https://github.com/opnsense/src/commit/6b79b52c.patch on stable/10, it was verified working on 11.0. Cheers, Franco (In reply to Mateusz Guzik from comment #35) Mateusz, Yes i realized many changes has been made on 11-STABLE related to this issue. I think it could be fixed as well. Anyway i have a server running with 11.1-BETA3 to confirm. Regarding FreeBSD 10, i can test it for you, just provide a patch and let me know. Thanks. (In reply to Mateusz Guzik from comment #35) Just to let you know my pfsense unit affected by this issue does not have an atom cpu. It has a celeron N3150 cpu. Can someone who runs into the problem on stable/10 test the following please: https://people.freebsd.org/~mjg/patches/rwlock-unlock.diff (In reply to Mateusz Guzik from comment #39) Mateusz, I can test it, but my 10-STABLE version has a different code: /* Release a write lock. */ #define __rw_wunlock(rw, tid, file, line) do { \ uintptr_t _tid = (uintptr_t)(tid); \ uintptr_t _v; \ \ if ((rw)->rw_recurse) \ (rw)->rw_recurse--; \ else { \ _v = (rw)->rw_lock; \ if (_v != _tid || !_rw_write_unlock_fetch((rw), &_v)) \ _rw_wunlock_hard((rw), _v, _tid, (file), (line));\ } \ } while (0) How shoukd i adapt your patch in this code? Thanks. This looks like leftovers from the previous patch. Just restore the original state of the tree and patch that. (In reply to Mateusz Guzik from comment #41) Ok you are right. I reverted all files touched by previous patch. Am i right? And then should i apply only this last patch? Yes, apply the patch to a clean tree. Thanks. (In reply to Mateusz Guzik from comment #43) Ok patch applied. I'll let you know the results. Thanks. Can we get the proposed patches included as attachments in this issue please, ideally, and if multiple versions are created over time, obsoleting the old ones. Also is this a candidate for upcoming releases? If so, please cc re@ I would hope this makes it into RELEASE 11.1, would be disappointed if that was released with a known problem. PFSense seem to have absolutely no motivation to apply the patches supplied here, so I am hoping they use 11.1 after its released and that has the patches applied. (In reply to Mateusz Guzik from comment #43) Dear Mateusz, I'm quite sure this patch has fixed the issue on 10-STABLE. My server is running 6 days without reboot. Please go ahead and commit it. Thank you for your support. Thanks for testing. I'll commit the patch later today with few other modifications. That said, I expect the problem to be gone on head starting with r313270 and on stable/11 starting with r315377, which also means it is included in all 11.1 builds. That said, if someone running into the crash on one of the above could confirm the newest version of their relevant branch makes the problem go away, that would be nice. I am running FreeBSD 12-CURRENT r319025 without any patches and can confirm the crashes do not occur for me anymore on my FreeNAS mini. Thanks for your help with this! A commit references this bug: Author: mjg Date: Tue Jul 11 03:45:48 UTC 2017 New revision: 320885 URL: https://svnweb.freebsd.org/changeset/base/320885 Log: Remove waiters check from the inline rw wunlock routine. This is a direct commit to stable/10. r310979 is a merge of depessimisation of locking primitives. The important part was getting rid of an attempt to grab the lock in the slow path immediately after the fast path failed. In addition to that temporary checks were added before all atomic ops. They have no impact on semantic nor correctness, they only avoid an atomic operation which is likely to fail. After the addition of atomic_fcmpset and further changes said checks became pessimal and got removed. This may get merged to stable/10. Reports started showing up about a crash in all branches having extra checks. The codepath is: .. -> vm_map_delete -> __rw_wunlock_hard -> turnstile_broadcast The kernel crashed trying to wake up nonexistent waiters. The lock value as found in the vmcore matches the panicking thread, so in particular there was no waiters bit set. The bit can only be cleared by the current owner. A debug patch was provided, which reportedly had a side effect of getting rid of the issue. Also one of the reporters said that reverting the patch which adds the extra checks makes the crash go away. It was also reported that head with the fcmpset changes (explicit checks removed) also stops crashing. Finally, one user tested crashing stable/10 variant with just the rw wunlock check removed. The common case in all but one reports was an Intel Atom cpu. Claiming a cpu bug at this point is bold and I'm going to refrain from it, but right now apart from cpu-specific optimisation made by the compiler on custom kernel compiles I don't see how this can be a software bug. This will have to be investigated more. Meanwhile, restore rw wunlock to pre-r310979 state. PR: 213903 Changes: stable/10/sys/sys/rwlock.h The change was committed to stable/10. To the best of my knowledge all branches with the original patch (head, stable/11, 11.1 and stable/10) have the issue resolved. Thanks everyone for reporting and testing. Correctly (+, not -) document merge completion Both patches in this PR have fixed my kernel panic issues on 11.0 I tested the debug patch which didnt aim to fix but did as a side affect, and I tested the later patch. I can confirm the same crash on FreeBSD 11.0-RELEASE-p1 (GENERIC) on the following hardware: Aug 12 11:57:04 gw kernel: CPU: Intel(R) Celeron(R) CPU J1900 @ 1.99GHz (2000.05-MHz K8-class CPU) Aug 12 11:57:04 gw kernel: Origin="GenuineIntel" Id=0x30678 Family=0x6 Model=0x37 Stepping=8 Aug 12 11:57:04 gw kernel: Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Aug 12 11:57:04 gw kernel: Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND> Aug 12 11:57:04 gw kernel: AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> Aug 12 11:57:04 gw kernel: AMD Features2=0x101<LAHF,Prefetch> Aug 12 11:57:04 gw kernel: Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG> Aug 12 11:57:04 gw kernel: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID Aug 12 11:57:04 gw kernel: TSC: P-state invariant, performance statistics Aug 12 11:57:04 gw kernel: real memory = 8589934592 (8192 MB) Aug 12 11:57:04 gw kernel: avail memory = 8137785344 (7760 MB) Aug 12 11:57:04 gw kernel: Event timer "LAPIC" quality 600 Aug 12 11:57:04 gw kernel: ACPI APIC Table: <ALASKA A M I > Aug 12 11:57:04 gw kernel: WARNING: L1 data cache covers less APIC IDs than a core Aug 12 11:57:04 gw kernel: 0 < 1 Aug 12 11:57:04 gw kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs Aug 12 11:57:04 gw kernel: FreeBSD/SMP: 1 package(s) x 4 core(s) Aug 12 11:57:04 gw kernel: random: unblocking device. Aug 12 11:57:04 gw kernel: ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650) Aug 12 11:57:04 gw kernel: WARNING: Bogus Interrupt Polarity. Assume CONFORMS more information from /var/log/messages and kgdb: Aug 12 11:57:04 gw kernel: Fatal trap 12: page fault while in kernel mode Aug 12 11:57:04 gw kernel: cpuid = 1; apic id = 02 Aug 12 11:57:04 gw kernel: fault virtual address = 0x30 Aug 12 11:57:04 gw kernel: fault code = supervisor read data, page not present Aug 12 11:57:04 gw kernel: instruction pointer = 0x20:0xffffffff80b3a89c Aug 12 11:57:04 gw kernel: stack pointer = 0x28:0xfffffe0232609440 Aug 12 11:57:04 gw kernel: frame pointer = 0x28:0xfffffe0232609470 Aug 12 11:57:04 gw kernel: code segment = base rx0, limit 0xfffff, type 0x1b Aug 12 11:57:04 gw kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 Aug 12 11:57:04 gw kernel: processor eflags = resume, IOPL = 0 Aug 12 11:57:04 gw kernel: current process = 18204 (telegraf) Aug 12 11:57:04 gw kernel: trap number = 12 Aug 12 11:57:04 gw kernel: panic: page fault Aug 12 11:57:04 gw kernel: cpuid = 1 Aug 12 11:57:04 gw kernel: KDB: stack backtrace: Aug 12 11:57:04 gw kernel: #0 0xffffffff80b24077 at kdb_backtrace+0x67 Aug 12 11:57:04 gw kernel: #1 0xffffffff80ad93e2 at vpanic+0x182 Aug 12 11:57:04 gw kernel: #2 0xffffffff80ad9253 at panic+0x43 Aug 12 11:57:04 gw kernel: #3 0xffffffff80fa0d31 at trap_fatal+0x351 Aug 12 11:57:04 gw kernel: #4 0xffffffff80fa0f23 at trap_pfault+0x1e3 Aug 12 11:57:04 gw kernel: #5 0xffffffff80fa04cc at trap+0x26c Aug 12 11:57:04 gw kernel: #6 0xffffffff80f84141 at calltrap+0x8 Aug 12 11:57:04 gw kernel: #7 0xffffffff80ad48cf at __rw_wunlock_hard+0x8f Aug 12 11:57:04 gw kernel: #8 0xffffffff80e1a75c at vm_map_delete+0x3dc Aug 12 11:57:04 gw kernel: #9 0xffffffff80e1c5f7 at vm_map_remove+0x47 Aug 12 11:57:04 gw kernel: #10 0xffffffff80a86c7f at exec_new_vmspace+0x22f Aug 12 11:57:04 gw kernel: #11 0xffffffff80a5bfe8 at exec_elf64_imgact+0xa58 Aug 12 11:57:04 gw kernel: #12 0xffffffff80a84d4d at kern_execve+0x7dd Aug 12 11:57:04 gw kernel: #13 0xffffffff80a841dc at sys_execve+0x4c Aug 12 11:57:04 gw kernel: #14 0xffffffff80fa168e at amd64_syscall+0x4ce Aug 12 11:57:04 gw kernel: #15 0xffffffff80f8442b at Xfast_syscall+0xfb ... (kgdb) list *0xffffffff80b3a89c 0xffffffff80b3a89c is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837). 832 833 /* 834 * Transfer the blocked list to the pending list. 835 */ 836 mtx_lock_spin(&td_contested_lock); 837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); 838 mtx_unlock_spin(&td_contested_lock); 839 840 /* 841 * Give a turnstile to each thread. The last thread gets Current language: auto; currently minimal (kgdb) backtrace #0 doadump (textdump=<value optimized out>) at pcpu.h:221 #1 0xffffffff80ad8e69 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366 #2 0xffffffff80ad941b in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80ad9253 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:690 #4 0xffffffff80fa0d31 in trap_fatal (frame=0xfffffe0232609390, eva=48) at /usr/src/sys/amd64/amd64/trap.c:841 #5 0xffffffff80fa0f23 in trap_pfault (frame=0xfffffe0232609390, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:691 #6 0xffffffff80fa04cc in trap (frame=0xfffffe0232609390) at /usr/src/sys/amd64/amd64/trap.c:442 #7 0xffffffff80f84141 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 #8 0xffffffff80b3a89c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837 #9 0xffffffff80ad48cf in __rw_wunlock_hard (c=0xfffff800437de858, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1027 #10 0xffffffff80e1a75c in vm_map_delete (map=<value optimized out>, start=<value optimized out>, end=<value optimized out>) at /usr/src/sys/vm/vm_map.c:2960 #11 0xffffffff80e1c5f7 in vm_map_remove (map=0xfffff80032b91000, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077 #12 0xffffffff80a86c7f in exec_new_vmspace (imgp=0xfffffe0232609860, sv=0xffffffff81a02720) at /usr/src/sys/kern/kern_exec.c:1095 #13 0xffffffff80a5bfe8 in exec_elf64_imgact (imgp=<value optimized out>) at /usr/src/sys/kern/imgact_elf.c:896 #14 0xffffffff80a84d4d in kern_execve (td=<value optimized out>, args=<value optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:602 #15 0xffffffff80a841dc in sys_execve (td=0xfffff801a5da3a00, uap=0xfffffe0232609b80) at /usr/src/sys/kern/kern_exec.c:218 #16 0xffffffff80fa168e in amd64_syscall (td=<value optimized out>, traced=0) at subr_syscall.c:135 #17 0xffffffff80f8442b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #18 0x000000000047da1f in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) (In reply to muxx.dev from comment #54)\ Looks to me like ts==0x0 was the starting value given to turnstile_broadcast in each backtrace listed in this buzilla bug (213903), for example: (kgdb) list *0xffffffff80b3a89c 0xffffffff80b3a89c is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837). 832 833 /* 834 * Transfer the blocked list to the pending list. 835 */ 836 mtx_lock_spin(&td_contested_lock); 837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); 838 mtx_unlock_spin(&td_contested_lock); 839 840 /* 841 * Give a turnstile to each thread. The last thread gets Current language: auto; currently minimal (kgdb) backtrace . . . #8 0xffffffff80b3a89c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837 #9 0xffffffff80ad48cf in __rw_wunlock_hard (c=0xfffff800437de858, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1027 . . . Note that ts==0x0 would be a problem for: TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); So I wonder if __rw_wunlock_hard is providing a bad ts value. If yes: the problem then occurs before the turnstile_broadcast call is made. batch change of PRs untouched in 2018 marked "in progress" back to open. Please do not put bugs on stable@, current@, hackers@, etc |