Bug 213903 - Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)
Summary: Kernel crashes from turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837)
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: Mateusz Guzik
URL:
Keywords: crash, needs-qa, patch
Depends on:
Blocks:
 
Reported: 2016-10-30 03:49 UTC by Ben Woods
Modified: 2023-04-07 21:13 UTC (History)
12 users (show)

See Also:
koobs: mfc-stable11+
koobs: mfc-stable10+

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Ben Woods freebsd_committer freebsd_triage 2016-10-30 03:49:30 UTC 
I am currently experiencing semi-regular kernel crashes on my FreeBSD 12-current machine. I am new to kernel debugging, and hoping someone can have a look at the debugging output below to point me in the direction of what the problem might be.

My machine is a FreeNAS-mini from iXsystems which I have formatted and installed stock FreeBSD onto. My kernel is the default generic-nodebug with the VIMAGE options added.

$ uname -a
FreeBSD freenas.woods.am 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r305311M: Sat Sep  3 12:29:01 AWST 2016     woodsb02@freenas.woods.am:/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE  amd64

$ cat /usr/src/sys/amd64/conf/GENERIC-NODEBUG-VIMAGE
# SPARTICUS -- WITNESS and INVARIANTS free kernel configuration file
#               for FreeBSD/amd64

include GENERIC-NODEBUG

ident   GENERIC-NODEBUG-VIMAGE

#nooptions      SCTP    # Stream Control Transmission Protocol
options         VIMAGE  # VNET/Vimage support


Output from kernel crash dump debug with kgdb below:

/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE)# kgdb kernel.debug /var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80b4d91c
stack pointer           = 0x28:0xfffffe046813a440
frame pointer           = 0x28:0xfffffe046813a470
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 33487 (sh)
Uptime: 15m16s
Dumping 1664 out of 16338 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug...
done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko
#0  doadump (textdump=1) at pcpu.h:221
221             __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) list *0xffffffff80b4d91c
0xffffffff80b4d91c is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837).
832
833             /*
834              * Transfer the blocked list to the pending list.
835              */
836             mtx_lock_spin(&td_contested_lock);
837             TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq);
838             mtx_unlock_spin(&td_contested_lock);
839
840             /*
841              * Give a turnstile to each thread.  The last thread gets
Current language:  auto; currently minimal
(kgdb) backtrace
#0  doadump (textdump=1) at pcpu.h:221
#1  0xffffffff80aea40e in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80aea9db in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80aea813 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff8039e197 in db_panic (addr=<value optimized out>, have_addr=false, count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:486
#5  0xffffffff8039d689 in db_command (cmd_table=<value optimized out>) at /usr/src/sys/ddb/db_command.c:453
#6  0xffffffff8039d3e4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:506
#7  0xffffffff803a053b in db_trap (type=<value optimized out>, code=<value optimized out>) at /usr/src/sys/ddb/db_main.c:251
#8  0xffffffff80b36b33 in kdb_trap (type=<value optimized out>, code=<value optimized out>, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
#9  0xffffffff80fdd441 in trap_fatal (frame=0xfffffe046813a390, eva=48) at /usr/src/sys/amd64/amd64/trap.c:836
#10 0xffffffff80fdd673 in trap_pfault (frame=0xfffffe046813a390, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:691
#11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at /usr/src/sys/amd64/amd64/trap.c:442
#12 0xffffffff80fbf491 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
#13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>)
    at /usr/src/sys/kern/kern_rwlock.c:1027
#15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>, start=<value optimized out>, end=<value optimized out>) at /usr/src/sys/vm/vm_map.c:2960
#16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077
#17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860, sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096
#18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at /usr/src/sys/kern/imgact_elf.c:896
#19 0xffffffff80a9670d in kern_execve (td=<value optimized out>, args=<value optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603
#20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00, uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219
#21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>, traced=0) at subr_syscall.c:135
#22 0xffffffff80fbf77b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
#23 0x0000000800b468ea in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) up 11
#11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at /usr/src/sys/amd64/amd64/trap.c:442
442                             (void) trap_pfault(frame, FALSE);
(kgdb) list
437
438                     KASSERT(cold || td->td_ucred != NULL,
439                         ("kernel trap doesn't have ucred"));
440                     switch (type) {
441                     case T_PAGEFLT:                 /* page fault */
442                             (void) trap_pfault(frame, FALSE);
443                             goto out;
444
445                     case T_DNA:
446                             if (PCB_USER_FPU(td->td_pcb))
(kgdb) print td
$1 = (struct thread *) 0xfffff8032893aa00
(kgdb) print td->td_ucred
$2 = (struct ucred *) 0xfffff8004005ec00
(kgdb) print type
$3 = 12
(kgdb) up
#12 0xffffffff80fbf491 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
236             call    trap_check
Current language:  auto; currently asm
(kgdb) up
#13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837
837             TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq);
Current language:  auto; currently minimal
(kgdb) up
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>)
    at /usr/src/sys/kern/kern_rwlock.c:1027
1027            turnstile_broadcast(ts, queue);
(kgdb) up
#15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>, start=<value optimized out>, end=<value optimized out>) at /usr/src/sys/vm/vm_map.c:2960
2960                    VM_OBJECT_WUNLOCK(object);
(kgdb) up
#16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077
3077            result = vm_map_delete(map, start, end);
(kgdb) up
#17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860, sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096
1096                    vm_map_remove(map, vm_map_min(map), vm_map_max(map));
(kgdb) up
#18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at /usr/src/sys/kern/imgact_elf.c:896
896             error = exec_new_vmspace(imgp, sv);
(kgdb) up
#19 0xffffffff80a9670d in kern_execve (td=<value optimized out>, args=<value optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603
603                     error = (*execsw[i]->ex_imgact)(imgp);
(kgdb) up
#20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00, uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219
219                     error = kern_execve(td, &args, NULL);
(kgdb) up
#21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>, traced=0) at subr_syscall.c:135
135                     error = (sa->callp->sy_call)(td, sa->args);
(kgdb) up
#22 0xffffffff80fbf77b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
396             call    amd64_syscall
Current language:  auto; currently asm
(kgdb) up
#23 0x0000000800b468ea in ?? ()
(kgdb) up
Initial frame selected; you cannot go up.
(kgdb) quit

After being requested for more info from Mateusz Guzik <mjguzik@gmail.com>, I also performed the following kgdb command:
(kgdb) f 14
#14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>)
    at /usr/src/sys/kern/kern_rwlock.c:1027
1027            turnstile_broadcast(ts, queue);
Current language:  auto; currently minimal
(kgdb) x/xg c
0xfffff803f886d960:     0xfffff8032893aa00
Comment 1 Shawn Webb 2017-02-02 15:11:42 UTC 
We at OPNsense have a user that might be affected by this as well. Here's the bug report on OPNsense's side: https://forum.opnsense.org/index.php?topic=4315.0

Any updates on this issue?
Comment 2 Ben Woods freebsd_committer freebsd_triage 2017-02-02 15:36:51 UTC 
I still experience the issue about 2-3 times a month. Not sure how to investigate further...
Comment 3 Franco Fichtner 2017-02-16 16:51:58 UTC 
We have over a dozen user reports on this collected in two weeks, some with daily crashes.  Trying to bring in a developer now...
Comment 4 Mateusz Guzik freebsd_committer freebsd_triage 2017-02-17 03:20:46 UTC 
Please reproduce with: https://people.freebsd.org/~mjg/patches/rwlock-debug.diff

the patch is against 11.0
Comment 5 Franco Fichtner 2017-02-17 07:09:27 UTC 
Thank you. A public call for testing will be out today based on your diff. :)


Cheers,
Franco
Comment 6 John Baldwin freebsd_committer freebsd_triage 2017-02-17 19:47:26 UTC 
In kgdb, can you go up to the frame that faulted (13) and print out 'ts' and 'queue'?
Comment 7 Franco Fichtner 2017-02-21 21:55:44 UTC 
So far we haven't heard back from users regarding the debug info. But we have a suspect: https://lists.freebsd.org/pipermail/freebsd-current/2015-November/058122.html

Some CPU info of previous user reports:

CPU: Intel(R) Atom(TM) CPU  C2758  @ 2.40GHz (2400.06-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16515608576 (15750 MB)


CPU: Intel(R) Celeron(R) CPU  N2930  @ 1.83GHz (1833.38-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30678  Family=0x6  Model=0x37  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 8589934592 (8192 MB)
avail memory = 8072794112 (7698 MB)


CPU: Intel(R) Celeron(R) CPU  J1900  @ 1.99GHz (2000.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x30678  Family=0x6  Model=0x37  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3961688064 (3778 MB)
Comment 8 Ben Woods freebsd_committer freebsd_triage 2017-02-25 02:13:36 UTC 
(In reply to John Baldwin from comment #6)
> In kgdb, can you go up to the frame that faulted (13) and print out 'ts' and 'queue'?

Apologies for the delay in replying, I had deleted this core, and moved poudriere to another box to avoid these crashes. I had to reinstall poudriere to recreate the high load on the server in order to reproduce this kernel crash.

I have printed the values you requested below. Please let me know if you would like any more info. I'll keep the core handy this time :)


# kgdb /usr/lib/debug/boot/kernel.GENERIC-NODEBUG-VIMAGE/kernel.debug /var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x30
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80b2db5c
stack pointer           = 0x28:0xfffffe04687ad8c0
frame pointer           = 0x28:0xfffffe04687ad8e0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 98303 (sh)

Reading symbols from /usr/lib/debug/boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done.
Loaded symbols for /usr/lib/debug/boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/cpuctl.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/cpuctl.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/cpuctl.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko
Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linprocfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linprocfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linprocfs.ko
#0  doadump (textdump=0) at pcpu.h:222
222     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump (textdump=0) at pcpu.h:222
#1  0xffffffff803a571c in db_fncall (dummy1=<value optimized out>, dummy2=<value optimized out>, dummy3=<value optimized out>, dummy4=<value optimized out>) at /usr/src/sys/ddb/db_command.c:581
#2  0xffffffff803a529f in db_command (cmd_table=<value optimized out>) at /usr/src/sys/ddb/db_command.c:453
#3  0xffffffff803a5014 in db_command_loop () at /usr/src/sys/ddb/db_command.c:506
#4  0xffffffff803a806f in db_trap (type=<value optimized out>, code=<value optimized out>) at /usr/src/sys/ddb/db_main.c:248
#5  0xffffffff80b168b3 in kdb_trap (type=<value optimized out>, code=<value optimized out>, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
#6  0xffffffff80fbb322 in trap_fatal (frame=0xfffffe04687ad800, eva=48) at /usr/src/sys/amd64/amd64/trap.c:796
#7  0xffffffff80fbb52c in trap_pfault (frame=0xfffffe04687ad800, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:658
#8  0xffffffff80fbabe0 in trap (frame=0xfffffe04687ad800) at /usr/src/sys/amd64/amd64/trap.c:421
#9  0xffffffff80f9d9f1 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
#10 0xffffffff80b2db5c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:840
#11 0xffffffff80ac693f in __rw_wunlock_hard (c=0xfffff803574082e8, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1051
#12 0xffffffff80e317ec in vm_map_delete (map=<value optimized out>, start=<value optimized out>, end=<value optimized out>) at /usr/src/sys/vm/vm_map.c:2956
#13 0xffffffff80e2f40e in vmspace_exit (td=<value optimized out>) at /usr/src/sys/vm/vm_map.c:3073
#14 0xffffffff80a7bd19 in exit1 (td=<value optimized out>, rval=<value optimized out>, signo=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:399
#15 0xffffffff80a7b4cd in sys_sys_exit (td=<value optimized out>, uap=<value optimized out>) at /usr/src/sys/kern/kern_exit.c:178
#16 0xffffffff80fbbcee in amd64_syscall (td=0xfffff801ec058a00, traced=0) at subr_syscall.c:135
#17 0xffffffff80f9dcdb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
#18 0x0000000800b661fa in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb) up 10
#10 0xffffffff80b2db5c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:840
840             TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq);
(kgdb) print ts
$1 = (struct turnstile *) 0x0
(kgdb) print queue
$2 = 1
(kgdb) %
Comment 9 Ben Woods freebsd_committer freebsd_triage 2017-02-25 02:18:01 UTC 
(In reply to Franco Fichtner from comment #7)
Indeed, my CPU is one of these. I am experiencing this on a FreeNAS mini bought from IxSystems.

Some details from my dmesg:
CPU: Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz (2400.14-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16534667264 (15768 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  TIANO   >
WARNING: L1 data cache covers less APIC IDs than a core
0 < 1
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
random: unblocking device.
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20170119/tbfadt-742)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
SMP: AP CPU #6 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Comment 10 Mateusz Guzik freebsd_committer freebsd_triage 2017-02-25 02:45:51 UTC 
turnstile of NULL value and the queue of 1 are consistent with the previous core showing the lock value being that of the curthread.

Can you please reproduce with this https://people.freebsd.org/~mjg/patches/rwlock-debug-head.diff
Comment 11 Cassiano Peixoto 2017-03-13 14:43:11 UTC 
Guys,

I'm having the same issue here on FreeBSD 10.3-STABLE. I'm using Atom C2758 as well. It has began after 10.3 update. It's very serious issue because many production servers are crashing. Can someone take a look please?

Thanks.
Comment 12 Franco Fichtner 2017-03-13 14:51:57 UTC 
r301157 was backported to 10-STABLE, but 10.3 is unaffected.  There is no 10.3-STABLE.  Which one did you mean?

From our experience r301157 is the bad commit as the panics have disappeared in our latest OPNsense version which reverted the rwlock bits of this particular patch.
Comment 13 Cassiano Peixoto 2017-03-13 19:34:46 UTC 
(In reply to Franco Fichtner from comment #12)
Hi Franco,

I don't know exactly which svn version i'm using, because when i run uname -a it doesn't show me. But anyway i updated my FreeBSD 10.3 on February 6th. Is it makes sense to you?  How can i revert this commit?

Thanks.
Comment 14 Franco Fichtner 2017-03-13 19:40:46 UTC 
Hi Cassiano,

What's your output of uname -v?

Can you make sure to include a backtrace here from ddb? type "bt" at the prompt when the panic happens. It may be related but not the same code path.


Cheers,
Franco
Comment 15 Cassiano Peixoto 2017-03-13 20:12:40 UTC 
(In reply to Franco Fichtner from comment #14)
Hi Franco,

Here it is:
FreeBSD 10.3-STABLE #4: Mon Feb  6 09:29:52 BRST 2017     root@bgp.server.us:/usr/obj/usr/src/sys/GENERIC

My debug bellow:

# kgdb kernel.debug /var/crash/vmcore.last 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 7; apic id = 0e
fault virtual address	= 0x30
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80b2b4fa
stack pointer	        = 0x28:0xfffffe0237a4f450
frame pointer	        = 0x28:0xfffffe0237a4f480
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= resume, IOPL = 0
current process		= 79530 (sh)
trap number		= 12
panic: page fault
cpuid = 7
KDB: stack backtrace:
#0 0xffffffff80b16230 at kdb_backtrace+0x60
#1 0xffffffff80ad7036 at vpanic+0x126
#2 0xffffffff80ad6f03 at panic+0x43
#3 0xffffffff80f810cd at trap_fatal+0x35d
#4 0xffffffff80f813e8 at trap_pfault+0x308
#5 0xffffffff80f80a2a at trap+0x47a
#6 0xffffffff80f661dc at calltrap+0x8
#7 0xffffffff80ad4d80 at __rw_wunlock_hard+0x90
#8 0xffffffff80dffd9a at vm_map_delete+0x33a
#9 0xffffffff80e01b47 at vm_map_remove+0x47
#10 0xffffffff80a96759 at exec_new_vmspace+0x1e9
#11 0xffffffff80a73284 at exec_elf64_imgact+0xa44
#12 0xffffffff80a94ec4 at kern_execve+0x7d4
#13 0xffffffff80a9438c at sys_execve+0x4c
#14 0xffffffff80f81b00 at amd64_syscall+0x450
#15 0xffffffff80f664cb at Xfast_syscall+0xfb
Uptime: 19h0m34s
Dumping 1063 out of 8149 MB: (CTRL-C to abort) ..2%..11%..22%..31%..41%..52%..61%..71%..82%..91%

Reading symbols from /boot/kernel.off/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel.off/coretemp.ko.symbols
Reading symbols from /boot/modules/plcm.ko...done.
Loaded symbols for /boot/modules/plcm.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) list *0xffffffff80b2b4fa
0xffffffff80b2b4fa is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:838).
833	
834		/*
835		 * Transfer the blocked list to the pending list.
836		 */
837		mtx_lock_spin(&td_contested_lock);
838		TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq);
839		mtx_unlock_spin(&td_contested_lock);
840	
841		/*
842		 * Give a turnstile to each thread.  The last thread gets
Current language:  auto; currently minimal
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff80ad6c53 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486
#2  0xffffffff80ad7075 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:889
#3  0xffffffff80ad6f03 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:818
#4  0xffffffff80f810cd in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:858
#5  0xffffffff80f813e8 in trap_pfault (frame=0xfffffe0237a4f3a0, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:681
#6  0xffffffff80f80a2a in trap (frame=0xfffffe0237a4f3a0) at /usr/src/sys/amd64/amd64/trap.c:447
#7  0xffffffff80f661dc in calltrap () at /usr/src/sys/amd64/amd64/exception.S:238
#8  0xffffffff80b2b4fa in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:838
#9  0xffffffff80ad4d80 in __rw_wunlock_hard (c=0xfffff8013a3b5318, tid=1, file=0xfffff80009947001 "8?\201????", line=1) at /usr/src/sys/kern/kern_rwlock.c:1027
#10 0xffffffff80dffd9a in vm_map_delete (map=0xfffff8000c22b8c0, start=<value optimized out>, end=140737488355328) at /usr/src/sys/vm/vm_map.c:2911
#11 0xffffffff80e01b47 in vm_map_remove (map=0xfffff8000c22b8c0, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3028
#12 0xffffffff80a96759 in exec_new_vmspace (imgp=0xfffffe0237a4f868, sv=0xffffffff819858e8) at /usr/src/sys/kern/kern_exec.c:1084
#13 0xffffffff80a73284 in exec_elf64_imgact (imgp=0xfffffe0237a4f868) at /usr/src/sys/kern/imgact_elf.c:881
#14 0xffffffff80a94ec4 in kern_execve (td=0xfffff80009947000, args=0xfffffe0237a4fa78, mac_p=<value optimized out>) at /usr/src/sys/kern/kern_exec.c:606
#15 0xffffffff80a9438c in sys_execve (td=0xfffff80009947000, uap=<value optimized out>) at /usr/src/sys/kern/kern_exec.c:222
#16 0xffffffff80f81b00 in amd64_syscall (td=0xfffff80009947000, traced=0) at subr_syscall.c:141
#17 0xffffffff80f664cb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:398
#18 0x0000000800d7a97a in ?? ()
Previous frame inner to this frame (corrupt stack?)

Thanks for your help.
Comment 16 Mateusz Guzik freebsd_committer freebsd_triage 2017-03-14 04:03:41 UTC 
Can you please reproduce with https://people.freebsd.org/~mjg/patches/rwlock-debug-10.diff appliled on top. E.g. like this:

cd /usr/src
fetch https://people.freebsd.org/~mjg/patches/rwlock-debug-10.diff
patch -p1 < rwlock-debug-10.diff
Comment 17 Cassiano Peixoto 2017-03-14 11:54:14 UTC 
(In reply to Mateusz Guzik from comment #16)
Hi Mateusz,

Sorry but i can't try this patch, i had to rollback the old kernel to avoid crashes. It's a production server and i can't let it down. :(
Comment 18 Cassiano Peixoto 2017-03-20 20:23:20 UTC 
(In reply to Cassiano Peixoto from comment #17)
Hi Mateusz,

I got another customer to try. Can i apply this patch and only replace the kernel or do i need to replace userland as well?

After that do you need only kgdb trace output?

Thanks.
Comment 19 Scott Larson 2017-03-21 17:33:47 UTC 
(In reply to Cassiano Peixoto from comment #15)

Tossing in that I've been seeing this panic on stock kernel 11.0-p8 machines running on Atom C2750s. The failure shown by kgdb output looks identical to what Cassiano posted for 10.3. In my case it doesn't require heavy traffic however, most often it's happening at periods of off-peak load.
Comment 20 Cassiano Peixoto 2017-03-24 12:05:41 UTC 
Hi guys,

After i applied Mateusz's patch it stopped crashing (2 days with no crash). So it seems worked. Mateusz are you planning to commit it?

Thank you for your help.
Comment 21 Franco Fichtner 2017-03-24 12:13:14 UTC 
This was only supposed to print further debug output, but something in here is probably responsible for fixing it as it touches the same lines as r301157 that's been known to cause the panic in the first place...

-	else if ((rw)->rw_lock != _tid || !_rw_write_unlock((rw), _tid))\
-		_rw_wunlock_hard((rw), _tid, (file), (line));		\
+	else {								\
+		_v = (rw)->rw_lock;					\
+		if (_v != _tid || !_rw_write_unlock_fetch((rw), &_v))	\
+			_rw_wunlock_hard((rw), _v, _tid, (file), (line));\
+	}
Comment 22 Cassiano Peixoto 2017-03-27 18:13:00 UTC 
(In reply to Franco Fichtner from comment #21)
Hi Franco, i agree with you. Now it's 6 days with no crashing anymore. Mateusz, can you take a look please?
Comment 23 Ben Woods freebsd_committer freebsd_triage 2017-03-27 22:28:13 UTC 
I have also been running a week with this patch, with no more crashes. It appears to have solved it - thank you!
Comment 24 Cassiano Peixoto 2017-03-29 18:34:18 UTC 
So guys, any committer available to commit this fix? It's really important. Thanks.
Comment 25 Mateusz Guzik freebsd_committer freebsd_triage 2017-03-29 20:06:40 UTC 
I'm working on an update on this, likely will be ready tomorrow.
Comment 26 Mateusz Guzik freebsd_committer freebsd_triage 2017-04-03 14:16:56 UTC 
So, can I please get confirmation that the problem is still present in current and still present in stable/11 past r315395? There were several changes which could have make the apparent cpu problem stop manifesting itself.

If so, I'll simply merge them to stable/10 as well.
Comment 27 Cassiano Peixoto 2017-04-03 14:34:36 UTC 
(In reply to Mateusz Guzik from comment #26)
Hi Mateuz, i can confirm only on stable/10. Maybe someone else can confirm it to you. After the patch is running 13 days without crash. Thanks.
Comment 28 Cassiano Peixoto 2017-04-06 14:36:38 UTC 
(In reply to Cassiano Peixoto from comment #27)
Hi Mateusz, any news about this issue?
Comment 29 Ben Woods freebsd_committer freebsd_triage 2017-04-11 15:43:17 UTC 
(In reply to Mateusz Guzik from comment #26)
I have been running with the patch with no crashes.

If you would like me to confirm whether the issue is still in CURRENT, I will have to upgrade to the latest CURRENT without the patch, and let you know if I get crashes again.

Let me know if this would be helpful for you.
Comment 30 Cassiano Peixoto 2017-05-02 16:37:12 UTC 
Hi Mateusz,

Any update about this issue?

Thanks.
Comment 31 Cassiano Peixoto 2017-05-29 13:00:59 UTC 
Hi Mateusz,

After some time i had a server with FreeBSD 11-p1 crashed with your patch applied. 

See bellow: 

(kgdb) list *0xffffffff80b317ac
0xffffffff80b317ac is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837).
832	
833		/*
834		 * Transfer the blocked list to the pending list.
835		 */
836		mtx_lock_spin(&td_contested_lock);
837		TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq);
838		mtx_unlock_spin(&td_contested_lock);
839	
840		/*
841		 * Give a turnstile to each thread.  The last thread gets
Current language:  auto; currently minimal
(kgdb) bt
#0  doadump (textdump=<value optimized out>) at pcpu.h:221
#1  0xffffffff80acfd69 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80ad031b in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80ad0153 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff80ee2d21 in trap_fatal (frame=0xfffffe02378e0850, eva=32) at /usr/src/sys/amd64/amd64/trap.c:841
#5  0xffffffff80ee2f13 in trap_pfault (frame=0xfffffe02378e0850, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:691
#6  0xffffffff80ee24bc in trap (frame=0xfffffe02378e0850) at /usr/src/sys/amd64/amd64/trap.c:442
#7  0xffffffff80ec5951 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
#8  0xffffffff80b317ac in turnstile_broadcast (ts=0x0, queue=0) at /usr/src/sys/kern/subr_turnstile.c:837
#9  0xffffffff80aabb70 in __mtx_unlock_sleep (c=0xffffffff81cab130, opts=<value optimized out>, file=0x1 <Address 0x1 out of bounds>, line=1) at /usr/src/sys/kern/kern_mutex.c:865
#10 0xffffffff80d42dd1 in swap_reserve_by_cred (incr=<value optimized out>, cred=<value optimized out>) at /usr/src/sys/vm/swap_pager.c:224
#11 0xffffffff80d58f49 in kmap_alloc_wait (map=0xfffff800061d7c30, size=266240) at /usr/src/sys/vm/vm_kern.c:437
#12 0xffffffff80a7cf5c in exec_copyin_args (args=0xfffffe02378e0a70, fname=0x801436b58 <Address 0x801436b58 out of bounds>, segflg=UIO_USERSPACE, argv=0x801436a98, envv=0x801436b20)
    at /usr/src/sys/kern/kern_exec.c:1326
#13 0xffffffff80a7cdb8 in sys_execve (td=0xfffff8013d03ba00, uap=0xfffffe02378e0b80) at /usr/src/sys/kern/kern_exec.c:215
#14 0xffffffff80ee367e in amd64_syscall (td=<value optimized out>, traced=0) at subr_syscall.c:135
#15 0xffffffff80ec5c3b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
#16 0x0000000800b45daa in ?? ()
Comment 32 Chris Collins 2017-06-07 00:49:49 UTC 
I am getting turnstile_broadcast() at turnstile_broadcast panic's on a PFSense unit which is based on FreeBSD 11.0-p8

The hardware is a intel celeron N3150.

Is it possible to provide a patch which I can pass onto the PFSense developers to see if the proposed fix works on that?

Thanks
Comment 33 Chris Collins 2017-06-08 18:25:22 UTC 
I have applied your patch now Mateusz on 11-p10 so will report back if it works.
Comment 34 Cassiano Peixoto 2017-06-27 12:26:43 UTC 
(In reply to Mateusz Guzik from comment #26)
Dear Mateusz,

Please, Can you confirm if it has been fixed in last STABLE? I've seen some commits but i'm no sure.

Thanks.
Comment 35 Mateusz Guzik freebsd_committer freebsd_triage 2017-06-29 14:00:12 UTC 
Hi there, sorry for late reply. This somehow fell through the cracks.

First of all there is no kernel bug per se that I can see, rather a bug in the atom cpu which started manifesting itself. There were several changes to the affected code path on stable/11 and in particular the condition which triggered the bug is gone. I don't have any reports, but I suspect stable/11 is now cleared. stable/10 will require a hack, but I'll need someone to test it as don't have the hardware to reproduce.
Comment 36 Franco Fichtner 2017-06-29 14:02:22 UTC 
You can use https://github.com/opnsense/src/commit/6b79b52c.patch on stable/10, it was verified working on 11.0.


Cheers,
Franco
Comment 37 Cassiano Peixoto 2017-06-29 14:07:01 UTC 
(In reply to Mateusz Guzik from comment #35)
Mateusz,

Yes i realized many changes has been made on 11-STABLE related to this issue. I think it could be fixed as well. Anyway i have a server running with 11.1-BETA3 to confirm.

Regarding FreeBSD 10, i can test it for you, just provide a patch and let me know.

Thanks.
Comment 38 Chris Collins 2017-06-29 16:38:20 UTC 
(In reply to Mateusz Guzik from comment #35)

Just to let you know my pfsense unit affected by this issue does not have an atom cpu.

It has a celeron N3150 cpu.
Comment 39 Mateusz Guzik freebsd_committer freebsd_triage 2017-07-01 02:09:19 UTC 
Can someone who runs into the problem on stable/10 test the following please: https://people.freebsd.org/~mjg/patches/rwlock-unlock.diff
Comment 40 Cassiano Peixoto 2017-07-03 18:07:47 UTC 
(In reply to Mateusz Guzik from comment #39)
Mateusz,

I can test it, but my 10-STABLE version has a different code:

/* Release a write lock. */
#define __rw_wunlock(rw, tid, file, line) do {                          \
        uintptr_t _tid = (uintptr_t)(tid);                              \
        uintptr_t _v;                                                   \
                                                                        \
        if ((rw)->rw_recurse)                                           \
                (rw)->rw_recurse--;                                     \
        else {                                                          \
                _v = (rw)->rw_lock;                                     \
                if (_v != _tid || !_rw_write_unlock_fetch((rw), &_v))   \
                        _rw_wunlock_hard((rw), _v, _tid, (file), (line));\
        }                                                               \
} while (0)

How shoukd i adapt your patch in this code?

Thanks.
Comment 41 Mateusz Guzik freebsd_committer freebsd_triage 2017-07-03 18:24:28 UTC 
This looks like leftovers from the previous patch. Just restore the original state of the tree and patch that.
Comment 42 Cassiano Peixoto 2017-07-03 19:27:49 UTC 
(In reply to Mateusz Guzik from comment #41)
Ok you are right. I reverted all files touched by previous patch. Am i right? And then should i apply only this last patch?
Comment 43 Mateusz Guzik freebsd_committer freebsd_triage 2017-07-03 20:15:57 UTC 
Yes, apply the patch to a clean tree. Thanks.
Comment 44 Cassiano Peixoto 2017-07-03 20:17:27 UTC 
(In reply to Mateusz Guzik from comment #43)
Ok patch applied. I'll let you know the results.

Thanks.
Comment 45 Kubilay Kocak freebsd_committer freebsd_triage 2017-07-05 05:50:04 UTC 
Can we get the proposed patches included as attachments in this issue please, ideally, and if multiple versions are created over time, obsoleting the old ones.

Also is this a candidate for upcoming releases? If so, please cc re@
Comment 46 Chris Collins 2017-07-05 06:14:03 UTC 
I would hope this makes it into RELEASE 11.1, would be disappointed if that was released with a known problem.  PFSense seem to have absolutely no motivation to apply the patches supplied here, so I am hoping they use 11.1 after its released and that has the patches applied.
Comment 47 Cassiano Peixoto 2017-07-10 11:40:48 UTC 
(In reply to Mateusz Guzik from comment #43)
Dear Mateusz,

I'm quite sure this patch has fixed the issue on 10-STABLE. My server is running 6 days without reboot. Please go ahead and commit it. Thank you for your support.
Comment 48 Mateusz Guzik freebsd_committer freebsd_triage 2017-07-10 15:36:52 UTC 
Thanks for testing. I'll commit the patch later today with few other modifications.

That said, I expect the problem to be gone on head starting with r313270 and on stable/11 starting with r315377, which also means it is included in all 11.1 builds.

That said, if someone running into the crash on one of the above could confirm the newest version of their relevant branch makes the problem go away, that would be nice.
Comment 49 Ben Woods freebsd_committer freebsd_triage 2017-07-10 22:35:48 UTC 
I am running FreeBSD 12-CURRENT r319025 without any patches and can confirm the crashes do not occur for me anymore on my FreeNAS mini. Thanks for your help with this!
Comment 50 commit-hook freebsd_committer freebsd_triage 2017-07-11 03:46:19 UTC 
A commit references this bug:

Author: mjg
Date: Tue Jul 11 03:45:48 UTC 2017
New revision: 320885
URL: https://svnweb.freebsd.org/changeset/base/320885

Log:
  Remove waiters check from the inline rw wunlock routine.

  This is a direct commit to stable/10.

  r310979 is a merge of depessimisation of locking primitives.
  The important part was getting rid of an attempt to grab the lock in the
  slow path immediately after the fast path failed. In addition to that
  temporary checks were added before all atomic ops. They have no impact on
  semantic nor correctness, they only avoid an atomic operation which is
  likely to fail.

  After the addition of atomic_fcmpset and further changes said checks
  became pessimal and got removed. This may get merged to stable/10.

  Reports started showing up about a crash in all branches having extra
  checks. The codepath is:
  .. -> vm_map_delete -> __rw_wunlock_hard -> turnstile_broadcast

  The kernel crashed trying to wake up nonexistent waiters. The lock value
  as found in the vmcore matches the panicking thread, so in particular
  there was no waiters bit set. The bit can only be cleared by the current
  owner.

  A debug patch was provided, which reportedly had a side effect of getting
  rid of the issue.

  Also one of the reporters said that reverting the patch which adds the
  extra checks makes the crash go away.

  It was also reported that head with the fcmpset changes (explicit checks
  removed) also stops crashing.

  Finally, one user tested crashing stable/10 variant with just the rw
  wunlock check removed.

  The common case in all but one reports was an Intel Atom cpu. Claiming
  a cpu bug at this point is bold and I'm going to refrain from it, but
  right now apart from cpu-specific optimisation made by the compiler on
  custom kernel compiles I don't see how this can be a software bug.

  This will have to be investigated more.

  Meanwhile, restore rw wunlock to pre-r310979 state.

  PR:		213903

Changes:
  stable/10/sys/sys/rwlock.h
Comment 51 Mateusz Guzik freebsd_committer freebsd_triage 2017-07-11 03:50:26 UTC 
The change was committed to stable/10. To the best of my knowledge all branches with the original patch (head, stable/11, 11.1 and stable/10) have the issue resolved.

Thanks everyone for reporting and testing.
Comment 52 Kubilay Kocak freebsd_committer freebsd_triage 2017-07-11 05:17:30 UTC 
Correctly (+, not -) document merge completion
Comment 53 Chris Collins 2017-07-12 10:21:34 UTC 
Both patches in this PR have fixed my kernel panic issues on 11.0

I tested the debug patch which didnt aim to fix but did as a side affect, and I tested the later patch.
Comment 54 muxx.dev 2017-08-12 12:03:24 UTC 
I can confirm the same crash on FreeBSD 11.0-RELEASE-p1 (GENERIC) on the following hardware:

Aug 12 11:57:04 gw kernel: CPU: Intel(R) Celeron(R) CPU  J1900  @ 1.99GHz (2000.05-MHz K8-class CPU)
Aug 12 11:57:04 gw kernel: Origin="GenuineIntel"  Id=0x30678  Family=0x6  Model=0x37  Stepping=8
Aug 12 11:57:04 gw kernel: Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Aug 12 11:57:04 gw kernel: Features2=0x41d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,RDRAND>
Aug 12 11:57:04 gw kernel: AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Aug 12 11:57:04 gw kernel: AMD Features2=0x101<LAHF,Prefetch>
Aug 12 11:57:04 gw kernel: Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
Aug 12 11:57:04 gw kernel: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Aug 12 11:57:04 gw kernel: TSC: P-state invariant, performance statistics
Aug 12 11:57:04 gw kernel: real memory  = 8589934592 (8192 MB)
Aug 12 11:57:04 gw kernel: avail memory = 8137785344 (7760 MB)
Aug 12 11:57:04 gw kernel: Event timer "LAPIC" quality 600
Aug 12 11:57:04 gw kernel: ACPI APIC Table: <ALASKA A M I >
Aug 12 11:57:04 gw kernel: WARNING: L1 data cache covers less APIC IDs than a core
Aug 12 11:57:04 gw kernel: 0 < 1
Aug 12 11:57:04 gw kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Aug 12 11:57:04 gw kernel: FreeBSD/SMP: 1 package(s) x 4 core(s)
Aug 12 11:57:04 gw kernel: random: unblocking device.
Aug 12 11:57:04 gw kernel: ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650)
Aug 12 11:57:04 gw kernel: WARNING: Bogus Interrupt Polarity. Assume CONFORMS

more information from /var/log/messages and kgdb:

Aug 12 11:57:04 gw kernel: Fatal trap 12: page fault while in kernel mode
Aug 12 11:57:04 gw kernel: cpuid = 1; apic id = 02
Aug 12 11:57:04 gw kernel: fault virtual address        = 0x30
Aug 12 11:57:04 gw kernel: fault code           = supervisor read data, page not present
Aug 12 11:57:04 gw kernel: instruction pointer  = 0x20:0xffffffff80b3a89c
Aug 12 11:57:04 gw kernel: stack pointer                = 0x28:0xfffffe0232609440
Aug 12 11:57:04 gw kernel: frame pointer                = 0x28:0xfffffe0232609470
Aug 12 11:57:04 gw kernel: code segment         = base rx0, limit 0xfffff, type 0x1b
Aug 12 11:57:04 gw kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Aug 12 11:57:04 gw kernel: processor eflags     = resume, IOPL = 0
Aug 12 11:57:04 gw kernel: current process              = 18204 (telegraf)
Aug 12 11:57:04 gw kernel: trap number          = 12
Aug 12 11:57:04 gw kernel: panic: page fault
Aug 12 11:57:04 gw kernel: cpuid = 1
Aug 12 11:57:04 gw kernel: KDB: stack backtrace:
Aug 12 11:57:04 gw kernel: #0 0xffffffff80b24077 at kdb_backtrace+0x67
Aug 12 11:57:04 gw kernel: #1 0xffffffff80ad93e2 at vpanic+0x182
Aug 12 11:57:04 gw kernel: #2 0xffffffff80ad9253 at panic+0x43
Aug 12 11:57:04 gw kernel: #3 0xffffffff80fa0d31 at trap_fatal+0x351
Aug 12 11:57:04 gw kernel: #4 0xffffffff80fa0f23 at trap_pfault+0x1e3
Aug 12 11:57:04 gw kernel: #5 0xffffffff80fa04cc at trap+0x26c
Aug 12 11:57:04 gw kernel: #6 0xffffffff80f84141 at calltrap+0x8
Aug 12 11:57:04 gw kernel: #7 0xffffffff80ad48cf at __rw_wunlock_hard+0x8f
Aug 12 11:57:04 gw kernel: #8 0xffffffff80e1a75c at vm_map_delete+0x3dc
Aug 12 11:57:04 gw kernel: #9 0xffffffff80e1c5f7 at vm_map_remove+0x47
Aug 12 11:57:04 gw kernel: #10 0xffffffff80a86c7f at exec_new_vmspace+0x22f
Aug 12 11:57:04 gw kernel: #11 0xffffffff80a5bfe8 at exec_elf64_imgact+0xa58
Aug 12 11:57:04 gw kernel: #12 0xffffffff80a84d4d at kern_execve+0x7dd
Aug 12 11:57:04 gw kernel: #13 0xffffffff80a841dc at sys_execve+0x4c
Aug 12 11:57:04 gw kernel: #14 0xffffffff80fa168e at amd64_syscall+0x4ce
Aug 12 11:57:04 gw kernel: #15 0xffffffff80f8442b at Xfast_syscall+0xfb

...

(kgdb) list *0xffffffff80b3a89c
0xffffffff80b3a89c is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837).
832
833             /*
834              * Transfer the blocked list to the pending list.
835              */
836             mtx_lock_spin(&td_contested_lock);
837             TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq);
838             mtx_unlock_spin(&td_contested_lock);
839
840             /*
841              * Give a turnstile to each thread.  The last thread gets
Current language:  auto; currently minimal
(kgdb) backtrace
#0  doadump (textdump=<value optimized out>) at pcpu.h:221
#1  0xffffffff80ad8e69 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff80ad941b in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff80ad9253 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:690
#4  0xffffffff80fa0d31 in trap_fatal (frame=0xfffffe0232609390, eva=48) at /usr/src/sys/amd64/amd64/trap.c:841
#5  0xffffffff80fa0f23 in trap_pfault (frame=0xfffffe0232609390, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:691
#6  0xffffffff80fa04cc in trap (frame=0xfffffe0232609390) at /usr/src/sys/amd64/amd64/trap.c:442
#7  0xffffffff80f84141 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236
#8  0xffffffff80b3a89c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837
#9  0xffffffff80ad48cf in __rw_wunlock_hard (c=0xfffff800437de858, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1027
#10 0xffffffff80e1a75c in vm_map_delete (map=<value optimized out>, start=<value optimized out>, end=<value optimized out>) at /usr/src/sys/vm/vm_map.c:2960
#11 0xffffffff80e1c5f7 in vm_map_remove (map=0xfffff80032b91000, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077
#12 0xffffffff80a86c7f in exec_new_vmspace (imgp=0xfffffe0232609860, sv=0xffffffff81a02720) at /usr/src/sys/kern/kern_exec.c:1095
#13 0xffffffff80a5bfe8 in exec_elf64_imgact (imgp=<value optimized out>) at /usr/src/sys/kern/imgact_elf.c:896
#14 0xffffffff80a84d4d in kern_execve (td=<value optimized out>, args=<value optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:602
#15 0xffffffff80a841dc in sys_execve (td=0xfffff801a5da3a00, uap=0xfffffe0232609b80) at /usr/src/sys/kern/kern_exec.c:218
#16 0xffffffff80fa168e in amd64_syscall (td=<value optimized out>, traced=0) at subr_syscall.c:135
#17 0xffffffff80f8442b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
#18 0x000000000047da1f in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
Comment 55 Mark Millard 2017-08-12 20:38:05 UTC 
(In reply to muxx.dev from comment #54)\

Looks  to me like ts==0x0 was the starting value
given to turnstile_broadcast in each backtrace
listed in this buzilla bug (213903), for example:

(kgdb) list *0xffffffff80b3a89c
0xffffffff80b3a89c is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837).
832
833             /*
834              * Transfer the blocked list to the pending list.
835              */
836             mtx_lock_spin(&td_contested_lock);
837             TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq);
838             mtx_unlock_spin(&td_contested_lock);
839
840             /*
841              * Give a turnstile to each thread.  The last thread gets
Current language:  auto; currently minimal
(kgdb) backtrace
. . .
#8  0xffffffff80b3a89c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837
#9  0xffffffff80ad48cf in __rw_wunlock_hard (c=0xfffff800437de858, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1027
. . .

Note that ts==0x0 would be a problem for:

TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq);

So I wonder if __rw_wunlock_hard is providing a bad ts
value. If yes: the problem then occurs before the
turnstile_broadcast call is made.
Comment 56 Eitan Adler freebsd_committer freebsd_triage 2018-05-23 10:27:47 UTC 
batch change of PRs untouched in 2018 marked "in progress" back to open.
Comment 57 Rodney W. Grimes freebsd_committer freebsd_triage 2019-02-13 01:56:50 UTC 
Please do not put bugs on stable@, current@, hackers@, etc