I recently tried upgrading a machine to 11-STABLE, but got a "Fatal trap 12: page fault while in kernel mode" after going multi-user, when it started syncthing 0.13.4: Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x10 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80d3cd3d stack pointer = 0x28:0xfffffe04538ef560 frame pointer = 0x28:0xfffffe04538ef5a0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 958 (syncthing) trap number = 12 panic: page fault cpuid = 2 KDB: stack backtrace: #0 0xffffffff80b2f917 at kdb_backtrace+0x67 #1 0xffffffff80ae6bb2 at vpanic+0x182 #2 0xffffffff80ae6a23 at panic+0x43 #3 0xffffffff80fb3020 at trap_fatal+0x350 #4 0xffffffff80fb3213 at trap_pfault+0x1e3 #5 0xffffffff80fb27bd at trap+0x26d #6 0xffffffff80f95311 at calltrap+0x8 #7 0xffffffff80d3c6ea at ip6_setpktopts+0x10a #8 0xffffffff80d54aa4 at udp6_send+0x364 #9 0xffffffff80b80672 at sosend_dgram+0x4d2 #10 0xffffffff80b88b7b at kern_sendit+0x22b #11 0xffffffff80b88f7f at sendit+0x19f #12 0xffffffff80b89031 at sys_sendmsg+0x61 #13 0xffffffff80fb397e at amd64_syscall+0x4ce #14 0xffffffff80f955fb at Xfast_syscall+0xfb Backtrace from kgdb: #0 __curthread () at ./machine/pcpu.h:221 #1 doadump (textdump=<optimized out>) at /home/dim/stable-11/sys/kern/kern_shutdown.c:298 #2 0xffffffff80ae663a in kern_reboot (howto=260) at /home/dim/stable-11/sys/kern/kern_shutdown.c:366 #3 0xffffffff80ae6beb in vpanic (fmt=<optimized out>, ap=0xfffffe04538ef1b0) at /home/dim/stable-11/sys/kern/kern_shutdown.c:759 #4 0xffffffff80ae6a23 in panic (fmt=<unavailable>) at /home/dim/stable-11/sys/kern/kern_shutdown.c:690 #5 0xffffffff80fb3020 in trap_fatal (frame=0xfffffe04538ef4b0, eva=16) at /home/dim/stable-11/sys/amd64/amd64/trap.c:841 #6 0xffffffff80fb3213 in trap_pfault (frame=0xfffffe04538ef4b0, usermode=0) at /home/dim/stable-11/sys/amd64/amd64/trap.c:691 #7 0xffffffff80fb27bd in trap (frame=0xfffffe04538ef4b0) at /home/dim/stable-11/sys/amd64/amd64/trap.c:442 #8 <signal handler called> #9 0xffffffff80d3cd3d in ip6_setpktopt (optname=<optimized out>, buf=0xfffff8000ff1f548 "", len=20, opt=0xfffffe04538ef698, cred=0xfffff800aa61b800, sticky=0, cmsg=<optimized out>, uproto=<optimized out>) at /home/dim/stable-11/sys/netinet6/ip6_output.c:2663 #10 0xffffffff80d3c6ea in ip6_setpktopts (control=<optimized out>, opt=<optimized out>, stickyopt=<optimized out>, cred=<optimized out>, uproto=<optimized out>) at /home/dim/stable-11/sys/netinet6/ip6_output.c:2557 #11 0xffffffff80d54aa4 in udp6_output (inp=0xfffff8002336fae0, m=<optimized out>, addr6=<optimized out>, control=0x0, td=<optimized out>) at /home/dim/stable-11/sys/netinet6/udp6_usrreq.c:695 #12 udp6_send (so=<optimized out>, flags=-2116399792, m=<optimized out>, addr=<optimized out>, control=<optimized out>, td=0xfffff800230f7a00) at /home/dim/stable-11/sys/netinet6/udp6_usrreq.c:1274 #13 0xffffffff80b80672 in sosend_dgram (so=0xfffff800aa24ea20, addr=<optimized out>, uio=<optimized out>, top=<optimized out>, control=<optimized out>, flags=<optimized out>, td=<optimized out>) at /home/dim/stable-11/sys/kern/uipc_socket.c:1174 #14 0xffffffff80b88b7b in kern_sendit (td=<optimized out>, s=<optimized out>, mp=<optimized out>, flags=0, control=0xfffff8000ff1f500, segflg=UIO_USERSPACE) at /home/dim/stable-11/sys/kern/uipc_syscalls.c:848 #15 0xffffffff80b88f7f in sendit (td=0xfffff800230f7a00, s=<optimized out>, mp=0xfffffe04538ef948, flags=<optimized out>) at /home/dim/stable-11/sys/kern/uipc_syscalls.c:775 #16 0xffffffff80b89031 in sys_sendmsg (td=0xfffff800230f7a00, uap=0xfffffe04538efa40) at /home/dim/stable-11/sys/kern/uipc_syscalls.c:977 #17 0xffffffff80fb397e in syscallenter (td=<optimized out>, sa=<optimized out>) at /home/dim/stable-11/sys/amd64/amd64/../../kern/subr_syscall.c:135 #18 amd64_syscall (td=<optimized out>, traced=0) at /home/dim/stable-11/sys/amd64/amd64/trap.c:942 #19 <signal handler called> #20 0x00000000004f9494 in ?? () In frame 9, ifp->if_afdata[AF_INET6] is NULL, which is the cause of the crash: (kgdb) frame 9 #9 0xffffffff80d3cd3d in ip6_setpktopt (optname=<optimized out>, buf=0xfffff8000ff1f548 "", len=20, opt=0xfffffe04538ef698, cred=0xfffff800aa61b800, sticky=0, cmsg=<optimized out>, uproto=<optimized out>) at /home/dim/stable-11/sys/netinet6/ip6_output.c:2663 2663 /home/dim/stable-11/sys/netinet6/ip6_output.c: No such file or directory. (kgdb) print ifp->if_afdata[28] $1 = (void *) 0x0 In fact, all if_afdata[] members except for AF_INET seem to be NULL: (kgdb) print ifp->if_afdata $4 = {0x0, 0x0, 0xfffff80008614c20, 0x0 <repeats 39 times>} It looks like syncthing is doing some sort of UDPv6 sending over all interfaces, or something. The Go is rather hard for me to digest, so what it exactly calls I don't know. In any case, the problem is also reproducible very easily on -CURRENT, simply by installing the syncthing port, then: kldload pflog syncthing and waiting for syncthing's initial startup to complete. It will panic almost immediately.
Bisection shows this was introduced by r271396 [1]. Specifically, this part that was added: 2572 if (ifp != NULL && ( 2573 ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED)) 2574 return (ENETDOWN); The problem is that ND_IFINFO(ifp) dereferences ifp->if_afdata[AF_INET6] unconditionally, so if that is NULL, a page fault occurs. Maybe a good fix is just the following? Index: sys/netinet6/ip6_output.c =================================================================== --- sys/netinet6/ip6_output.c (revision 271396) +++ sys/netinet6/ip6_output.c (working copy) @@ -2569,7 +2569,7 @@ if (ifp == NULL) return (ENXIO); } - if (ifp != NULL && ( + if (ifp != NULL && ifp->if_afdata[AF_INET6] != NULL && ( ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED)) return (ENETDOWN); [1] https://svnweb.freebsd.org/base?view=revision&revision=271396
I prefer this patch, can you test? --- ip6_output.c (revision 302315) +++ ip6_output.c (working copy) @@ -2659,8 +2659,8 @@ ip6_setpktopt(int optname, u_char *buf, int len, s if (ifp == NULL) return (ENXIO); } - if (ifp != NULL && ( - ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED)) + if (ifp != NULL && (ifp->if_afdata[AF_INET6] == NULL || + (ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED) != 0)) return (ENETDOWN); if (ifp != NULL &&
(In reply to Andrey V. Elsukov from comment #2) > I prefer this patch, can you test? Yes, that works fine too.
So Andrey, shall I commit your version from comment 2, with "Reviewed by: ae"?
A commit references this bug: Author: dim Date: Wed Jul 13 19:41:19 UTC 2016 New revision: 302784 URL: https://svnweb.freebsd.org/changeset/base/302784 Log: Fix a page fault in ip6_setpktopt(), occurring when the pflog module is loaded, and syncthing is started, which uses setsockopt(IPV6_PKGINFO). This is because pflog interfaces do not normally have an IPv6 address, causing the ND_IFINFO() macro to dereference a NULL pointer. Reviewed by: ae PR: 210943 MFC after: 3 days Changes: head/sys/netinet6/ip6_output.c
Assign to committer that resolved. Re-open for MFC to stable/11, stable/10 Please set flag mfc-stable* to + if/when committed, or - if not appropriate with comment
A commit references this bug: Author: dim Date: Sat Jul 16 10:50:28 UTC 2016 New revision: 302934 URL: https://svnweb.freebsd.org/changeset/base/302934 Log: MFC r302784: Fix a page fault in ip6_setpktopt(), occurring when the pflog module is loaded, and syncthing is started, which uses setsockopt(IPV6_PKGINFO). This is because pflog interfaces do not normally have an IPv6 address, causing the ND_IFINFO() macro to dereference a NULL pointer. Approved by: re (kib) Reviewed by: ae PR: 210943 Changes: _U stable/11/ stable/11/sys/netinet6/ip6_output.c
Merged to stable/11 in time for 11.0-RELEASE. Not applicable to stable/10.