Bug 210943

Summary:	Page fault in ip6_setpktopts when syncthing is started with pflog loaded
Product:	Base System	Reporter:	Dimitry Andric <dim>
Component:	kern	Assignee:	Dimitry Andric <dim>
Status:	Closed FIXED
Severity:	Affects Some People	CC:	ae, emaste
Priority:	---	Keywords:	patch
Version:	CURRENT	Flags:	dim: mfc-stable11+ dim: mfc-stable10-
Hardware:	Any
OS:	Any

Description Dimitry Andric freebsd_committer

2016-07-09 13:29:36 UTC

I recently tried upgrading a machine to 11-STABLE, but got a "Fatal trap 12: page fault while in kernel mode" after going multi-user, when it started syncthing 0.13.4:

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address	= 0x10
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d3cd3d
stack pointer	        = 0x28:0xfffffe04538ef560
frame pointer	        = 0x28:0xfffffe04538ef5a0
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 958 (syncthing)
trap number		= 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff80b2f917 at kdb_backtrace+0x67
#1 0xffffffff80ae6bb2 at vpanic+0x182
#2 0xffffffff80ae6a23 at panic+0x43
#3 0xffffffff80fb3020 at trap_fatal+0x350
#4 0xffffffff80fb3213 at trap_pfault+0x1e3
#5 0xffffffff80fb27bd at trap+0x26d
#6 0xffffffff80f95311 at calltrap+0x8
#7 0xffffffff80d3c6ea at ip6_setpktopts+0x10a
#8 0xffffffff80d54aa4 at udp6_send+0x364
#9 0xffffffff80b80672 at sosend_dgram+0x4d2
#10 0xffffffff80b88b7b at kern_sendit+0x22b
#11 0xffffffff80b88f7f at sendit+0x19f
#12 0xffffffff80b89031 at sys_sendmsg+0x61
#13 0xffffffff80fb397e at amd64_syscall+0x4ce
#14 0xffffffff80f955fb at Xfast_syscall+0xfb

Backtrace from kgdb:

#0  __curthread () at ./machine/pcpu.h:221
#1  doadump (textdump=<optimized out>) at /home/dim/stable-11/sys/kern/kern_shutdown.c:298
#2  0xffffffff80ae663a in kern_reboot (howto=260) at /home/dim/stable-11/sys/kern/kern_shutdown.c:366
#3  0xffffffff80ae6beb in vpanic (fmt=<optimized out>, ap=0xfffffe04538ef1b0) at /home/dim/stable-11/sys/kern/kern_shutdown.c:759
#4  0xffffffff80ae6a23 in panic (fmt=<unavailable>) at /home/dim/stable-11/sys/kern/kern_shutdown.c:690
#5  0xffffffff80fb3020 in trap_fatal (frame=0xfffffe04538ef4b0, eva=16) at /home/dim/stable-11/sys/amd64/amd64/trap.c:841
#6  0xffffffff80fb3213 in trap_pfault (frame=0xfffffe04538ef4b0, usermode=0) at /home/dim/stable-11/sys/amd64/amd64/trap.c:691
#7  0xffffffff80fb27bd in trap (frame=0xfffffe04538ef4b0) at /home/dim/stable-11/sys/amd64/amd64/trap.c:442
#8  <signal handler called>
#9  0xffffffff80d3cd3d in ip6_setpktopt (optname=<optimized out>, buf=0xfffff8000ff1f548 "", len=20, opt=0xfffffe04538ef698, cred=0xfffff800aa61b800, sticky=0, cmsg=<optimized out>, uproto=<optimized out>) at /home/dim/stable-11/sys/netinet6/ip6_output.c:2663
#10 0xffffffff80d3c6ea in ip6_setpktopts (control=<optimized out>, opt=<optimized out>, stickyopt=<optimized out>, cred=<optimized out>, uproto=<optimized out>) at /home/dim/stable-11/sys/netinet6/ip6_output.c:2557
#11 0xffffffff80d54aa4 in udp6_output (inp=0xfffff8002336fae0, m=<optimized out>, addr6=<optimized out>, control=0x0, td=<optimized out>) at /home/dim/stable-11/sys/netinet6/udp6_usrreq.c:695
#12 udp6_send (so=<optimized out>, flags=-2116399792, m=<optimized out>, addr=<optimized out>, control=<optimized out>, td=0xfffff800230f7a00) at /home/dim/stable-11/sys/netinet6/udp6_usrreq.c:1274
#13 0xffffffff80b80672 in sosend_dgram (so=0xfffff800aa24ea20, addr=<optimized out>, uio=<optimized out>, top=<optimized out>, control=<optimized out>, flags=<optimized out>, td=<optimized out>) at /home/dim/stable-11/sys/kern/uipc_socket.c:1174
#14 0xffffffff80b88b7b in kern_sendit (td=<optimized out>, s=<optimized out>, mp=<optimized out>, flags=0, control=0xfffff8000ff1f500, segflg=UIO_USERSPACE) at /home/dim/stable-11/sys/kern/uipc_syscalls.c:848
#15 0xffffffff80b88f7f in sendit (td=0xfffff800230f7a00, s=<optimized out>, mp=0xfffffe04538ef948, flags=<optimized out>) at /home/dim/stable-11/sys/kern/uipc_syscalls.c:775
#16 0xffffffff80b89031 in sys_sendmsg (td=0xfffff800230f7a00, uap=0xfffffe04538efa40) at /home/dim/stable-11/sys/kern/uipc_syscalls.c:977
#17 0xffffffff80fb397e in syscallenter (td=<optimized out>, sa=<optimized out>) at /home/dim/stable-11/sys/amd64/amd64/../../kern/subr_syscall.c:135
#18 amd64_syscall (td=<optimized out>, traced=0) at /home/dim/stable-11/sys/amd64/amd64/trap.c:942
#19 <signal handler called>
#20 0x00000000004f9494 in ?? ()

In frame 9, ifp->if_afdata[AF_INET6] is NULL, which is the cause of the crash:

(kgdb) frame 9
#9  0xffffffff80d3cd3d in ip6_setpktopt (optname=<optimized out>, buf=0xfffff8000ff1f548 "", len=20, opt=0xfffffe04538ef698, cred=0xfffff800aa61b800, sticky=0, cmsg=<optimized out>, uproto=<optimized out>) at /home/dim/stable-11/sys/netinet6/ip6_output.c:2663
2663    /home/dim/stable-11/sys/netinet6/ip6_output.c: No such file or directory.
(kgdb) print ifp->if_afdata[28]
$1 = (void *) 0x0

In fact, all if_afdata[] members except for AF_INET seem to be NULL:

(kgdb) print ifp->if_afdata
$4 =   {0x0,
  0x0,
  0xfffff80008614c20,
  0x0 <repeats 39 times>}

It looks like syncthing is doing some sort of UDPv6 sending over all interfaces, or something.  The Go is rather hard for me to digest, so what it exactly calls I don't know.

In any case, the problem is also reproducible very easily on -CURRENT, simply by installing the syncthing port, then:

kldload pflog
syncthing

and waiting for syncthing's initial startup to complete.  It will panic almost immediately.

Comment 1 Dimitry Andric freebsd_committer

2016-07-10 16:10:16 UTC

Bisection shows this was introduced by r271396 [1].  Specifically, this part that was added:

  2572                  if (ifp != NULL && (
  2573                      ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED))
  2574                          return (ENETDOWN);

The problem is that ND_IFINFO(ifp) dereferences ifp->if_afdata[AF_INET6] unconditionally, so if that is NULL, a page fault occurs.

Maybe a good fix is just the following?

Index: sys/netinet6/ip6_output.c
===================================================================
--- sys/netinet6/ip6_output.c	(revision 271396)
+++ sys/netinet6/ip6_output.c	(working copy)
@@ -2569,7 +2569,7 @@
 			if (ifp == NULL)
 				return (ENXIO);
 		}
-		if (ifp != NULL && (
+		if (ifp != NULL && ifp->if_afdata[AF_INET6] != NULL && (
 		    ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED))
 			return (ENETDOWN);


[1] https://svnweb.freebsd.org/base?view=revision&revision=271396

Comment 2 Andrey V. Elsukov freebsd_committer

2016-07-11 11:21:40 UTC

I prefer this patch, can you test?

--- ip6_output.c	(revision 302315)
+++ ip6_output.c	(working copy)
@@ -2659,8 +2659,8 @@ ip6_setpktopt(int optname, u_char *buf, int len, s
 			if (ifp == NULL)
 				return (ENXIO);
 		}
-		if (ifp != NULL && (
-		    ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED))
+		if (ifp != NULL && (ifp->if_afdata[AF_INET6] == NULL ||
+		    (ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED) != 0))
 			return (ENETDOWN);
 
 		if (ifp != NULL &&

Comment 3 Dimitry Andric freebsd_committer

2016-07-11 16:36:40 UTC

(In reply to Andrey V. Elsukov from comment #2)
> I prefer this patch, can you test?

Yes, that works fine too.

Comment 4 Dimitry Andric freebsd_committer

2016-07-11 21:12:03 UTC

So Andrey, shall I commit your version from comment 2, with "Reviewed by: ae"?

Comment 5 commit-hook freebsd_committer

2016-07-13 19:42:09 UTC

A commit references this bug:

Author: dim
Date: Wed Jul 13 19:41:19 UTC 2016
New revision: 302784
URL: https://svnweb.freebsd.org/changeset/base/302784

Log:
  Fix a page fault in ip6_setpktopt(), occurring when the pflog module is
  loaded, and syncthing is started, which uses setsockopt(IPV6_PKGINFO).

  This is because pflog interfaces do not normally have an IPv6 address,
  causing the ND_IFINFO() macro to dereference a NULL pointer.

  Reviewed by:	ae
  PR:		210943
  MFC after:	3 days

Changes:
  head/sys/netinet6/ip6_output.c

Comment 6 Kubilay Kocak freebsd_committer

2016-07-14 07:25:36 UTC

Assign to committer that resolved.

Re-open for MFC to stable/11, stable/10

Please set flag mfc-stable* to + if/when committed, or - if not appropriate with comment

Comment 7 commit-hook freebsd_committer

2016-07-16 10:51:28 UTC

A commit references this bug:

Author: dim
Date: Sat Jul 16 10:50:28 UTC 2016
New revision: 302934
URL: https://svnweb.freebsd.org/changeset/base/302934

Log:
  MFC r302784:

  Fix a page fault in ip6_setpktopt(), occurring when the pflog module is
  loaded, and syncthing is started, which uses setsockopt(IPV6_PKGINFO).

  This is because pflog interfaces do not normally have an IPv6 address,
  causing the ND_IFINFO() macro to dereference a NULL pointer.

  Approved by:	re (kib)
  Reviewed by:	ae
  PR:		210943

Changes:
_U  stable/11/
  stable/11/sys/netinet6/ip6_output.c

Comment 8 Dimitry Andric freebsd_committer

2016-07-16 10:53:10 UTC

Merged to stable/11 in time for 11.0-RELEASE.  Not applicable to stable/10.