Bug 231728 - panic: udp6_output: sin6(0x...)->sin6_addr is v4mapped which we should have handled
Summary: panic: udp6_output: sin6(0x...)->sin6_addr is v4mapped which we should have h...
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Bjoern A. Zeeb
URL:
Keywords: panic, patch, regression
Depends on:
Blocks:
 
Reported: 2018-09-26 10:04 UTC by Jeremy Faulkner
Modified: 2018-10-02 17:31 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jeremy Faulkner 2018-09-26 10:04:46 UTC
Panic when attempting to use qbitorrent-nox to download a torrent.

https://flic.kr/p/PiZmzy
Comment 1 Andriy Gapon freebsd_committer 2018-09-27 05:55:05 UTC
Please take at least a minimal effort to transcribe what you see on the screen into a useful bug description.  Developers' time is as valuable as yours.
Comment 2 Andriy Gapon freebsd_committer 2018-09-27 06:02:58 UTC
Quick and dirty OCR (unrefined):
panic: udp6_output: sin6(8xfffff988937eb838)->sin6_addr is v4mapped which m-
ld have handled.
cpuid = 14
time = 153?955884
KDB: stack backtrace:
db_trace_self_mrapper() at db_trace_se1f_wrapper+8x2b1frame 8xfffffe8Bea7eb6
vpanic() at vpanic+8x1a31frame 8xfffffe889a?eb6b8
panic() at panic+8x43fframe 8xfffff9889679b718
udp6_send() at udp6_send+8xab5/frame 8xfffffeBBea?eb8d8
sosendfidgram() at sosend_dgram+8x350!frame 8xfffff988ea?eb938
sosend() at sosend+8x6d1frame 8xfffffeBBea?eb9?8
kern_sendit() at kern_sendit+8x2881frame 8xfffffeBBea?eb318
senditt) at sendit+8x1992frame 8xfffffe88ea?eb668
sgs_sendmsg() at SQS;sendmsg+8x61!frame 8xfffffe889a79bacfi
amd64_sgscall() at amd64_sgscall+8x2802frame 8xfffff988ea?ebbf8
fast_sgscall_common() at fast_sgscall_common+8x181/frame 8xfffffe88ea?ebbf8
sgscall (28, FreeBSD ELF64, sgs_sendmsg), rip = 8x881684f3a, rsp = ax?ff'
fo278, rbp = 8x?fffdfdfo2b8
KDB: enter: panic
I thread pid 61986 tid 182288
Stopped at kdb_enter+8x3b: movq $8,kdb_mhg
Comment 3 Bjoern A. Zeeb freebsd_committer 2018-09-27 07:51:41 UTC
Seems like my bug
Comment 4 Bjoern A. Zeeb freebsd_committer 2018-09-27 08:03:27 UTC
Did you get a core dump by any chance?
Comment 5 Jeremy Faulkner 2018-09-27 13:58:20 UTC
I setup a dumpdev and paniced the system. How best to make it available? I wouldn't want to waste anyone's time.
Comment 6 Jeremy Faulkner 2018-09-27 14:22:55 UTC
tar.gz of info.0 core.txt.0 vmcore.0

https://drive.google.com/open?id=1VkAkCxd4PPJoz3ptjrafz7vJzyoNnyZ-
Comment 7 Bjoern A. Zeeb freebsd_committer 2018-09-27 21:23:31 UTC
(In reply to Jeremy Faulkner from comment #6)

thanks;  could you also please tgz /boot/kernel and put it there as well?
Comment 8 Jeremy Faulkner 2018-09-27 21:47:54 UTC
FreeBSD constans 12.0-ALPHA7 FreeBSD 12.0-ALPHA7 r338924 GENERIC  amd64

https://drive.google.com/open?id=1TQiELc1Q8vR7bjIDVdnPiwTLKDQYxWPF
Comment 9 Bjoern A. Zeeb freebsd_committer 2018-09-27 22:20:00 UTC
Sorry;  kgdb is still acting up here with an internal assertion and other things; not sure what's going on there.

Can you investigate the core file (assuming you have not reinstalled the kernel); might have to install the gdb-811 or gdb-82 port:

kgdb /boot/kernel/kernel vmcore.0

select the frame for udp6_send() using frame #n

and then print out the the following arguments (the full structures not just the addresses):
     - so
     - so->so_pcb
     - addr


and paste the results in here?
Comment 10 Jeremy Faulkner 2018-09-27 23:12:13 UTC
constans# kgdb /boot/kernel/kernel /var/crash/vmcore.0
GNU gdb (GDB) 8.2 [GDB v8.2 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug..                      .done.
done.

Unread portion of the kernel message buffer:
panic: udp6_output: sin6(0xfffffe00febc3830)->sin6_addr is v4mapped which we should have handled.
cpuid = 1
time = 1538056540
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00febc3650
vpanic() at vpanic+0x1a3/frame 0xfffffe00febc36b0
panic() at panic+0x43/frame 0xfffffe00febc3710
udp6_send() at udp6_send+0xab5/frame 0xfffffe00febc38d0
sosend_dgram() at sosend_dgram+0x35c/frame 0xfffffe00febc3930
sosend() at sosend+0x6d/frame 0xfffffe00febc3970
kern_sendit() at kern_sendit+0x200/frame 0xfffffe00febc3a10
sendit() at sendit+0x19e/frame 0xfffffe00febc3a60
sys_sendmsg() at sys_sendmsg+0x61/frame 0xfffffe00febc3ac0
amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe00febc3bf0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00febc3bf0
--- syscall (28, FreeBSD ELF64, sys_sendmsg), rip = 0x801604f3a, rsp = 0x7fffdfdfc278, rbp = 0x7fffdfdfc2b0 ---
KDB: enter: panic

__curthread () at ./machine/pcpu.h:230
230             __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) frame #0xfffffe00febc38d0
Invalid character '#' in expression.
(kgdb) frame 0xfffffe00febc38d0
#0  0x0000000000000000 in ?? ()
(kgdb) set print pretty on
(kgdb) print so
No symbol "so" in current context.
(kgdb) print so->so_pcb
No symbol "so" in current context.
(kgdb) print addr
No symbol "addr" in current context.
(kgdb) bt
#0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:366
#2  0xffffffff8043b98b in db_dump (dummy=<optimized out>, dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>)
    at /usr/src/sys/ddb/db_command.c:574
#3  0xffffffff8043b759 in db_command (last_cmdp=<optimized out>, cmd_table=<optimized out>, dopager=<optimized out>)
    at /usr/src/sys/ddb/db_command.c:481
#4  0xffffffff8043b4f4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:534
#5  0xffffffff8043e6df in db_trap (type=<optimized out>, code=<optimized out>) at /usr/src/sys/ddb/db_main.c:252
#6  0xffffffff80bcc683 in kdb_trap (type=3, code=0, tf=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:693
#7  0xffffffff8105468f in trap (frame=0xfffffe00febc3580) at /usr/src/sys/amd64/amd64/trap.c:619
#8  <signal handler called>
#9  kdb_enter (why=0xffffffff812dcd1e "panic", msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:479
#10 0xffffffff80b84760 in vpanic (fmt=<optimized out>, ap=0xfffffe00febc36f0) at /usr/src/sys/kern/kern_shutdown.c:861
#11 0xffffffff80b84503 in panic (fmt=0xffffffff81e8e988 <cnputs_mtx> "X&*\201\377\377\377\377")
    at /usr/src/sys/kern/kern_shutdown.c:799
#12 0xffffffff80df8f05 in udp6_output (so=<optimized out>, flags_arg=<optimized out>, m=<optimized out>,
    addr6=<optimized out>, control=<optimized out>, td=<optimized out>) at /usr/src/sys/netinet6/udp6_usrreq.c:844
#13 udp6_send (so=0xfffff8027983d000, flags=17, m=0xfffff809a00e7100, addr=<optimized out>, control=0x0, td=0xfffff80206eae580)
    at /usr/src/sys/netinet6/udp6_usrreq.c:1343
#14 0xffffffff80c1d6ac in sosend_dgram (so=0xfffff8027983d000, addr=0xfffff809820794e0, uio=0xfffffe00febc3990, top=0x0,
    control=<optimized out>, flags=0, td=0xfffff80206eae580) at /usr/src/sys/kern/uipc_socket.c:1391
#15 0xffffffff80c1dded in sosend (so=0xffffffff81e8e988 <cnputs_mtx>, addr=0x80, uio=0xfffffe00febc3540, top=0x80,
    control=0x0, flags=464, td=0xfffff80206eae580) at /usr/src/sys/kern/uipc_socket.c:1623
#16 0xffffffff80c24c80 in kern_sendit (td=0xfffff80206eae580, s=<optimized out>, mp=0xfffffe00febc3a70, flags=0,
    control=<optimized out>, segflg=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:796
#17 0xffffffff80c24fee in sendit (td=0xfffff80206eae580, s=40, mp=0xfffffe00febc3a70, flags=0)
    at /usr/src/sys/kern/uipc_syscalls.c:721
#18 0xffffffff80c250c1 in sys_sendmsg (td=0xfffff80206eae580, uap=0xfffff80206eae940) at /usr/src/sys/kern/uipc_syscalls.c:897
#19 0xffffffff8105542c in syscallenter (td=0xfffff80206eae580) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#20 amd64_syscall (td=0xfffff80206eae580, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1050
#21 <signal handler called>
#22 0x0000000801604f3a in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffdfdfc278
(kgdb) frame #12
Invalid character '#' in expression.
(kgdb) frame 12
#12 0xffffffff80df8f05 in udp6_output (so=<optimized out>, flags_arg=<optimized out>, m=<optimized out>,
    addr6=<optimized out>, control=<optimized out>, td=<optimized out>) at /usr/src/sys/netinet6/udp6_usrreq.c:844
844                     KASSERT(!IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr),
(kgdb) print so
$1 = <optimized out>
(kgdb) print so->so_pcb
value has been optimized out
(kgdb) print addr
No symbol "addr" in current context.
(kgdb)
Comment 11 Andrey V. Elsukov freebsd_committer 2018-09-28 09:20:45 UTC
Probably you can trigger this assert when (inp->inp_flags & IN6P_IPV6_V6ONLY) != 0 and destination address is V4MAPPED.
Comment 12 Bjoern A. Zeeb freebsd_committer 2018-09-29 14:02:55 UTC
(In reply to Andrey V. Elsukov from comment #11)

Yupp.  That was the obvious one.  I kept wondering if we can hit this otherwise and came to the conclusion only if bind or connect do something wrong.  Would you agree?

Can you try this patch?

Index: sys/netinet6/udp6_usrreq.c
===================================================================
--- sys/netinet6/udp6_usrreq.c  (revision 338966)
+++ sys/netinet6/udp6_usrreq.c  (working copy)
@@ -784,8 +784,20 @@ udp6_output(struct socket *so, int flags_arg, stru
                        return ((*pru->pru_send)(so, flags_arg, m,
                            (struct sockaddr *)sin6, control, td));
                }
+       } else
+#endif
+       if (IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) {
+               /*
+                * Given this is either an IPv6-only socket or not INET is
+                * supported we will fail the send if the given destination
+                * address is a v4mapped address.
+                */
+               if (unlock_inp == UH_WLOCKED)
+                       INP_WUNLOCK(inp);
+               else
+                       INP_RUNLOCK(inp);
+               return (EINVAL);
        }
-#endif

        if (control) {
                if ((error = ip6_setpktopts(control, &opt,
Comment 13 Jeremy Faulkner 2018-09-29 22:52:15 UTC
I was able to download a torrent that had previously paniced the system. While it was downloading I noticed that it had some unusual behaviour, it would disconnect from all peers and the transfer speed would of course drop to 0. I assume this is the application handling the error being returned when it tries to send packets to an ipv6 address. The torrent I was using to test this had almost 9000 peers, to get the best odds of finding a torrent with IPv6 peers. I don't know for certain that it actually attempted to connect to any IPv6 peers though.
Comment 14 Bjoern A. Zeeb freebsd_committer 2018-09-29 23:23:04 UTC
(In reply to Jeremy Faulkner from comment #13)

So just for reference:  you might have an app that apparently uses v4mapped sockets without checking and properly dealing with errors.

What's your current setting of
    sysctl net.inet6.ip6.v6only

and do you set ipv6_ipv4mapping="YES" in rc.conf?

If the answer is 0 (and/or no), could you try either setting the sysctl to 1 (which is the same as setting the rc.conf variable to YES and letting the rc script run, e.g. with a boot) and see if the behaviour changes?
Comment 15 Bjoern A. Zeeb freebsd_committer 2018-09-29 23:25:00 UTC
(In reply to Bjoern A. Zeeb from comment #14)

and if you manually change the sysctl, you should re-start the application afterwards.
Comment 16 Jeremy Faulkner 2018-09-30 00:16:41 UTC
It was already set to 1 and I have no IPv6 address other than loopback. No ipv6 settings in rc.conf or sysctl.conf

$ sysctl -a |grep inet6
kern.features.inet6: 1
net.inet6.ip6.forwarding: 0
net.inet6.ip6.redirect: 1
net.inet6.ip6.hlim: 64
net.inet6.ip6.maxfragpackets: 31702
net.inet6.ip6.accept_rtadv: 0
net.inet6.ip6.log_interval: 5
net.inet6.ip6.hdrnestlimit: 15
net.inet6.ip6.dad_count: 1
net.inet6.ip6.auto_flowlabel: 1
net.inet6.ip6.defmcasthlim: 1
net.inet6.ip6.gifhlim: 30
net.inet6.ip6.kame_version: FreeBSD
net.inet6.ip6.use_deprecated: 1
net.inet6.ip6.rr_prune: 5
net.inet6.ip6.v6only: 1
net.inet6.ip6.use_tempaddr: 0
net.inet6.ip6.temppltime: 86400
net.inet6.ip6.tempvltime: 604800
net.inet6.ip6.auto_linklocal: 1
net.inet6.ip6.prefer_tempaddr: 0
net.inet6.ip6.use_defaultzone: 0
net.inet6.ip6.maxfrags: 31702
net.inet6.ip6.mcast_pmtu: 0
net.inet6.ip6.no_radr: 0
net.inet6.ip6.norbit_raif: 0
net.inet6.ip6.rfc6204w3: 0
net.inet6.ip6.intr_queue_maxlen: 256
net.inet6.ip6.maxfragsperpacket: 64
net.inet6.ip6.maxfragbucketsize: 61
net.inet6.ip6.deembed_scopeid: 1
net.inet6.ip6.dad_enhanced: 1
net.inet6.ip6.mcast.loop: 1
net.inet6.ip6.mcast.maxsocksrc: 128
net.inet6.ip6.mcast.maxgrpsrc: 512
net.inet6.ipsec6.def_policy: 1
net.inet6.ipsec6.esp_trans_deflev: 1
net.inet6.ipsec6.esp_net_deflev: 1
net.inet6.ipsec6.ah_trans_deflev: 1
net.inet6.ipsec6.ah_net_deflev: 1
net.inet6.ipsec6.ecn: 0
net.inet6.ipsec6.debug: 0
net.inet6.ipsec6.filtertunnel: 0
net.inet6.icmp6.rediraccept: 1
net.inet6.icmp6.redirtimeout: 600
net.inet6.icmp6.nd6_prune: 1
net.inet6.icmp6.nd6_delay: 5
net.inet6.icmp6.nd6_umaxtries: 3
net.inet6.icmp6.nd6_mmaxtries: 3
net.inet6.icmp6.nd6_useloopback: 1
net.inet6.icmp6.nodeinfo: 3
net.inet6.icmp6.errppslimit: 100
net.inet6.icmp6.nd6_maxnudhint: 0
net.inet6.icmp6.nd6_debug: 0
net.inet6.icmp6.nd6_maxqueuelen: 1
net.inet6.icmp6.nodeinfo_oldmcprefix: 1
net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0
net.inet6.icmp6.nd6_gctimer: 86400
net.inet6.mld.use_allow: 1
net.inet6.mld.v1enable: 1
net.inet6.mld.gsrdelay: 10


$ cat /etc/rc.conf
hostname="ryzen-current"
ifconfig_re0="DHCP"
sshd_enable="YES"
ntpdate_enable="YES"
#powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
iscsid_enable="YES"

dbus_enable="YES"
microcode_update_enable="YES"
vboxnet_enable="yes"
sndiod_enable="yes"


$ cat /etc/sysctl.conf
# $FreeBSD: head/sbin/sysctl/sysctl.conf 337624 2018-08-11 13:28:03Z brd $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
$ ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
	ether 88:d7:f6:7d:ea:68
	inet 192.168.1.114 netmask 0xffffff00 broadcast 192.168.1.255 
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
	inet 127.0.0.1 netmask 0xff000000 
	groups: lo 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
$
Comment 17 Bjoern A. Zeeb freebsd_committer 2018-09-30 06:14:48 UTC
(In reply to Jeremy Faulkner from comment #16)

Sorry my fault in the middle of the night, the rc.conf variable goes to YES or the sysctl goes to 0;  should be v6only=0.
Comment 18 Jeremy Faulkner 2018-09-30 20:27:51 UTC
v6only = 0 had no change to the app behaviour, it still doesn't panic so I'd say the patch probably works. I don't know if the unusual behaviour of the app warrants further investigation or not.
Comment 19 Bjoern A. Zeeb freebsd_committer 2018-10-01 10:16:34 UTC
Please note that the suggested patch needs a sin6 && check; otherwise another kernel panic might occur if you are actually using IPv6.
Comment 20 commit-hook freebsd_committer 2018-10-02 17:30:44 UTC
A commit references this bug:

Author: bz
Date: Tue Oct  2 17:29:56 UTC 2018
New revision: 339091
URL: https://svnweb.freebsd.org/changeset/base/339091

Log:
  After r338257 is was possible to trigger a KASSERT() in ud6_output()
  using an application trying to use a v4mapped destination address on a
  kernel without INET support or on a v6only socket.
  Catch this case and prevent the packet from going anywhere;
  else, without the KASSERT() armed, a v4mapped destination
  address might go out on the wire or other undefined behaviour
  might happen, while with the KASSERT() we panic.

  PR:		231728
  Reported by:	Jeremy Faulkner (gldisater gmail.com)
  Approved by:	re (kib)

Changes:
  head/sys/netinet6/udp6_usrreq.c
Comment 21 Bjoern A. Zeeb freebsd_committer 2018-10-02 17:31:13 UTC
Thanks a lot for reporting and testing.