Panic when attempting to use qbitorrent-nox to download a torrent. https://flic.kr/p/PiZmzy
Please take at least a minimal effort to transcribe what you see on the screen into a useful bug description. Developers' time is as valuable as yours.
Quick and dirty OCR (unrefined): panic: udp6_output: sin6(8xfffff988937eb838)->sin6_addr is v4mapped which m- ld have handled. cpuid = 14 time = 153?955884 KDB: stack backtrace: db_trace_self_mrapper() at db_trace_se1f_wrapper+8x2b1frame 8xfffffe8Bea7eb6 vpanic() at vpanic+8x1a31frame 8xfffffe889a?eb6b8 panic() at panic+8x43fframe 8xfffff9889679b718 udp6_send() at udp6_send+8xab5/frame 8xfffffeBBea?eb8d8 sosendfidgram() at sosend_dgram+8x350!frame 8xfffff988ea?eb938 sosend() at sosend+8x6d1frame 8xfffffeBBea?eb9?8 kern_sendit() at kern_sendit+8x2881frame 8xfffffeBBea?eb318 senditt) at sendit+8x1992frame 8xfffffe88ea?eb668 sgs_sendmsg() at SQS;sendmsg+8x61!frame 8xfffffe889a79bacfi amd64_sgscall() at amd64_sgscall+8x2802frame 8xfffff988ea?ebbf8 fast_sgscall_common() at fast_sgscall_common+8x181/frame 8xfffffe88ea?ebbf8 sgscall (28, FreeBSD ELF64, sgs_sendmsg), rip = 8x881684f3a, rsp = ax?ff' fo278, rbp = 8x?fffdfdfo2b8 KDB: enter: panic I thread pid 61986 tid 182288 Stopped at kdb_enter+8x3b: movq $8,kdb_mhg
Seems like my bug
Did you get a core dump by any chance?
I setup a dumpdev and paniced the system. How best to make it available? I wouldn't want to waste anyone's time.
tar.gz of info.0 core.txt.0 vmcore.0 https://drive.google.com/open?id=1VkAkCxd4PPJoz3ptjrafz7vJzyoNnyZ-
(In reply to Jeremy Faulkner from comment #6) thanks; could you also please tgz /boot/kernel and put it there as well?
FreeBSD constans 12.0-ALPHA7 FreeBSD 12.0-ALPHA7 r338924 GENERIC amd64 https://drive.google.com/open?id=1TQiELc1Q8vR7bjIDVdnPiwTLKDQYxWPF
Sorry; kgdb is still acting up here with an internal assertion and other things; not sure what's going on there. Can you investigate the core file (assuming you have not reinstalled the kernel); might have to install the gdb-811 or gdb-82 port: kgdb /boot/kernel/kernel vmcore.0 select the frame for udp6_send() using frame #n and then print out the the following arguments (the full structures not just the addresses): - so - so->so_pcb - addr and paste the results in here?
constans# kgdb /boot/kernel/kernel /var/crash/vmcore.0 GNU gdb (GDB) 8.2 [GDB v8.2 for FreeBSD] Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd12.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug.. .done. done. Unread portion of the kernel message buffer: panic: udp6_output: sin6(0xfffffe00febc3830)->sin6_addr is v4mapped which we should have handled. cpuid = 1 time = 1538056540 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00febc3650 vpanic() at vpanic+0x1a3/frame 0xfffffe00febc36b0 panic() at panic+0x43/frame 0xfffffe00febc3710 udp6_send() at udp6_send+0xab5/frame 0xfffffe00febc38d0 sosend_dgram() at sosend_dgram+0x35c/frame 0xfffffe00febc3930 sosend() at sosend+0x6d/frame 0xfffffe00febc3970 kern_sendit() at kern_sendit+0x200/frame 0xfffffe00febc3a10 sendit() at sendit+0x19e/frame 0xfffffe00febc3a60 sys_sendmsg() at sys_sendmsg+0x61/frame 0xfffffe00febc3ac0 amd64_syscall() at amd64_syscall+0x28c/frame 0xfffffe00febc3bf0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00febc3bf0 --- syscall (28, FreeBSD ELF64, sys_sendmsg), rip = 0x801604f3a, rsp = 0x7fffdfdfc278, rbp = 0x7fffdfdfc2b0 --- KDB: enter: panic __curthread () at ./machine/pcpu.h:230 230 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) frame #0xfffffe00febc38d0 Invalid character '#' in expression. (kgdb) frame 0xfffffe00febc38d0 #0 0x0000000000000000 in ?? () (kgdb) set print pretty on (kgdb) print so No symbol "so" in current context. (kgdb) print so->so_pcb No symbol "so" in current context. (kgdb) print addr No symbol "addr" in current context. (kgdb) bt #0 __curthread () at ./machine/pcpu.h:230 #1 doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:366 #2 0xffffffff8043b98b in db_dump (dummy=<optimized out>, dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>) at /usr/src/sys/ddb/db_command.c:574 #3 0xffffffff8043b759 in db_command (last_cmdp=<optimized out>, cmd_table=<optimized out>, dopager=<optimized out>) at /usr/src/sys/ddb/db_command.c:481 #4 0xffffffff8043b4f4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:534 #5 0xffffffff8043e6df in db_trap (type=<optimized out>, code=<optimized out>) at /usr/src/sys/ddb/db_main.c:252 #6 0xffffffff80bcc683 in kdb_trap (type=3, code=0, tf=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:693 #7 0xffffffff8105468f in trap (frame=0xfffffe00febc3580) at /usr/src/sys/amd64/amd64/trap.c:619 #8 <signal handler called> #9 kdb_enter (why=0xffffffff812dcd1e "panic", msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:479 #10 0xffffffff80b84760 in vpanic (fmt=<optimized out>, ap=0xfffffe00febc36f0) at /usr/src/sys/kern/kern_shutdown.c:861 #11 0xffffffff80b84503 in panic (fmt=0xffffffff81e8e988 <cnputs_mtx> "X&*\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:799 #12 0xffffffff80df8f05 in udp6_output (so=<optimized out>, flags_arg=<optimized out>, m=<optimized out>, addr6=<optimized out>, control=<optimized out>, td=<optimized out>) at /usr/src/sys/netinet6/udp6_usrreq.c:844 #13 udp6_send (so=0xfffff8027983d000, flags=17, m=0xfffff809a00e7100, addr=<optimized out>, control=0x0, td=0xfffff80206eae580) at /usr/src/sys/netinet6/udp6_usrreq.c:1343 #14 0xffffffff80c1d6ac in sosend_dgram (so=0xfffff8027983d000, addr=0xfffff809820794e0, uio=0xfffffe00febc3990, top=0x0, control=<optimized out>, flags=0, td=0xfffff80206eae580) at /usr/src/sys/kern/uipc_socket.c:1391 #15 0xffffffff80c1dded in sosend (so=0xffffffff81e8e988 <cnputs_mtx>, addr=0x80, uio=0xfffffe00febc3540, top=0x80, control=0x0, flags=464, td=0xfffff80206eae580) at /usr/src/sys/kern/uipc_socket.c:1623 #16 0xffffffff80c24c80 in kern_sendit (td=0xfffff80206eae580, s=<optimized out>, mp=0xfffffe00febc3a70, flags=0, control=<optimized out>, segflg=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:796 #17 0xffffffff80c24fee in sendit (td=0xfffff80206eae580, s=40, mp=0xfffffe00febc3a70, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:721 #18 0xffffffff80c250c1 in sys_sendmsg (td=0xfffff80206eae580, uap=0xfffff80206eae940) at /usr/src/sys/kern/uipc_syscalls.c:897 #19 0xffffffff8105542c in syscallenter (td=0xfffff80206eae580) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135 #20 amd64_syscall (td=0xfffff80206eae580, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1050 #21 <signal handler called> #22 0x0000000801604f3a in ?? () Backtrace stopped: Cannot access memory at address 0x7fffdfdfc278 (kgdb) frame #12 Invalid character '#' in expression. (kgdb) frame 12 #12 0xffffffff80df8f05 in udp6_output (so=<optimized out>, flags_arg=<optimized out>, m=<optimized out>, addr6=<optimized out>, control=<optimized out>, td=<optimized out>) at /usr/src/sys/netinet6/udp6_usrreq.c:844 844 KASSERT(!IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr), (kgdb) print so $1 = <optimized out> (kgdb) print so->so_pcb value has been optimized out (kgdb) print addr No symbol "addr" in current context. (kgdb)
Probably you can trigger this assert when (inp->inp_flags & IN6P_IPV6_V6ONLY) != 0 and destination address is V4MAPPED.
(In reply to Andrey V. Elsukov from comment #11) Yupp. That was the obvious one. I kept wondering if we can hit this otherwise and came to the conclusion only if bind or connect do something wrong. Would you agree? Can you try this patch? Index: sys/netinet6/udp6_usrreq.c =================================================================== --- sys/netinet6/udp6_usrreq.c (revision 338966) +++ sys/netinet6/udp6_usrreq.c (working copy) @@ -784,8 +784,20 @@ udp6_output(struct socket *so, int flags_arg, stru return ((*pru->pru_send)(so, flags_arg, m, (struct sockaddr *)sin6, control, td)); } + } else +#endif + if (IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) { + /* + * Given this is either an IPv6-only socket or not INET is + * supported we will fail the send if the given destination + * address is a v4mapped address. + */ + if (unlock_inp == UH_WLOCKED) + INP_WUNLOCK(inp); + else + INP_RUNLOCK(inp); + return (EINVAL); } -#endif if (control) { if ((error = ip6_setpktopts(control, &opt,
I was able to download a torrent that had previously paniced the system. While it was downloading I noticed that it had some unusual behaviour, it would disconnect from all peers and the transfer speed would of course drop to 0. I assume this is the application handling the error being returned when it tries to send packets to an ipv6 address. The torrent I was using to test this had almost 9000 peers, to get the best odds of finding a torrent with IPv6 peers. I don't know for certain that it actually attempted to connect to any IPv6 peers though.
(In reply to Jeremy Faulkner from comment #13) So just for reference: you might have an app that apparently uses v4mapped sockets without checking and properly dealing with errors. What's your current setting of sysctl net.inet6.ip6.v6only and do you set ipv6_ipv4mapping="YES" in rc.conf? If the answer is 0 (and/or no), could you try either setting the sysctl to 1 (which is the same as setting the rc.conf variable to YES and letting the rc script run, e.g. with a boot) and see if the behaviour changes?
(In reply to Bjoern A. Zeeb from comment #14) and if you manually change the sysctl, you should re-start the application afterwards.
It was already set to 1 and I have no IPv6 address other than loopback. No ipv6 settings in rc.conf or sysctl.conf $ sysctl -a |grep inet6 kern.features.inet6: 1 net.inet6.ip6.forwarding: 0 net.inet6.ip6.redirect: 1 net.inet6.ip6.hlim: 64 net.inet6.ip6.maxfragpackets: 31702 net.inet6.ip6.accept_rtadv: 0 net.inet6.ip6.log_interval: 5 net.inet6.ip6.hdrnestlimit: 15 net.inet6.ip6.dad_count: 1 net.inet6.ip6.auto_flowlabel: 1 net.inet6.ip6.defmcasthlim: 1 net.inet6.ip6.gifhlim: 30 net.inet6.ip6.kame_version: FreeBSD net.inet6.ip6.use_deprecated: 1 net.inet6.ip6.rr_prune: 5 net.inet6.ip6.v6only: 1 net.inet6.ip6.use_tempaddr: 0 net.inet6.ip6.temppltime: 86400 net.inet6.ip6.tempvltime: 604800 net.inet6.ip6.auto_linklocal: 1 net.inet6.ip6.prefer_tempaddr: 0 net.inet6.ip6.use_defaultzone: 0 net.inet6.ip6.maxfrags: 31702 net.inet6.ip6.mcast_pmtu: 0 net.inet6.ip6.no_radr: 0 net.inet6.ip6.norbit_raif: 0 net.inet6.ip6.rfc6204w3: 0 net.inet6.ip6.intr_queue_maxlen: 256 net.inet6.ip6.maxfragsperpacket: 64 net.inet6.ip6.maxfragbucketsize: 61 net.inet6.ip6.deembed_scopeid: 1 net.inet6.ip6.dad_enhanced: 1 net.inet6.ip6.mcast.loop: 1 net.inet6.ip6.mcast.maxsocksrc: 128 net.inet6.ip6.mcast.maxgrpsrc: 512 net.inet6.ipsec6.def_policy: 1 net.inet6.ipsec6.esp_trans_deflev: 1 net.inet6.ipsec6.esp_net_deflev: 1 net.inet6.ipsec6.ah_trans_deflev: 1 net.inet6.ipsec6.ah_net_deflev: 1 net.inet6.ipsec6.ecn: 0 net.inet6.ipsec6.debug: 0 net.inet6.ipsec6.filtertunnel: 0 net.inet6.icmp6.rediraccept: 1 net.inet6.icmp6.redirtimeout: 600 net.inet6.icmp6.nd6_prune: 1 net.inet6.icmp6.nd6_delay: 5 net.inet6.icmp6.nd6_umaxtries: 3 net.inet6.icmp6.nd6_mmaxtries: 3 net.inet6.icmp6.nd6_useloopback: 1 net.inet6.icmp6.nodeinfo: 3 net.inet6.icmp6.errppslimit: 100 net.inet6.icmp6.nd6_maxnudhint: 0 net.inet6.icmp6.nd6_debug: 0 net.inet6.icmp6.nd6_maxqueuelen: 1 net.inet6.icmp6.nodeinfo_oldmcprefix: 1 net.inet6.icmp6.nd6_onlink_ns_rfc4861: 0 net.inet6.icmp6.nd6_gctimer: 86400 net.inet6.mld.use_allow: 1 net.inet6.mld.v1enable: 1 net.inet6.mld.gsrdelay: 10 $ cat /etc/rc.conf hostname="ryzen-current" ifconfig_re0="DHCP" sshd_enable="YES" ntpdate_enable="YES" #powerd_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="AUTO" zfs_enable="YES" iscsid_enable="YES" dbus_enable="YES" microcode_update_enable="YES" vboxnet_enable="yes" sndiod_enable="yes" $ cat /etc/sysctl.conf # $FreeBSD: head/sbin/sysctl/sysctl.conf 337624 2018-08-11 13:28:03Z brd $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Uncomment this to prevent users from seeing information about processes that # are being run under another UID. #security.bsd.see_other_uids=0 $ ifconfig re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 88:d7:f6:7d:ea:68 inet 192.168.1.114 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> $
(In reply to Jeremy Faulkner from comment #16) Sorry my fault in the middle of the night, the rc.conf variable goes to YES or the sysctl goes to 0; should be v6only=0.
v6only = 0 had no change to the app behaviour, it still doesn't panic so I'd say the patch probably works. I don't know if the unusual behaviour of the app warrants further investigation or not.
Please note that the suggested patch needs a sin6 && check; otherwise another kernel panic might occur if you are actually using IPv6.
A commit references this bug: Author: bz Date: Tue Oct 2 17:29:56 UTC 2018 New revision: 339091 URL: https://svnweb.freebsd.org/changeset/base/339091 Log: After r338257 is was possible to trigger a KASSERT() in ud6_output() using an application trying to use a v4mapped destination address on a kernel without INET support or on a v6only socket. Catch this case and prevent the packet from going anywhere; else, without the KASSERT() armed, a v4mapped destination address might go out on the wire or other undefined behaviour might happen, while with the KASSERT() we panic. PR: 231728 Reported by: Jeremy Faulkner (gldisater gmail.com) Approved by: re (kib) Changes: head/sys/netinet6/udp6_usrreq.c
Thanks a lot for reporting and testing.