Hello, The following panic occurs on an INVARIANTS kernel, a reproducer is attached at the end of this message. The problem started to happen quite some months ago on 14-CURRENT while using the net-p2p/rtorrent application. I believe the reproducer to be similar to what rtorrent does when it connects to an udp tracker, .e.g. by using this magnet link: magnet:?xt=urn:btih:a9164e99d5181cfef0c23c209334103619080908&dn=debian-12.1.0-amd64-netinst.iso&tr=udp%3A%2F%2Fbttracker.debian.org%3A6969%2Fannounce From my very incomplete understanding, the in_pcb_lport_dest function isn't propery setting the laddr in case of ipv4-mapped ipv6 sendto address due to the following check on line 758 of sys/netinet/in_pcb.c: laddr.s_addr = INADDR_ANY; /* used by INET6+INET below too */ if ((inp->inp_vflag & (INP_IPV4|INP_IPV6)) == INP_IPV4) { if (lsa != NULL) laddr = ((struct sockaddr_in *)lsa)->sin_addr; In this case, the inp_vlags is equals to INP_IPV4|INP_IPV6|INP_IPV6PROTO (see below), however the ipv4 code is later followed on, hence the assertion in laddr.s_addr != INADDR_ANY in_pcblookup_hash_locked fails. Please let me know if any more information is needed. (kgdb) bt #0 kdb_enter (why=<optimized out>, msg=<optimized out>) at /usr/home/benja/src/freebsd/src/sys/kern/subr_kdb.c:556 #1 0xffffffff80b4f6c3 in vpanic (fmt=0xffffffff811c9e45 "%s: invalid local address", ap=ap@entry=0xfffffe00695f9790) at /usr/home/benja/src/freebsd/src/sys/kern/kern_shutdown.c:958 #2 0xffffffff80b4f4a3 in panic (fmt=0xffffffff8196c800 <cnputs_mtx> "G\246\024\201\377\377\377\377") at /usr/home/benja/src/freebsd/src/sys/kern/kern_shutdown.c:894 #3 0xffffffff80d22c16 in in_pcblookup_hash_locked (pcbinfo=pcbinfo@entry=0xfffffe000a362dd0, faddr=..., fport_arg=2166026500, fport_arg@entry=256, laddr=..., lport_arg=lport_arg@entry=22652, lookupflags=16, numa_domain=255 '\377') at /usr/home/benja/src/freebsd/src/sys/netinet/in_pcb.c:2350 #4 0xffffffff80d228d6 in in_pcb_lport_dest (inp=inp@entry=0xfffff800077ffc40, lsa=lsa@entry=0xfffffe00695f9920, lportp=lportp@entry=0xfffffe00695f98fe, fsa=fsa@entry=0xfffffe00695f9910, fport=fport@entry=256, cred=0xfffff800074b3400, lookupflags=1) at /usr/home/benja/src/freebsd/src/sys/netinet/in_pcb.c:794 #5 0xffffffff80d2339b in in_pcbconnect_setup (inp=inp@entry=0xfffff800077ffc40, sin=sin@entry=0xfffffe00695f9b78, laddrp=laddrp@entry=0xfffffe00695f9a00, lportp=lportp@entry=0xfffffe00695f9a1c, faddrp=faddrp@entry=0xfffffe00695f9998, fportp=fportp@entry=0xfffffe00695f9a0e, cred=0xfffff800074b3400) at /usr/home/benja/src/freebsd/src/sys/netinet/in_pcb.c:1376 #6 0xffffffff80d63392 in udp_send (so=so@entry=0xfffff800076813c0, flags=flags@entry=16, m=m@entry=0xfffff80007bfd900, addr=addr@entry=0xfffffe00695f9b78, control=<optimized out>, control@entry=0x0, td=td@entry=0xfffffe0072393740) at /usr/home/benja/src/freebsd/src/sys/netinet/udp_usrreq.c:1253 #7 0xffffffff80d9fba2 in udp6_send (so=0xfffff800076813c0, flags_arg=0, m=0xfffff80007bfd900, addr6=<optimized out>, control=0x0, td=0xfffffe0072393740) at /usr/home/benja/src/freebsd/src/sys/netinet6/udp6_usrreq.c:757 #8 0xffffffff80c00221 in sosend_dgram (so=0xfffff800076813c0, addr=0xfffff8000377bd80, uio=<optimized out>, top=<optimized out>, control=0x0, flags=0, td=0xfffffe0072393740) at /usr/home/benja/src/freebsd/src/sys/kern/uipc_socket.c:1553 #9 0xffffffff80c00e19 in sousrsend (so=0xffffffff8196c800 <cnputs_mtx>, so@entry=0xfffff800076813c0, addr=0x80, uio=0xffffffff811af104, uio@entry=0xfffffe00695f9ce8, control=0x10, control@entry=0x0, flags=flags@entry=0, userproc=userproc@entry=0x0) at /usr/home/benja/src/freebsd/src/sys/kern/uipc_socket.c:1892 #10 0xffffffff80c076e0 in kern_sendit (td=td@entry=0xfffffe0072393740, s=3, mp=mp@entry=0xfffffe00695f9dc0, flags=0, control=0x0, segflg=segflg@entry=UIO_USERSPACE) at /usr/home/benja/src/freebsd/src/sys/kern/uipc_syscalls.c:789 #11 0xffffffff80c078e7 in sendit (td=0xfffffe0072393740, s=128, mp=mp@entry=0xfffffe00695f9dc0, flags=-2128530671) at /usr/home/benja/src/freebsd/src/sys/kern/uipc_syscalls.c:714 #12 0xffffffff80c0781d in sys_sendto (td=0xffffffff8196c800 <cnputs_mtx>, uap=<optimized out>) at /usr/home/benja/src/freebsd/src/sys/kern/uipc_syscalls.c:820 #13 0xffffffff8104e67f in syscallenter (td=0xfffffe0072393740) at /usr/home/benja/src/freebsd/src/sys/amd64/amd64/../../kern/subr_syscall.c:187 #14 amd64_syscall (td=0xfffffe0072393740, traced=0) at /usr/home/benja/src/freebsd/src/sys/amd64/amd64/trap.c:1194 #15 <signal handler called> #16 0x000000082302e98a in ?? () #17 0x000000000020183f in ?? () #18 0x0000185a00000001 in ?? () #19 0x0000000001001c00 in ?? () #20 0x0000000000000000 in ?? () (kgdb) frame 4 #4 0xffffffff80d228d6 in in_pcb_lport_dest (inp=inp@entry=0xfffff800077ffc40, lsa=lsa@entry=0xfffffe00695f9920, lportp=lportp@entry=0xfffffe00695f98fe, fsa=fsa@entry=0xfffffe00695f9910, fport=fport@entry=256, cred=0xfffff800074b3400, lookupflags=1) at /usr/home/benja/src/freebsd/src/sys/netinet/in_pcb.c:794 794 tmpinp = in_pcblookup_hash_locked(pcbinfo, (kgdb) p/x inp->inp_vflag $1 = 0x7 (kgdb) p/x ((struct sockaddr_in *)lsa)->sin_addr $3 = {s_addr = 0x100007f} The reproducer: #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <arpa/inet.h> int main() { int optval=0; int sock; struct sockaddr_in6 sa6 = { 0 }; sa6.sin6_family = AF_INET6; sa6.sin6_port = htons(1); inet_pton(AF_INET6, "::ffff:127.0.0.1", &(sa6.sin6_addr)); sock = socket (PF_INET6, SOCK_DGRAM, 0); setsockopt (sock, IPPROTO_IPV6, IPV6_V6ONLY, &optval, sizeof(optval)); sendto (sock, "abcd", 4, 0, (struct sockaddr *)&sa6, sizeof(sa6)); return 0; }
This problem is also found by syzkaller in https://syzkaller.appspot.com/bug?extid=81ccc423a2737ed031ac.
(In reply to Michael Tuexen from comment #1) if that is that the case, so we have to cc markj@? ;)
(In reply to Mina Galić from comment #2) I don't think he is responsible for all bug found by syzkaller... You can add him, but I think glebius@ did a lot of improvements in the inp area and he is on the bug.
I can take a look. I guess, https://syzkaller.appspot.com/bug?extid=c8e3dac881bba85bc029 is also related. Both seem to be a consequence of not handling mapped addresses correctly.
^Triage: former assignee net@ added to the CC list.
Is this the same problem that I tried to fix in https://reviews.freebsd.org/D39215? I lost track of that revision, sorry. Note that there's a test case in D39216.
(In reply to Mark Johnston from comment #6) Hi Mark, thanks for the pointers, wasn't aware of it. I'll plan to work on the PR this Friday and will let you know. Best regards Michael
(In reply to Mark Johnston from comment #6) I tested your patch in review D39215 and it resolves the issues. However, in other places inp_vflag is modified when using mapped addresses. I have done this dance in review D42031 and it resolves the panic, too. This seems to be consistent. I'm not sure which one is the correct approach. Is the intention of review D39215 than such a inp_vflag is not necessary anymore and can be removed from the other places?
(In reply to Mark Johnston from comment #6) Hi, yes it does seem to be that same issue. (In reply to Michael Tuexen from comment #8) My 2 cents: the version flag is indeed tricky because - as noted by Mark in its revision - an AF_INET6 UDP socket can transition back and forth between v4 and v6 (either by using connect() and/or sendto). I'm not sure either that getting rid of it is the right approach because the code ends up having to pass around an extra flag argument all over the place. But there are also some unclear locking rules, as stated in the comment around the in_pcb stuff, which makes the whole concept far from trivial for me to understand :) Nonetheless, I made a patch in a way for me to have something working. But it does seem all very hacky and ugly to carry an argument for "it is actually a v4-mapped" flag to all callers, and callers of callers, of the in_pcb_lport_dest. Also I did not completely understood the implication w.r.t. the handling of wildcard addresses. And possible concurrency issues are likely not addressed. Anyway, that might be of interest to you. Side note: it is trivial to trigger the bug using "sysctl net.inet6.ip6.v6only=0; drill @::ffff:8.8.8.8 freebsd.org"
Created attachment 245375 [details] Adds a flag argument to in_pcb_lport_dest to support v4-mapped
What about adding extra inp_vflag for mapped pcbs? So that in_pcb.c code can tell regular IPv4 inpcb from mapped one?
(In reply to Gleb Smirnoff from comment #11) Why? Don't we know the state right now from inp_vflag? We just adapt the value to the usage of the inp.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=abca3ae7734f664ee9c5edc7a9d3a17e29180bdb commit abca3ae7734f664ee9c5edc7a9d3a17e29180bdb Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2023-10-07 13:56:00 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2023-10-07 13:56:00 +0000 udp: fix sending of IPv4-mapped addresses The inp_vflags field must be adjusted during the call of in_pcbbind_setup(). This is consistent with the other places in the code, but not elegant at all. PR: 274009 Reported by: syzbot+81ccc423a2737ed031ac@syzkaller.appspotmail.com Reported by: syzbot+c8e3dac881bba85bc029@syzkaller.appspotmail.com Reviewed by: markj, rrs, rscheff MFC after: 3 days Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D42031 sys/netinet/udp_usrreq.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8209af15a6994d52276be05380e0c0b7ae117cea commit 8209af15a6994d52276be05380e0c0b7ae117cea Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2023-10-07 13:56:00 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2023-10-13 20:13:44 +0000 udp: fix sending of IPv4-mapped addresses The inp_vflags field must be adjusted during the call of in_pcbbind_setup(). This is consistent with the other places in the code, but not elegant at all. PR: 274009 Reported by: syzbot+81ccc423a2737ed031ac@syzkaller.appspotmail.com Reported by: syzbot+c8e3dac881bba85bc029@syzkaller.appspotmail.com Reviewed by: markj, rrs, rscheff Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D42031 (cherry picked from commit abca3ae7734f664ee9c5edc7a9d3a17e29180bdb) sys/netinet/udp_usrreq.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
A commit in branch releng/14.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f28965bcaa1f7f1bd3d855f5c025daeb445d07a7 commit f28965bcaa1f7f1bd3d855f5c025daeb445d07a7 Author: Michael Tuexen <tuexen@FreeBSD.org> AuthorDate: 2023-10-07 13:56:00 +0000 Commit: Michael Tuexen <tuexen@FreeBSD.org> CommitDate: 2023-10-13 21:29:47 +0000 udp: fix sending of IPv4-mapped addresses The inp_vflags field must be adjusted during the call of in_pcbbind_setup(). This is consistent with the other places in the code, but not elegant at all. PR: 274009 Approved by: re (gjb) Reported by: syzbot+81ccc423a2737ed031ac@syzkaller.appspotmail.com Reported by: syzbot+c8e3dac881bba85bc029@syzkaller.appspotmail.com Reviewed by: markj, rrs, rscheff Sponsored by: Netflix, Inc. Differential Revision: https://reviews.freebsd.org/D42031 (cherry picked from commit abca3ae7734f664ee9c5edc7a9d3a17e29180bdb) (cherry picked from commit 8209af15a6994d52276be05380e0c0b7ae117cea) sys/netinet/udp_usrreq.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)