I'm on CURRENT on 27c6009e7237597a502521e0255a74164ff6bea7 with our internal irdma driver (newer than in vanilla FreeBSD, but with no other modifications). I do (yes, wrong IP in rping): ifconfig ice0 100.0.0.117/24 rping -svda 100.0.0.116 Instead of the usual error, there is an instant panic after: created cm_id 0x128dd9c99000 The panic is: (kgdb) #0 __curthread () at /usr/237bd1/sys/amd64/include/pcpu_aux.h:59 #1 dump_savectx () at /usr/237bd1/sys/kern/kern_shutdown.c:405 #2 0xffffffff80bf0495 in dumpsys (di=0x0) at /usr/237bd1/sys/x86/include/dump.h:87 #3 doadump (textdump=1) at /usr/237bd1/sys/kern/kern_shutdown.c:434 #4 kern_reboot (howto=260) at /usr/237bd1/sys/kern/kern_shutdown.c:541 #5 0xffffffff80bf08de in vpanic (fmt=<optimized out>, ap=ap@entry=0xfffffe029d759a50) at /usr/237bd1/sys/kern/kern_shutdown.c:979 #6 0xffffffff80bf0643 in panic (fmt=<unavailable>) at /usr/237bd1/sys/kern/kern_shutdown.c:903 #7 0xffffffff81160a29 in trap_fatal (frame=0xfffffe029d759b40, eva=40) at /usr/237bd1/sys/amd64/amd64/trap.c:955 #8 0xffffffff81160adb in trap_pfault (frame=0xfffffe029d759b40, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /usr/237bd1/sys/amd64/amd64/trap.c:763 #9 <signal handler called> #10 0xffffffff80e8de19 in cma_loopback_addr (addr=0xfffffe029d759c78) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_cma.c:1060 #11 cma_any_addr (addr=0xfffffe029d759c78) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_cma.c:1075 #12 rdma_bind_addr (id=0xfffff80156479c00, addr=addr@entry=0xfffffe029d759c78) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_cma.c:3459 #13 0xffffffff80eb179f in ucma_bind_ip (linux_file=0xfffff80127d96380, inbuf=<optimized out>, in_len=<optimized out>, out_len=<optimized out>) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_ucma.c:650 #14 0xffffffff80eb0c82 in ucma_write (filp=<optimized out>, buf=<optimized out>, len=48, pos=<optimized out>) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_ucma.c:1635 #15 0xffffffff80e66ea7 in linux_file_write (file=0xfffff801220afaf0, uio=0xfffffe029d759da0, active_cred=<optimized out>, flags=<optimized out>, td=<optimized out>) at /usr/237bd1/sys/compat/linuxkpi/common/src/linux_compat.c:1592 #16 0xffffffff80c67001 in fo_write (fp=0xfffff801220afaf0, uio=0xfffffe029d759da0, active_cred=0x74000064061c0210, td=0xfffffe00f35cd720, flags=<optimized out>) at /usr/237bd1/sys/sys/file.h:349 #17 dofilewrite (td=td@entry=0xfffffe00f35cd720, fd=fd@entry=3, fp=0xfffff801220afaf0, auio=auio@entry=0xfffffe029d759da0, offset=offset@entry=-1, flags=flags@entry=0) at /usr/237bd1/sys/kern/sys_generic.c:565 #18 0xffffffff80c66b30 in kern_writev (td=0xfffffe00f35cd720, fd=3, auio=0xfffffe029d759da0) at /usr/237bd1/sys/kern/sys_generic.c:492 #19 sys_write (td=0xfffffe00f35cd720, uap=<optimized out>) at /usr/237bd1/sys/kern/sys_generic.c:407 #20 0xffffffff811613de in syscallenter (td=<optimized out>) at /usr/237bd1/sys/amd64/amd64/../../kern/subr_syscall.c:189 #21 amd64_syscall (td=0xfffffe00f35cd720, traced=0) at /usr/237bd1/sys/amd64/amd64/trap.c:1200 #22 <signal handler called> #23 0x0000128dd91f413a in ?? () which suggests an issue in the OFED code.
Can you go to frame #10 and dump all the variables, including: print *addr --HPS
Like this: frame 10 info local print /x *(struct sockaddr_in6 *)addr --HPS
I think I can help; I discovered this already: (we had a little Twitter thread) https://twitter.com/kaevans91/status/1557167097813278722 The tl;dr is that the V_in_loopback_mask variable (that gets accessed from within cma_loopback_addr via the ipv4 loopback check) got added in a previous commit but that RDMA code doesn't have the currvnet set, so the vnet that gets checked in V_in_loopback_mask is NULL and the kernel panics. IPv6 code in RDMA doesn't seem to use anything VIMAGE-related so that isn't affected by this issue; you can IPv6 ping as a workaround for now. But krping is IPv4 only, so that's currently unusable. I think there are other IPv4 places in OFED that also need fixing, like validate_ipv4_net_dev(). I tried looking around, but never got around to properly finishing a patch that I was working on.
(In reply to Eric Joyner from comment #3) > The tl;dr is that the V_in_loopback_mask variable (that gets accessed from within > cma_loopback_addr via the ipv4 loopback check) got added in a previous commit > but that RDMA code doesn't have the currvnet set, so the vnet that gets checked > in V_in_loopback_mask is NULL and the kernel panics. Ouch. I hadn't anticipated this situation. Shouldn't the default vnet be set if nothing has changed it?
(In reply to Mike Karels from comment #4) That's what I would assume! But I don't know how VIMAGE works, so I am not the right person to respond to this question.
(In reply to Mike Karels from comment #4) Negative; it's NULL by default. Not sure the original rationale, but enforcing vnet context to be set reaching into this kind of stuff is somewhat useful for catching bugs that would be harder to track down if we assume vnet0.
I wonder if this mask needs to be made compile time instead! +#define IN_LOOPBACK(i) \ + (((in_addr_t)(i) & V_in_loopback_mask) == 0x7f000000) +#define IN_LOOPBACK_MASK_DFLT 0xff000000 --HPS
Mike: ibcore uses this variable before it knows the actual network interface, so there is no vnet to set, but the default ...
(In reply to Hans Petter Selasky from comment #7 and #8) > I wonder if this mask needs to be made compile time instead! That would essentially undo the change; the whole point is to be able to change it at runtime. Otherwise, changes require a custom kernel, and obviously cannot be done per vnet. > ibcore uses this variable before it knows the actual network interface, so there is no vnet to set, but the default ... In that case, the default is probably appropriate. It seems strange to do any processing with no context or interface though.
I think debugnet w/ INVARIANTS enabled is also broken. Cannot see that debugnet cares about VNET's sys/net/debugnet_inet.c: if ((IN_LOOPBACK(ntohl(ip->ip_dst.s_addr)) || sys/net/debugnet_inet.c: IN_LOOPBACK(ntohl(ip->ip_src.s_addr))) && --HPS
Created attachment 236257 [details] Patch to try Hi, Can you test / review this patch? --HPS
This patch seems to fix the reported panic, although we still get the panic in our copy of irdma driver, but this is likely a different issue.
Do you have a backtrace of the irdma panic?
Sure: panic: page fault cpuid = 48 time = 1661961498 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe02ad5766f0 vpanic() at vpanic+0x151/frame 0xfffffe02ad576740 panic() at panic+0x43/frame 0xfffffe02ad5767a0 trap_fatal() at trap_fatal+0x409/frame 0xfffffe02ad576800 trap_pfault() at trap_pfault+0xab/frame 0xfffffe02ad576860 calltrap() at calltrap+0x8/frame 0xfffffe02ad576860 --- trap 0xc, rip = 0xffffffff80ed3cd4, rsp = 0xfffffe02ad576930, rbp = 0xfffffe02ad576930 --- irdma_ipv4_is_lpb() at irdma_ipv4_is_lpb+0x14/frame 0xfffffe02ad576930 irdma_create_ah() at irdma_create_ah+0x36d/frame 0xfffffe02ad576a10 _ib_create_ah() at _ib_create_ah+0xc1/frame 0xfffffe02ad576a50 cm_alloc_msg() at cm_alloc_msg+0x104/frame 0xfffffe02ad576a90 ib_send_cm_req() at ib_send_cm_req+0x24c/frame 0xfffffe02ad576ad0 rdma_connect() at rdma_connect+0x62d/frame 0xfffffe02ad576b50 ucma_connect() at ucma_connect+0x101/frame 0xfffffe02ad576cb0 ucma_write() at ucma_write+0x102/frame 0xfffffe02ad576cf0 linux_file_write() at linux_file_write+0x107/frame 0xfffffe02ad576d40 dofilewrite() at dofilewrite+0x81/frame 0xfffffe02ad576d90 sys_write() at sys_write+0xc0/frame 0xfffffe02ad576e00 amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe02ad576f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe02ad576f30 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x374528d613a, rsp = 0x3744c4de968, rbp = 0x3744c4de9a0 --- Uptime: 4m11s Dumping 5202 out of 130936 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /usr/237bd1/sys/amd64/include/pcpu_aux.h:59 59 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) #0 __curthread () at /usr/237bd1/sys/amd64/include/pcpu_aux.h:59 #1 dump_savectx () at /usr/237bd1/sys/kern/kern_shutdown.c:405 #2 0xffffffff80bf0495 in dumpsys (di=0x0) at /usr/237bd1/sys/x86/include/dump.h:87 #3 doadump (textdump=1) at /usr/237bd1/sys/kern/kern_shutdown.c:434 #4 kern_reboot (howto=260) at /usr/237bd1/sys/kern/kern_shutdown.c:541 #5 0xffffffff80bf08de in vpanic (fmt=<optimized out>, ap=ap@entry=0xfffffe02ad576780) at /usr/237bd1/sys/kern/kern_shutdown.c:979 #6 0xffffffff80bf0643 in panic (fmt=<unavailable>) at /usr/237bd1/sys/kern/kern_shutdown.c:903 #7 0xffffffff8115fa29 in trap_fatal (frame=0xfffffe02ad576870, eva=40) at /usr/237bd1/sys/amd64/amd64/trap.c:955 #8 0xffffffff8115fadb in trap_pfault (frame=0xfffffe02ad576870, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /usr/237bd1/sys/amd64/amd64/trap.c:763 #9 <signal handler called> #10 0xffffffff80ed3cd4 in irdma_ipv4_is_lpb (loc_addr=1677721717, rem_addr=1677721717) at /usr/237bd1/sys/dev/irdma/irdma_cm.c:2060 #11 0xffffffff80ef084d in irdma_fill_ah_info (ah_info=0xfffff81143c20b20, sgid_addr=0xfffffe02ad576980, dgid_addr=0xfffffe02ad5769b8, dmac=0xfffffe02ad5769e0 "", sgid_attr=<optimized out>, net_type=<optimized out>) at /usr/237bd1/sys/dev/irdma/irdma_kcompat.c:344 #12 irdma_create_ah (ib_ah=0xfffff81143c20b00, attr=0xfffff81143c60ec8, flags=0, udata=0x0) at /usr/237bd1/sys/dev/irdma/irdma_kcompat.c:496 #13 0xffffffff80ec8781 in _ib_create_ah (pd=0xfffff810c1869ec0, ah_attr=0x64000075, ah_attr@entry=0xfffff81143c60ec8, flags=flags@entry=0, udata=udata@entry=0x0) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_verbs.c:380 #14 0xffffffff80ec86b9 in ib_create_ah (pd=0x64000075, ah_attr=0x64000075, ah_attr@entry=0xfffff81143c60ec8, flags=279, flags@entry=0) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_verbs.c:406 #15 0xffffffff80e83884 in cm_alloc_msg ( cm_id_priv=cm_id_priv@entry=0xfffff81143c60e00, msg=msg@entry=0xfffff81143c60ea0) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_cm.c:354 #16 0xffffffff80e8337c in ib_send_cm_req (cm_id=0xfffff81143c60e00, param=param@entry=0xfffffe02ad576ae0) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_cm.c:1386 #17 0xffffffff80e8ee1d in cma_connect_ib (id_priv=0xfffff81143c18c00, conn_param=0xfffffe02ad576b68) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_cma.c:3762 #18 rdma_connect (id=0xfffff81143c18c00, conn_param=conn_param@entry=0xfffffe02ad576b68) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_cma.c:3833 #19 0xffffffff80eb17a1 in ucma_connect (linux_file=<optimized out>, inbuf=<optimized out>, in_len=<optimized out>, out_len=<optimized out>) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_ucma.c:1053 #20 0xffffffff80eb0752 in ucma_write (filp=<optimized out>, buf=<optimized out>, len=288, pos=<optimized out>) at /usr/237bd1/sys/ofed/drivers/infiniband/core/ib_ucma.c:1635 #21 0xffffffff80e66ea7 in linux_file_write (file=0xfffff810ca58f410, uio=0xfffffe02ad576da0, active_cred=<optimized out>, flags=<optimized out>, td=<optimized out>) at /usr/237bd1/sys/compat/linuxkpi/common/src/linux_compat.c:1592 #22 0xffffffff80c67001 in fo_write (fp=0xfffff810ca58f410, uio=0xfffffe02ad576da0, active_cred=0x117, td=0xfffffe029cce6000, flags=<optimized out>) at /usr/237bd1/sys/sys/file.h:349 #23 dofilewrite (td=td@entry=0xfffffe029cce6000, fd=fd@entry=3, fp=0xfffff810ca58f410, auio=auio@entry=0xfffffe02ad576da0, offset=offset@entry=-1, flags=flags@entry=0) at /usr/237bd1/sys/kern/sys_generic.c:565 #24 0xffffffff80c66b30 in kern_writev (td=0xfffffe029cce6000, fd=3, auio=0xfffffe02ad576da0) at /usr/237bd1/sys/kern/sys_generic.c:492 #25 sys_write (td=0xfffffe029cce6000, uap=<optimized out>) at /usr/237bd1/sys/kern/sys_generic.c:407 #26 0xffffffff811603de in syscallenter (td=<optimized out>) at /usr/237bd1/sys/amd64/amd64/../../kern/subr_syscall.c:189 #27 amd64_syscall (td=0xfffffe029cce6000, traced=0) at /usr/237bd1/sys/amd64/amd64/trap.c:1200 #28 <signal handler called> #29 0x00000374528d613a in ?? () Backtrace stopped: Cannot access memory at address 0x3744c4de968
I believe my patch contains a fix for that! Did you rebuild the irdma driver? --HPS
Yes, this configuration involves built-in irdma: include GENERIC ident VALIDATION options MROUTING options DDB options KDB options KDB_UNATTENDED options KDB_TRACE # iWARP support options OFED options OFED_DEBUG_INIT options COMPAT_LINUXKPI options SDP options IPOIB_CM device netmap nodevice ixl device ice device ice_ddp device irdma
Can you add a printf in irdma_cm_create_ah() to see if the vnet I extract is NULL? You are sure you re-built the kernel, and that both patches were correctly applied? No .rej files? --HPS
(In reply to Hans Petter Selasky from comment #17) It now looks like: static int irdma_cm_create_ah(struct irdma_cm_node *cm_node, bool wait) { #ifdef VIMAGE struct rdma_cm_id *rdma_id = (struct rdma_cm_id *)cm_node->cm_id->context; struct vnet *vnet = rdma_id->route.addr.dev_addr.net; printf("%vnet address: p\n", vnet); #endif But there's nothing in the dmesg with that string. Isn't that function called later? I also recompiled the kernel to use ice and irdma as modules - same result. There are no .rej files, the patch applied cleanly.
Created attachment 236319 [details] Try this patch Try this patch instead (revert previous one). I see some missed cases VNET cases.
Ping - any updates? --HPS
Sorry, not yet, I'm busy with other work.
Could you test this today? --HPS
Created attachment 236554 [details] Try this patch (v2) Found a missed case. Can you test?
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=f50274674ebf3a51a77708f569d38a238cbad6ec commit f50274674ebf3a51a77708f569d38a238cbad6ec Author: Hans Petter Selasky <hselasky@FreeBSD.org> AuthorDate: 2022-08-31 10:37:28 +0000 Commit: Hans Petter Selasky <hselasky@FreeBSD.org> CommitDate: 2022-09-23 11:42:03 +0000 ibcore: The use of IN_LOOPBACK() now requires a valid VNET context. Make sure the VNET is set before using this macro. Fixes: efe58855f3ea2cfc24cb705aabce3bc0fe1fb6d5 PR: 266054 MFC after: 1 week Sponsored by: NVIDIA Networking sys/dev/irdma/fbsd_kcompat.c | 4 +- sys/dev/irdma/irdma_cm.c | 15 ++++-- sys/dev/irdma/irdma_cm.h | 2 +- sys/dev/irdma/irdma_kcompat.c | 9 ++-- sys/ofed/drivers/infiniband/core/ib_cma.c | 81 ++++++++++++++++++++----------- 5 files changed, 76 insertions(+), 35 deletions(-)
Please test!
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3b65af50fc2aefbb5950cc48af0de6753be69762 commit 3b65af50fc2aefbb5950cc48af0de6753be69762 Author: Hans Petter Selasky <hselasky@FreeBSD.org> AuthorDate: 2022-08-31 10:37:28 +0000 Commit: Eric Joyner <erj@FreeBSD.org> CommitDate: 2023-02-08 00:23:56 +0000 ibcore: The use of IN_LOOPBACK() now requires a valid VNET context. Make sure the VNET is set before using this macro. Fixes: efe58855f3ea2cfc24cb705aabce3bc0fe1fb6d5 PR: 266054 Sponsored by: NVIDIA Networking (cherry picked from commit f50274674ebf3a51a77708f569d38a238cbad6ec) sys/dev/irdma/fbsd_kcompat.c | 4 +- sys/dev/irdma/irdma_cm.c | 15 ++++-- sys/dev/irdma/irdma_cm.h | 2 +- sys/dev/irdma/irdma_kcompat.c | 9 ++-- sys/ofed/drivers/infiniband/core/ib_cma.c | 81 ++++++++++++++++++++----------- 5 files changed, 76 insertions(+), 35 deletions(-)
Assign to committer who resolved.