The last few test runs have been failing as follows, with panics in strncmp() managing (what I assume is epair or tun) interfaces via the ioctl handler. There are some questionable LoR messages also printed out in the pf code about lock holding, but this is likely not the root cause. From https://ci.freebsd.org/job/FreeBSD-head-amd64-test/11014/consoleText : sys/netpfil/pf/pfsync:basic -> lock order reversal: (sleepable after non-sleepable) 1st 0xfffff8003a43f070 pfsync (pfsync) @ /usr/src/sys/netpfil/pf/if_pfsync.c:1402 2nd 0xffffffff820c48c0 in_multi_sx (in_multi_sx) @ /usr/src/sys/netinet/in_mcast.c:1251 stack backtrace: #0 0xffffffff80c47773 at witness_debugger+0x73 #1 0xffffffff80c474bd at witness_checkorder+0xa7d #2 0xffffffff80be7ed8 at _sx_xlock+0x68 #3 0xffffffff80d65271 at in_joingroup+0x31 #4 0xffffffff82839086 at pfsyncioctl+0x6e6 #5 0xffffffff80d60116 at in_control+0x376 #6 0xffffffff80ce168b at ifioctl+0x57b #7 0xffffffff80c4c6ba at kern_ioctl+0x28a #8 0xffffffff80c4c3bd at sys_ioctl+0x15d #9 0xffffffff810b2e16 at amd64_syscall+0x276 #10 0xffffffff8108b5fd at fast_syscall_common+0x101 passed [2.316s] sys/netpfil/pf/pfsync:defer -> passed [2.248s] sys/netpfil/pf/rdr:basic -> Apr 29 18:28:59 kernel: nd6_dad_timer: cancel DAD on epair3a because of ND6_IFF_IFDISABLED. passed [4.153s] sys/netpfil/pf/route_to:v4 -> passed [3.195s] sys/netpfil/pf/route_to:v6 -> Apr 29 18:29:07 kernel: nd6_dad_timer: called with non-tentative address fe80:7::39:3ff:fe4c:500a(epair4a) Apr 29 18:29:07 kernel: nd6_dad_timer: called with non-tentative address fe80:3::35:96ff:fe61:640b(epair3b) Apr 29 18:29:07 kernel: nd6_dad_timer: called with non-tentative address fe80:4::39:3ff:fe4c:500b(epair4b) Apr 29 18:29:07 kernel: nd6_dad_timer: called with non-tentative address fe80:5::35:96ff:fe61:640a(epair3a) passed [3.181s] sys/netpfil/pf/set_skip:set_skip_group -> passed [0.097s] sys/netpfil/pf/set_skip:set_skip_group_lo -> passed [0.113s] sys/netpfil/pf/set_tos:v4 -> passed [8.702s] sys/netpfil/pf/synproxy:synproxy -> passed [0.161s] Fatal trap 9: general protection fault while in kernel mode cpuid = 0; apic id = 00 instruction pointer = 0x20:0xffffffff80ccb525 stack pointer = 0x28:0xfffffe0030ec9740 frame pointer = 0x28:0xfffffe0030ec9740 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 69669 (ifconfig) trap number = 9 panic: general protection fault cpuid = 0 time = 1556562559 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0030ec9450 vpanic() at vpanic+0x19d/frame 0xfffffe0030ec94a0 panic() at panic+0x43/frame 0xfffffe0030ec9500 trap_fatal() at trap_fatal+0x394/frame 0xfffffe0030ec9560 trap() at trap+0x6c/frame 0xfffffe0030ec9670 calltrap() at calltrap+0x8/frame 0xfffffe0030ec9670 --- trap 0x9, rip = 0xffffffff80ccb525, rsp = 0xfffffe0030ec9740, rbp = 0xfffffe0030ec9740 --- strncmp() at strncmp+0x15/frame 0xfffffe0030ec9740 ifunit_ref() at ifunit_ref+0x51/frame 0xfffffe0030ec9780 ifioctl() at ifioctl+0x508/frame 0xfffffe0030ec9850 kern_ioctl() at kern_ioctl+0x28a/frame 0xfffffe0030ec98c0 sys_ioctl() at sys_ioctl+0x15d/frame 0xfffffe0030ec9990 amd64_syscall() at amd64_syscall+0x276/frame 0xfffffe0030ec9ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0030ec9ab0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x80048531a, rsp = 0x7fffffffe458, rbp = 0x7fffffffe4c0 --- KDB: enter: panic [ thread pid 69669 tid 100143 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db:0:kdb.enter.panic> show pcpu cpuid = 0 dynamic pcpu = 0xb84800 curthread = 0xfffff80004f305a0: pid 69669 tid 100143 "ifconfig" curpcb = 0xfffffe0030ec9b80 fpcurthread = 0xfffff80004f305a0: pid 69669 "ifconfig" idlethread = 0xfffff80003272000: tid 100003 "idle: cpu0" curpmap = 0xfffff8003e6fb130 tssp = 0xffffffff821cd320 commontssp = 0xffffffff821cd320 rsp0 = 0xfffffe0030ec9b80 gs32p = 0xffffffff821d3f58 ldt = 0xffffffff821d3f98 tss = 0xffffffff821d3f88 tlb gen = 455364 curvnet = 0xfffff8000307aec0 spin locks held: db:0:kdb.enter.panic> alltrace Tracing command ifconfig pid 69669 tid 100143 td 0xfffff80004f305a0 (CPU 0) kdb_enter() at kdb_enter+0x3b/frame 0xfffffe0030ec9450 vpanic() at vpanic+0x1ba/frame 0xfffffe0030ec94a0 panic() at panic+0x43/frame 0xfffffe0030ec9500 trap_fatal() at trap_fatal+0x394/frame 0xfffffe0030ec9560 trap() at trap+0x6c/frame 0xfffffe0030ec9670 calltrap() at calltrap+0x8/frame 0xfffffe0030ec9670 --- trap 0x9, rip = 0xffffffff80ccb525, rsp = 0xfffffe0030ec9740, rbp = 0xfffffe0030ec9740 --- strncmp() at strncmp+0x15/frame 0xfffffe0030ec9740 ifunit_ref() at ifunit_ref+0x51/frame 0xfffffe0030ec9780 ifioctl() at ifioctl+0x508/frame 0xfffffe0030ec9850 kern_ioctl() at kern_ioctl+0x28a/frame 0xfffffe0030ec98c0 sys_ioctl() at sys_ioctl+0x15d/frame 0xfffffe0030ec9990 amd64_syscall() at amd64_syscall+0x276/frame 0xfffffe0030ec9ab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0030ec9ab0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x80048531a, rsp = 0x7fffffffe458, rbp = 0x7fffffffe4c0 ---
I can't seem to reproduce this with current versions. It also doesn't appear to have affected any of the recent ci.freebsd.org test runs. I've reassigned this to net@, because it's not a pf problem, merely exposed by the pf tests.
I also don't see this panic after last update. Let's close this one and reopen when we see this panic again in the CI system.