When upgrading from 13.1 to 13.2, I'm running into the panic below. The panic is triggered by running "fsck_ffs -p" on a filesystem via iSCSI where the iSCSI is protedted using IPSec (strongswan ESP:AES_GCM_16-128). Fatal trap 12: page fault while in kernel mode cpuid = 7; apic id = 07 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff810adf16 stack pointer = 0x28:0xfffffe0568cba730 frame pointer = 0x28:0xfffffe0568cba730 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (iscsitx) trap number = 12 panic: page fault cpuid = 7 time = 1684066170 KDB: stack backtrace: #0 0xffffffff80c53dc5 at kdb_backtrace+0x65 #1 0xffffffff80c06741 at vpanic+0x151 #2 0xffffffff80c065e3 at panic+0x43 #3 0xffffffff810b1fa7 at trap_fatal+0x387 #4 0xffffffff810b1fff at trap_pfault+0x4f #5 0xffffffff81088e78 at calltrap+0x8 #6 0xffffffff80c9cb87 at m_unshare+0x297 #7 0xffffffff8288e4b3 at esp_output+0x183 #8 0xffffffff8288af13 at ipsec4_perform_request+0x3b3 #9 0xffffffff8288b063 at ipsec4_common_output+0x83 #10 0xffffffff80e3970c at ipsec_kmod_output+0x2c #11 0xffffffff80dbcf84 at ip_output+0xb64 #12 0xffffffff80dd43af at tcp_output+0x1dbf #13 0xffffffff80de638d at tcp_usr_send+0x17d #14 0xffffffff80ca7807 at sosend_generic+0x617 #15 0xffffffff80ca7d20 at sosend+0x50 #16 0xffffffff828a0899 at icl_send_thread+0x499 #17 0xffffffff80bc2fce at fork_exit+0x7e Uptime: 7m7s Dumping 5022 out of 130720 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:396 #2 0xffffffff80c0630a in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:484 #3 0xffffffff80c067ae in vpanic (fmt=<optimized out>, ap=ap@entry=0xfffffe0568cba580) at /usr/src/sys/kern/kern_shutdown.c:923 #4 0xffffffff80c065e3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:847 #5 0xffffffff810b1fa7 in trap_fatal (frame=0xfffffe0568cba670, eva=0) at /usr/src/sys/amd64/amd64/trap.c:942 #6 0xffffffff810b1fff in trap_pfault (frame=0xfffffe0568cba670, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:761 #7 <signal handler called> #8 memcpy_erms () at /usr/src/sys/amd64/amd64/support.S:553 #9 0xffffffff80c9cb87 in m_unshare (m0=m0@entry=0xfffff8028a647b00, how=how@entry=1) at /usr/src/sys/kern/uipc_mbuf.c:2082 #10 0xffffffff8288e4b3 in esp_output (m=0xfffff801ea078800, sp=<optimized out>, sav=0xfffff801ea8d8c00, idx=0, skip=20, protoff=9) at /usr/src/sys/netipsec/xform_esp.c:770 #11 0xffffffff8288af13 in ipsec4_perform_request (m=0xfffff801ea078800, m@entry=0xfffff8028a647b00, sp=0x0, sp@entry=0xfffff801ea8d8800, inp=inp@entry=0xfffff8037760b5d0, idx=idx@entry=0) at /usr/src/sys/netipsec/ipsec_output.c:275 #12 0xffffffff8288b063 in ipsec4_process_packet (m=0xfffff8028a647b00, sp=0xfffff801ea8d8800, inp=0xfffff8037760b5d0) at /usr/src/sys/netipsec/ipsec_output.c:292 #13 ipsec4_common_output (m=0xfffff8028a647b00, inp=0xfffff8037760b5d0, forwarding=<optimized out>) at /usr/src/sys/netipsec/ipsec_output.c:340 #14 0xffffffff80e3970c in ipsec_kmod_output (sc=sc@entry=0xffffffff81d172a8 <ipv4_ipsec>, m=0xfffff801ea078800, inp=0x517, inp@entry=0xfffff8037760b5d0) at /usr/src/sys/netipsec/subr_ipsec.c:369 #15 0xffffffff80dbcf84 in ip_output (m=0x0, m@entry=0xfffff8028a647b00, opt=<optimized out>, ro=0xfffff8037760b760, flags=0, imo=imo@entry=0x0, inp=0x517) at /usr/src/sys/netinet/ip_output.c:680 #16 0xffffffff80dd43af in tcp_output (tp=0xfffffe0568a0f000) at /usr/src/sys/netinet/tcp_output.c:1553 #17 0xffffffff80de638d in tcp_usr_send (so=0xfffff8017674f760, flags=0, m=0x0, nam=0x0, control=<optimized out>, td=0xfffffe020d66c020) at /usr/src/sys/netinet/tcp_usrreq.c:1178 #18 0xffffffff80ca7807 in sosend_generic (so=0xfffff8017674f760, addr=0x517, uio=0x0, top=0x517, control=0xfffff80357919b00, flags=128, td=0xfffffe020d66c020) at /usr/src/sys/kern/uipc_socket.c:1759 #19 0xffffffff80ca7d20 in sosend (so=0xfffff801ea078800, so@entry=0xfffff8017674f760, addr=addr@entry=0x0, uio=0x517, uio@entry=0x0, top=0x517, control=control@entry=0x0, flags=1469161464, flags@entry=128, td=0xfffffe020d66c020) at /usr/src/sys/kern/uipc_socket.c:1809 #20 0xffffffff828a0899 in icl_conn_send_pdus (isc=0xfffff8039b839500, queue=0xfffffe0568cbaeb8) at /usr/src/sys/dev/iscsi/icl_soft.c:989 #21 icl_send_thread (arg=arg@entry=0xfffff8039b839500) at /usr/src/sys/dev/iscsi/icl_soft.c:1027 #22 0xffffffff80bc2fce in fork_exit (callout=0xffffffff828a0400 <icl_send_thread>, arg=0xfffff8039b839500, frame=0xfffffe0568cbaf40) at /usr/src/sys/kern/kern_fork.c:1093 #23 <signal handler called> #24 0x000000082f05ad2c in ?? () Backtrace stopped: Cannot access memory at address 0x84c4f4e08 (kgdb) frame 9 #9 0xffffffff80c9cb87 in m_unshare (m0=m0@entry=0xfffff8028a647b00, how=how@entry=1) at /usr/src/sys/kern/uipc_mbuf.c:2082 2082 memcpy(mtod(n, caddr_t), mtod(m, caddr_t) + off, cc); (kgdb) q
I suspect that this is the same problem as PR 272616. It should be possible to work around this by setting kern.ipc.mb_use_ext_pgs = 0.
(In reply to Mark Johnston from comment #1) Apparently 272616 now has a commit as of 20230728. To submitter: can you try that to see if it fixes the problem?
(In reply to Mark Linimon from comment #2) Gave the 13.2 upgrade another try. I was going to report that the issue persists, but then I double-checked and found that this fix is not included in 13.2-RELEASE-p3. Now I'm running 13.2-STABLE and the system has an uptime of 20mins, which is longer than it was able to make it before. I'll report back in case it does end up crashing again, but it's looking good so far.
Since the commit has been MFCed to 13-STABLE, I don't think there is anything else to do here.