Bug 271409

Summary:	kernel panic triggered by iscsi via IPSec
Product:	Base System	Reporter:	noah.bergbauer
Component:	kern	Assignee:	Mark Linimon <linimon>
Status:	Closed FIXED
Severity:	Affects Only Me	CC:	crest, linimon, markj
Priority:	---	Keywords:	crash
Version:	13.2-RELEASE
Hardware:	Any
OS:	Any
See Also:	https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=271393 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272073

Description noah.bergbauer 2023-05-14 12:34:15 UTC

When upgrading from 13.1 to 13.2, I'm running into the panic below.
The panic is triggered by running "fsck_ffs -p" on a filesystem via iSCSI where the iSCSI is protedted using IPSec (strongswan ESP:AES_GCM_16-128).


Fatal trap 12: page fault while in kernel mode
cpuid = 7; apic id = 07
fault virtual address	= 0x0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff810adf16
stack pointer	        = 0x28:0xfffffe0568cba730
frame pointer	        = 0x28:0xfffffe0568cba730
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (iscsitx)
trap number		= 12
panic: page fault
cpuid = 7
time = 1684066170
KDB: stack backtrace:
#0 0xffffffff80c53dc5 at kdb_backtrace+0x65
#1 0xffffffff80c06741 at vpanic+0x151
#2 0xffffffff80c065e3 at panic+0x43
#3 0xffffffff810b1fa7 at trap_fatal+0x387
#4 0xffffffff810b1fff at trap_pfault+0x4f
#5 0xffffffff81088e78 at calltrap+0x8
#6 0xffffffff80c9cb87 at m_unshare+0x297
#7 0xffffffff8288e4b3 at esp_output+0x183
#8 0xffffffff8288af13 at ipsec4_perform_request+0x3b3
#9 0xffffffff8288b063 at ipsec4_common_output+0x83
#10 0xffffffff80e3970c at ipsec_kmod_output+0x2c
#11 0xffffffff80dbcf84 at ip_output+0xb64
#12 0xffffffff80dd43af at tcp_output+0x1dbf
#13 0xffffffff80de638d at tcp_usr_send+0x17d
#14 0xffffffff80ca7807 at sosend_generic+0x617
#15 0xffffffff80ca7d20 at sosend+0x50
#16 0xffffffff828a0899 at icl_send_thread+0x499
#17 0xffffffff80bc2fce at fork_exit+0x7e
Uptime: 7m7s
Dumping 5022 out of 130720 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55		__asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:396
#2  0xffffffff80c0630a in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:484
#3  0xffffffff80c067ae in vpanic (fmt=<optimized out>, ap=ap@entry=0xfffffe0568cba580) at /usr/src/sys/kern/kern_shutdown.c:923
#4  0xffffffff80c065e3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:847
#5  0xffffffff810b1fa7 in trap_fatal (frame=0xfffffe0568cba670, eva=0) at /usr/src/sys/amd64/amd64/trap.c:942
#6  0xffffffff810b1fff in trap_pfault (frame=0xfffffe0568cba670, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:761
#7  <signal handler called>
#8  memcpy_erms () at /usr/src/sys/amd64/amd64/support.S:553
#9  0xffffffff80c9cb87 in m_unshare (m0=m0@entry=0xfffff8028a647b00, how=how@entry=1) at /usr/src/sys/kern/uipc_mbuf.c:2082
#10 0xffffffff8288e4b3 in esp_output (m=0xfffff801ea078800, sp=<optimized out>, sav=0xfffff801ea8d8c00, idx=0, skip=20, protoff=9)
    at /usr/src/sys/netipsec/xform_esp.c:770
#11 0xffffffff8288af13 in ipsec4_perform_request (m=0xfffff801ea078800, m@entry=0xfffff8028a647b00, sp=0x0, sp@entry=0xfffff801ea8d8800,
    inp=inp@entry=0xfffff8037760b5d0, idx=idx@entry=0) at /usr/src/sys/netipsec/ipsec_output.c:275
#12 0xffffffff8288b063 in ipsec4_process_packet (m=0xfffff8028a647b00, sp=0xfffff801ea8d8800, inp=0xfffff8037760b5d0) at /usr/src/sys/netipsec/ipsec_output.c:292
#13 ipsec4_common_output (m=0xfffff8028a647b00, inp=0xfffff8037760b5d0, forwarding=<optimized out>) at /usr/src/sys/netipsec/ipsec_output.c:340
#14 0xffffffff80e3970c in ipsec_kmod_output (sc=sc@entry=0xffffffff81d172a8 <ipv4_ipsec>, m=0xfffff801ea078800, inp=0x517, inp@entry=0xfffff8037760b5d0)
    at /usr/src/sys/netipsec/subr_ipsec.c:369
#15 0xffffffff80dbcf84 in ip_output (m=0x0, m@entry=0xfffff8028a647b00, opt=<optimized out>, ro=0xfffff8037760b760, flags=0, imo=imo@entry=0x0, inp=0x517)
    at /usr/src/sys/netinet/ip_output.c:680
#16 0xffffffff80dd43af in tcp_output (tp=0xfffffe0568a0f000) at /usr/src/sys/netinet/tcp_output.c:1553
#17 0xffffffff80de638d in tcp_usr_send (so=0xfffff8017674f760, flags=0, m=0x0, nam=0x0, control=<optimized out>, td=0xfffffe020d66c020)
    at /usr/src/sys/netinet/tcp_usrreq.c:1178
#18 0xffffffff80ca7807 in sosend_generic (so=0xfffff8017674f760, addr=0x517, uio=0x0, top=0x517, control=0xfffff80357919b00, flags=128, td=0xfffffe020d66c020)
    at /usr/src/sys/kern/uipc_socket.c:1759
#19 0xffffffff80ca7d20 in sosend (so=0xfffff801ea078800, so@entry=0xfffff8017674f760, addr=addr@entry=0x0, uio=0x517, uio@entry=0x0, top=0x517,
    control=control@entry=0x0, flags=1469161464, flags@entry=128, td=0xfffffe020d66c020) at /usr/src/sys/kern/uipc_socket.c:1809
#20 0xffffffff828a0899 in icl_conn_send_pdus (isc=0xfffff8039b839500, queue=0xfffffe0568cbaeb8) at /usr/src/sys/dev/iscsi/icl_soft.c:989
#21 icl_send_thread (arg=arg@entry=0xfffff8039b839500) at /usr/src/sys/dev/iscsi/icl_soft.c:1027
#22 0xffffffff80bc2fce in fork_exit (callout=0xffffffff828a0400 <icl_send_thread>, arg=0xfffff8039b839500, frame=0xfffffe0568cbaf40)
    at /usr/src/sys/kern/kern_fork.c:1093
#23 <signal handler called>
#24 0x000000082f05ad2c in ?? ()
Backtrace stopped: Cannot access memory at address 0x84c4f4e08
(kgdb) frame 9
#9  0xffffffff80c9cb87 in m_unshare (m0=m0@entry=0xfffff8028a647b00, how=how@entry=1) at /usr/src/sys/kern/uipc_mbuf.c:2082
2082				memcpy(mtod(n, caddr_t), mtod(m, caddr_t) + off, cc);
(kgdb) q

Comment 1 Mark Johnston freebsd_committer

2023-07-28 13:41:34 UTC

I suspect that this is the same problem as PR 272616.  It should be possible to work around this by setting kern.ipc.mb_use_ext_pgs = 0.

Comment 2 Mark Linimon freebsd_committer

2023-07-30 06:10:01 UTC

(In reply to Mark Johnston from comment #1)

Apparently 272616 now has a commit as of 20230728.  To submitter: can you try that to see if it fixes the problem?

Comment 3 noah.bergbauer 2023-09-09 16:34:32 UTC

(In reply to Mark Linimon from comment #2)

Gave the 13.2 upgrade another try. I was going to report that the issue persists, but then I double-checked and found that this fix is not included in 13.2-RELEASE-p3.
Now I'm running 13.2-STABLE and the system has an uptime of 20mins, which is longer than it was able to make it before.
I'll report back in case it does end up crashing again, but it's looking good so far.

Comment 4 Mark Linimon freebsd_committer

2024-02-05 02:14:09 UTC

Since the commit has been MFCed to 13-STABLE, I don't think there is anything else to do here.