Bug 281831 - Reproducible kernel panic; double fault in AES_GCM_ENCRYPT
Summary: Reproducible kernel panic; double fault in AES_GCM_ENCRYPT
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 14.1-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Kristof Provost
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2024-10-03 06:15 UTC by Alexey Vyskubov
Modified: 2024-11-21 07:40 UTC (History)
2 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Vyskubov 2024-10-03 06:15:22 UTC 
Hello, I have a reproducible kernel panic on 14.1-p5. Backtrace follows.

#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:405
#2  0xffffffff80b32767 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:523
#3  0xffffffff80b32c3e in vpanic (fmt=0xffffffff811cf137 "double fault", ap=ap@entry=0xfffffe006e62ee40) at /usr/src/sys/kern/kern_shutdown.c:967
#4  0xffffffff80b32a93 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:891
#5  0xffffffff81000e7e in dblfault_handler (frame=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:1012
#6  <signal handler called>
#7  0xffffffff80edb4fc in AES_GCM_encrypt (in=in@entry=0xfffff8075b658c00 "E", out=out@entry=0xfffff8075b658c00 "E", addt=addt@entry=0xfffff800029dc468 "H",
    ivec=ivec@entry=0xfffffe0265038180 "", tag=tag@entry=0xfffffe0265038170 "", nbytes=1003, abytes=8, ibytes=12,
    key=0xfffff8002ca4b850 "df|)\372\3745\342Q\274\371\304H%\263j6\005~\006߯\225%\2228\023\326Z\321\022Z[\257\302\227\241S\367u\360\357\016\261\270ʽ\333Zq\004\277\205ޑ\232\027\346\202LM7\220\026\303υtb\234r\001\222s|\260*\271\301k\277'|\300:\371\355Z-\037o\026`(\377", nr=14) at /usr/src/sys/crypto/aesni/aesni_ghash.c:381
#8  0xffffffff80ed9da1 in aesni_cipher_crypt (ses=0xfffff8002ca4b848, crp=0xfffff8057a42d348, csp=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:717
#9  0xffffffff80ed9668 in aesni_cipher_process (ses=0xfffff8002ca4b848, crp=0xfffff8057a42d348) at /usr/src/sys/crypto/aesni/aesni.c:611
#10 aesni_process (dev=<optimized out>, crp=0xfffff8057a42d348, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:318
#11 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42d348, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#12 crypto_dispatch (crp=crp@entry=0xfffff8057a42d348) at /usr/src/sys/opencrypto/crypto.c:1448
#13 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff800029dc400, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe0265038280) at /usr/src/sys/net/if_ovpn.c:1955
#14 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff800029dc400, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#15 0xffffffff80cf36ac in ip_output (m=0xfffff800029dc400, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#16 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff800029dc400) at /usr/src/sys/net/if_ovpn.c:2045
#17 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42d460) at /usr/src/sys/net/if_ovpn.c:1501
#18 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42d460, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#19 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42d460, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#20 crypto_dispatch (crp=crp@entry=0xfffff8057a42d460) at /usr/src/sys/opencrypto/crypto.c:1448
#21 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff800029dc400, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe0265038590) at /usr/src/sys/net/if_ovpn.c:1955
#22 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff800029dc400, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#23 0xffffffff80cf36ac in ip_output (m=0xfffff800029dc400, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#24 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff800029dc400) at /usr/src/sys/net/if_ovpn.c:2045
#25 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42d578) at /usr/src/sys/net/if_ovpn.c:1501
#26 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42d578, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#27 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42d578, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#28 crypto_dispatch (crp=crp@entry=0xfffff8057a42d578) at /usr/src/sys/opencrypto/crypto.c:1448
#29 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff800029dc400, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe02650388a0) at /usr/src/sys/net/if_ovpn.c:1955
#30 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff800029dc400, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#31 0xffffffff80cf36ac in ip_output (m=0xfffff800029dc400, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#32 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=0xfffff800029dc400, m@entry=0xfffff8003b08c300)
    at /usr/src/sys/net/if_ovpn.c:2045
#33 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42d690) at /usr/src/sys/net/if_ovpn.c:1501
#34 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42d690, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#35 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42d690, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#36 crypto_dispatch (crp=crp@entry=0xfffff8057a42d690) at /usr/src/sys/opencrypto/crypto.c:1448
#37 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8003b08c300, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe0265038bb0) at /usr/src/sys/net/if_ovpn.c:1955
#38 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8003b08c300, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#39 0xffffffff80cf36ac in ip_output (m=0xfffff8003b08c300, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#40 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff8003b08c300) at /usr/src/sys/net/if_ovpn.c:2045
#41 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42d7a8) at /usr/src/sys/net/if_ovpn.c:1501
#42 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42d7a8, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#43 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42d7a8, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#44 crypto_dispatch (crp=crp@entry=0xfffff8057a42d7a8) at /usr/src/sys/opencrypto/crypto.c:1448
#45 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8003b08c300, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe0265038ec0) at /usr/src/sys/net/if_ovpn.c:1955
#46 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8003b08c300, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#47 0xffffffff80cf36ac in ip_output (m=0xfffff8003b08c300, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#48 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff8003b08c300) at /usr/src/sys/net/if_ovpn.c:2045
#49 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42d8c0) at /usr/src/sys/net/if_ovpn.c:1501
#50 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42d8c0, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#51 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42d8c0, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#52 crypto_dispatch (crp=crp@entry=0xfffff8057a42d8c0) at /usr/src/sys/opencrypto/crypto.c:1448
#53 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8003b08c300, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe02650391d0) at /usr/src/sys/net/if_ovpn.c:1955
#54 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8003b08c300, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#55 0xffffffff80cf36ac in ip_output (m=0xfffff8003b08c300, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#56 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=0xfffff8003b08c300, m@entry=0xfffff8003b87bb00)
    at /usr/src/sys/net/if_ovpn.c:2045
#57 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42d9d8) at /usr/src/sys/net/if_ovpn.c:1501
#58 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42d9d8, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#59 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42d9d8, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#60 crypto_dispatch (crp=crp@entry=0xfffff8057a42d9d8) at /usr/src/sys/opencrypto/crypto.c:1448
#61 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8003b87bb00, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe02650394e0) at /usr/src/sys/net/if_ovpn.c:1955
#62 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8003b87bb00, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#63 0xffffffff80cf36ac in ip_output (m=0xfffff8003b87bb00, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#64 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff8003b87bb00) at /usr/src/sys/net/if_ovpn.c:2045
#65 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42daf0) at /usr/src/sys/net/if_ovpn.c:1501
#66 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42daf0, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#67 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42daf0, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#68 crypto_dispatch (crp=crp@entry=0xfffff8057a42daf0) at /usr/src/sys/opencrypto/crypto.c:1448
#69 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8003b87bb00, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe02650397f0) at /usr/src/sys/net/if_ovpn.c:1955
#70 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8003b87bb00, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#71 0xffffffff80cf36ac in ip_output (m=0xfffff8003b87bb00, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#72 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff8003b87bb00) at /usr/src/sys/net/if_ovpn.c:2045
#73 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42dc08) at /usr/src/sys/net/if_ovpn.c:1501
#74 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42dc08, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#75 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42dc08, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#76 crypto_dispatch (crp=crp@entry=0xfffff8057a42dc08) at /usr/src/sys/opencrypto/crypto.c:1448
#77 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8003b87bb00, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe0265039b00) at /usr/src/sys/net/if_ovpn.c:1955
#78 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8003b87bb00, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#79 0xffffffff80cf36ac in ip_output (m=0xfffff8003b87bb00, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#80 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=0xfffff8003b87bb00, m@entry=0xfffff80790f8d200)
    at /usr/src/sys/net/if_ovpn.c:2045
#81 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42dd20) at /usr/src/sys/net/if_ovpn.c:1501
#82 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42dd20, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#83 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42dd20, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#84 crypto_dispatch (crp=crp@entry=0xfffff8057a42dd20) at /usr/src/sys/opencrypto/crypto.c:1448
#85 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff80790f8d200, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe0265039e10) at /usr/src/sys/net/if_ovpn.c:1955
#86 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff80790f8d200, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#87 0xffffffff80cf36ac in ip_output (m=0xfffff80790f8d200, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#88 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff80790f8d200) at /usr/src/sys/net/if_ovpn.c:2045
#89 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42de38) at /usr/src/sys/net/if_ovpn.c:1501
#90 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42de38, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#91 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42de38, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#92 crypto_dispatch (crp=crp@entry=0xfffff8057a42de38) at /usr/src/sys/opencrypto/crypto.c:1448
#93 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff80790f8d200, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe026503a120) at /usr/src/sys/net/if_ovpn.c:1955
#94 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff80790f8d200, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#95 0xffffffff80cf36ac in ip_output (m=0xfffff80790f8d200, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#96 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff80790f8d200) at /usr/src/sys/net/if_ovpn.c:2045
#97 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42c000) at /usr/src/sys/net/if_ovpn.c:1501
#98 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42c000, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#99 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42c000, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#100 crypto_dispatch (crp=crp@entry=0xfffff8057a42c000) at /usr/src/sys/opencrypto/crypto.c:1448
#101 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff80790f8d200, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe026503a430) at /usr/src/sys/net/if_ovpn.c:1955
#102 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff80790f8d200, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#103 0xffffffff80cf36ac in ip_output (m=0xfffff80790f8d200, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#104 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=0xfffff80790f8d200, m@entry=0xfffff8000502f100)
    at /usr/src/sys/net/if_ovpn.c:2045
#105 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42c118) at /usr/src/sys/net/if_ovpn.c:1501
#106 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42c118, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#107 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42c118, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#108 crypto_dispatch (crp=crp@entry=0xfffff8057a42c118) at /usr/src/sys/opencrypto/crypto.c:1448
#109 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8000502f100, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe026503a740) at /usr/src/sys/net/if_ovpn.c:1955
#110 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8000502f100, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#111 0xffffffff80cf36ac in ip_output (m=0xfffff8000502f100, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#112 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff8000502f100) at /usr/src/sys/net/if_ovpn.c:2045
#113 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42c230) at /usr/src/sys/net/if_ovpn.c:1501
#114 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42c230, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#115 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42c230, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#116 crypto_dispatch (crp=crp@entry=0xfffff8057a42c230) at /usr/src/sys/opencrypto/crypto.c:1448
#117 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8000502f100, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe026503aa50) at /usr/src/sys/net/if_ovpn.c:1955
#118 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8000502f100, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#119 0xffffffff80cf36ac in ip_output (m=0xfffff8000502f100, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#120 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff8000502f100) at /usr/src/sys/net/if_ovpn.c:2045
#121 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42c348) at /usr/src/sys/net/if_ovpn.c:1501
#122 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42c348, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#123 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42c348, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#124 crypto_dispatch (crp=crp@entry=0xfffff8057a42c348) at /usr/src/sys/opencrypto/crypto.c:1448
#125 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8000502f100, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe026503ad60) at /usr/src/sys/net/if_ovpn.c:1955
#126 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8000502f100, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#127 0xffffffff80cf36ac in ip_output (m=0xfffff8000502f100, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#128 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=0xfffff8000502f100, m@entry=0xfffff8003bd3c500)
    at /usr/src/sys/net/if_ovpn.c:2045
#129 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42c460) at /usr/src/sys/net/if_ovpn.c:1501
#130 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42c460, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#131 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42c460, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#132 crypto_dispatch (crp=crp@entry=0xfffff8057a42c460) at /usr/src/sys/opencrypto/crypto.c:1448
#133 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8003bd3c500, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe026503b070) at /usr/src/sys/net/if_ovpn.c:1955
#134 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8003bd3c500, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#135 0xffffffff80cf36ac in ip_output (m=0xfffff8003bd3c500, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#136 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff8003bd3c500) at /usr/src/sys/net/if_ovpn.c:2045
#137 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42c578) at /usr/src/sys/net/if_ovpn.c:1501
#138 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42c578, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#139 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42c578, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#140 crypto_dispatch (crp=crp@entry=0xfffff8057a42c578) at /usr/src/sys/opencrypto/crypto.c:1448
#141 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8003bd3c500, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe026503b380) at /usr/src/sys/net/if_ovpn.c:1955
#142 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8003bd3c500, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#143 0xffffffff80cf36ac in ip_output (m=0xfffff8003bd3c500, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#144 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=m@entry=0xfffff8003bd3c500) at /usr/src/sys/net/if_ovpn.c:2045
#145 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42c690) at /usr/src/sys/net/if_ovpn.c:1501
#146 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42c690, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#147 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42c690, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#148 crypto_dispatch (crp=crp@entry=0xfffff8057a42c690) at /usr/src/sys/opencrypto/crypto.c:1448
#149 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff8003bd3c500, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe026503b690) at /usr/src/sys/net/if_ovpn.c:1955
#150 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff8003bd3c500, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#151 0xffffffff80cf36ac in ip_output (m=0xfffff8003bd3c500, opt=opt@entry=0x0, ro=<optimized out>, ro@entry=0x0, flags=flags@entry=0, imo=imo@entry=0x0,
    inp=inp@entry=0x0) at /usr/src/sys/netinet/ip_output.c:798
#152 0xffffffff8422a63c in ovpn_encap (sc=sc@entry=0xfffff802c2686300, peerid=<optimized out>, m=0xfffff8003bd3c500, m@entry=0xfffff800029e0600)
    at /usr/src/sys/net/if_ovpn.c:2045
#153 0xffffffff8422a70a in ovpn_encrypt_tx_cb (crp=0xfffff8057a42c7a8) at /usr/src/sys/net/if_ovpn.c:1501
#154 0xffffffff80ed9675 in aesni_process (dev=<optimized out>, crp=0xfffff8057a42c7a8, hint=<optimized out>) at /usr/src/sys/crypto/aesni/aesni.c:321
#155 0xffffffff80df7280 in crypto_dispatch_one (crp=crp@entry=0xfffff8057a42c7a8, hint=0) at /usr/src/sys/opencrypto/crypto.c:1432
#156 crypto_dispatch (crp=crp@entry=0xfffff8057a42c7a8) at /usr/src/sys/opencrypto/crypto.c:1448
#157 0xffffffff8422a0f1 in ovpn_transmit_to_peer (ifp=<optimized out>, m=m@entry=0xfffff800029e0600, peer=peer@entry=0xfffff80037028400,
    _ovpn_lock_trackerp=_ovpn_lock_trackerp@entry=0xfffffe026503b9a0) at /usr/src/sys/net/if_ovpn.c:1955
#158 0xffffffff842270cd in ovpn_output (ifp=0xfffff80256708000, m=0xfffff800029e0600, dst=<optimized out>, ro=<optimized out>) at /usr/src/sys/net/if_ovpn.c:2142
#159 0xffffffff80cf36ac in ip_output (m=0xfffff800029e0600, opt=<optimized out>, ro=<optimized out>, flags=64, imo=0x0, inp=inp@entry=0xfffff80037a65380)
    at /usr/src/sys/netinet/ip_output.c:798
#160 0xffffffff80d24710 in udp_send (so=<optimized out>, flags=<optimized out>, m=0xfffff800029e0600, addr=<optimized out>, control=<optimized out>,
    td=<optimized out>) at /usr/src/sys/netinet/udp_usrreq.c:1454
#161 0xffffffff80bd73f0 in sosend_dgram (so=<optimized out>, addr=<optimized out>, uio=<optimized out>, top=<optimized out>, control=0x0, flags=<optimized out>,
    td=0xfffff807c83c8740) at /usr/src/sys/kern/uipc_socket.c:1557
#162 0xffffffff80bd82cf in sousrsend (so=0x13, so@entry=0xfffff80037c22780, addr=0xfffff8075b658c00, uio=0xfffff800029dc468, uio@entry=0xfffffe026503bce8,
    control=0xe, control@entry=0x0, flags=flags@entry=0, userproc=userproc@entry=0x0) at /usr/src/sys/kern/uipc_socket.c:1896
#163 0xffffffff80bdeb5e in kern_sendit (td=0xfffff807c83c8740, s=5, mp=0xfffffe026503bdc0, flags=<optimized out>, control=control@entry=0x0,
    segflg=segflg@entry=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:789
#164 0xffffffff80bdee61 in sendit (td=0x13, s=1533381632, mp=0xfffff800029dc468, mp@entry=0xfffffe026503bdc0, flags=748992816)
    at /usr/src/sys/kern/uipc_syscalls.c:714
#165 0xffffffff80bdeccd in sys_sendto (td=0x13, uap=<optimized out>) at /usr/src/sys/kern/uipc_syscalls.c:821
#166 0xffffffff810011c0 in syscallenter (td=0xfffff807c83c8740) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:188
#167 amd64_syscall (td=0xfffff807c83c8740, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1194
#168 <signal handler called>
#169 0x000000082745662a in ?? ()
Backtrace stopped: Cannot access memory at address 0x820867db8

Some background: I have OpenVPN running; establishing a connection does not break anything. My system is tuned to go through VPN on fib 1; fib 0 is normal internet. As soon as I do something like "setfib 1 w3m ..." the system crashes. Removing "cipher AES-128-CBC" (sic, CBC) line from config helped. I do not control the other end of the VPN, and it worked previously. Probably something has changed on the other side. No changes were made on my side. I do not run IPv6; the only entries in IPv4 routing table in fib 1 say "default goes to tun1" and 127.0.0.1 goes to lo0.
Comment 1 Alexey Vyskubov 2024-10-03 06:52:16 UTC 
I was privately asked for the following information:

(kgdb) frame 7                                                              #7  0xffffffff80edb4fc in AES_GCM_encrypt (
    in=in@entry=0xfffff8075b658c00 "E",
    out=out@entry=0xfffff8075b658c00 "E",
    addt=addt@entry=0xfffff800029dc468 "H",
    ivec=ivec@entry=0xfffffe0265038180 "",
    tag=tag@entry=0xfffffe0265038170 "", nbytes=1003, abytes=8, ibytes=12,
    key=0xfffff8002ca4b850 "df|)\372\3745\342Q\274\371\304H%\263j6\005~\006߯225%\2228\023\326Z\321\022Z[\257\302\227\241S\367u\360\357\016\261\270ʽ\333Zq\004\277\205ޑ\232\027\346\202LM7\220\026\303υtb\234r\001\222s|\260*\271\301k\277'|\300:\371\355Z-\037o\026`(\377", nr=14)
    at /usr/src/sys/crypto/aesni/aesni_ghash.c:381
381             ctr2 = _mm_add_epi64(ctr1, ONE);
(kgdb) info frame                                                           Stack level 7, frame at 0xfffffe0265038100:
 rip = 0xffffffff80edb4fc in AES_GCM_encrypt
    (/usr/src/sys/crypto/aesni/aesni_ghash.c:381);
    saved rip = 0xffffffff80ed9da1
 called by frame at 0xfffffe02650381d0,
    caller of frame at 0xfffffe006e62eff0
 source language c.
 Arglist at 0xfffffe02650380f0, args: in=in@entry=0xfffff8075b658c00 "E",
    out=out@entry=0xfffff8075b658c00 "E",
    addt=addt@entry=0xfffff800029dc468 "H",
    ivec=ivec@entry=0xfffffe0265038180 "",
    tag=tag@entry=0xfffffe0265038170 "", nbytes=1003, abytes=8, ibytes=12,
    key=0xfffff8002ca4b850 "df|)\372\3745\342Q\274\371\304H%\263j6\005~\006߯\225%\2228\023\326Z\321\022Z[\257\302\227\241S\367u\360\357\016\261\270ʽ\333Zq\004\277\205ޑ\232\027\346\202LM7\220\026\303υtb\234r\001\222s|\260*\271\301k\277'|\300:\371\355Z-\037o\026`(\377", nr=14
 Locals at 0xfffffe02650380f0, Previous frame's sp is 0xfffffe0265038100
 Saved registers:
  rbx at 0xfffffe02650380c8, rbp at 0xfffffe02650380f0,
  r12 at 0xfffffe02650380d0, r13 at 0xfffffe02650380d8,
  r14 at 0xfffffe02650380e0, r15 at 0xfffffe02650380e8,
  rip at 0xfffffe02650380f8
(kgdb) frame 167
#167 amd64_syscall (td=0xfffff807c83c8740, traced=0)
    at /usr/src/sys/amd64/amd64/trap.c:1194
1194            syscallenter(td);
(kgdb) info frame
Stack level 167, frame at 0xfffffe026503bf40:
 rip = 0xffffffff810011c0 in amd64_syscall
    (/usr/src/sys/amd64/amd64/trap.c:1194); saved rip = 0xffffffff80fd765b
 called by frame at 0xfffffe026503c000,
    caller of frame at 0xfffffe026503bf40
 source language c.
 Arglist at 0xfffffe026503bf30, args: td=0xfffff807c83c8740, traced=0
 Locals at 0xfffffe026503bf30, Previous frame's sp is 0xfffffe026503bf40
 Saved registers:
  rbx at 0xfffffe026503bf08, rbp at 0xfffffe026503bf30,
  r12 at 0xfffffe026503bf10, r13 at 0xfffffe026503bf18,
  r14 at 0xfffffe026503bf20, r15 at 0xfffffe026503bf28,
  rip at 0xfffffe026503bf38
(kgdb)
Comment 2 Gleb Smirnoff freebsd_committer freebsd_triage 2024-10-03 07:43:32 UTC 
Although already was 99% sure, but just wanted to confirm this is stack overflow:

(gdb) p 0xfffffe026503bf40 - 0xfffffe0265038100
$2 = 15936

Indeed, and kernel stack is 16k.
Comment 3 Kristof Provost freebsd_committer freebsd_triage 2024-10-03 08:05:30 UTC 
(In reply to Gleb Smirnoff from comment #2)
Yeah, it's a stack overflow because we're looping though if_ovpn.

The immediate cause for that is a misconfiguration routing the tunnelled OpenVPN traffic back through the tunnel. 
That shouldn't panic, of course, but it does have to be fixed as well.

The panic is already fixed in https://cgit.freebsd.org/src/commit/?id=59a6666ec91d71f97aaae5195bbfafd9d422db2e (main) and https://cgit.freebsd.org/src/commit/?id=18c38eda39bb8162ddce9acc4de4d6e4bbf48ef0 (stable/14)
Comment 4 Alexey Vyskubov 2024-10-04 06:17:35 UTC 
I'll check if things work with STABLE a bit later today.

Can I please have some more information on routing misconfiguration? I am not sure what I am doing wrong.
Comment 5 Alexey Vyskubov 2024-10-04 06:58:30 UTC 
Something very strange is happening.

I have the script, which runs on OpenVPN startup. Among other things, it does "ifconfig tun1 fib 1", basically. When the system goes up, in fib 1 I have only two routes in fib 1 (default and 127.0.0.1), and that's probably where recursion happens. However, if I restart OpenVPN manually (with service restart) I get two more routes in the table: to both ends of the VPN tunnel via link# -- and no crash. I'll investigate what is going on.

Meanwhile, I'm compiling STABLE.
Comment 6 Alexey Vyskubov 2024-10-04 12:35:06 UTC 
I can confirm that 14-STABLE does not crash, even though I still do not understand what is happening.

The routing table I get after boot is

❯ setfib 1 netstat -4rn
Routing tables (fib: 1)
                                                                                                        Internet:
Destination        Gateway            Flags     Netif Expire
default            <remote OpenVPN>   UGS        tun1
127.0.0.1          link#3             UHS         lo0

and it does not look correct to me. I still get connectivity in fib 1, and it indeed goes through VPN:

❯ ping -q -c 1 google.com
PING google.com (142.250.74.142): 56 data bytes
                                                                                                        --- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 18.471/18.471/18.471/0.000 ms
❯ setfib 1 ping -q -c 1 google.com
PING google.com (172.217.168.46): 56 data bytes
                                                                                                        --- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 50.299/50.299/50.299/0.000 ms
❯ REAL_IP=$(curl -s ifconfig.co) && VPN_IP=$(setfib 1 curl -s ifconfig.co) && if [[ $REAL_IP != $VPN_IP ]]; then printf "\nGot VPN\n\n"; fi

Got VPN

After restarting OpenVPN with service restart the routing table looks better:

❯ setfib 1 netstat -4rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            <remote OpenVPN>   UGS        tun1
<my end>           link#3             UHS         lo0
<remote OpenVPN>   link#12            UH         tun1
127.0.0.1          link#3             UHS         lo0

and I still get connectivity.

Anyway, this is another kettle of fish, and I think the bug can be closed as there is no kernel panic in STABLE.
Comment 7 FiLiS 2024-11-21 07:40:18 UTC 
is that fix going to be in 14.2 RELEASE?