When using IPv6, I random but high frequently got kernel panic: Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 06 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff811525b5 stack pointer = 0x28:0xfffffe00c5f895e0 frame pointer = 0x28:0xfffffe00c5f895e0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (if_io_tqg_3) trap number = 12 panic: page fault cpuid = 3 time = 1692374078 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c5f893a0 vpanic() at vpanic+0x151/frame 0xfffffe00c5f893f0 panic() at panic+0x43/frame 0xfffffe00c5f89450 trap_fatal() at trap_fatal+0x387/frame 0xfffffe00c5f894b0 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c5f89510 calltrap() at calltrap+0x8/frame 0xfffffe00c5f89510 --- trap 0xc, rip = 0xffffffff811525b5, rsp = 0xfffffe00c5f895e0, rbp = 0xfffffe00c5f895e0 --- memmove_erms() at memmove_erms+0xe5/frame 0xfffffe00c5f895e0 SipHash_Update() at SipHash_Update+0x56/frame 0xfffffe00c5f89620 pf_syncookie_mac() at pf_syncookie_mac+0xac/frame 0xfffffe00c5f896a0 pf_syncookie_check() at pf_syncookie_check+0x49/frame 0xfffffe00c5f896c0 pf_test_state_tcp() at pf_test_state_tcp+0x353/frame 0xfffffe00c5f89820 pf_test6() at pf_test6+0xb9b/frame 0xfffffe00c5f899a0 pf_check6_in() at pf_check6_in+0x5b/frame 0xfffffe00c5f899d0 pfil_run_hooks() at pfil_run_hooks+0x97/frame 0xfffffe00c5f89a10 ip6_tryforward() at ip6_tryforward+0x1e3/frame 0xfffffe00c5f89a90 ip6_input() at ip6_input+0x5e4/frame 0xfffffe00c5f89b70 netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00c5f89bc0 ether_demux() at ether_demux+0x159/frame 0xfffffe00c5f89bf0 ether_nh_input() at ether_nh_input+0x36b/frame 0xfffffe00c5f89c50 netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00c5f89ca0 ether_input() at ether_input+0x69/frame 0xfffffe00c5f89d00 iflib_rxeof() at iflib_rxeof+0xbcb/frame 0xfffffe00c5f89e00 _task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00c5f89e40 gtaskqueue_run_locked() at gtaskqueue_run_locked+0x15d/frame 0xfffffe00c5f89ec0 gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc3/frame 0xfffffe00c5f89ef0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00c5f89f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c5f89f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Timeout initializing vt_vga Uptime: 6m48s Dumping 1010 out of 16211 MB:..2%..12%..21%..31%..42%..51%..61%..72%..81%..91% Dump complete OPNsense 23.7.1_3-amd64 FreeBSD 13.2-RELEASE-p2 OpenSSL 1.1.1v 1 Aug 2023 Intel(R) Xeon(R) CPU E3-1225 v5 @ 3.30GHz (4 cores, 4 threads) Also https://github.com/opnsense/src/issues/184
Does this problem happen on base FreeBSD?
I haven't had a chance to test it on base FreeBSD yet, which would require me to turn off the gateway for a while.
(In reply to Rin Cat from comment #2) Okay, I'll close this until you've tested on FreeBSD.
I can confirm it also happened in the base kernel `FreeBSD XXX 13.2-RELEASE-p1 FreeBSD 13.2-RELEASE-p1 releng/13.2-n254621-08b87f63a04 SMP amd64` ``` Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 02 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8114d675 stack pointer = 0x28:0xfffffe00c5fc38b0 frame pointer = 0x28:0xfffffe00c5fc38b0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi1: netisr 0) trap number = 12 panic: page fault cpuid = 1 time = 1692394742 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00c5fc3670 vpanic() at vpanic+0x151/frame 0xfffffe00c5fc36c0 panic() at panic+0x43/frame 0xfffffe00c5fc3720 trap_fatal() at trap_fatal+0x387/frame 0xfffffe00c5fc3780 trap_pfault() at trap_pfault+0x65/frame 0xfffffe00c5fc37e0 calltrap() at calltrap+0x8/frame 0xfffffe00c5fc37e0 --- trap 0xc, rip = 0xffffffff8114d675, rsp = 0xfffffe00c5fc38b0, rbp = 0xfffffe00c5fc38b0 --- memmove_erms() at memmove_erms+0xe5/frame 0xfffffe00c5fc38b0 SipBuf() at SipBuf+0x49/frame 0xfffffe00c5fc38e0 SipHash_Update() at SipHash_Update+0x40/frame 0xfffffe00c5fc3910 pf_syncookie_mac() at pf_syncookie_mac+0xda/frame 0xfffffe00c5fc3990 pf_syncookie_check() at pf_syncookie_check+0x78/frame 0xfffffe00c5fc39b0 pf_test_state_tcp() at pf_test_state_tcp+0x353/frame 0xfffffe00c5fc3b10 pf_test6() at pf_test6+0xd8a/frame 0xfffffe00c5fc3c90 pf_check6_in() at pf_check6_in+0x6b/frame 0xfffffe00c5fc3cc0 pfil_run_hooks() at pfil_run_hooks+0xb7/frame 0xfffffe00c5fc3d00 ip6_input() at ip6_input+0x625/frame 0xfffffe00c5fc3de0 swi_net() at swi_net+0x1a1/frame 0xfffffe00c5fc3e60 ithread_loop() at ithread_loop+0x259/frame 0xfffffe00c5fc3ef0 fork_exit() at fork_exit+0x80/frame 0xfffffe00c5fc3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c5fc3f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Timeout initializing vt_vga Uptime: 47s Dumping 1065 out of 16209 MB:..2%..11%..22%..31%..41%..52%..61%..71%..82%..91% Dump complete ---<<BOOT>>--- ```
Okay, so with what ruleset? What non-default sysctls? What traffic?
Changed sysctl: debug.debugger_on_panic="0" dev.mce.0.rx_pauseframe_control="0" dev.mce.1.rx_pauseframe_control="0" hw.ibrs_disable="0" hw.ixl.enable_head_writeback="0" hw.syscons.kbd_reboot="0" kern.ipc.maxsockbuf="4262144" kern.ipc.mb_use_ext_pgs="0" kern.ipc.nmbclusters="1000000" kern.randompid="1" net.enc.in.ipsec_bpf_mask="2" net.enc.in.ipsec_filter_mask="2" net.enc.out.ipsec_bpf_mask="1" net.enc.out.ipsec_filter_mask="1" net.inet.icmp.drop_redirect="1" net.inet.icmp.icmplim="0" net.inet.icmp.log_redirect="0" net.inet.icmp.reply_from_interface="1" net.inet.ip.accept_sourceroute="0" net.inet.ip.forwarding="1" net.inet.ip.intr_queue_maxlen="1000" net.inet.ip.portrange.first="1024" net.inet.ip.random_id="1" net.inet.ip.redirect="0" net.inet.ip.sourceroute="0" net.inet.tcp.blackhole="2" net.inet.tcp.delayed_ack="0" net.inet.tcp.drop_synfin="1" net.inet.tcp.log_debug="0" net.inet.tcp.recvspace="65228" net.inet.tcp.sendspace="65228" net.inet.tcp.syncookies="0" net.inet.tcp.tso="0" net.inet.udp.blackhole="1" net.inet.udp.checksum="1" net.inet.udp.maxdgram="57344" net.inet6.ip6.dad_count="0" net.inet6.ip6.forwarding="1" net.inet6.ip6.intr_queue_maxlen="1000" net.inet6.ip6.prefer_tempaddr="1" net.inet6.ip6.redirect="0" net.inet6.ip6.use_tempaddr="1" net.link.bridge.pfil_bridge="1" net.link.bridge.pfil_local_phys="0" net.link.bridge.pfil_member="0" net.link.bridge.pfil_onlyip="0" net.link.ether.inet.log_arp_movements="1" net.link.ether.inet.log_arp_wrong_iface="1" net.link.tap.user_open="1" net.link.vlan.mtag_pcp="1" net.local.dgram.maxdgram="8192" net.pf.share_forward="0" net.pf.share_forward6="0" net.route.multipath="0" security.bsd.see_other_gids="0" security.bsd.see_other_uids="0" vfs.read_max="32" vm.pmap.pti="1" pf ruleset: Very basic since I just set it up for a few days. It has multiple NICs and one of them connect to 10G switch with VLANs, works as gateway. scrub on igb1 all random-id fragment reassemble scrub on mce1_vlan10 all random-id fragment reassemble scrub on mce1_vlan4 all random-id fragment reassemble scrub on mce1_vlan3 all random-id fragment reassemble scrub on mce1_vlan2 all random-id fragment reassemble scrub on mce1_vlan6 all random-id fragment reassemble scrub on mce1_vlan5 all random-id fragment reassemble scrub on mce1_vlan1 all random-id fragment reassemble scrub on igb0 all random-id fragment reassemble block drop in log on mce1_vlan10 inet6 from fe80::ee0d:9aff:fea6:bfff to any block drop in log on mce1_vlan4 inet6 from fe80::ee0d:9aff:fea6:bfff to any block drop in log on mce1_vlan2 inet6 from fe80::ee0d:9aff:fea6:bfff to any block drop in log on mce1_vlan6 inet6 from fe80::ee0d:9aff:fea6:bfff to any block drop in log on mce1_vlan5 inet6 from fe80::ee0d:9aff:fea6:bfff to any block drop in log on ! igb1 inet6 from 2001:1970:5642:b400::/64 to any block drop in log on igb1 inet6 from fe80::a236:9fff:fe85:4ee5 to any block drop in log inet6 from <__automatic_6aadc26c_1> to any block drop in log on ! mce1_vlan10 inet6 from 2605:59c8:X:A::/64 to any block drop in log on ! mce1_vlan4 inet6 from 2605:59c8:X:B::/64 to any block drop in log on ! mce1_vlan2 inet6 from 2605:59c8:X:C::/64 to any block drop in log on ! mce1_vlan6 inet6 from 2605:59c8:X:D::/64 to any block drop in log on ! mce1_vlan5 inet6 from 2605:59c8:X:E::/64 to any block drop in log on ! igb0 inet6 from 2605:59c8:2200:25cf::/64 to any block drop in log on igb0 inet6 from fe80::a236:9fff:fe85:4ee4 to any block drop in log on ! igb1 inet from 192.168.0.0/24 to any block drop in log inet from <__automatic_6aadc26c_0> to any block drop in log on ! mce1_vlan10 inet from 10.1.50.0/24 to any block drop in log on ! mce1_vlan4 inet from 10.1.3.0/24 to any block drop in log on ! mce1_vlan3 inet from 10.1.2.0/24 to any block drop in log on ! mce1_vlan2 inet from 10.1.1.0/24 to any block drop in log on ! mce1_vlan6 inet from 10.1.6.0/24 to any block drop in log on ! mce1_vlan5 inet from 10.1.5.0/24 to any block drop in log on ! mce1_vlan1 inet from 10.1.0.0/24 to any block drop in log on ! igb0 inet from 100.64.0.0/10 to any block drop in log inet all block drop in log inet6 all pass in log quick inet6 proto ipv6-icmp all icmp6-type unreach keep state pass in log quick inet6 proto ipv6-icmp all icmp6-type toobig keep state pass in log quick inet6 proto ipv6-icmp all icmp6-type neighbrsol keep state pass in log quick inet6 proto ipv6-icmp all icmp6-type neighbradv keep state pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type echoreq keep state pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type echoreq keep state pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type echorep keep state pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type echorep keep state pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type routersol keep state pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type routersol keep state pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type routeradv keep state pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type routeradv keep state pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type neighbrsol keep state pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type neighbrsol keep state pass out log quick inet6 proto ipv6-icmp from (self) to fe80::/10 icmp6-type neighbradv keep state pass out log quick inet6 proto ipv6-icmp from (self) to ff02::/16 icmp6-type neighbradv keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type echoreq keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type echoreq keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routersol keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routersol keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type routeradv keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type routeradv keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbrsol keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbrsol keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type neighbradv keep state pass in log quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type neighbradv keep state pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type echoreq keep state pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routersol keep state pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type routeradv keep state pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbrsol keep state pass in log quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type neighbradv keep state pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type echoreq keep state pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type routersol keep state pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type routeradv keep state pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type neighbrsol keep state pass in log quick inet6 proto ipv6-icmp from :: to ff02::/16 icmp6-type neighbradv keep state block drop in log quick inet proto tcp from any port = 0 to any block drop in log quick inet proto udp from any port = 0 to any block drop in log quick inet6 proto tcp from any port = 0 to any block drop in log quick inet6 proto udp from any port = 0 to any block drop in log quick inet proto tcp from any to any port = 0 block drop in log quick inet proto udp from any to any port = 0 block drop in log quick inet6 proto tcp from any to any port = 0 block drop in log quick inet6 proto udp from any to any port = 0 pass log quick inet6 proto carp from any to ff02::12 keep state pass log quick inet proto carp from any to 224.0.0.18 keep state block drop in log quick proto tcp from <sshlockout> to (self) port = ssh block drop in log quick proto tcp from <sshlockout> to (self) port = https block drop in log quick from <virusprot> to any pass in log quick on igb1 inet6 proto udp from fe80::/10 port = dhcpv6-client to fe80::/10 port = dhcpv6-client keep state pass in log quick on igb1 proto udp from any port = dhcpv6-server to any port = dhcpv6-client keep state pass out log quick on igb1 proto udp from any port = dhcpv6-client to any port = dhcpv6-server keep state pass in log quick on igb1 proto udp from any port = bootps to any port = bootpc keep state pass out log quick on igb1 proto udp from any port = bootpc to any port = bootps keep state pass in log quick on mce1_vlan10 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state pass in log quick on mce1_vlan10 proto udp from any port = bootpc to (self) port = bootps keep state pass out log quick on mce1_vlan10 proto udp from (self) port = bootps to any port = bootpc keep state pass in log quick on mce1_vlan10 inet6 proto udp from fe80::/10 to fe80::/10 port = dhcpv6-client keep state pass in log quick on mce1_vlan10 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-client keep state pass in log quick on mce1_vlan10 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-server keep state pass in log quick on mce1_vlan10 inet6 proto udp from ff02::/16 to fe80::/10 port = dhcpv6-server keep state pass in log quick on mce1_vlan10 inet6 proto udp from fe80::/10 to (self) port = dhcpv6-client keep state pass out log quick on mce1_vlan10 inet6 proto udp from (self) port = dhcpv6-server to fe80::/10 keep state pass in log quick on mce1_vlan4 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state pass in log quick on mce1_vlan4 proto udp from any port = bootpc to (self) port = bootps keep state pass out log quick on mce1_vlan4 proto udp from (self) port = bootps to any port = bootpc keep state pass in log quick on mce1_vlan4 inet6 proto udp from fe80::/10 to fe80::/10 port = dhcpv6-client keep state pass in log quick on mce1_vlan4 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-client keep state pass in log quick on mce1_vlan4 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-server keep state pass in log quick on mce1_vlan4 inet6 proto udp from ff02::/16 to fe80::/10 port = dhcpv6-server keep state pass in log quick on mce1_vlan4 inet6 proto udp from fe80::/10 to (self) port = dhcpv6-client keep state pass out log quick on mce1_vlan4 inet6 proto udp from (self) port = dhcpv6-server to fe80::/10 keep state pass in log quick on mce1_vlan3 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state pass in log quick on mce1_vlan3 proto udp from any port = bootpc to (self) port = bootps keep state pass out log quick on mce1_vlan3 proto udp from (self) port = bootps to any port = bootpc keep state pass in log quick on mce1_vlan2 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state pass in log quick on mce1_vlan2 proto udp from any port = bootpc to (self) port = bootps keep state pass out log quick on mce1_vlan2 proto udp from (self) port = bootps to any port = bootpc keep state pass in log quick on mce1_vlan2 inet6 proto udp from fe80::/10 to fe80::/10 port = dhcpv6-client keep state pass in log quick on mce1_vlan2 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-client keep state pass in log quick on mce1_vlan2 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-server keep state pass in log quick on mce1_vlan2 inet6 proto udp from ff02::/16 to fe80::/10 port = dhcpv6-server keep state pass in log quick on mce1_vlan2 inet6 proto udp from fe80::/10 to (self) port = dhcpv6-client keep state pass out log quick on mce1_vlan2 inet6 proto udp from (self) port = dhcpv6-server to fe80::/10 keep state pass in log quick on mce1_vlan6 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state pass in log quick on mce1_vlan6 proto udp from any port = bootpc to (self) port = bootps keep state pass out log quick on mce1_vlan6 proto udp from (self) port = bootps to any port = bootpc keep state pass in log quick on mce1_vlan6 inet6 proto udp from fe80::/10 to fe80::/10 port = dhcpv6-client keep state pass in log quick on mce1_vlan6 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-client keep state pass in log quick on mce1_vlan6 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-server keep state pass in log quick on mce1_vlan6 inet6 proto udp from ff02::/16 to fe80::/10 port = dhcpv6-server keep state pass in log quick on mce1_vlan6 inet6 proto udp from fe80::/10 to (self) port = dhcpv6-client keep state pass out log quick on mce1_vlan6 inet6 proto udp from (self) port = dhcpv6-server to fe80::/10 keep state pass in log quick on mce1_vlan5 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state pass in log quick on mce1_vlan5 proto udp from any port = bootpc to (self) port = bootps keep state pass out log quick on mce1_vlan5 proto udp from (self) port = bootps to any port = bootpc keep state pass in log quick on mce1_vlan5 inet6 proto udp from fe80::/10 to fe80::/10 port = dhcpv6-client keep state pass in log quick on mce1_vlan5 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-client keep state pass in log quick on mce1_vlan5 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-server keep state pass in log quick on mce1_vlan5 inet6 proto udp from ff02::/16 to fe80::/10 port = dhcpv6-server keep state pass in log quick on mce1_vlan5 inet6 proto udp from fe80::/10 to (self) port = dhcpv6-client keep state pass out log quick on mce1_vlan5 inet6 proto udp from (self) port = dhcpv6-server to fe80::/10 keep state pass in log quick on mce1_vlan1 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state pass in log quick on mce1_vlan1 proto udp from any port = bootpc to (self) port = bootps keep state pass out log quick on mce1_vlan1 proto udp from (self) port = bootps to any port = bootpc keep state pass in log quick on igb0 inet6 proto udp from fe80::/10 port = dhcpv6-client to fe80::/10 port = dhcpv6-client keep state pass in log quick on igb0 proto udp from any port = dhcpv6-server to any port = dhcpv6-client keep state pass out log quick on igb0 proto udp from any port = dhcpv6-client to any port = dhcpv6-server keep state pass in log quick on igb0 proto udp from any port = bootps to any port = bootpc keep state pass out log quick on igb0 proto udp from any port = bootpc to any port = bootps keep state block drop in log quick on igb1 inet from <bogons> to any block drop in log quick on igb1 inet6 from <bogonsv6> to any block drop in log quick on igb0 inet from <bogons> to any block drop in log quick on igb0 inet6 from <bogonsv6> to any pass in quick on lo0 all no state pass out log all flags S/SA keep state allow-opts pass in log quick on mce1_vlan2 proto tcp from any to (self) port = ssh flags S/SA keep state pass in log quick on mce1_vlan2 proto tcp from any to (self) port = http flags S/SA keep state pass in log quick on mce1_vlan2 proto tcp from any to (self) port = https flags S/SA keep state pass out log route-to (igb1 192.168.0.1) inet from (igb1) to ! (igb1:network) flags S/SA keep state allow-opts pass out log route-to (igb1 fe80::481d:70ff:feaf:b2) inet6 from (igb1) to ! (igb1:network) flags S/SA keep state allow-opts pass out log route-to (igb0 100.64.0.1) inet from (igb0) to ! (igb0:network) flags S/SA keep state allow-opts pass out log route-to (igb0 fe80::200:5eff:fe00:101) inet6 from (igb0) to ! (igb0:network) flags S/SA keep state allow-opts pass in quick on mce1_vlan2 inet from (mce1_vlan2:network) to any flags S/SA keep state pass in quick on mce1_vlan2 inet6 from (mce1_vlan2:network) to any flags S/SA keep state pass in quick on mce1_vlan2 inet6 from fe80::/10 to any flags S/SA keep state pass in quick on mce1_vlan1 inet from (mce1_vlan1:network) to any flags S/SA keep state pass in quick on mce1_vlan3 inet from (mce1_vlan3:network) to any flags S/SA keep state pass in quick on mce1_vlan4 inet from (mce1_vlan4:network) to any flags S/SA keep state pass in quick on mce1_vlan4 inet6 from (mce1_vlan4:network) to any flags S/SA keep state pass in quick on mce1_vlan4 inet6 from fe80::/10 to any flags S/SA keep state pass in quick on mce1_vlan5 inet from (mce1_vlan5:network) to any flags S/SA keep state pass in quick on mce1_vlan5 inet6 from (mce1_vlan5:network) to any flags S/SA keep state pass in quick on mce1_vlan5 inet6 from fe80::/10 to any flags S/SA keep state pass in quick on mce1_vlan6 inet from (mce1_vlan6:network) to any flags S/SA keep state pass in quick on mce1_vlan6 inet6 from (mce1_vlan6:network) to any flags S/SA keep state pass in quick on mce1_vlan6 inet6 from fe80::/10 to any flags S/SA keep state pass in quick on mce1_vlan10 inet from (mce1_vlan10:network) to any flags S/SA keep state pass in quick on mce1_vlan10 inet6 from (mce1_vlan10:network) to any flags S/SA keep state pass in quick on mce1_vlan10 inet6 from fe80::/10 to any flags S/SA keep state There is no special traffic (40+ normal devices (servers/PC/phones)), but this panic will only happen if I configured IPv6, used for a few days on IPv4 only without any issue. I can have like 90% chance to make it panic if I run IPv6 test on https://test-ipv6.com/ . It does seem to be related to IPv6 new connections, not loads.
How are you getting a syncookie panic if 'set syncookies` is not part of the ruleset? Also, what is net.pf.share_forward? I can't seem to find that in the code. Does this happen on 14?
I copied the configurations from OPNsense, so some of them may not exists in base FreeBSD, and it shouldn't affect anything. From the backtrace I think the syncookies are from incoming / forwarding connections, not pf it self.
(In reply to Rin Cat from comment #8) > From the backtrace I think the syncookies are from incoming / forwarding connections, not pf it self. No, I mean, how can we be hitting syncookie code if the feature isn't enabled?
I don't know either, I also tried set net.inet.tcp.syncookies="0", but it still panic.
That pf_syncookie_check() is in a if statement which doesn't matter if we enabled the feature or not. https://github.com/freebsd/freebsd-src/blob/57a3b81785c0f7f458789d0baa5c8265ecfd5bac/sys/netpfil/pf/pf.c#L5716
(kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=textdump@entry=1) at /usr/src/sys/kern/kern_shutdown.c:396 #2 0xffffffff80c72724 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:484 #3 0xffffffff80c72b8e in vpanic (fmt=<optimized out>, ap=ap@entry=0xfffffe00c5f643f0) at /usr/src/sys/kern/kern_shutdown.c:923 #4 0xffffffff80c72913 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:847 #5 0xffffffff811519c7 in trap_fatal (frame=0xfffffe00c5f644e0, eva=0) at /usr/src/sys/amd64/amd64/trap.c:942 #6 0xffffffff81151a35 in trap_pfault (frame=0xfffffe00c5f644e0, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:761 #7 <signal handler called> #8 memmove_erms () at /usr/src/sys/amd64/amd64/support.S:539 #9 0xffffffff804b6ce9 in SipBuf (ctx=ctx@entry=0xfffffe00c5f64618, src=src@entry=0xfffffe00c5f645e0, len=len@entry=2, final=2, final@entry=0) at /usr/src/sys/crypto/siphash/siphash.c:103 #10 0xffffffff804b6b20 in SipHash_Update (ctx=ctx@entry=0xfffffe00c5f64618, src=<optimized out>, len=len@entry=2) at /usr/src/sys/crypto/siphash/siphash.c:139 #11 0xffffffff823eac5a in pf_syncookie_mac (pd=pd@entry=0xfffffe00c5f64870, cookie=..., seq=<optimized out>) at /usr/src/sys/netpfil/pf/pf_syncookies.c:444 #12 0xffffffff823eab48 in pf_syncookie_check (pd=pd@entry=0xfffffe00c5f64870) at /usr/src/sys/netpfil/pf/pf_syncookies.c:321 #13 0xffffffff823b7e93 in pf_test_state_tcp (state=state@entry=0xfffffe00c5f64948, direction=direction@entry=1, kif=kif@entry=0xfffff80001fed500, m=m@entry=0xfffff80125df2400, off=off@entry=40, h=<optimized out>, pd=pd@entry=0xfffffe00c5f64870, reason=0xfffffe00c5f64954) at /usr/src/sys/netpfil/pf/pf.c:4958 #14 0xffffffff823c0bca in pf_test6 (dir=dir@entry=1, pflags=65536, ifp=0xfffff80001fe1800, m0=m0@entry=0xfffffe00c5f64a30, inp=0x0) at /usr/src/sys/netpfil/pf/pf.c:6947 #15 0xffffffff823d66ab in pf_check6_in (m=0xfffffe00c5f64a30, ifp=<optimized out>, flags=0, ruleset=<optimized out>, inp=0x2) at /usr/src/sys/netpfil/pf/pf_ioctl.c:5604 #16 0xffffffff80dbc537 in pfil_run_hooks (head=<optimized out>, p=..., ifp=ifp@entry=0xfffff80001fe1800, flags=flags@entry=65536, inp=inp@entry=0x0) at /usr/src/sys/net/pfil.c:187 #17 0xffffffff80e97828 in ip6_tryforward (m=0xfffff80125df2400) at /usr/src/sys/netinet6/ip6_fastfwd.c:167 #18 0xffffffff80e99889 in ip6_input (m=0xfffffe00c5f64638) at /usr/src/sys/netinet6/ip6_input.c:723 #19 0xffffffff80db8ca3 in netisr_dispatch_src (proto=<optimized out>, source=source@entry=0, m=0xfffff80125df2400) at /usr/src/sys/net/netisr.c:1194 #20 0xffffffff80db8e6f in netisr_dispatch (proto=3321251384, m=0x2) at /usr/src/sys/net/netisr.c:1234 #21 0xffffffff80d9aecc in ether_demux (ifp=ifp@entry=0xfffff80001fe1800, m=0x0) at /usr/src/sys/net/if_ethersubr.c:921 #22 0xffffffff80d9c51d in ether_input_internal (ifp=0xfffff80001fe1800, m=0x0) at /usr/src/sys/net/if_ethersubr.c:707 #23 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:737 #24 0xffffffff80db8b11 in netisr_dispatch_src (proto=proto@entry=5, source=source@entry=0, m=m@entry=0xfffff80125df2400) at /usr/src/sys/net/netisr.c:1143 #25 0xffffffff80db8e6f in netisr_dispatch (proto=3321251384, proto@entry=5, m=0x2, m@entry=0xfffff80125df2400) at /usr/src/sys/net/netisr.c:1234 #26 0xffffffff80d9b379 in ether_input (ifp=0xfffff80001fe1800, m=0xfffff80125df2400) at /usr/src/sys/net/if_ethersubr.c:828 #27 0xffffffff80db4631 in iflib_rxeof (rxq=rxq@entry=0xfffff80001fd4000, budget=<optimized out>) at /usr/src/sys/net/iflib.c:3048 #28 0xffffffff80dae5aa in _task_fn_rx (context=0xfffff80001fd4000) at /usr/src/sys/net/iflib.c:4122 #29 0xffffffff80cbe947 in gtaskqueue_run_locked (queue=queue@entry=0xfffff80001962a00) at /usr/src/sys/kern/subr_gtaskqueue.c:371 #30 0xffffffff80cbe772 in gtaskqueue_thread_loop (arg=arg@entry=0xfffffe001fff1008) at /usr/src/sys/kern/subr_gtaskqueue.c:547 #31 0xffffffff80c2b830 in fork_exit (callout=0xffffffff80cbe6b0 <gtaskqueue_thread_loop>, arg=0xfffffe001fff1008, frame=0xfffffe00c5f64f40) at /usr/src/sys/kern/kern_fork.c:1093 #32 <signal handler called> #33 0x0f04f983480f74cb in ?? ()
(kgdb) frame 9 #9 0xffffffff804b6ce9 in SipBuf (ctx=ctx@entry=0xfffffe00c5fc38b8, src=src@entry=0xfffffe00c5fc3880, len=len@entry=2, final=2, final@entry=0) at /usr/src/sys/crypto/siphash/siphash.c:103 (kgdb) p ctx $1 = (SIPHASH_CTX *) 0xfffffe00c5fc38b8 (kgdb) p *ctx $2 = {v = {12591897065586319223, 3410300342546510073, 4381067403221538949, 14075035438097627711}, buf = {b64 = 0, b8 = "\000\000\000\000\000\000\000"}, bytes = 34, buflen = 0 '\000', rounds_compr = 2 '\002', rounds_final = 4 '\004', initialized = 2 '\002'} (kgdb) p src $3 = (const uint8_t **) 0xfffffe00c5fc3880 (kgdb) p *src $4 = (const uint8_t *) 0x0 (kgdb) p **src Cannot access memory at address 0x0 (kgdb) p len $5 = 2 (kgdb) p final $6 = 2 (kgdb) p x $7 = 2 (kgdb) p ctx->buf.b64 $8 = 0 (kgdb) p &ctx->buf.b8 $9 = (uint8_t (*)[8]) 0xfffffe00c5fc38d8 (kgdb) p ctx->buf.b8 $10 = "\000\000\000\000\000\000\000" (kgdb) p ctx->buflen $11 = 0 '\000' (kgdb)
So bcopy(*src, &ctx->buf.b8[ctx->buflen], x) become bcopy(NULL, 0xfffffe00c5fc38d8, 2)
#11 0xffffffff82abac5a in pf_syncookie_mac (pd=pd@entry=0xfffffe00c5fc3b10, cookie=..., seq=<optimized out>) at /usr/src/sys/netpfil/pf/pf_syncookies.c:444 (kgdb) p pd $19 = (struct pf_pdesc *) 0xfffffe00c5fc3b10 (kgdb) p *pd $20 = {lookup = {done = 0, uid = 0, gid = 0}, tot_len = 72, hdr = {tcp = {th_sport = 47873, th_dport = 6321, th_seq = 3504939527, th_ack = 2232635287, th_x2 = 0 '\000', th_off = 8 '\b', th_flags = 16 '\020', th_win = 9729, th_sum = 49434, th_urp = 0}, udp = { uh_sport = 47873, uh_dport = 6321, uh_ulen = 8711, uh_sum = 53481}, icmp = {icmp_type = 1 '\001', icmp_code = 187 '\273', icmp_cksum = 6321, icmp_hun = {ih_pptr = 7 '\a', ih_gwaddr = {s_addr = 3504939527}, ih_idseq = {icd_id = 8711, icd_seq = 53481}, ih_void = -790027769, ih_pmtu = {ipm_void = 8711, ipm_nextmtu = 53481}, ih_rtradv = {irt_num_addrs = 7 '\a', irt_wpa = 34 '"', irt_lifetime = 53481}}, icmp_dun = {id_ts = {its_otime = 2232635287, its_rtime = 637603968, its_ttime = 49434}, id_ip = {idi_ip = { ip_hl = 7 '\a', ip_v = 9 '\t', ip_tos = 79 'O', ip_len = 34067, ip_id = 4224, ip_off = 9729, ip_ttl = 26 '\032', ip_p = 193 '\301', ip_sum = 0, ip_src = {s_addr = 0}, ip_dst = {s_addr = 0}}}, id_radv = {ira_addr = 2232635287, ira_preference = 637603968}, id_mask = 2232635287, id_data = "\227"}}, any = 0xfffffe00c5fc3b28 "\001\273\261\030\a\"\351ЗO\023\205\200\020\001&\032\301"}, nat_rule = 0x0, src = 0xfffff801d35cc06e, dst = 0xfffff801d35cc07e, sport = 0x0, dport = 0x0, pf_mtag = 0x0, act = {qid = 0, pqid = 0}, p_len = 0, ip_sum = 0x0, proto_sum = 0x0, flags = 0, af = 28 '\034', proto = 6 '\006', tos = 0 '\000', dir = 1 '\001', sidx = 0 '\000', didx = 1 '\001'} (kgdb) p pd->src $21 = (struct pf_addr *) 0xfffff801d35cc06e (kgdb) p *pd->src $22 = {pfa = {v4 = {s_addr = 2969XXXXX}, v6 = {__u6_addr = {__u6_addr8 = "&\a\370\260@\006\b XXXXX000\000\000\000 \n", __u6_addr16 = {XXXX, XXXXX, XXXX, XXX, 0, 0, 0, XXXX}, __u6_addr32 = {XXXXXXXXXX, XXXXXXXX, XXXXXXXXXX, XXXXXXXXX}}}, addr8 = "&\a\370\260@\006\b \000\000\000\000\000\000 \n", addr16 = {XXXXXXX, XXXXX, XXXX, XXXXX, XXXX, XXXX, XXXXX, XXXXXX}, addr32 = {XXXXXXX, XXXXXX, XXXXXX, XXXXXXX}}}
Not sure what's happened, SipHash_Update(&ctx, pd->src, sizeof(pd->src->v6)); pd->src (struct pf_addr *) 0xfffff8010a3cae6e become (const uint8_t *) 0x0 in SipHash_Update Before if (ctx->buflen > 0 || len < 8) len -= SipBuf(ctx, &s, len, 0); And you cannot copy from NULL. bcopy(*src, &ctx->buf.b8[ctx->buflen], x);
> Resolution: --- → Not Accepted I'm sorry but where is this policy written? It is not helpful smashing legitimate bug reports which happens to be your feature... Cheers, Franco
(In reply to Franco Fichtner from comment #17) That would be the bit where this website says “freebsd.org” rather than opnsense. Supporting opnsense is your job, not mine. You don’t get to just throw bugs over the wall without doing any actual testing on freebsd.
(In reply to Rin Cat from comment #16) Can you print pd->sport and pd->dport, because I suspect those are the issue, not src or dst. Also, you may be interested in following https://reviews.freebsd.org/D41502 because Kajetan found other issues with syncookies and IPv6, so even when this issue is fixed it may still not do what you expect.
(kgdb) p pd->dport $2 = (u_int16_t *) 0x0 (kgdb) p pd->sport $3 = (u_int16_t *) 0x0
The TCP and UDP hdr th_sport = uh_sport = 61525 and th_dport = uh_dport = 47873 looks correct. ip_src = {s_addr = 0}, ip_dst = {s_addr = 0}. And this one is a TCP connection proto = 6 '\006'
(In reply to Rin Cat from comment #20) Yeah, that’s what I expected. It’s pretty trivial to fix, and I’ll land it along with the other fixes in the review above. For extra fun this is only a problem on 13.
OK, any ETA of the patch? I would like give it a try before merge.
> That would be the bit where this website says “freebsd.org” rather than opnsense. I get you are trying to sound assertive, but the fact of the matter is this is unmodified code and obviously introduced by yourself. I don't know why you would think it is not. I just know it's not helpful. If you can show me the policy for bugs.freebsd.org that says "close issues immediately because reporters fail to report on a FreeBSD release version for 30 minutes". That policy doesn't exist and you make a pretty bad example of how now to communicate over this bug tracker. > Supporting opnsense is your job, not mine. You don’t get to just throw bugs over the wall without doing any actual testing on freebsd. I don't think this statement aged very well. And we are happy to clean up bugs that you bring to FreeBSD. I know I have before. Cheers, Franco
(In reply to Franco Fichtner from comment #24) Franco, I will continue to close bugs which have not been demonstrated on freebsd. I will not continue this argument.
Fair enough. Apology accepted. :)
For reference: https://docs.freebsd.org/en/articles/pr-guidelines/ "Jane Random BugBuster confirms that the bug report has sufficient information to be reproducible. ***If not, she goes back and forth with the reporter to obtain the needed information.*** At this point the bug is set to the Open state." So I don't think it requires closing a report in a short period of time you would want to require the reporter installing a debug kernel instead to provide more information. It could easily be misinterpreted and we don't want that. ;) Cheers, Franco
> It’s pretty trivial to fix > For extra fun this is only a problem on 13. I can assure you it's not fun to wait for weeks for a trivial fix for a production release like FreeBSD 13.2. "Affects only me" is NOT the scope this is in. Cheers, Franco
^Triage (hat on): * status * keywords * severity * URLs, including (closed) D41502 * …
Hat off. Concerning comment 27: > For reference: https://docs.freebsd.org/en/articles/pr-guidelines/ > > … That is a _terribly_ outdated point of reference. I wish for someone to continue the work that I began in D37671 | <https://github.com/freebsd/freebsd-doc/pull/100>. If this is of interest, please discuss in GitHub.
Yeah, never mind me quoting official resources. :) To be honest it would be nice if someone with knowledge of the actual code could confirm this was fixed with which particular commit. I don't think it is out of scope to ask after being scolded for raising a bug ticket that shows a panic in 13.2-RELEASE? Please guys, if you ask to raise the bar... it is bug tracker 101 to reply if an issue was fixed or not. Cheers, Franco
Anyone got an update on this? Thanks, Franco
(In reply to Franco Fichtner from comment #31) The commit appears to be ae0512bb02ccd3d878b1d0cf6ee5c77942120a21 on stable/13.
I'm sure the assignee can confirm and close. I suppose errata for 13.2-RELEASE is out of the question?
^Triage: assign to committer that resolved.
^Triage: can any of the submitters/commenters confirm that this is fixed in 13-STABLE?
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=81eb1a733dacc201a8264908cc0bb7053fdaa8e3 commit 81eb1a733dacc201a8264908cc0bb7053fdaa8e3 Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2024-05-08 08:39:08 +0000 Commit: Baptiste Daroussin <bapt@FreeBSD.org> CommitDate: 2024-05-29 15:02:08 +0000 net/miniupnpd: update to 2.3.6 PR: 273207 PR: https://redmine.pfsense.org/issues/15470 Sponsored by: Rubicon Communications, LLC ("Netgate") net/miniupnpd/Makefile | 5 +- net/miniupnpd/distinfo | 6 +- net/miniupnpd/files/patch-pf_obsdrdr.c | 164 ++++++++++++++++++++++++------- net/miniupnpd/files/patch-pf_pfpinhole.c | 88 +++++++++++++---- 4 files changed, 201 insertions(+), 62 deletions(-)