Since 14.0 (starting with 14.1) I have had an intermittent problem with my server which, previously, ran for weeks without a problem. Now it spontaneously reboots. This is the dump info I get (every time): Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 04 fault virtual address = 0x1d fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80d3c002 stack pointer = 0x0:0xfffffe00828d2b40 frame pointer = 0x0:0xfffffe00828d2bc0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2 (clock (0)) rdi: 0000000000000000 rsi: 0000000000000000 rdx: fffffe00828d2d28 rcx: 0000000000000000 r8: 0000000000000000 r9: 0000000000000000 rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe00828d2bc0 r10: 00000000000004fc r11: 000000000000144a r12: fffff801cdea36d0 r13: 0000000000000034 r14: fffff80041ed2000 r15: 00000000000004fc trap number = 12 panic: page fault cpuid = 2 time = 1755367015 KDB: stack backtrace: #0 0xffffffff80ba8f1d at kdb_backtrace+0x5d #1 0xffffffff80b5aa11 at vpanic+0x161 #2 0xffffffff80b5a8a3 at panic+0x43 #3 0xffffffff8104db7a at trap_pfault+0x3da #4 0xffffffff81023d88 at calltrap+0x8 #5 0xffffffff80d3a2de at tcp_default_output+0x121e #6 0xffffffff80d4b0cd at tcp_timer_delack+0x6d #7 0xffffffff80d4a064 at tcp_timer_enter+0x104 #8 0xffffffff80b78c79 at softclock_call_cc+0x129 #9 0xffffffff80b7a445 at softclock_thread+0xe5 #10 0xffffffff80b13641 at fork_exit+0x81 #11 0xffffffff81024dee at fork_trampoline+0xe Uptime: 11d15h43m51s Dumping 986 out of 8042 MB:..2%..12%..22%..31%..41%..51%..62%..72%..82%..91% Dump complete Automatic reboot in 15 seconds - press a key on the console to abort ---<<BOOT>>--- I have tried changing the RAM and changing network adapter. I have swapped in a backup server with the same hardware. None of that has fixed it. However, I put in a backup with 14.0 and it did not have the problem. Something seems to have changed with 14.1 (and going forward through 14.3). Because it's intermittent it takes days, sometimes weeks for the problem to recur so it's hard to know if a change makes a difference without waiting. I have found no way to otherwise replicate the reboot. Thanks...
Created attachment 263053 [details] non-debug kernel crashinfo I seem to have noticed the same kernel crashes. At the moment I have no clue how to reproduce this crash, but it seems to happen randomly. For example, based on 1 machine: Before the first crash, the machine uptime was about 1 month. The second crash happened only 6 days after the first crash. I have attached a crashinfo text file (unfortunately from a non-debug kernel dump). Hopefully it helps a little bit.
Is it possible for you to enable Black Box Logging for all TCP connections and when the core happens extract it and send it to me? I will code some memory when running. Black Box Logging provides information about the system calls, the timers and the packets being sent and received without containing any payload. It just provides the IP address of the peer. So it should be OK from a security perspective. Let me know if it is possible and I will provide instructions for enabling it. If I have it, I might be able to reproduce it and then fix it.
(In reply to Michael Tuexen from comment #2) Thank you for your response. I think it's possible to enable this black box logging, so could you please provide the instructions for enabling it? I will try to set this up on some machine and provide the logs when the next crash occurs.
(In reply to Eero Hänninen from comment #3) Thanks for helping to debug the problem. I put the instructions up in the WiKi: https://wiki.freebsd.org/MichaelTuexen If you have any questions or suggestions to improve the instructions, please let me know.
Comment #5 Hi Michael... Thanks for your help with this frustrating problem. I have set up a 14.3 box with the instructions you provided for enabling TCP Blackbox logging. In my case where I'm running several websites and an Internet based streaming radio station I don't want this to be my main server. If/when it reboots it knocks all of my listeners off the station. Do you know of a way to stress the Internet connection in an attempt to force a reboot? In my experience this problem is more likely when the connection is busy. As it stands now, this "experimental" server isn't seeing much traffic. (In reply to Michael Tuexen from comment #4)
(In reply to Allan Sniffen from comment #5) Since I don't know the problem, I don't know how to trigger it. But when the panic happens, you have to write a core file. If it just reboots, we don't have the core to extract the information from. If you don't have a memory constrained server, you might enable BBLog on it. If you run into any problem, you can issue `sysctl net.inet.tcp.bb.disable_all=1` and BBlog is disabled.
Minor Update: The 14.3 "Test Server" has not had a panic after 12 days so I am unable to provide a trace as requested. I'm not sure it will fail since it's not doing much. When it's the primary server it manages multiple web sites (Apache) and a radio station (streaming servers). My current primary server is set back to 14.0 and has not had a panic in 20 days under typical load. Whatever the issue, it's a result of something being changed between 14.0 and 14.1. I'd look there to see what was changed. In the meantime I'll keep the test server up (on 14.3) and hope it has a panic at some point.
*** Bug 289666 has been marked as a duplicate of this bug. ***
Has any progress been made with addressing this bug? I would like to upgrade past 14.0. I don't feel comfortable doing so if the page fault problem is still present. Thanks to all who are trying!
(In reply to Allan Sniffen from comment #9) No progress has been made since no new information could be gathered.
The problem might be known and fixed by bffebc336f4ece4d18774c1ab8f555802cebf961. But this revision didn't make it to stable/14 and it depends on other changes, that are too invasive for MFC. Can you please try to switch to 15.0-RELEASE?
(In reply to Gleb Smirnoff from comment #11) Gleb, based on this request I am in the process of setting up a 15.0 server. It takes some time and then has to be put online. Based on my experience with this problem the server has to be under load so simply having it running locally isn't enough. I'll let you know when it's online -- probably in about a week. It will then be a waiting game to see what happens. It can take weeks for the problem to occur on v 14.3. Thanks.
(In reply to Allan Sniffen from comment #12) As suggested I installed v 15.0 and have now been running it online for 30 days (and counting) without a crash. On 14.3 it would have crashed by now. I therefore conclude that Gleb's assessment that this problem was corrected in v 15 is correct. Should the crash recur I'll repost (I doubt it). Thanks to those who helped investigate the problem!
Thanks for update, Allan! Eero, you also reported similar crashes. Are you still having the problem? May I advice upgrade to 15.0-RELEASE?
(In reply to Gleb Smirnoff from comment #14) For me, the crashes disappeared after updating to 14.3-p5. Maybe I was hitting a different bug? I haven't tested 15.0-RELEASE on heavily loaded web servers yet, so I can't say whether it's stable under high load. Low and medium loaded servers have been running fine on 15.0 though.
I had a similar crash today with 15.0-RELEASE-p9: May 30 12:37:38 hades dhclient[97393]: My address (192.168.0.121) was deleted, dhclient exiting May 30 12:37:38 hades dhclient[97393]: connection closed May 30 12:37:38 hades dhclient[97393]: exiting. May 30 12:37:38 hades wpa_supplicant[97627]: wlan0: CTRL-EVENT-DISCONNECTED bssid=ec:a8:1f:bc:8 6:98 reason=3 locally_generated=1 May 30 12:37:38 hades wpa_supplicant[97627]: wlan0: CTRL-EVENT-DSCP-POLICY clear_all May 30 12:37:38 hades syslogd: last message repeated 1 times May 30 12:37:38 hades wpa_supplicant[97627]: wlan0: CTRL-EVENT-TERMINATING May 30 12:37:38 hades rtsold[13529]: <rtsock_input_ifannounce> interface wlan0 removed May 30 12:37:38 hades kernel: wlan0: link state changed to DOWN May 30 12:38:48 hades syslogd: kernel boot file is /boot/kernel/kernel May 30 12:38:48 hades kernel: May 30 12:38:48 hades syslogd: last message repeated 1 times May 30 12:38:48 hades kernel: Fatal trap 12: page fault while in kernel mode May 30 12:38:48 hades kernel: cpuid = 1; apic id = 01 May 30 12:38:48 hades kernel: fault virtual address = 0x10 May 30 12:38:48 hades kernel: fault code = supervisor read data, page not present May 30 12:38:48 hades kernel: instruction pointer = 0x20:0xffffffff80d8e2f5 May 30 12:38:48 hades kernel: stack pointer = 0x28:0xfffffe013da118d0 May 30 12:38:48 hades kernel: frame pointer = 0x28:0xfffffe013da11900 May 30 12:38:48 hades kernel: code segment = base rx0, limit 0xfffff, type 0x1b May 30 12:38:48 hades kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 May 30 12:38:48 hades kernel: processor eflags = interrupt enabled, resume, IOPL = 0 May 30 12:38:48 hades kernel: current process = 25609 (Socket Thread) May 30 12:38:48 hades kernel: rdi: fffff8000e9b7000 rsi: 000000000000001c rdx: fffff801b3e30a78 May 30 12:38:48 hades kernel: rcx: fffff8000e9b7000 r8: 00000000ffffffbd r9: 0000000000000000 May 30 12:38:48 hades kernel: rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe013da11900 May 30 12:38:48 hades kernel: r10: fffffe013da11880 r11: 0000000000000008 r12: 0000000000018700 May 30 12:38:48 hades kernel: r13: 0000000000000000 r14: fffffe013da118dc r15: fffff8002cc26900 May 30 12:38:48 hades kernel: trap number = 12 May 30 12:38:48 hades kernel: panic: page fault May 30 12:38:48 hades kernel: cpuid = 1 May 30 12:38:48 hades kernel: time = 1780137458 May 30 12:38:48 hades kernel: KDB: stack backtrace: May 30 12:38:48 hades kernel: #0 0xffffffff80bbe18d at kdb_backtrace+0x5d May 30 12:38:48 hades kernel: #1 0xffffffff80b71506 at vpanic+0x136 May 30 12:38:48 hades kernel: #2 0xffffffff80b713c3 at panic+0x43 May 30 12:38:48 hades kernel: #3 0xffffffff81079f99 at trap_pfault+0x3c9 May 30 12:38:48 hades kernel: #4 0xffffffff81050018 at calltrap+0x8 May 30 12:38:48 hades kernel: #5 0xffffffff80d5a1be at tcp_default_output+0x1c8e May 30 12:38:48 hades kernel: #6 0xffffffff80d6bcda at tcp_usr_send+0x1fa May 30 12:38:48 hades kernel: #7 0xffffffff80c1960f at sosend_generic_locked+0x5ef May 30 12:38:48 hades kernel: #8 0xffffffff80c18ef1 at sosend_generic+0x61 May 30 12:38:48 hades kernel: #9 0xffffffff80c19b6f at sousrsend+0x5f May 30 12:38:48 hades kernel: #10 0xffffffff80c21b60 at kern_sendit+0x1c0 May 30 12:38:48 hades kernel: #11 0xffffffff80c21e88 at sendit+0x1a8 May 30 12:38:48 hades kernel: #12 0xffffffff80c21ccd at sys_sendto+0x4d May 30 12:38:48 hades kernel: #13 0xffffffff8107a8d6 at amd64_syscall+0x126 May 30 12:38:48 hades kernel: #14 0xffffffff8105090b at fast_syscall_common+0xf8 May 30 12:38:48 hades kernel: Uptime: 3h19m57s May 30 12:38:48 hades kernel: May 30 12:38:48 hades kernel: Dump failed. Partition too small (about 2414MB were needed this time). May 30 12:38:48 hades kernel: Automatic reboot in 15 seconds - press a key on the console to abort May 30 12:38:48 hades kernel: --> Press a key on the console to reboot, May 30 12:38:48 hades kernel: --> or switch off the system now. May 30 12:38:48 hades kernel: ---<<BOOT>>--- This happend while I run "service netif restart wlan0"[0]. I'm not sure if this is connected and if this is the same bug (because the stack trace is a bit different). [0] I sometimes have the problem that ipv6 stops working and restarting the interface works a workaround. But this is another problem.
I have seen this on my FreeBSD 15.0-p9 too: #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80b55969 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:519 #3 0xffffffff80b55e77 in vpanic (fmt=0xffffffff811a72af "%s", ap=ap@entry=0xfffffe011f345730) at /usr/src/sys/kern/kern_shutdown.c:974 #4 0xffffffff80b55ca3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:887 #5 0xffffffff81053c08 in trap_fatal (frame=<optimized out>, eva=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:969 #6 0xffffffff81053c08 in trap_pfault (frame=0xfffffe011f3457b0, usermode=false, signo=<optimized out>, ucode=<optimized out>) #7 <signal handler called> #8 m_copydata (m=0x0, m@entry=0xfffff80b6ea88100, off=0, len=len@entry=1, cp=<optimized out>) at /usr/src/sys/kern/uipc_mbuf.c:694 #9 0xffffffff80d35cb8 in tcp_default_output (tp=0xfffff801d7df1000) at /usr/src/sys/netinet/tcp_output.c:1080 #10 0xffffffff80d2dce3 in tcp_output (tp=0xfffff80b6ea88100, tp@entry=0xfffff801d7df1000) at /usr/src/sys/netinet/tcp_var.h:662 #11 0xffffffff80d2c372 in tcp_do_segment (tp=0xfffff801d7df1000, m=<optimized out>, th=0xfffff8014447c296, drop_hdrlen=100, tlen=0, iptos=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:2842 #12 0xffffffff80d29b8c in tcp_input_with_port (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>, port=port@entry=0) at /usr/src/sys/netinet/tcp_input.c:1160 #13 0xffffffff80d29fff in tcp6_input_with_port (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>, port=0) at /usr/src/sys/netinet/tcp_input.c:582 #14 tcp6_input (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:589 #15 0xffffffff80d6d14e in ip6_input (m=0x0) at /usr/src/sys/netinet6/ip6_input.c:963 #16 0xffffffff80c9b668 in netisr_process_workstream_proto (nwsp=0xfffffe00469d5e00, proto=6) at /usr/src/sys/net/netisr.c:926 #17 swi_net (arg=0xfffffe00469d5e00) at /usr/src/sys/net/netisr.c:973 #18 0xffffffff80b11709 in intr_event_execute_handlers (ie=0xfffff80001482a00, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1191 #19 ithread_execute_handlers (ie=0xfffff80001482a00, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1204 #20 ithread_loop (arg=arg@entry=0xfffff80001401c20) at /usr/src/sys/kern/kern_intr.c:1297 #21 0xffffffff80b0dcab in fork_exit (callout=0xffffffff80b114d0 <ithread_loop>, arg=0xfffff80001401c20, frame=0xfffffe011f345f40) at /usr/src/sys/kern/kern_fork.c:1153 I have a full memory dump, so I can provide additional information.
Note that in the original bug report the stack trace is a TCP timer, not a response to TCP input. Kajetan's and satanist's panics are a different one. I would suggest to open a new bug report to reduce confusion.
Sure, no problem https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=295870 .
I have the panic with the similar backtrace, but the system is based on 15.0-CURRENT: kern.osreldate: 1500025 Not sure it is worth investigating, I'll keep the core for a while. Unread portion of the kernel message buffer: panic: error while FIN with SACK rxmit cpuid = 6 time = 1781851312 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00e41f0a10 vpanic() at vpanic+0x13f/frame 0xfffffe00e41f0b40 panic() at panic+0x43/frame 0xfffffe00e41f0ba0 tcp_default_output() at tcp_default_output+0x2e14/frame 0xfffffe00e41f0d70 tcp_timer_rexmt() at tcp_timer_rexmt+0x6b6/frame 0xfffffe00e41f0dd0 tcp_timer_enter() at tcp_timer_enter+0x15e/frame 0xfffffe00e41f0e10 softclock_call_cc() at softclock_call_cc+0x139/frame 0xfffffe00e41f0ec0 softclock_thread() at softclock_thread+0xc6/frame 0xfffffe00e41f0ef0 fork_exit() at fork_exit+0x82/frame 0xfffffe00e41f0f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00e41f0f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 69d22h48m39s (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 #1 doadump (textdump=textdump@entry=1) at /usr/src/sys/kern/kern_shutdown.c:404 #2 0xffffffff80b534c0 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:524 #3 0xffffffff80b539dc in vpanic (fmt=0xffffffff8129750b "error while FIN with SACK rxmit", ap=ap@entry=0xfffffe00e41f0b80) at /usr/src/sys/kern/kern_shutdown.c:979 #4 0xffffffff80b53823 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:892 #5 0xffffffff80d57604 in tcp_default_output (tp=0xfffff80fcd2f6a80, tp@entry=<error reading variable: value is not available>) at /usr/src/sys/netinet/tcp_output.c:1675 #6 0xffffffff80d69636 in tcp_output_locked (tp=0xfffff80fcd2f6a80) at /usr/src/sys/netinet/tcp_timer.c:298 #7 tcp_timer_rexmt (tp=0xfffff80fcd2f6a80) at /usr/src/sys/netinet/tcp_timer.c:815 #8 0xffffffff80d68b5e in tcp_timer_enter (xtp=0xfffff80fcd2f6a80) at /usr/src/sys/netinet/tcp_timer.c:890 #9 0xffffffff80b71f99 in softclock_call_cc (c=0xfffff80fcd2f6c20, cc=cc@entry=0xffffffff8398a300, direct=direct@entry=0) at /usr/src/sys/kern/kern_timeout.c:714 #10 0xffffffff80b73606 in softclock_thread (arg=arg@entry=0xffffffff8398a300) at /usr/src/sys/kern/kern_timeout.c:846 #11 0xffffffff80b08052 in fork_exit (callout=0xffffffff80b73540 <softclock_thread>, arg=0xffffffff8398a300, frame=0xfffffe00e41f0f40) at /usr/src/sys/kern/kern_fork.c:1151
Andrey, supposed fix bffebc336f4ece4d18774c1ab8f555802cebf961 landed after f3dbef108212460489ae68b4c47e20b73984f433, which bumped version to 1500026.
Feel free to re-open if anybody reproduces the problem with 15.0-RELEASE or newer.
(In reply to Gleb Smirnoff from comment #21) Could you please describe the exact timeline of the race fixed by bffebc336f4ece4d18774c1ab8f555802cebf961?
(In reply to firk from comment #23) nvm, the question is resolved
(In reply to Gleb Smirnoff from comment #11) Looks like it is PR 276761 (tcp_close() vs tcp_discardcb() timers desync). And yes, it was not MFCed to 14.x. The commit bffebc336f4ece4d18774c1ab8f555802cebf961 is related because it fixes a race introduced by PR 276761 fix, but it is not a direct source of these panics. I made a small patch which is expected to fix the problem in 14.x branch. Did not tested it much for now. I don't think it is a good idea to keep supported 14.4 release in unusable state. --- sys/netinet/tcp_subr.c.orig 2026-08-01 23:05:43.997108869 +0300 +++ sys/netinet/tcp_subr.c 2026-08-05 11:51:31.431921701 +0300 @@ -2598,6 +2598,16 @@ tcp_fastopen_decrement_counter(tp->t_tfo_pending); tp->t_tfo_pending = NULL; } + + /* + * tcp_timer_stop() may drop INP_WLOCK internally + * this may lead to racy two-thread closing in rare cases + * so just reset tp->t_timers + * tcp_timer_enter() is aware of this + */ + for (tt_which i = 0; i < TT_N; i++) + tp->t_timers[i] = SBT_MAX; + if (tp->t_fb->tfb_tcp_timer_stop_all != NULL) tp->t_fb->tfb_tcp_timer_stop_all(tp); in_pcbdrop(inp); --- sys/netinet/tcp_timer.c.orig 2026-08-01 23:05:44.001110870 +0300 +++ sys/netinet/tcp_timer.c 2026-08-05 12:17:39.251384522 +0300 @@ -874,6 +874,11 @@ curthread->td_pflags |= TDP_INTCPCALLOUT; which = tcp_timer_next(tp, NULL); + if (which == TT_N) { /* see tcp_close() comment about stoping timers */ + INP_WUNLOCK(inp); + curthread->td_pflags &= ~TDP_INTCPCALLOUT; + return; + } MPASS(which < TT_N); tp->t_timers[which] = SBT_MAX; tp->t_precisions[which] = 0; @@ -909,6 +914,10 @@ #endif INP_WLOCK_ASSERT(inp); + if (tp->t_state == TCPS_CLOSED) { +/* printf("tcp_timer_activate(%p,%d,%u) for TCPS_CLOSED connection, ignoring\n", tp, (int)which, delta);*/ + return; + } if (delta > 0) { what = TT_STARTING; The commented printf() will often print this warnings for TT_REXMT (0) if enabled, but I don't think these silently skipped timer activations will harm (they seems fixed in 15.x in some other place).
> I don't think it is a good idea to keep supported 14.4 release in unusable state. "Unusable" is too far fetched description for a race that only very few people hit. Those people who experience the panic can upgrade to 15.1-RELEASE. I do not object for an ad-hoc fix in the stable/14 branch. btw, I briefly reviewed your patch and it seems to be okay. But you need to find a committer who is willing to take responsibility. I don't have extra time for responsibilities in legacy stable branch. This person doesn't need to be a TCP expert, just be responsible for any kind of fallout that could be caused by a patch that has not been run through the main branch.