# if_pfsync: pfsync_sendout() silently and permanently fails to transmit when pfsync MTU exceeds MJUMPAGESIZE ## Synopsis If a `pfsync` interface's MTU is set above `MJUMPAGESIZE` (4096 bytes on amd64), `pfsync_sendout()` fails every packet allocation, never transmits again, and cannot recover without destroying and recreating the interface (i.e., a reboot, since `pfsync0` cannot be destroyed at runtime). ## Environment - FreeBSD 14.3-RELEASE-p16 (`stable/26.1-n272152-9b6eef552f24`, via OPNsense 26.1.11_10, amd64) - Believed applicable to any FreeBSD version with the current `pfsync_sendout()`/`m_get2()` implementation - Two-node CARP/pfsync pair, `pfsync0` MTU set to 8930 to match a jumbo-framed sync interface ## Description `pfsync_sendout()` (`sys/netpfil/pf/if_pfsync.c`) allocates its outgoing packet as: ```c m = m_get2(max_linkhdr + b->b_len, M_NOWAIT, MT_DATA, M_PKTHDR); if (m == NULL) { if_inc_counter(sc->sc_ifp, IFCOUNTER_OERRORS, 1); V_pfsyncstats.pfsyncs_onomem++; return; } ``` `m_get2()` (`sys/kern/kern_mbuf.c`) returns NULL for any request above `MJUMPAGESIZE` (`PAGE_SIZE`, 4096 on amd64): ```c if (size <= MCLBYTES) return (uma_zalloc_arg(zone_pack, &args, how)); if (size > MJUMPAGESIZE) return (NULL); ``` `pfsync_q_ins()` accumulates `b->b_len` up to the pfsync interface's configured MTU before calling `pfsync_sendout()`. If that MTU is above ~4080 bytes (`MJUMPAGESIZE - max_linkhdr`), every flush requests an allocation `m_get2()` will always refuse. **No pfsync packet is ever constructed**, silently — the failure is counted (`pfsyncs_onomem`) but nothing is logged, and `bpf_mtap()` (which would make the failure visible to `tcpdump -i pfsync0`) is never reached, since it only runs after a successful allocation. **The failure is permanent, not transient.** The `m_get2() == NULL` branch returns without resetting `b->b_len`. The only reset (`b->b_len = PFSYNC_MINPKT`) happens after a successful send. The only other path that clears it is `pfsync_drop()`, reachable only from `pfsync_clone_destroy()`. Once `b_len` exceeds the threshold, every subsequent insert pushes it further past it, and every subsequent flush fails identically — indefinitely, surviving config reloads and even, in our case, a full VM restart with the interface still misconfigured (recreating the interface with a *corrected* MTU does resolve it, since that resets `b_len`; changing the MTU on the live, already-latched instance does not). **Symptom chain observed:** with pfsync effectively unable to transmit anything, the pfsync bulk-update mechanism (used e.g. at boot, or when CARP failover occurs) also fails every attempt (times out after ~65–90s), and because `pfsync bulk start` self-demotes the requesting node's CARP priority (`carp_demote_adj_p`) until a successful `pfsync bulk done` clears it, a bulk transfer that can never succeed causes a full, repeating CARP mastership handoff to the peer every time it's attempted. ## Steps to reproduce 1. Configure `pfsync0` (via `ifconfig pfsync0 mtu <N>` or by setting it to match a syncdev interface with `N > ~4080`) with an MTU above `MJUMPAGESIZE - max_linkhdr`. 2. Generate enough pf state churn that a pfsync send bucket's `b_len` exceeds ~4080 bytes before a flush (in practice: any nontrivial amount of real traffic). 3. Observe `netstat -s -p pfsync`: `failures due to mbuf memory error` climbs continuously; `state inserts sent` and related send counters stay frozen. 4. Observe `pfsync bulk start` always followed by `pfsync bulk fail`, never `pfsync bulk done`, if CARP is in use. 5. Confirmed via DTrace (`fbt::pfsync_sendout:entry`, `fbt::pfsync_tx:entry`, `fbt::m_get2:entry`): `pfsync_sendout` executes continuously (tens of thousands of calls per few seconds under load), `pfsync_tx` is never reached, and `m_get2` is repeatedly called with a size in the 8192–16384 range, all returning NULL. ## Expected result In descending order of desirability: 1. **Best:** a pfsync MTU above `MJUMPAGESIZE` actually works — the allocator path supports it, and jumbo pfsync frames are transmitted as configured. 2. **Acceptable:** the interface self-clamps to the largest MTU the allocator can actually satisfy, so an operator's oversized request degrades to a working configuration rather than a broken one. 3. **Minimum acceptable:** an MTU the allocator cannot satisfy is rejected at set time (`SIOCSIFMTU` / clone-create) with a visible error, rather than accepted and left to fail silently at every subsequent send. Independent of which of the above is implemented, `pfsync_sendout()` should not permanently latch on an allocation failure — the queue-reset/drop behavior described below is defense-in-depth that's worth having regardless of whether (1), (2), or (3) is also fixed, since it protects against any other path that could produce an oversized `b_len`. ## Actual result Silent, total, permanent loss of pfsync transmission with no log output and no visible indication beyond the `pfsyncs_onomem` counter, until the interface is destroyed and recreated (reboot, in practice). ## Suggested fixes In order matching the cascade above: 1. **Support the configured MTU.** Use an allocation path capable of satisfying requests above `MJUMPAGESIZE` (e.g. `m_getjcl`/`m_getm2`) so a large, deliberately-configured pfsync MTU actually works rather than being silently rejected. This is the best outcome and removes the bug's precondition entirely. 2. **Self-clamp if (1) isn't feasible.** If pfsync's send path is intentionally bounded by `MJUMPAGESIZE`, have the MTU-setting path clamp any requested value down to `MJUMPAGESIZE - max_linkhdr` rather than accepting a larger value it cannot serve. 3. **Reject loudly, at minimum.** If clamping is undesirable (e.g. because a silently-lowered MTU could itself be confusing), reject an MTU above the allocator's ceiling at `SIOCSIFMTU`/clone-create time with a clear error, so the misconfiguration fails at the moment it's introduced instead of the first time traffic volume triggers a flush. 4. **Defense-in-depth, regardless of (1)–(3):** reset or drop the accumulated queue state on the `m_get2() == NULL` path in `pfsync_sendout()`, so an allocation failure — from this cause or any other — degrades gracefully (drops this batch, recovers next cycle) instead of permanently wedging the send path until the interface is recreated. ## Additional notes - This was reached downstream of an OPNsense-specific issue where `pfsync0`'s MTU is unconditionally copied from the sync interface with no upper bound (`interfaces_pfsync_configure()`, OPNsense `src/etc/inc/interfaces.inc`, referencing an earlier fix at commit `15acbad935` for the reverse problem). That OPNsense-side issue is what makes this reachable via normal configuration (any jumbo-framed interface assigned to pfsync duty) rather than requiring a deliberately unusual manual MTU. It's reported separately to OPNsense; this report is about the underlying FreeBSD kernel behavior, which affects any FreeBSD/pfsync consumer that can end up with an oversized pfsync MTU by any means. - `netstat -m` will show no mbuf/cluster exhaustion during this failure — `m_get2()`'s size check rejects the request before it reaches UMA, so the usual mbuf-pressure diagnostics look completely clean while the failure is ongoing. This is worth calling out since it's an easy place for a diagnosis to go astray (as happened in our own investigation before the `pfsyncs_onomem` counter and the source were correlated). ## How this was found (AI transparency) This root cause was identified through a collaborative debugging session between the reporter and two Claude models (Anthropic): Claude Sonnet for interactive troubleshooting, and Claude Opus for the source-level analysis once the fault was narrowed to a single function. All commands, DTrace invocations, and configuration changes were run by the reporter on live hardware; the AI proposed hypotheses and interpreted output but never had direct system access. The cause was located by: (1) using DTrace (`fbt` provider) to trace `pfsync_sendout`, `pfsync_tx`, and related functions by call count, showing `pfsync_sendout` executing continuously while `pfsync_tx` was never reached despite a nonempty, actively-filling send queue; (2) fetching and reading the actual `if_pfsync.c`/`kern_mbuf.c` source for the exact FreeBSD version in use, which located the `m_get2()` call and the `MJUMPAGESIZE` ceiling; and (3) confirming `pfsyncs_onomem` incremented at a rate matching the DTrace call volume. The fix was then confirmed empirically: lowering the sync interface's MTU and rebooting produced the three outcomes predicted from the source — send counters resuming, the failure counter reaching zero, and bulk transfers completing — observed together on the reporter's production hardware. The source excerpts, `MJUMPAGESIZE` value, DTrace results, and before/after counters in this report reflect real system output and real source code, not paraphrase or invention.
I've not yet managed to reproduce this, but your analysis appears correct. Fortunately the fix should be trivial: diff --git a/sys/netpfil/pf/if_pfsync.c b/sys/netpfil/pf/if_pfsync.c index a64d0ef64a61..db7db987197b 100644 --- a/sys/netpfil/pf/if_pfsync.c +++ b/sys/netpfil/pf/if_pfsync.c @@ -2039,7 +2039,7 @@ pfsync_sendout(int schedswi, int c) return; } - m = m_get2(max_linkhdr + b->b_len, M_NOWAIT, MT_DATA, M_PKTHDR); + m = m_get3(max_linkhdr + b->b_len, M_NOWAIT, MT_DATA, M_PKTHDR); if (m == NULL) { if_inc_counter(sc->sc_ifp, IFCOUNTER_OERRORS, 1); V_pfsyncstats.pfsyncs_onomem++; That's against main, but I expect it to just apply to 14 and 15 as well.
Glad it's so trivial to just make it work, at least up to normal jumbo frame limits (16k would be the new cap?). What do you think about also failing less badly if that cap is exceeded or allocation fails for some other reason?
(In reply to Mike Bishop from comment #2) I don't think changing the error handling buys us anything, but it will add extra risk to the change. It doesn't really matter if we drop sync updates at the head or at the tail of the queue.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=7e2781fdcfdbe489cc07572d33dc36bca06a342d commit 7e2781fdcfdbe489cc07572d33dc36bca06a342d Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2026-08-07 14:07:09 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2026-08-08 09:02:50 +0000 pfsync: handle large MTU pfsync interfaces pfsync packets were allocated with m_get2(), which can't return packets larger than MJUMPAGESIZE. As a result 9k MTU pfsync interfaces simply didn't work. Use m_get3(), which can allocate sufficiently large mbufs. Extend the pfsync:bulk test case to provoke this problem. PR: 297307 MFC after: 2 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") sys/netpfil/pf/if_pfsync.c | 2 +- tests/sys/netpfil/pf/pfsync.sh | 11 ++++++++--- 2 files changed, 9 insertions(+), 4 deletions(-)
After 7e2781fdcfdbe489cc07572d33dc36bca06a342d, the sys/netpfil/pf/pfsync:bulk testcase becomes flaky in CI and fails often. This is reproducible locally with the following command, using 200+ consecutive runs for good measure: bricoler freebsd-regression-test-suite --freebsd-src-git-checkout/url=/usr/src --freebsd-src-git-checkout/branch= --freebsd-regression-test-suite/count=200 --freebsd-regression-test-suite/tests=sys/netpfil/pf/pfsync:bulk
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=4a8bf4de7b1e5537dc576f8ad91419d324f151da commit 4a8bf4de7b1e5537dc576f8ad91419d324f151da Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2026-08-13 15:49:25 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2026-08-13 16:20:00 +0000 pfsync test: reduce the number of states to bulk-sync We created so many states that our bulk-sync occasionally caused epair to drop packets, which in turn caused the test to fail. That's not what we're testing here, make it more robust by creating fewer states. PR: 297307 Sponsored by: Rubicon Communications, LLC ("Netgate") tests/sys/netpfil/pf/pfsync.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b92d13ab60dc0eb929e66edddf6f389e5b56e50d commit b92d13ab60dc0eb929e66edddf6f389e5b56e50d Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2026-08-07 14:07:09 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2026-08-24 07:17:33 +0000 pfsync: handle large MTU pfsync interfaces pfsync packets were allocated with m_get2(), which can't return packets larger than MJUMPAGESIZE. As a result 9k MTU pfsync interfaces simply didn't work. Use m_get3(), which can allocate sufficiently large mbufs. Extend the pfsync:bulk test case to provoke this problem. PR: 297307 MFC after: 2 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") (cherry picked from commit 7e2781fdcfdbe489cc07572d33dc36bca06a342d) sys/netpfil/pf/if_pfsync.c | 2 +- tests/sys/netpfil/pf/pfsync.sh | 11 ++++++++--- 2 files changed, 9 insertions(+), 4 deletions(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=ef676ecff1c36b9273394c5628313e65ba420a20 commit ef676ecff1c36b9273394c5628313e65ba420a20 Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2026-08-07 14:07:09 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2026-08-24 07:17:28 +0000 pfsync: handle large MTU pfsync interfaces pfsync packets were allocated with m_get2(), which can't return packets larger than MJUMPAGESIZE. As a result 9k MTU pfsync interfaces simply didn't work. Use m_get3(), which can allocate sufficiently large mbufs. Extend the pfsync:bulk test case to provoke this problem. PR: 297307 MFC after: 2 weeks Sponsored by: Rubicon Communications, LLC ("Netgate") (cherry picked from commit 7e2781fdcfdbe489cc07572d33dc36bca06a342d) sys/netpfil/pf/if_pfsync.c | 2 +- tests/sys/netpfil/pf/pfsync.sh | 11 ++++++++--- 2 files changed, 9 insertions(+), 4 deletions(-)
The test still seems to be quite flaky even after 4a8bf4de7b1e5537dc576f8ad91419d324f151da: https://ci.freebsd.org/job/FreeBSD-main-amd64-test/29414/testReport/sys.netpfil.pf/pfsync/bulk/history/
Any updates at looking into this? pfsync:bulk still fails occasionally in CI: https://ci.freebsd.org/job/FreeBSD-main-amd64-test/29823/testReport/sys.netpfil.pf/pfsync/bulk/ (from 8 hours ago).
(In reply to Siva Mahadevan from comment #10) So far I’ve been completely unable to reproduce this, which makes fixing it a bit difficult. I just started another build in a test vm to try again.
Created attachment 276666 [details] kyua test report I can reproduce it very consistently within a few iterations of a 200+ count run. Here's the reproducer on amd64+bhyve with Bricoler: bricoler freebsd-regression-test-suite --freebsd-src-git-checkout/url=/usr/src --freebsd-src-git-checkout/branch= --freebsd-regression-test-suite/count=200 --freebsd-regression-test-suite/tests=sys/netpfil/pf/pfsync:bulk I've attached the resulting test report from this local run.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c6a520af95b206d6e6a603e95314398d31f57def commit c6a520af95b206d6e6a603e95314398d31f57def Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2026-10-30 09:48:26 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2026-10-30 14:14:13 +0000 pfsync: improve handling of large packets On setups with pfsync over large MTUs (> 2K) we could fail to parse packets because m_pulldown() fails on packets larger than MCLBYTES. Instead use m_copydata() to copy out the individual chunks we're interested in. This should also fix the flaky pfsync:bulk test case. Depending on the hash seed states would or would not end up synced. PR: 297307 Sponsored by: Rubicon Communications, LLC ("Netgate") sys/netpfil/pf/if_pfsync.c | 212 ++++++++++++++++++++------------------------- 1 file changed, 93 insertions(+), 119 deletions(-)