Bug 289628 - sys/netpfil/pf/nat:endpoint_independent_pass test fails intermittently in CI
Summary: sys/netpfil/pf/nat:endpoint_independent_pass test fails intermittently in CI
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: tests (show other bugs)
Version: 16.0-CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Tom Jones
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2025-09-15 17:30 UTC by Siva Mahadevan
Modified: 2026-04-28 00:13 UTC (History)
3 users (show)

See Also:


Attachments
kyua test report (2.50 KB, text/plain)
2025-09-15 17:30 UTC, Siva Mahadevan
no flags Details
[PATCH] tests/pf/nat: temporarily skip endpoint_independent_pass testcase (1.03 KB, patch)
2025-10-23 13:04 UTC, Siva Mahadevan
no flags Details | Diff
[PATCH v2] tests/pf/nat: temporarily skip endpoint_independent_{compat,pass} testcases (1.17 KB, patch)
2025-11-06 18:24 UTC, Siva Mahadevan
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Siva Mahadevan freebsd_committer freebsd_triage 2025-09-15 17:30:24 UTC
Created attachment 263827 [details]
kyua test report

The 'sys/netpfil/pf/nat:endpoint_independent_pass' testcase fails intermittently on aarch64 in CI:


===> sys/netpfil/pf/nat:endpoint_independent_pass
Result:     failed: server1 did not receive connection from client (default)

Standard output:
net.inet.ip.forwarding: 0 -> 1
add net default: gateway 192.0.2.1

Standard error:
pf enabled
Ethernet rules cleared
rules cleared
nat cleared
0 tables deleted.
rules cleared
0 states cleared
source tracking entries cleared
pf: statistics cleared
pf: interface flags reset
tcpdump: listening on epair3a, link-type EN10MB (Ethernet), snapshot length 262144 bytes
tcpdump: listening on epair2a, link-type EN10MB (Ethernet), snapshot length 262144 bytes
8 packets captured9 packets captured
18 packets received by filter
0 packets dropped by kernel

16 packets received by filter
0 packets dropped by kernel
tcpdump: truncated dump file; tried to read 4 file header bytes, only got 0
reading from file /tmp/kyua.c0arWO/2/work/server2.pcap, link-type EN10MB (Ethernet), snapshot length 262144



I have reproduced this error using Bricoler with 100 runs for good measure and attached the failing test report in this bug:

$ bricoler run freebsd-src-regression-suite --param freebsd-src:url=/usr/src --param freebsd-src:branch= --param freebsd-src-regression-suite:hypervisor=bhyve --param freebsd-src-regression-suite:memory=4096 --param freebsd-src-regression-suite:ncpus=2 --param freebsd-src-regression-suite:parallelism=1 --param freebsd-src-regression-suite:count=100 --param freebsd-src-regression-suite:tests='sys/netpfil/pf/nat:endpoint_independent_pass'

@ks, since you made the last significant edit to the tests in https://cgit.freebsd.org/src/commit/?id=e0fe26691fc98a16cdda9d4f4beea9c5698ac64a (pf: Add modern NAT syntax), please triage as necessary.
Comment 1 Kristof Provost freebsd_committer freebsd_triage 2025-09-15 18:12:18 UTC
Assign to thj@ who committed the endpoint independent nat code.
Comment 2 Tom Jones freebsd_committer freebsd_triage 2025-09-16 08:29:29 UTC
It looks like an issue with the pcap, maybe we are killing tcpdump too early. I'll have a look.
Comment 3 Siva Mahadevan freebsd_committer freebsd_triage 2025-10-08 14:30:46 UTC
I was able to repro this on amd64 as well, so it is not aarch64 specific. Updating bug metdata as required.
Comment 4 Siva Mahadevan freebsd_committer freebsd_triage 2025-10-23 13:04:24 UTC
Created attachment 264804 [details]
[PATCH] tests/pf/nat: temporarily skip endpoint_independent_pass testcase

To help get CI for main and stable/15 clean before the 15.0 release, I'm attaching a patch to temporarily skip the testcase against this PR. We can look into fixing this afterwards.
Comment 5 Siva Mahadevan freebsd_committer freebsd_triage 2025-11-06 18:24:22 UTC
Created attachment 265212 [details]
[PATCH v2] tests/pf/nat: temporarily skip endpoint_independent_{compat,pass} testcases

The 'endpoint_independent_compat' testcase also intermittently fails, so I'm adding a skip for it in v2.
Comment 6 Siva Mahadevan freebsd_committer freebsd_triage 2026-04-02 14:04:17 UTC
https://reviews.freebsd.org/D56145 submitted a "fix" here that matches the sleeps found in other test cases. This seems to solve the flakiness.
Comment 7 commit-hook freebsd_committer freebsd_triage 2026-04-23 13:14:03 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=60000d94b4e85caf931230a8a7ae86888db00d01

commit 60000d94b4e85caf931230a8a7ae86888db00d01
Author:     Siva Mahadevan <siva@FreeBSD.org>
AuthorDate: 2026-04-22 17:20:38 +0000
Commit:     Siva Mahadevan <siva@FreeBSD.org>
CommitDate: 2026-04-23 13:11:08 +0000

    tests/pf/nat: endpoint_independent: wait for tcpdump to start

    This matches pflog.sh tests and fixes flaky tests found in CI[0].

    [0] https://ci.freebsd.org/job/FreeBSD-main-amd64-test/28126/testReport/junit/sys.netpfil.pf/nat/endpoint_independent_exhaust/

    PR:     289628
    Reviewed by:    glebius
    Approved by:    lwhsu (mentor)
    Sponsored by:   The FreeBSD Foundation
    MFC after:      3 days
    Differential Revision:  https://reviews.freebsd.org/D56145

 tests/sys/netpfil/pf/nat.sh | 44 ++++++++++++++++++++++++++++----------------
 1 file changed, 28 insertions(+), 16 deletions(-)
Comment 8 commit-hook freebsd_committer freebsd_triage 2026-04-28 00:10:50 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=8c84a6ebbe166710be42815d29da683363b385a8

commit 8c84a6ebbe166710be42815d29da683363b385a8
Author:     Siva Mahadevan <siva@FreeBSD.org>
AuthorDate: 2026-04-22 17:20:38 +0000
Commit:     Siva Mahadevan <siva@FreeBSD.org>
CommitDate: 2026-04-27 20:15:16 +0000

    tests/pf/nat: endpoint_independent: wait for tcpdump to start

    This matches pflog.sh tests and fixes flaky tests found in CI[0].

    [0] https://ci.freebsd.org/job/FreeBSD-main-amd64-test/28126/testReport/junit/sys.netpfil.pf/nat/endpoint_independent_exhaust/

    PR:     289628
    Reviewed by:    glebius
    Approved by:    lwhsu (mentor)
    Sponsored by:   The FreeBSD Foundation
    MFC after:      3 days
    Differential Revision:  https://reviews.freebsd.org/D56145

    (cherry picked from commit 60000d94b4e85caf931230a8a7ae86888db00d01)

 tests/sys/netpfil/pf/nat.sh | 44 ++++++++++++++++++++++++++++----------------
 1 file changed, 28 insertions(+), 16 deletions(-)
Comment 9 Siva Mahadevan freebsd_committer freebsd_triage 2026-04-28 00:13:49 UTC
Closing as fixed.