Created attachment 263728 [details] kyua test report The 'sys/netpfil/pf/route_to:prefer_ipv6_nexthop_mixed_af_random_table_ipv4' testcase fails intermittently in CI: ===> sys/netpfil/pf/route_to:prefer_ipv6_nexthop_mixed_af_random_table_ipv4 Result: failed: Target 2001:db8:4202::42:4 not selected after 10 attempts! I have reproduced this error using Bricoler with 100 runs for good measure and attached the failing test report in this bug: $ bricoler run freebsd-src-regression-suite --param freebsd-src:url=/usr/src --param freebsd-src:branch= --param freebsd-src-regression-suite:hypervisor=bhyve --param freebsd-src-regression-suite:memory=4096 --param freebsd-src-regression-suite:ncpus=2 --param freebsd-src-regression-suite:parallelism=1 --param freebsd-src-regression-suite:count=100 --param freebsd-src-regression-suite:tests='sys/netpfil/pf/route_to:prefer_ipv6_nexthop_mixed_af_random_table_ipv4' @ks, since you made the last significant edit to the tests in https://cgit.freebsd.org/src/commit/?id=65c318630123fcf2b6f491bf4d02a5cad3031d20 (pf: Add prefer-ipv6-nexthop option for route-to pools), please triage as necessary.
The problem with this test and a few other similar ones is that they test random selection of pf loadbalancing. The test uses a route-to action with lists of good and bad nexthops. For each good nexthop it attempts a tcp connection up to 10 times and checks if this good nexthop was selected. Since it's testing the *random* algorithm it can occasionally fail. Even if I raise the amount of connections being made it could still fail. Any idea how to better test a random algorithm?
(In reply to Kajetan Staszkiewicz from comment #1) I'm afraid I can't think of anything better than 'ensure the sample size is large enough to make failure very unlikely'.
These tests are continuing to fail in CI (https://ci.freebsd.org/view/Test/job/FreeBSD-main-amd64-test/26918/), so if there is no other way to deterministically seed the random algorithm, then please go ahead and increase the sample size to something like 100 instead.
Any updates on this? This is one of the only remaining failing tests in CI. Can we go ahead and increase the sample size as per Comment 2?
Ping on this bug, can we get confirmation on Comment 2? These tests are still intermittently failing in CI: https://ci.freebsd.org/view/Test/job/FreeBSD-main-amd64-test/27448/testReport/sys.netpfil.pf/route_to/random_table/
(In reply to Siva Mahadevan from comment #5) I'd just leave the test as is and wait for https://reviews.freebsd.org/D54105 to land. It's an obvious candidate for that annotation. As are a number of the dummynet tests. Anything probabilistic is going to have this problem, and Igor has a more systematic solution to it.
Created attachment 271422 [details] flakiness fix patch Not sure if I should bump this bug to report this issue, but while attempting to fix the determinism of the route_to "random_check" testcases, I can consistently repro a panic. I've attached a patch for testing, and I used this command line to reproduce it (with the latest Bricoler): bricoler freebsd-regression-test-suite --freebsd-src-git-checkout/url=/usr/src --freebsd-src-git-checkout/branch= --freebsd-regression-test-suite/memory=3072 --freebsd-regression-test-suite/ncpus=2 --freebsd-regression-test-suite/parallelism=1 --freebsd-regression-test-suite/count=300 --freebsd-regression-test-suite/tests='sys/netpfil/pf/route_to:random_table' Panic: sys/netpfil/pf/route_to:random_table -> panic: failed to reclaim memory cpuid = 1 time = 1780399737 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame 0xfffffe005d38f940 vpanic() at vpanic+0x149/frame 0xfffffe005d38fa70 panic() at panic+0x43/frame 0xfffffe005d38fad0 vm_pageout_oom() at vm_pageout_oom+0x6b8/frame 0xfffffe005d38fb30 vm_pageout_worker() at vm_pageout_worker+0x760/frame 0xfffffe005d38feb0 vm_pageout() at vm_pageout+0x1d7/frame 0xfffffe005d38fef0 fork_exit() at fork_exit+0x82/frame 0xfffffe005d38ff30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe005d38ff30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- KDB: enter: panic [ thread pid 6 tid 100062 ]
(In reply to Siva Mahadevan from comment #7) Hi Siva! This panic should be reported as a separate issue and I would coordinate developing a regression test with Peter Holm (pho).
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1294f332aec02e111b47d69e0d91e32dc2624001 commit 1294f332aec02e111b47d69e0d91e32dc2624001 Author: Siva Mahadevan <siva@FreeBSD.org> AuthorDate: 2026-06-17 16:52:00 +0000 Commit: Siva Mahadevan <siva@FreeBSD.org> CommitDate: 2026-06-17 16:53:14 +0000 tests/pf/route_to: check_random: rely on test timeout to fail This test is known to be flaky if 10 attempts happens to be not enough for the test to pass. Usually, this test passes in ~4-6 attempts. See https://ci.freebsd.org/job/FreeBSD-main-amd64-test/28664/testReport/junit/sys.netpfil.pf/route_to/random_table/ Rely on the timeout (default 5 mins) to catch a failure here, rather than just 10 attempts. It's very unlikely that 5 mins worth of attempts still isn't enough. For a history of flakiness, see https://ci.freebsd.org/job/FreeBSD-main-amd64-test/lastSuccessfulBuild/testReport/junit/sys.netpfil.pf/route_to/random_table/history/ PR: 289477 Reviewed by: kp MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D57408 tests/sys/netpfil/pf/route_to.sh | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=a13936fcdc8dc3f37125614daa424a9cf1ae380d commit a13936fcdc8dc3f37125614daa424a9cf1ae380d Author: Siva Mahadevan <siva@FreeBSD.org> AuthorDate: 2026-06-17 16:52:00 +0000 Commit: Siva Mahadevan <siva@FreeBSD.org> CommitDate: 2026-06-26 15:16:43 +0000 tests/pf/route_to: check_random: rely on test timeout to fail This test is known to be flaky if 10 attempts happens to be not enough for the test to pass. Usually, this test passes in ~4-6 attempts. See https://ci.freebsd.org/job/FreeBSD-main-amd64-test/28664/testReport/junit/sys.netpfil.pf/route_to/random_table/ Rely on the timeout (default 5 mins) to catch a failure here, rather than just 10 attempts. It's very unlikely that 5 mins worth of attempts still isn't enough. For a history of flakiness, see https://ci.freebsd.org/job/FreeBSD-main-amd64-test/lastSuccessfulBuild/testReport/junit/sys.netpfil.pf/route_to/random_table/history/ PR: 289477 Reviewed by: kp MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D57408 (cherry picked from commit 1294f332aec02e111b47d69e0d91e32dc2624001) tests/sys/netpfil/pf/route_to.sh | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-)