I run multi-fib IPv6-only desktop/router/firewall/virtualization all-in-one host and I see two regressions after updating post base d05d1f256082. I'm now at base 4d37c2ea3c86 (with cherry-picked base 30d4d3db431a build fix) with base be42ac4dceac and base d05d1f256082 reverted, the latter revert unbreaks bridge IPv6 forwarding for me. 1) Starting a VM with something like: #!/bin/sh UEFI=/usr/local/share/uefi-firmware/BHYVE_UEFI.fd MEM=8G VM="windoze" IF="$(ifconfig tap create up)" MAC="mac=00:15:17:d2:c0:11" ifconfig bridge0 addm $IF while true ; do bhyve \ -AHPSw \ -s 0,hostbridge \ -s 4,nvme,/usr/vm/$VM/$VM-0.img,sectorsize=4096 \ -s 10,e1000,$IF,$MAC \ -s 14,virtio-rnd \ -s 31,lpc \ -o cpus="1" \ -o cores="1" \ -o threads="1" \ -o sockets="1" \ -l com1,stdio \ -l bootrom,$UEFI,fwcfg=qemu \ -m $MEM \ $VM RES=$? bhyvectl --destroy --vm=$VM if [ $RES -eq 1 ] ; then ifconfig $IF destroy exit 1 fi sleep 5 done Causes the IPv6 EUI-64 address of guest to become unreachable, I get "destination unreachable no route to host" when pinging guest from host. 2) After I start VM with the script above (tap0 becomes member of bridge0), IPv6-only nodes behind host (bridge0 -> em1 -> dumb wireless AP -> nodes (e.g. Linux/Android/iOS nodes) no longer can route packets to the Internet. I didn't try to ping default gateway on them (host) or ff02::1, but I can reboot into affected environment on request later. - I have a bit unusual of a setup to reproduce, but I'll try to include some details: ifconfig_bridge0_name="brer" ifconfig_brer="addm em1 up" ifconfig_brer_ipv6="inet6 2001:db8:b0d7:fed5::1/64" ifconfig_brer_alias0="inet6 2001:db8:b0d7:fed5:38b8:42d:c42a:da00 prefer_source" ifconfig_brer_alias1="inet6 2001:db8:b0d7:fed5::123/64" root@desktop:~ # cat /etc/rtadvd.conf brer:\ :raflags="h":\ :rdnss="2001:4860:4860::6464":rdnssltime#604800:mtu#1492:\ :pref64="64:ff9b::": I get IPv6 Internet access with this convoluted DHCPv6-PD setup of net/mpd5 + net/dhcpcd that I didn't have time to figure out if right or wrong, yet: root@desktop:~ # cat /usr/local/etc/mpd5/ipv6_up.sh: #!/bin/sh route add -inet6 default -interface $1 exit 0 root@desktop:~ # cat /usr/local/etc/dhcpcd.conf: allowinterfaces ng0 brer duid ipv6only noipv6rs waitip 6 timeout 15 interface ng0 ia_pd ng0 brer/0 Rrouting table, it is the same with working/regressed base: root@desktop:~ # netstat -nrF0 -6 Routing tables Internet6: Destination Gateway Flags Netif Expire ::/96 link#4 URS lo0 default link#9 US ng0 ::1 link#4 UHS lo0 ::ffff:0.0.0.0/96 link#4 URS lo0 2001:db8:b0d7:ef92::/64 link#5 U brer 2001:db8:b0d7:ef92::1 link#4 UHS lo0 2001:db8:b0d7:ef92::123 link#4 UHS lo0 2001:db8:b0d7:fed5:38b8:42d:c42a:da00 link#4 UHS lo0 fe80::%lo0/10 link#4 URS lo0 fe80::%lo0/64 link#4 U lo0 fe80::1%lo0 link#4 UHS lo0 fe80::%brer/64 link#5 U brer fe80::6361:2dcc:e3a3:2fed%lo0 link#4 UHS lo0 fe80::%ng0/64 link#9 U ng0 fe80::5a9c:fcff:fe10:9a95%lo0 link#4 UHS lo0 ff02::/16 link#4 URS lo0 Except that, when I do "ndp -a" with working base: 2001:db8:b0d7:fed5:215:17ff:fed2:c011 00:15:17:d2:c0:11 brer 23h59m18s S fe80::215:17ff:fed2:c011%brer 00:15:17:d2:c0:11 brer 23h59m29s S With regressed base: fe80::215:17ff:fed2:c011%brer (incomplete) brer permanent N P.S. I might be slow to provide feedback this time, I'm unfortunately very busy with something else now.
(In reply to Evgenii Khramtsov from comment #0) err, I've got lost a bit with bridge0 -> brer rename, I have multiple of bridges. When I mention "bridge0" assume that it is brer.
(In reply to Evgenii Khramtsov from comment #0) > rdnssltime#604800 That is a workaround to avoid iOS losing RDNSS, there is a patch on Bugzilla already to remove limit for rtadvd RDNSS lifetime for this to work. Sorry for adding more details in new messages, for some reason I didn't see that before I posted this PR.
Hi, Could you please post the output of these commands? ``` netstat -rn6W netstat -on6W netstat -On6W ifconfig tapX ``` Please replace tapX with your $IF interface.
(In reply to Pouria Mousavizadeh Tehrani from comment #3) I need those output with regressed base. Also, please post the output of your bridge0 interface too: ifconfig bridge0
Created attachment 272977 [details] output for c3 and c4 (In reply to Pouria Mousavizadeh Tehrani from comment #4) Inside: revert/: base 4d37c2ea3c86 output (with cherry-picked base 30d4d3db431a build fix) with base be42ac4dceac and base d05d1f256082 reverted (works) regressed/: above but base d05d1f256082 reapplied (forwarding dies after VM starts) 1.txt -> netstat -rn6W 2.txt -> netstat -on6W 3.txt -> netstat -On6W 4.txt -> ifconfig tap0 5.txt -> ifconfig brer ("brer" is bridge0 renamed, that tap0 gets attached to) I removed interfaces from fibs unrelated and their routing entries from output (they do not touch brer's network in any way / are irrelevant to this PR, and are also banned by firewall in case of local misconfiguration). I redacted my real GUA /64 prefix with "2001:db8:b0d7:fed5" (prior mention of "ef92" was myopic redacting of GUA prefix).
(In reply to Evgenii Khramtsov from comment #5) Looks normal, you don't have a nexthop group so be42ac4dceac is not the problem. I will try to reproduce your setup. Just to retrigger the nhop subscription event: 1. Could you please manually set your guest interfaces (brer and tap0) down and then up? (for example: ifconfig tap0 down && ifconfig tap0 up) I suspect the link_state capability of tap0 might have some issues. 2. Also, would it be possible to test with the virtio-net driver instead of e1000 for the guest VM?
(In reply to Pouria Mousavizadeh Tehrani from comment #6) I suspected that this is not related to tap0, so instead I tried first: ifconfig brer deletem em1 ifconfig brer addm em1 -> no forwarding afterwards ifconfig down em1 ifconfig up em1 -> still no forwarding ifconfig down brer ifconfig up brer -> still no forwarding I tried the sequence above with tap0 and the result was the same. I didn't bother with trying virtio-net as I doubt anything would change. Doing the same sequence (all steps) works fine on non-regressed base. > I will try to reproduce your setup. em0 faces PPPoE ISP, ng0 gets created by net/mpd5 (set bundle enable ipv6cp + ipv6_up.sh for LL communication with ISP), then net/dhcpcd requests DHCPv6-PD via LL, then delegates the GUA prefix (/64) to the bridge. I only added /64 to bridge in rc.conf because the /64 is static for me and I want to bind sockets (e.g. dns/unbound) before net/dhcpcd gets to delegate GUA to bridge.
Do I need to apply D58347 and post new output with regressed base or not?
(In reply to Evgenii Khramtsov from comment #8) I was waiting for my tests to complete. For the first case, I attempted to replicate your setup using a Windows VM with UEFI and an e1000 tap interface as a bridge member. I didn't encounter any issues. For the second case, I have an almost identical setup to yours, except I don't use mpd5 + ng_iface. My current implementation uses bridge0 → bge0 → AP → nodes, plus rtadvd on bridge0 and dhcpcd on another interface. This setup worked fine (with that revision applied) long before I commit that. I suspect the issue might be with the bridge driver (since it lacks LINKSTATE capability BUT reports its link status based on its members) or possibly ng_iface. Could you please apply the patch (netstat only) and run the command below on the regressed kernel, then share the results? ``` netstat -on6W --libxo json ``` The netstat output with XML/JSON should now emit nhop flags with an "nh_flags_pretty" key. If the regression was caused by d05d1f256082, there must be "invalid" flag somewhere in your nexthop lists. This should help identify the issue. Could can filter it with jq too: ``` netstat -on6W --libxo json | jq '.statistics."route-nhop-information"."nhop-table"."rt-family".[]."nh-entry".[] | select(.nh_flags_pretty | index("invalid"))' ```
Created attachment 273026 [details] output for c9 (In reply to Pouria Mousavizadeh Tehrani from comment #9) > Could you please apply the patch (netstat only) and run the command below on the regressed kernel, then share the results? Attached output for regressed kernel. pre: booted and routing is working, no deletem and no addm1 done yet post-{deletem,addm}: self-describing for em1
(In reply to Evgenii Khramtsov from comment #10) Hmm after now I've got this for filter and post-addm, not sure if it's inside archive: { "index": 2, "type_str": "v6/resolve", "ifa": "2001:db8:b0d7:fed5::1", "gateway": "brer/resolve", "flags": "", "rt_flags_pretty": [], "nh_flags_pretty": [ "invalid" ], "use": 0, "mtu": 1500, "interface-name": "brer", "address-interface-name": "", "refcount": 2 }
(In reply to Evgenii Khramtsov from comment #11) Interesting, Thank you! Could you please tell me which routes are using index 13? It might get change so use point me the routes with the nhop index that are invalid (post addm and post deletem). ``` netstat -rn6W ``` (look for "Nhop#" column") Also which interface do you mean by addm/deletem? (tapX or emX?) And after deletem, do you make the bridge empty or is there a member left in that bridge? The scope of problem is now limited to bridge driver. But it might be the problem of its member that incorrectly state its linkstate. So I need to know which member is it?
(In reply to Evgenii Khramtsov from comment #11) No it was not there. include routes with that index too please. Again, it might change, verify that it's invalid.
Also, please include ifconfig result of your lo0 interface too. Not sure why but it says it's down.
(In reply to Pouria Mousavizadeh Tehrani from comment #12) > Also which interface do you mean by addm/deletem? (tapX or emX?) em1, as I stopped attaching tap0 since I can reproduce this without bhyve. > And after deletem, do you make the bridge empty or is there a member left in that bridge? brer has no members after deletem em1 afterwards. > Could you please tell me which routes are using index 13? > include routes with that index too please. > ifconfig result of your lo0 interface too I'll be able to reboot next time tomorrow.
(In reply to Evgenii Khramtsov from comment #15) > brer has no members after deletem em1 afterwards. Ok, so it's normal to have invalid flag on empty bridge. Because it's empty. The problem is it reports the lo0 and also your bridge with em0 is down.
Created attachment 273061 [details] output for c12+ I simply decided to omit almost all of unrelated interfaces in rc.conf and boot with the minimum that includes the affected brer, so I barely have to redact anything. I've included the following output for every step: netstat -nrF0 -6 ndp -a netstat -rn6W netstat -on6W netstat -On6W ifconfig em1 ifconfig brer ifconfig em0 ifconfig ng0 ifconfig lo0 netstat -on6W --libxo json netstat -on6W --libxo json | jq '.statistics."route-nhop-information"."nhop-table"."rt-family".[]."nh-entry".[] | select(.nh_flags_pretty | index("invalid"))' You can also see "get.sh" inside archive. The only redaction now is: find . -type f -exec sed -i '' 's/real-gua-64/2001:db8:b0d7:fed5/g' {} + + the same for globally routable IPv4 /32 on ng0 I'll be extra clear: All the logs inside archive are with regressed kernel boot + Phabricator patch. after_boot_and_1_min -> I booted, then I was using the Internet on nodes for ~1m, forwarding OK, then I run get.sh *immediately -> I did the command mentioned in the name of directory and then I run get.sh immediately *1min -> 1 minute passed after the related *immediate, then I run get.sh again FWIW, SLAAC still works even when routing on brer is dead. > your bridge with em0 As previously mentioned: brer adds em1 in rc.conf on boot, rtadvd runs on brer in rc.conf. em0 faces ISP for net/mpd5 to create ng0, then net/dhcpcd delegates /64 to brer, brer never adds em0.
(In reply to Evgenii Khramtsov from comment #17) AFAICU: I see your em1 become `status: no carrier` in "04_after_addm_em1_immediately". Its media is also (100baseTX ). When you move it under a bridge, it resets and reports its link_state as down in: 2308 } else if (!link_check && (sc->link_active == 1)) { 2309 sc->link_speed = 0; 2310 sc->link_duplex = 0; 2311 sc->link_active = 0; 2312 iflib_link_state_change(ctx, LINK_STATE_DOWN, 0); 2313 } However, it only reports the link as up when TSO (automasked) is set, and it won't set TSO (automasked) for media under 1G. (This should not be related to TSO, but the code logic does it anyway.) 2300 /* Only do TSO on gigabit for older chips due to errata */ 2301 if (hw->mac.type < igb_mac_min) 2302 automasked = em_automask_tso(ctx); 2303 2304 /* Automasking resets the interface so don't mark it up yet */ 2305 if (!automasked) 2306 iflib_link_state_change(ctx, LINK_STATE_UP, 2307 IF_Mbps(sc->link_speed)); So em reports the link state as down, but it won't report it back as up again when it transitions to 100 Mbps media. The original patch that *might* cause this bug is https://reviews.freebsd.org/D41170. Could you please try setting this sysctl tunable: ``` sysctl hw.em.unsupported_tso=1 ``` and resetting your em1 interface? (might not be enough, but worth trying)
(In reply to Pouria Mousavizadeh Tehrani from comment #18) > Its media is also (100baseTX ) It's expected, the AP is dirt cheap (100baseTX is the max). > Could you please try setting this sysctl tunable: > ``` > sysctl hw.em.unsupported_tso=1 > ``` > and resetting your em1 interface? (might not be enough, but worth trying) Resetting means the down/up em1 from Comment 17 but with hw.em.unsupported_tso=1 set before doing that? If I guessed it right I might be able to do that tomorrow. I'll also try to do the same with em1 connected to something with a GbE.
(In reply to Evgenii Khramtsov from comment #19) > Resetting means the down/up em1 from Comment 17 but with hw.em.unsupported_tso=1 set before doing that? If I guessed it right I might be able to do that tomorrow. Looks like, 'hw.em.unsupported_tso' is a read only tunable. So, As you know, you should set in /boot/loader.conf Therefore, you have to reboot, that should be enough :) > I'll also try to do the same with em1 connected to something with a GbE. Awesome! Thank you for your report and your testing.
Created attachment 273079 [details] c18 output Inside: 01_em1_GbE/0* - self-describing 02_unsup_tso_GbE/0* - above but booted with hw.em.unsupported_tso="1" in loader.conf 03_unsup_tso_FE/0* - above (including unsup tso) but this time it is 100baseTX 04_no_em1_tap0_virtio/0* - brer has no members after boot, then I start an "open-webui" VM (virtio-net in bhyve start script, attached tap0, Linux guest), and I can't ping the VM from host (UDP destination no route, routing died again), EUI-64 of VM has reverse zone entry so it resolves as "open-webui" in output FWIW: I could ping LL when routing died. It is unlikely that I can provide more logs for the rest of this week.
(In reply to Evgenii Khramtsov from comment #21) Hi, please try again with this commit: 0bd6a167c1561f01c227b1c428a3d8adf0e38833 (it's in CURRENT) I understand you were able to reproduce without e1000, but I'm curious if this solves em1 problem or not. Thank you for your time.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=5286cbea9118f79b43fb7416d9fc4f081c400949 commit 5286cbea9118f79b43fb7416d9fc4f081c400949 Author: Pouria Mousavizadeh Tehrani <pouria@FreeBSD.org> AuthorDate: 2026-07-28 08:16:14 +0000 Commit: Pouria Mousavizadeh Tehrani <pouria@FreeBSD.org> CommitDate: 2026-07-28 08:17:32 +0000 netstat(1): Emit nhop flags for encoding styles with Wflag Show nhop flags like invalid nexthop to debug cases like the PR below. PR: 296883 Reviewed by: glebius Differential Revision: https://reviews.freebsd.org/D58347 usr.bin/netstat/nhops.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
(In reply to Pouria Mousavizadeh Tehrani from comment #22) > Hi, please try again with this commit: 0bd6a167c1561f01c227b1c428a3d8adf0e38833 > (it's in CURRENT) > I understand you were able to reproduce without e1000, but I'm curious if this > solves em1 problem or not. No change with 0bd6a167c156 cherry-picked. How bothersome would it be to have a chicken switch to previous routing behavior? My -CURRENT is getting stale, cherry-picking security fixes would not be sustainable, -RELEASE and -STABLE branches often lack desktop stability fixes and features from -CURRENT. I've never found non-CURRENT to be usable for a semi-modern desktop. I don't think my PPPoE DHCPv6-PD bridge setup was robust in the first place (e.g. net/dhcpcd receives PD, then needs to timeout before forking to background), and it would be a while to debug this either by you (no reproducer) or by me (free time strangled).
(In reply to Evgenii Khramtsov from comment #24) > How bothersome would it be to have a chicken switch to previous routing behavior? Don't bother, I think I can figure it out on my own which hunk causes this, later when I would have time.
No need for a chicken switch! The d05d1f256082 doesn't change any behavior intentionally, it just exposed a bug. Your setup is quite complex, so please be patient and provide info to Pouria. The bug is going to be fixed.