Good day all, I run a set of VM's on an if_bridge: bridge103: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:ff:f5 inet 192.168.103.1 netmask 0xffffff00 broadcast 192.168.103.255 inet6 fe80::5a9c:fcff:fe10:fff5%bridge103 prefixlen 64 scopeid 0x4 inet6 2a01:4f8:190:1183::103:1 prefixlen 64 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap1033 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000000 member: tap1032 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000000 member: tap1031 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 11 priority 128 path cost 2000000 member: tap1030 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 10 priority 128 path cost 2000000 groups: bridge nd6 options=63<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_RADR> IPv4 functionality is working as expected, and ipv6 on the host works perfectly. However no guest may ping6 its closest host inet6 address (tap1033->bridge103) and get a response. tcpdump reveals that bridge103 received the request, but does not appear to do anything about it: root@de1:/usr/venv/bhyve/init # tcpdump -vvi bridge103 icmp6 tcpdump: listening on bridge103, link-type EN10MB (Ethernet), capture size 262144 bytes 20:11:53.073960 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2a01:4f8:190:1183::103:2 > ff02::1:ff03:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2a01:4f8:190:1183::103:1 source link-address option (1), length 8 (1): 00:d3:4d:be:3f:ab 0x0000: 00d3 4dbe 3fab 20:11:54.120586 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2a01:4f8:190:1183::103:2 > ff02::1:ff03:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2a01:4f8:190:1183::103:1 source link-address option (1), length 8 (1): 00:d3:4d:be:3f:ab 0x0000: 00d3 4dbe 3fab The same is true for all other VM's that are connected via virtio/tap to an if_bridge, bearing in mind the ICMP request is seen perhaps the problem is with if_bridge. Unfortunatly I am now somewhat out of my depth with bhyve/if_bridge_tap, I have tried everything I can possible think of and am hoping someone here knows what is causing such an issue. Notable the inet6 that is directly assigned to each bridge is reachable from the outside, that would be these: em0: inet6 2a01:4f8:190:1183::1:1 prefixlen 64 lo0: inet6 ::1 prefixlen 128 bridge102: inet6 2a01:4f8:190:1183::102:1 prefixlen 64 bridge103: inet6 2a01:4f8:190:1183::103:1 prefixlen 64 bridge104: inet6 2a01:4f8:190:1183::104:1 prefixlen 64 As the VM in the above examples was 2a01:4f8:190:1183::103:2 assigned to bridge 103, there should be no way it failed to get a response that I can see
I run an OpenBSD PF firewall guest in bhyve with taps to various bridges and IPv6 native on the firewall to the internet and internal machines. From an internal machine, I achieve a 20/20 on ipv6-test.com and that wouldn't be possible if there was an IPv6 issue in any chain of the FreeBSD 13.0-RELEASE/lagg/bridge/tap/bhyve/OpenBSD configuration. I know it is possible to put IPs on bridge interfaces but have you tried without this? I typically IP the interface from the host that is attached to the bridge. Can you post kldstat please?
(In reply to Jason Tubnor from comment #1) I removed the alias, please see below for my test: I destroy bridge 102/104 as there not in use for the moment, so really to simplify things root@de1:/usr/home/paul.webster # ifconfig bridge102 destroy root@de1:/usr/home/paul.webster # ifconfig bridge104 destroy Then re-show the host bridge103: root@de1:/usr/home/paul.webster # ifconfig bridge103 bridge103: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:ff:f5 inet 192.168.103.1 netmask 0xffffff00 broadcast 192.168.103.255 inet6 fe80::5a9c:fcff:fe10:fff5%bridge103 prefixlen 64 scopeid 0x4 inet6 2a01:4f8:190:1183::103:1 prefixlen 64 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap1033 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000000 member: tap1032 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000000 member: tap1031 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 11 priority 128 path cost 2000000 member: tap1030 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 10 priority 128 path cost 2000000 groups: bridge nd6 options=63<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_RADR> Let's remove that ipv6 addr: root@de1:/usr/home/paul.webster # ifconfig bridge103 inet6 2a01:4f8:190:1183::103:1 -alias root@de1:/usr/home/paul.webster # ifconfig bridge103 bridge103: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:ff:f5 inet 192.168.103.1 netmask 0xffffff00 broadcast 192.168.103.255 inet6 fe80::5a9c:fcff:fe10:fff5%bridge103 prefixlen 64 scopeid 0x4 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap1033 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000000 member: tap1032 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000000 member: tap1031 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 11 priority 128 path cost 2000000 member: tap1030 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 10 priority 128 path cost 2000000 groups: bridge nd6 options=63<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_RADR> Ok dokey with that bit done lets repeat a test: From the host: root@de1:~ # tcpdump -vvi bridge103 ip6 tcpdump: listening on bridge103, link-type EN10MB (Ethernet), capture size 262144 bytes 07:59:02.139781 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2a01:4f8:190:1183::103:2 > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1 source link-address option (1), length 8 (1): 00:d3:4d:be:3f:ab 0x0000: 00d3 4dbe 3fab 07:59:03.195030 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2a01:4f8:190:1183::103:2 > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1 source link-address option (1), length 8 (1): 00:d3:4d:be:3f:ab 0x0000: 00d3 4dbe 3fab 07:59:04.249361 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2a01:4f8:190:1183::103:2 > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1 source link-address option (1), length 8 (1): 00:d3:4d:be:3f:ab 0x0000: 00d3 4dbe 3fab 07:59:05.369920 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2a01:4f8:190:1183::103:2 > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1 source link-address option (1), length 8 (1): 00:d3:4d:be:3f:ab 0x0000: 00d3 4dbe 3fab From the client: root@sitehost:~ # netstat -6rn Routing tables Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 default fe80::1%vtnet0 UGS vtnet0 ::1 link#2 UHS lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 2a01:4f8:190:1183::/64 link#1 U vtnet0 2a01:4f8:190:1183::103:2 link#1 UHS lo0 fe80::/10 ::1 UGRS lo0 fe80::%vtnet0/64 link#1 U vtnet0 fe80::2d3:4dff:febe:3fab%vtnet0 link#1 UHS lo0 fe80::%lo0/64 link#2 U lo0 fe80::1%lo0 link#2 UHS lo0 ff02::/16 ::1 UGRS lo0 root@sitehost:~ # ping6 ipv6.google.com PING6(56=40+8+8 bytes) 2a01:4f8:190:1183::103:2 --> 2a00:1450:4001:829::200e ^C --- ipv6.l.google.com ping6 statistics --- 4 packets transmitted, 0 packets received, 100.0% packet loss
Can you try removing all inet4/6 from the bridge? This is my bridge: vm-main: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 62:c7:77:2d:20:1b id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: e3a_jail1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 20 priority 128 path cost 2000 member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 27 priority 128 path cost 2000000 member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 22 priority 128 path cost 2000000 member: e1a_jail2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 21 priority 128 path cost 2000 member: e0a_jail3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 19 priority 128 path cost 2000 member: e2a_jail4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 16 priority 128 path cost 2000 member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000000 groups: bridge vm-switch viid-9f274@ nd6 options=9<PERFORMNUD,IFDISABLED> vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=4000001<RXCSUM,NOMAP> ether 00:1f:3a:aa:ff:cc inet 192.168.1.200 netmask 0xffffff00 broadcast 192.168.1.255 groups: vlan vlan: 1 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> ** NOTE: I don't have IPv6 in the host, it is all passed between guests and the internet also attached to bridges.
(In reply to Jason Tubnor from comment #3) That actually appears to make things worse ... now the host does not even see ipv6 traffic at all (according to tcpdump) root@de1:~ # ifconfig bridge103 bridge103: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:ff:f5 inet 192.168.103.1 netmask 0xffffff00 broadcast 192.168.103.255 inet6 fe80::5a9c:fcff:fe10:fff5%bridge103 prefixlen 64 scopeid 0x4 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap1033 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000000 member: tap1032 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000000 member: tap1031 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 11 priority 128 path cost 2000000 member: tap1030 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 10 priority 128 path cost 2000000 groups: bridge nd6 options=63<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_RADR> root@de1:~ # ifconfig bridge103 inet6 fe80::5a9c:fcff:fe10:fff5%bridge103 -alias root@de1:~ # ifconfig bridge103 inet 192.168.103.1 -alias root@de1:~ # ifconfig bridge103 bridge103: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:10:ff:f5 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap1033 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 9 priority 128 path cost 2000000 member: tap1032 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000000 member: tap1031 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 11 priority 128 path cost 2000000 member: tap1030 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 10 priority 128 path cost 2000000 groups: bridge nd6 options=63<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL,NO_RADR> root@de1:~ # The vm: root@sitehost:~ # ping6 2a00:1450:4001:829::200e PING6(56=40+8+8 bytes) 2a01:4f8:190:1183::103:2 --> 2a00:1450:4001:829::200e ^C --- 2a00:1450:4001:829::200e ping6 statistics --- 30 packets transmitted, 0 packets received, 100.0% packet loss The host: root@de1:~ # tcpdump -vvi bridge103 ip6 tcpdump: listening on bridge103, link-type EN10MB (Ethernet), capture size 262144 bytes ^C 0 packets captured 13 packets received by filter 0 packets dropped by kernel root@de1:~ #
I just found this as I was looking for another tun/bhyve inet6 issue. Did you ever solve this?
I have similar configuration on FreeBSD-13 and it works well. Host system has if_bridge with configured fe80::1 as IPv6 LLA and some IPv6 GA, it uses two tap interfaces. Bhyve VMs use fe80::1%vtnet0 as default gateway and IPv6 GA from the same prefix as GA from if_bridge.
No problem whatsoever. If you place all layer 3 addresses on the bridge interfaces and not on any member. This is mandatory according to the handbook section on bridging and has been discussed in half a dozen of other tickets. FreeBSD 13.3 and up available for testing, but used to "just work" in 11.x, 12.x, too. Kind regards, Patrick
Seems to resonate with my experiences detailed in comment 2 of bug 287146. -- Martin
I've got this problem on 14.3p5 (but not involving tap). Host has a /64, guests get other IPs in that /64, and I've got a bridge set up for the jails to connect to. auto_linklocal is turned on on both sides. tcpdump on the bridge shows well-formed neighbor solicitation requests sent to the solicited multicast address. They're never answered and don't appear in ndp output. net.inet6.icmp6.nd6_onlink_ns_rfc4861 doesn't make any difference.
(In reply to Brad Ackerman from comment #9) > tcpdump on the bridge shows well-formed neighbor solicitation requests > sent to the solicited multicast address. They're never answered and > don't appear in ndp output. show tcpdump output, please. also, show ifconfig for the bridge and all member interfaces.
(In reply to ivy from comment #10) So the problem is that I'm not quite figuring out how to share a /64 with the jails. I've got a /64 from which I'd like to assign both jails and the host; if I remove the address from the host's external interface NDP on the bridge starts working — but telling ifconfig that the prefixlen is 128 on all the global-scope addresses (and then setting up a far gateway on the jail) doesn't seem to make a difference. jails: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=0 ether 58:9c:fc:10:80:39 inet 172.21.188.1 netmask 0xfffffc00 broadcast 172.21.191.255 inet6 fe80::5a9c:fcff:fe10:8039%jails prefixlen 64 scopeid 0x4 inet6 2604:2dc0:200:2116::ff:ffff prefixlen 64 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: e0a_powerdns flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000 groups: bridge nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> e0a_powerdns: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: vnet0 host interface for Bastille jail powerdns options=8<VLAN_MTU> ether 02:57:e9:cd:fc:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Inside jail, the other side of the epair: vnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:57:e9:cd:fc:0b inet6 2604:2dc0:200:2116::4 prefixlen 64 inet6 fe80::57:e9ff:fecd:fc0b%vnet0 prefixlen 64 scopeid 0x9 groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> $ sudo tcpdump -nvvve -i jails tcpdump: listening on jails, link-type EN10MB (Ethernet), snapshot length 262144 bytes 02:13:07.789039 02:cb:b3:d7:b8:0b > 33:33:ff:ff:ff:ff, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2604:2dc0:200:2116::4 > ff02::1:ffff:ffff: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2604:2d c0:200:2116::ff:ffff source link-address option (1), length 8 (1): 02:cb:b3:d7:b8:0b 0x0000: 02cb b3d7 b80b 02:13:08.789166 02:cb:b3:d7:b8:0b > 33:33:ff:ff:ff:ff, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2604:2dc0:200:2116::4 > ff02::1:ffff:ffff: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2604:2d c0:200:2116::ff:ffff source link-address option (1), length 8 (1): 02:cb:b3:d7:b8:0b 0x0000: 02cb b3d7 b80b 02:13:09.816291 02:cb:b3:d7:b8:0b > 33:33:ff:ff:ff:ff, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payl oad length: 32) 2604:2dc0:200:2116::4 > ff02::1:ffff:ffff: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2604:2dc0:200:2116::ff:ffff source link-address option (1), length 8 (1): 02:cb:b3:d7:b8:0b 0x0000: 02cb b3d7 b80b 02:13:15.163167 02:cb:b3:d7:b8:0b > 33:33:ff:ff:ff:ff, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2604:2dc0:200:2116::4 > ff02::1:ffff:ffff: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2604:2d c0:200:2116::ff:ffff source link-address option (1), length 8 (1): 02:cb:b3:d7:b8:0b 0x0000: 02cb b3d7 b80b
Have you tried: ifconfig e0a_powerdns inet6 -ifdisabled and see what happens?
(In reply to Brad Ackerman from comment #11) > I've got a /64 from which I'd like to assign both jails and the host; if I remove the address from the host's external interface NDP on the bridge starts working Did you assign the host's address to the bridge interface or to the physical interface that is a bridge member? Any layer 3 address - IPv4 and IPv6 *must* be configured on the bridge, never a member interface. Or the other way round - any interface that is a member of a bridge *must not* have a layer 3 address. HTH, Patrick
(In reply to punkt.de Hosting Team from comment #13) Only the bridge has IP addresses and not its members. Removing the ifdisabled flag had no effect.
(In reply to Brad Ackerman from comment #14) The physical interface that's a member of the bridge does have a link local address. Does AUTO_LINKLOCAL need to be disabled on bridge members?
Yes. No layer 3 address whatsoever on a bridge member. As soon as one is present, multicast breaks. Which would explain ND failing.
In rc.conf I have this for part of my dual stack config on bhyve host ifconfig_bridge0_ipv6="inet6 -ifdisabled auto_linklocal accept_rtadv" linux bhyve guests using taps on the bridge get static ipv4 I manually set within the vm and automatically get ipv6 via slaac
(In reply to punkt.de Hosting Team from comment #16) That resolves my issue. I just rebuilt the environment in question and bridges are working as I expect them to.
i'm closing this bug since it's become confused with multiple, unrelated issues posted by different people. IPv6 definitely works over a bridge, including via tap devices. if anyone has issues with this, please post to the freebsd-net mailing list, or file a new bug if you have a specific, reproducible issue.