Running FreeBSD in AWS EC2, it is not possible to join the ena0 interface to a bridge at the same time as an epair interface. The commands below result in the same error on FreeBSD 13.1, 12.3, 11.4. With FreeBSD 10.4 in AWS EC2, the xn driver is used instead of the ena driver, and this same sequence of commands then works without any issue. This same sequence of commands also works in Vultr which uses the vtnet driver instead of the ena driver. It also works on my laptop with the em driver instead of the ena driver. ec2-user@freebsd:~ $ su root@freebsd:/home/ec2-user # ifconfig bridge create bridge0 root@freebsd:/home/ec2-user # ifconfig epair create epair0a root@freebsd:/home/ec2-user # ifconfig bridge0 addm ena0 root@freebsd:/home/ec2-user # ifconfig bridge0 addm epair0a ifconfig: BRDGADD epair0a: Invalid argument root@freebsd:/home/ec2-user # ifconfig bridge0 deletem ena0 root@freebsd:/home/ec2-user # ifconfig bridge0 addm epair0a root@freebsd:/home/ec2-user # ifconfig bridge0 addm ena0 ifconfig: BRDGADD ena0: Invalid argument root@freebsd:/home/ec2-user # ifconfig ena0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9001 options=422<TXCSUM,JUMBO_MTU,LRO> ether xxx inet6 xxx%ena0 prefixlen 64 scopeid 0x1 inet xxx netmask 0xfffff000 broadcast xxx media: Ethernet autoselect (Unknown <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 58:9c:fc:00:20:47 inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 4 priority 128 path cost 2000 groups: bridge nd6 options=3<PERFORMNUD,ACCEPT_RTADV> epair0a: flags=8962<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:3c:b7:88:ff:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> epair0b: flags=8862<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:3c:b7:88:ff:0b groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
I suspect the problem here is one of MTU. if_bridge sets its MTU to that of the first interface added to it. That's either 9001 (ena) or 1500 (epair). When you try to add the second interface it the MTU does not match and the interface is not added. There should be a log in `dmesg` saying 'bridge0: invalid MTU: 9001(ena0) != 1500' or something like it. You should be able to fix that by changing the ena or epair MTU prior to adding it to the bridge. You should also know that the behaviour of if_bridge has been changed in 14, where it will change the MTU of newly added members, rather than rejecting them.
Hi Kristof, thanks for the quick response! Indeed, I had a dmesg entry with the information I needed - exactly as you described, but didn't see it as I was not on the console. I just tested it on the latest snapshot FreeBSD-14.0-CURRENT-amd64-20220617-c692ea40265-256174-disc1.iso. addm mtu9001 = bridge changed its mtu to 9001 addm mtu1500 = same error How hard would it be to have the stderr output match the dmesg error, so that future people like me using ssh who don't think to check dmesg get well informed of the issue?
So it turns out I'd misremembered the change in 14. That one was to allow MTU changes on a bridge (by changing all of the member interface MTUs). That's useful, but not what you need. This allows interfaces with different MTUs to be added to a bridge as well: https://reviews.freebsd.org/D35597 I'm not going to MFC that change, but in 13.1 and earlier you can work around the problem by doing `ifconfig epair0a mtu 9001 ; ifconfig bridge0 addm epair0a`.
Thanks for that change! In the mean time, I'm closing this as "not a bug" as it was the expected behaviour if the interfaces have different MTUs.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1e824ee84f46b4a086bc3c7c528c0762e388bc66 commit 1e824ee84f46b4a086bc3c7c528c0762e388bc66 Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2022-06-25 11:47:31 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2022-06-27 06:27:28 +0000 if_bridge tests: test adding interfaces with different MTU We now allow new bridge members to be added with a different MTU from the bridge. We change the interface MTU to match the bridge. Test this. PR: 264883 tests/sys/net/if_bridge_test.sh | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1865ebfb12ddaf3d0ff1458e6152b3cb1f1bdee8 commit 1865ebfb12ddaf3d0ff1458e6152b3cb1f1bdee8 Author: Kristof Provost <kp@FreeBSD.org> AuthorDate: 2022-06-25 11:39:44 +0000 Commit: Kristof Provost <kp@FreeBSD.org> CommitDate: 2022-06-27 06:27:27 +0000 if_bridge: change MTU for new members Rather than reject new bridge members because they have the wrong MTU change it to match the bridge. If that fails, reject the new interface. PR: 264883 Different Revision: https://reviews.freebsd.org/D35597 sys/net/if_bridge.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1879f7affbf80c701c76081b2b8511f4c7dd81f5 commit 1879f7affbf80c701c76081b2b8511f4c7dd81f5 Author: Eugene Grosbein <eugen@FreeBSD.org> AuthorDate: 2024-04-09 10:49:05 +0000 Commit: Eugene Grosbein <eugen@FreeBSD.org> CommitDate: 2024-04-09 10:50:20 +0000 MFC: if_bridge: change MTU for new members Rather than reject new bridge members because they have the wrong MTU change it to match the bridge. If that fails, reject the new interface. PR: 264883 Different Revision: https://reviews.freebsd.org/D35597 (cherry picked from commit 1865ebfb12ddaf3d0ff1458e6152b3cb1f1bdee8) sys/net/if_bridge.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-)