Bug 264883 - ena and epair interfaces cannot be members of the same bridge
Summary: ena and epair interfaces cannot be members of the same bridge
Status: Closed Not A Bug
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.1-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: Kristof Provost
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-06-25 06:40 UTC by Ben Woods
Modified: 2022-06-27 08:29 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ben Woods freebsd_committer freebsd_triage 2022-06-25 06:40:26 UTC
Running FreeBSD in AWS EC2, it is not possible to join the ena0 interface to a bridge at the same time as an epair interface.

The commands below result in the same error on FreeBSD 13.1, 12.3, 11.4.

With FreeBSD 10.4 in AWS EC2, the xn driver is used instead of the ena driver, and this same sequence of commands then works without any issue.

This same sequence of commands also works in Vultr which uses the vtnet driver instead of the ena driver. It also works on my laptop with the em driver instead of the ena driver.

ec2-user@freebsd:~ $ su
root@freebsd:/home/ec2-user # ifconfig bridge create
bridge0
root@freebsd:/home/ec2-user # ifconfig epair create
epair0a
root@freebsd:/home/ec2-user # ifconfig bridge0 addm ena0
root@freebsd:/home/ec2-user # ifconfig bridge0 addm epair0a
ifconfig: BRDGADD epair0a: Invalid argument

root@freebsd:/home/ec2-user # ifconfig bridge0 deletem ena0
root@freebsd:/home/ec2-user # ifconfig bridge0 addm epair0a
root@freebsd:/home/ec2-user # ifconfig bridge0 addm ena0
ifconfig: BRDGADD ena0: Invalid argument

root@freebsd:/home/ec2-user # ifconfig
ena0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9001
	options=422<TXCSUM,JUMBO_MTU,LRO>
	ether xxx
	inet6 xxx%ena0 prefixlen 64 scopeid 0x1
	inet xxx netmask 0xfffff000 broadcast xxx
	media: Ethernet autoselect (Unknown <full-duplex>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
	inet 127.0.0.1 netmask 0xff000000
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 58:9c:fc:00:20:47
	inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 4 priority 128 path cost 2000
	groups: bridge
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
epair0a: flags=8962<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:3c:b7:88:ff:0a
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair0b: flags=8862<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:3c:b7:88:ff:0b
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Comment 1 Kristof Provost freebsd_committer freebsd_triage 2022-06-25 07:44:13 UTC
I suspect the problem here is one of MTU.

if_bridge sets its MTU to that of the first interface added to it. That's either 9001 (ena) or 1500 (epair). When you try to add the second interface it the MTU does not match and the interface is not added.

There should be a log in `dmesg` saying 'bridge0: invalid MTU: 9001(ena0) != 1500' or something like it.

You should be able to fix that by changing the ena or epair MTU prior to adding it to the bridge.

You should also know that the behaviour of if_bridge has been changed in 14, where it will change the MTU of newly added members, rather than rejecting them.
Comment 2 Ben Woods freebsd_committer freebsd_triage 2022-06-25 12:59:48 UTC
Hi Kristof, thanks for the quick response!

Indeed, I had a dmesg entry with the information I needed - exactly as you described, but didn't see it as I was not on the console.

I just tested it on the latest snapshot FreeBSD-14.0-CURRENT-amd64-20220617-c692ea40265-256174-disc1.iso.

addm mtu9001 = bridge changed its mtu to 9001
addm mtu1500 = same error

How hard would it be to have the stderr output match the dmesg error, so that future people like me using ssh who don't think to check dmesg get well informed of the issue?
Comment 3 Kristof Provost freebsd_committer freebsd_triage 2022-06-25 17:39:58 UTC
So it turns out I'd misremembered the change in 14. That one was to allow MTU changes on a bridge (by changing all of the member interface MTUs).
That's useful, but not what you need.

This allows interfaces with different MTUs to be added to a bridge as well: https://reviews.freebsd.org/D35597

I'm not going to MFC that change, but in 13.1 and earlier you can work around the problem by doing `ifconfig epair0a mtu 9001 ; ifconfig bridge0 addm epair0a`.
Comment 4 Ben Woods freebsd_committer freebsd_triage 2022-06-25 19:08:32 UTC
Thanks for that change!
In the mean time, I'm closing this as "not a bug" as it was the expected behaviour if the interfaces have different MTUs.
Comment 5 commit-hook freebsd_committer freebsd_triage 2022-06-27 08:29:33 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1e824ee84f46b4a086bc3c7c528c0762e388bc66

commit 1e824ee84f46b4a086bc3c7c528c0762e388bc66
Author:     Kristof Provost <kp@FreeBSD.org>
AuthorDate: 2022-06-25 11:47:31 +0000
Commit:     Kristof Provost <kp@FreeBSD.org>
CommitDate: 2022-06-27 06:27:28 +0000

    if_bridge tests: test adding interfaces with different MTU

    We now allow new bridge members to be added with a different MTU from
    the bridge. We change the interface MTU to match the bridge.
    Test this.

    PR:     264883

 tests/sys/net/if_bridge_test.sh | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
Comment 6 commit-hook freebsd_committer freebsd_triage 2022-06-27 08:29:34 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1865ebfb12ddaf3d0ff1458e6152b3cb1f1bdee8

commit 1865ebfb12ddaf3d0ff1458e6152b3cb1f1bdee8
Author:     Kristof Provost <kp@FreeBSD.org>
AuthorDate: 2022-06-25 11:39:44 +0000
Commit:     Kristof Provost <kp@FreeBSD.org>
CommitDate: 2022-06-27 06:27:27 +0000

    if_bridge: change MTU for new members

    Rather than reject new bridge members because they have the wrong MTU
    change it to match the bridge. If that fails, reject the new interface.

    PR:     264883
    Different Revision:     https://reviews.freebsd.org/D35597

 sys/net/if_bridge.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)