The same issue is present in 15.0-RC3.

Comment 2 Zhenlei Huang 2025-11-23 13:40:23 UTC

(In reply to peter.sopko+freebsd.org from comment #0)
> nics using if_qlnxe can't be used as laggdev in a lagg interface, unless they are
> put to promiscuous mode.

It appears the interface is filtering out the LACP packets ( slow proto, aka ether proto 0x8809 ), so the promiscuous mode comes a rescue.

Can you do traffic sniff on one of the lagg member say ql0 ? Please do not enable the promiscuous mode, you can do that with

```
# tcpdump --no-promiscuous-mode -envi ql0
```

and share the captured traffic ?

I tried capturing traffic on the ql0 interface when it is not set to promisc mode using the command you suggested, but there appears to be absolutely no traffic captured at all...

I took a look at the switch where the port is connected and the interface is in an initializing state (but as soon as I put it to promisc mode it is up and running perfectly)....

relevant switch configuration for reference:
> interface port-channel30
>   speed 10000
>   description mysql01
> 
> interface Ethernet1/41
>   lacp rate fast
>   description mysql01 - 1
>   channel-group 30 mode active
> 
> interface Ethernet1/42
>   lacp rate fast
>   description mysql-01 -2
>   channel-group 30 mode active

show interface output (~ 1 minute after I disabled promisc mode, enabled tcpdump and cleared counters):
> Ethernet1/42 is down (initializing)
>  Dedicated Interface 
>   Belongs to Po30
>   Hardware: 100/1000/10000 Ethernet, address: a44c.11b8.6ef1 (bia a44c.11b8.6ef1)
>   Description: mysql-01 -2
>   MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec
>   reliability 255/255, txload 1/255, rxload 1/255
>   Encapsulation ARPA
>   Port mode is access
>   auto-duplex, 10 Gb/s, media type is 10G
>   Beacon is turned off
>   Input flow-control is off, output flow-control is off
>   Rate mode is dedicated
>   Switchport monitor is off 
>   EtherType is 0x8100 
>   Last link flapped 00:00:02
>   Last clearing of "show interface" counters 00:01:09
>   2 interface resets
>   Load-Interval #1: 30 seconds
>   30 seconds input rate 80 bits/sec, 0 packets/sec
>   30 seconds output rate 488 bits/sec, 0 packets/sec
>   Load-Interval #2: 5 minute (300 seconds)
>     input rate 80 bps, 0 pps; output rate 208 bps, 0 pps
>   RX
>     0 unicast packets  5 multicast packets  0 broadcast packets
>     5 input packets  762 bytes
>     0 jumbo packets  0 storm suppression packets
>     0 runts  0 giants  0 CRC  0 no buffer
>     0 input error  0 short frame  0 overrun   0 underrun  0 ignored
>     0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
>     0 input with dribble  0 input discard
>     0 Rx pause
>   TX
>     0 unicast packets  53 multicast packets  0 broadcast packets
>     53 output packets  5343 bytes
>     0 jumbo packets
>     0 output errors  0 collision  0 deferred  0 late collision
>     0 lost carrier  0 no carrier  0 babble 0 output discard
>     0 Tx pause

ifconfig -vvvv ql0 output:
> ql0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
> 	options=8d07ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,HWSTATS>
> 	ether 94:f1:28:b3:07:96
> 	media: Ethernet autoselect (10Gbase-SR <full-duplex>)
> 	status: active
> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> 	drivername: ql0
> 	plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
> 	vendor: OEM PN: SFP-10G85-SR SN: CB220919746 DATE: 2022-09-20
> 	module temperature: 36.42 C voltage: 3.26 Volts
> 	lane 1: RX power: 0.54 mW (-2.68 dBm) TX bias: 6.40 mA
> 
> 	SFF8472 DUMP (0xA0 0..127 range):
> 	 03 04 07 10 00 00 00 00 00 00 00 06 67 00 00 00
> 	 08 02 00 1e 4f 45 4d 20 20 20 20 20 20 20 20 20
> 	 20 20 20 20 00 00 00 00 53 46 50 2d 31 30 47 38
> 	 35 2d 53 52 20 20 20 20 41 20 20 20 03 52 00 a7
> 	 00 1a 00 00 43 42 32 32 30 39 31 39 37 34 36 20
> 	 20 20 20 20 32 32 30 39 32 30 20 20 68 f0 03 e1
> 	 00 00 11 b7 ed 31 a4 15 d3 b2 f4 14 38 ed c2 33
> 	 f0 3c 11 00 00 00 00 00 00 00 00 00 a0 be dc ab

Comment 4 Zhenlei Huang 2025-11-26 03:07:32 UTC

(In reply to peter.sopko+freebsd.org from comment #3)
> I tried capturing traffic on the ql0 interface when it is not set to promisc mode
> using the command you suggested, but there appears to be absolutely no traffic
> captured at all...

Actually the interface should at least send out LACP packets when it is a member of lagg interface.

I have managed to repeat this issue. There're two issue with qlnxe(4) currently are identified.

The first one is, the out path ( if_transmit ) lacks ETHER_BPF_MTAP and thus bpf(4) does not have a chance to see the outgoing packets.

The second one is, there's a small initialization bug, that when the qlnxe(4) is not flagged up, aka `ifconfig ql0 up`, setup the multicast addresses ( aka SIOCADDMULTI ioctl ) is a noop, and the interface will not be programmed with desired multicast filter, hence the slow protocol multicast traffic ( dest 01:80:c2:00:00:02 ) are all dropped by the interface. That can be identified by the statistic,

```
# sysctl dev.ql.0.hwstat.mac_filter_discards
dev.ql.0.hwstat.mac_filter_discards: 284
```

To workaround the second issue, you may want to put the qlnxe(4) interface into up status before adding it to lagg(4),

```
# ifconfig ql0 up
# ifconfig ql1 up
# ifconfig lagg0 create
# ifconfig lagg0 laggproto lacp laggport ql0 laggport ql1
# ifconfig lagg0 up
# ifconfig lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=8d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,HWSTATS>
	ether f4:e9:d4:xx:xx:xx
	hwaddr 00:00:00:00:00:00
	laggproto lacp lagghash l2,l3,l4
	laggport: ql0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	laggport: ql1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
	groups: lagg
	media: Ethernet autoselect
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```

I'm preparing the fix.

(In reply to Zhenlei Huang from comment #4)
Thanks, just one more thing (not that in changes anything you wrote, just wanted to let you know for clarity) - I have the following in my rc.conf for testing purposes:

> ifconfig_ql0="mtu 9000 up"
> ifconfig_ql1="mtu 9000 promisc up"
> 
> cloned_interfaces="lagg0"
> 
> ifconfig_lagg0="inet 172.16.200.30 netmask 255.255.255.0 laggproto lacp laggport ql0 laggport ql1 lacp_fast_timeout up"

after a clean boot, ifconfig output clearly states that ql0 has no hwaddr, just ether (which is the same as ql1), and "tcpdump -A -a -nn -s 0 --no-promiscuous-mode -envi ql0" gives no captured traffic at all. executing "ifconfig ql0 up" afterwards makes no differece for me. 

> ql0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
> 	 options=8d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,HWSTATS>
> 	ether 94:f1:28:b3:07:96
> 	media: Ethernet autoselect (10Gbase-SR <full-duplex>)
> 	status: active
> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> ql1: flags=1028943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC,LOWER_UP> metric 0 mtu 9000
> 	options=8d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,HWSTATS>
> 	ether 94:f1:28:b3:07:96
> 	hwaddr 94:f1:28:b3:07:97
> 	media: Ethernet autoselect (10Gbase-SR <full-duplex>)
> 	status: active
> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
> 	options=8d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,HWSTATS>
> 	ether 94:f1:28:b3:07:96
> 	hwaddr 00:00:00:00:00:00
> 	inet 172.16.200.30 netmask 0xffffff00 broadcast 172.16.200.255
> 	laggproto lacp lagghash l2,l3,l4
> 	laggport: ql0 flags=0<>
> 	laggport: ql1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> 	groups: lagg
> 	media: Ethernet autoselect
> 	status: active
> 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>