Bug 236983 - bnxt(4) VLAN not operational unless explicit "ifconfig promisc" is used on the physical IF
Summary: bnxt(4) VLAN not operational unless explicit "ifconfig promisc" is used on th...
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.2-STABLE
Hardware: amd64 Any
: Normal Affects Some People
Assignee: freebsd-net mailing list
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2019-04-03 09:29 UTC by punkt.de Hosting Team
Modified: 2019-11-04 11:14 UTC (History)
6 users (show)

See Also:
koobs: mfc-stable11?
koobs: mfc-stable12?


Attachments
Fix handling of -vlanhwfilter (1.34 KB, patch)
2019-04-04 19:42 UTC, Stephen Hurd
no flags Details | Diff
Update to disable vlanhwfilter option on hwrm if > 1.8 (1.78 KB, patch)
2019-04-04 21:02 UTC, Stephen Hurd
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description punkt.de Hosting Team 2019-04-03 09:29:31 UTC
ifconfig bnxt0 up
ifconfig vlan1 create
ifconfig vlan1 vlan 1 vlandev bnxt0
ifconfig vlan1 inet 192.168.0.1/24

Result: no connection on VLAN1, no packets visible if

tcpdump -n -i vlan1


Workaround:

ifconfig bnxt0 promisc

Working as expected.


Kind regards,
Patrick
Comment 1 Eugene Grosbein freebsd_committer 2019-04-03 10:04:44 UTC
Please show output of: ifconfig bnxt0

Does it help if you do the following?

ifconfig bnxt0 -vlanhwtag -vlanhwfilter

If it helps, you can put this as temporary workaround to /etc/rc.conf:

ifconfig_bnxt0="-vlanhwtag -vlanhwfilter"
Comment 2 punkt.de Hosting Team 2019-04-03 10:11:52 UTC
Nope - that was my first attempt.

"up promisc" in rc.conf works, "up -vlanhwtag -vlanhwfilter" doesn't.

Kind regards
Patrick
Comment 3 punkt.de Hosting Team 2019-04-03 10:12:46 UTC
This is the non-working case:

bnxt0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=e427ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:25:90:5f:9a:82
	hwaddr 00:25:90:5f:9a:82
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
	status: active
Comment 4 Eugene Grosbein freebsd_committer 2019-04-03 10:22:50 UTC
(In reply to punkt.de Hosting Team from comment #2)

The order can matter. Try putting "up" to the end and disable all vlan* and *csum hardware offloads.
Comment 5 Rodney W. Grimes freebsd_committer 2019-04-03 10:50:29 UTC
Are you sure the switch is configured to tag vlan 1 packets?  Normally vlan 1 is the default and untagged management vlan of the switch and often you have to set a port to "general" mode and force vlan 1 tagged on it.
Comment 6 punkt.de Hosting Team 2019-04-03 11:24:41 UTC
Yes, the switch is configured with

switchport trunk native vlan 1001

VLAN 1001 is my "throwaway" VLAN for untagged traffic ...


As I wrote: everything is working as expected the moment I issue

ifconfig bnxt0 promisc


I'll try disabling the other options and putting "up" at the end and report back.

Patrick
Comment 7 punkt.de Hosting Team 2019-04-03 11:51:33 UTC
This is my current config line:

-vlanhwtag -vlanhwfilter -vlanmtu -vlanhwcsum -vlanhwtso -rxcsum -rxcsum6 -txcsum -txcsum6 up

This is the resulting IF config after a complete reboot:

bnxt0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8027a0<JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC>
	ether 00:25:90:5f:9a:82
	hwaddr 00:25:90:5f:9a:82
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
	status: active

No traffic on VLAN 1.


For some reason VLAN_HWCSUM is not removed. I tried repeatedly on the command line, ifconfig does not change.


ifconfig bnxt0 promisc --> bingo!


BTW: you might want to look at this:
https://redmine.ixsystems.com/issues/83019

The ix Systems colleague suspects a driver bug.


Kind regards,
Patrick
Comment 8 Eugene Grosbein freebsd_committer 2019-04-03 12:37:33 UTC
It is obviously driver bug.

Does it continue to work if you disable promisc. mode after it's enabled once?
Comment 9 Rodney W. Grimes freebsd_committer 2019-04-03 12:51:03 UTC
(In reply to punkt.de Hosting Team from comment #6)
Can you show me tagged vlan traffic via tcpdump with the command:
tcpdump -n -l -i bnxt0 -e vlan

Thanks,
Rod
Comment 10 punkt.de Hosting Team 2019-04-03 13:22:07 UTC
ifconfig bnxt0 -promisc
packets stop

ifconfig bnxt0 promisc
packets resume

Some SSH traffic:

5:18:48.715538 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603871 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.715586 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603871 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.715632 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603871 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.716515 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603872 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.717626 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603873 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.717678 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603873 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0


Kind regards,
Patrick


P.S. I have setups like this in production by the dozen. With em(4), igb(4), ix(4), ixl(4), bge(4) ... this is our first system with bnxt(4) and the only system with this problem. I really strongly suspect a driver problem.

P.P.S. In the linked ix Systems ticket there is another link to a forum post that deals with precisely this issue with bnxt(4) *only*. That user wonders why it seems to work the moment he starts tcpdump on the interface ... go figure ;-)
Comment 11 Rodney W. Grimes freebsd_committer 2019-04-03 15:57:16 UTC
The tcpdump looks good, thanks

sysctl dev.bnxt.X.vlan_only=1

Try that instead of promisc and see if that makes a difference.
Comment 12 punkt.de Hosting Team 2019-04-03 17:55:03 UTC
Nope :-)
Comment 13 Stephen Hurd freebsd_committer 2019-04-04 02:07:59 UTC
(In reply to punkt.de Hosting Team from comment #12)

What are your values of dev.bnxt.0.ver.hwrm_fw and dev.bnxt.0.ver.hwrm_if?

I don't think vlans were tested with hwrm interfaces greater than 1.8.1.  If hwrm_if is greater than that, I may need to get a new hsi_struct_defs.h from Broadcom.
Comment 14 punkt.de Hosting Team 2019-04-04 07:34:57 UTC
root@freenas01[~]# sysctl dev.bnxt.0.ver.hwrm_fw
dev.bnxt.0.ver.hwrm_fw: 212.0.162
root@freenas01[~]# sysctl dev.bnxt.0.ver.hwrm_if
dev.bnxt.0.ver.hwrm_if: 1.9.1
root@freenas01[~]# uname -a
FreeBSD freenas01.intern.punkt.de 11.2-STABLE FreeBSD 11.2-STABLE #0 r325575+9a3c7d8b53f(HEAD): Wed Mar 27 12:41:58 EDT 2019     root@mp20.tn.ixsystems.com:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64  amd64
Comment 15 Stephen Hurd freebsd_committer 2019-04-04 19:42:43 UTC
Created attachment 203392 [details]
Fix handling of -vlanhwfilter

It appears that when VLAN HW filter is disabled, the NIC does not pass any vlan tagged traffic.  This patch should pass all tagged and untagged traffic to the stack when -vlanhwfilter is in effect.

This should fix the bug when only -vlanhwfilter is set rather than requiring promisc.  Actual HW vlan filtering is still broken.

This patch is against CURRENT, but should apply cleanly.  If this works, I'm inclined to disable the vlanhwfilter capability until it's fixed for all firmware revisions.
Comment 16 Stephen Hurd freebsd_committer 2019-04-04 21:02:10 UTC
Created attachment 203393 [details]
Update to disable vlanhwfilter option on hwrm if > 1.8

Updated patch to remove vlanhwfilter capability if hwrm if is > 1.8.
Comment 17 Bhargava Chenna. Marreddy 2019-11-04 11:14:11 UTC
Stephen Hurd, 

Thank you for the patch, Can you please let me know if this issue got resolved?