Bug 236983
| Summary: | bnxt(4): VLAN not operational unless explicit "ifconfig promisc" is enabled on physical interface | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Base System | Reporter: | punkt.de Hosting Team <mops> | ||||||||
| Component: | kern | Assignee: | Tom Jones <thj> | ||||||||
| Status: | Closed FIXED | ||||||||||
| Severity: | Affects Some People | CC: | allanjude, bhargava.marreddy, dgeo, emaste, eugen, foo, franco, j.kelly.hays, jlduran, misho, nagy.attila, net, otis, pi, rgrimes, shurd, thj | ||||||||
| Priority: | Normal | Keywords: | needs-qa | ||||||||
| Version: | 11.2-STABLE | Flags: | koobs:
maintainer-feedback?
(shurd) kp: maintainer-feedback- koobs: mfc-stable13? koobs: mfc-stable12? koobs: mfc-stable11? |
||||||||
| Hardware: | amd64 | ||||||||||
| OS: | Any | ||||||||||
| Attachments: |
|
||||||||||
|
Description
punkt.de Hosting Team
2019-04-03 09:29:31 UTC
Please show output of: ifconfig bnxt0 Does it help if you do the following? ifconfig bnxt0 -vlanhwtag -vlanhwfilter If it helps, you can put this as temporary workaround to /etc/rc.conf: ifconfig_bnxt0="-vlanhwtag -vlanhwfilter" Nope - that was my first attempt. "up promisc" in rc.conf works, "up -vlanhwtag -vlanhwfilter" doesn't. Kind regards Patrick This is the non-working case: bnxt0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e427ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:25:90:5f:9a:82 hwaddr 00:25:90:5f:9a:82 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>) status: active (In reply to punkt.de Hosting Team from comment #2) The order can matter. Try putting "up" to the end and disable all vlan* and *csum hardware offloads. Are you sure the switch is configured to tag vlan 1 packets? Normally vlan 1 is the default and untagged management vlan of the switch and often you have to set a port to "general" mode and force vlan 1 tagged on it. Yes, the switch is configured with switchport trunk native vlan 1001 VLAN 1001 is my "throwaway" VLAN for untagged traffic ... As I wrote: everything is working as expected the moment I issue ifconfig bnxt0 promisc I'll try disabling the other options and putting "up" at the end and report back. Patrick This is my current config line: -vlanhwtag -vlanhwfilter -vlanmtu -vlanhwcsum -vlanhwtso -rxcsum -rxcsum6 -txcsum -txcsum6 up This is the resulting IF config after a complete reboot: bnxt0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8027a0<JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC> ether 00:25:90:5f:9a:82 hwaddr 00:25:90:5f:9a:82 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>) status: active No traffic on VLAN 1. For some reason VLAN_HWCSUM is not removed. I tried repeatedly on the command line, ifconfig does not change. ifconfig bnxt0 promisc --> bingo! BTW: you might want to look at this: https://redmine.ixsystems.com/issues/83019 The ix Systems colleague suspects a driver bug. Kind regards, Patrick It is obviously driver bug. Does it continue to work if you disable promisc. mode after it's enabled once? (In reply to punkt.de Hosting Team from comment #6) Can you show me tagged vlan traffic via tcpdump with the command: tcpdump -n -l -i bnxt0 -e vlan Thanks, Rod ifconfig bnxt0 -promisc
packets stop
ifconfig bnxt0 promisc
packets resume
Some SSH traffic:
5:18:48.715538 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603871 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.715586 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603871 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.715632 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603871 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.716515 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603872 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.717626 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603873 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
15:18:48.717678 00:19:e7:94:5b:6a > 00:25:90:5f:9a:82, ethertype 802.1Q (0x8100), length 98: vlan 1, p 0, ethertype IPv4, 217.29.44.151.63305 > 217.29.45.32.22: Flags [.], ack 3274335, win 2048, options [nop,nop,TS val 506603873 ecr 3213738477,nop,nop,sack 3 {3280555:3280679}{3278667:3280291}{3287327:3292827}], length 0
Kind regards,
Patrick
P.S. I have setups like this in production by the dozen. With em(4), igb(4), ix(4), ixl(4), bge(4) ... this is our first system with bnxt(4) and the only system with this problem. I really strongly suspect a driver problem.
P.P.S. In the linked ix Systems ticket there is another link to a forum post that deals with precisely this issue with bnxt(4) *only*. That user wonders why it seems to work the moment he starts tcpdump on the interface ... go figure ;-)
The tcpdump looks good, thanks sysctl dev.bnxt.X.vlan_only=1 Try that instead of promisc and see if that makes a difference. Nope :-) (In reply to punkt.de Hosting Team from comment #12) What are your values of dev.bnxt.0.ver.hwrm_fw and dev.bnxt.0.ver.hwrm_if? I don't think vlans were tested with hwrm interfaces greater than 1.8.1. If hwrm_if is greater than that, I may need to get a new hsi_struct_defs.h from Broadcom. root@freenas01[~]# sysctl dev.bnxt.0.ver.hwrm_fw dev.bnxt.0.ver.hwrm_fw: 212.0.162 root@freenas01[~]# sysctl dev.bnxt.0.ver.hwrm_if dev.bnxt.0.ver.hwrm_if: 1.9.1 root@freenas01[~]# uname -a FreeBSD freenas01.intern.punkt.de 11.2-STABLE FreeBSD 11.2-STABLE #0 r325575+9a3c7d8b53f(HEAD): Wed Mar 27 12:41:58 EDT 2019 root@mp20.tn.ixsystems.com:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64 Created attachment 203392 [details]
Fix handling of -vlanhwfilter
It appears that when VLAN HW filter is disabled, the NIC does not pass any vlan tagged traffic. This patch should pass all tagged and untagged traffic to the stack when -vlanhwfilter is in effect.
This should fix the bug when only -vlanhwfilter is set rather than requiring promisc. Actual HW vlan filtering is still broken.
This patch is against CURRENT, but should apply cleanly. If this works, I'm inclined to disable the vlanhwfilter capability until it's fixed for all firmware revisions.
Created attachment 203393 [details]
Update to disable vlanhwfilter option on hwrm if > 1.8
Updated patch to remove vlanhwfilter capability if hwrm if is > 1.8.
Stephen Hurd, Thank you for the patch, Can you please let me know if this issue got resolved? FWIW, this behaviour has been observed with lagg interface containing 4 bnxt cards. LACP won't come up until promisc is enabled. (In reply to Juraj Lutter from comment #18) To add: dev.bnxt.0.ver.hwrm_min_ver: 1.2.2 dev.bnxt.0.ver.hwrm_if: 1.10.1 flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> FreeBSD 12.2-STABLE r367273 With physical interfaces in promisc: lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether bc:97:e1:7d:d4:70 laggproto lacp lagghash l3,l4 laggport: bnxt0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: bnxt1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: bnxt2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: bnxt3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> Otherwise: bnxt3: Interface stopped DISTRIBUTING, possible flapping With the patch attached, following was emitted into log: kernel: debugnet_any_ifnet_update: Bad dn_init result from bnxt2 (ifp 0xfffff823ebdbc000), ignoring. I can see the same with fresh versions of stable/12 and /13. dev.bnxt.0.ver.hwrm_fw: 214.4.91 This patch fix promisc mode issue about VLAN on bnxt interface diff --git a/sys/dev/bnxt/if_bnxt.c b/sys/dev/bnxt/if_bnxt.c index 7811f4fdebf..f4a53cd4d93 100644 --- a/sys/dev/bnxt/if_bnxt.c +++ b/sys/dev/bnxt/if_bnxt.c @@ -588,7 +588,8 @@ bnxt_rx_queues_alloc(if_ctx_t ctx, caddr_t *vaddrs, softc->vnic_info.def_ring_grp = (uint16_t)HWRM_NA_SIGNATURE; softc->vnic_info.cos_rule = (uint16_t)HWRM_NA_SIGNATURE; softc->vnic_info.lb_rule = (uint16_t)HWRM_NA_SIGNATURE; - softc->vnic_info.rx_mask = HWRM_CFA_L2_SET_RX_MASK_INPUT_MASK_BCAST; + softc->vnic_info.rx_mask = HWRM_CFA_L2_SET_RX_MASK_INPUT_MASK_BCAST | + HWRM_CFA_L2_SET_RX_MASK_INPUT_MASK_ANYVLAN_NONVLAN; softc->vnic_info.mc_list_count = 0; softc->vnic_info.flags = BNXT_VNIC_FLAG_DEFAULT; rc = iflib_dma_alloc(ctx, BNXT_MAX_MC_ADDRS * ETHER_ADDR_LEN, @@ -1392,8 +1393,7 @@ bnxt_promisc_set(if_ctx_t ctx, int flags) HWRM_CFA_L2_SET_RX_MASK_INPUT_MASK_ANYVLAN_NONVLAN; else softc->vnic_info.rx_mask &= - ~(HWRM_CFA_L2_SET_RX_MASK_INPUT_MASK_PROMISCUOUS | - HWRM_CFA_L2_SET_RX_MASK_INPUT_MASK_ANYVLAN_NONVLAN); + ~(HWRM_CFA_L2_SET_RX_MASK_INPUT_MASK_PROMISCUOUS); rc = bnxt_hwrm_cfa_l2_set_rx_mask(softc, &softc->vnic_info); Created attachment 229498 [details]
Patch for fix issue with promisc mode and vlans
Comment on attachment 229498 [details]
Patch for fix issue with promisc mode and vlans
@Stephen Are you able to review this?
^Triage: Kristof may also be able to assist here (In reply to Kubilay Kocak from comment #25) I have no specific insight in Broadcom nics in general or the bnxt driver. Same problem here: $ uname -a FreeBSD ... 13.0-STABLE FreeBSD 13.0-STABLE stable/22.1-n248053-232cb14f501 SMP amd64 $ sysctl dev.bnxt.0.ver.hwrm_if dev.bnxt.0.ver.hwrm_if: 1.10.0 $ sysctl dev.bnxt.0.ver.hwrm_fw dev.bnxt.0.ver.hwrm_fw: 214.0.222 Is it possible to estimate when the patch will be released/officially integrated? Thank you for an information, the patch and your tireless work. still happens on 13.1-RC1 (In reply to Kurt Jaeger from comment #28) dev.bnxt.0.ver.hwrm_if: 1.10.1 dev.bnxt.0.ver.hwrm_fw: 214.4.91 (In reply to misho from comment #23) tested the patch on 13.1-RC1, works. (In reply to Kurt Jaeger from comment #30) asked re@ if it can be integrated into 13.1-REL Thanks Kurt Jaeger for feedback. I know that is works well on every version of driver from 12.x till 14-current Because we working with such cards of recently shipped HP chassis Gen10. We working with FreeBSD patched FreeBSD kernels from 12.1 to current as well. I needed to patch bnxt driver ASAP few months ago on the fly, because in production was delivered such chassis servers from HP without tests and approval in our new location. Similar was situation/reason before few years ago when I had to patch mellanox mlx5en driver ASAP. I was in FOSDEM during this time and be very unpleasant to working set on fire in one of the rooms these two days., instead to talking with people. :( I have created a review with the combination of the two patches. It wasn't 100% clear to me what has been tested, but they didn't seem to conflict so I took both. The review is here: https://reviews.freebsd.org/D34824 If someone can test and let me know the results in the next day or so I will persue getting this into the next 13.1 RC. (In reply to Tom Jones from comment #34) See review. Tested the version from the review, problem still: interface works only in promisc mode. The patch from https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236983#c23 works fine, without the promisc. (In reply to Kurt Jaeger from comment #35) Just to test, can you try the patch from the review, but with: ifconfig bnxt0 -vlanhwfilter (In reply to Allan Jude from comment #36) tried, does not work either. I have not looked at the code what it behaves like this... A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=0c6a2fa33e36ac0b5d51cbae39a9c5564ad61788 commit 0c6a2fa33e36ac0b5d51cbae39a9c5564ad61788 Author: Misho <misho@elwix.org> AuthorDate: 2022-04-21 08:24:04 +0000 Commit: Tom Jones <thj@FreeBSD.org> CommitDate: 2022-04-21 08:31:51 +0000 if_bnxt: Allow bnxt interfaces to use vlans When VLAN HW filter is disabled, the NIC does not pass any vlan tagged traffic. Setting these flags on the device allows vlan tagged traffic to pass. PR: 236983 Tested by: pi Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D34824 sys/dev/bnxt/if_bnxt.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e54505277c39cad32f999fd72033f6fa9594b130 commit e54505277c39cad32f999fd72033f6fa9594b130 Author: Misho <misho@elwix.org> AuthorDate: 2022-04-21 08:24:04 +0000 Commit: Tom Jones <thj@FreeBSD.org> CommitDate: 2022-04-27 13:14:40 +0000 if_bnxt: Allow bnxt interfaces to use vlans When VLAN HW filter is disabled, the NIC does not pass any vlan tagged traffic. Setting these flags on the device allows vlan tagged traffic to pass. PR: 236983 Tested by: pi Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D34824 (cherry picked from commit 0c6a2fa33e36ac0b5d51cbae39a9c5564ad61788) sys/dev/bnxt/if_bnxt.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) A commit in branch releng/13.1 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=5acfa48bb2585adfb8ceeacb695c6e6cc660894d commit 5acfa48bb2585adfb8ceeacb695c6e6cc660894d Author: Misho <misho@elwix.org> AuthorDate: 2022-04-21 08:24:04 +0000 Commit: Tom Jones <thj@FreeBSD.org> CommitDate: 2022-04-27 13:55:24 +0000 if_bnxt: Allow bnxt interfaces to use vlans When VLAN HW filter is disabled, the NIC does not pass any vlan tagged traffic. Setting these flags on the device allows vlan tagged traffic to pass. Approved by: re (gjb) PR: 236983 Tested by: pi Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D34824 (cherry picked from commit 0c6a2fa33e36ac0b5d51cbae39a9c5564ad61788) (cherry picked from commit e54505277c39cad32f999fd72033f6fa9594b130) sys/dev/bnxt/if_bnxt.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) |
