Bug 240576 - mlx4en(4) driver unusable with MT26448 unless IPV6 is built into the kernel
Summary: mlx4en(4) driver unusable with MT26448 unless IPV6 is built into the kernel
Status: New
Alias: None
Product: Documentation
Classification: Unclassified
Component: Manual Pages (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-net (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-09-14 04:25 UTC by Ian
Modified: 2019-11-07 14:18 UTC (History)
3 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Ian 2019-09-14 04:25:58 UTC 
I've installed a Mellanox MT26448 based card, which appears to be recognized:

# dmesg|grep mlx4
mlx4_core0: <mlx4_core> mem 0xdf200000-0xdf2fffff,0xdf800000-0xdfffffff irq 22 at device 0.0 on pci4
mlx4_core: Mellanox ConnectX core driver v3.4.1 (October 2017)
mlx4_core: Initializing mlx4_core
mlx4_core0: Unable to determine PCI device chain minimum BW

# pciconf -lcbv mlx4_core0@pci0:4:0:0:
mlx4_core0@pci0:4:0:0:  class=0x020000 card=0x002115b3 chip=0x675015b3 rev=0xb0 hdr=0x00
    vendor     = 'Mellanox Technologies'
    device     = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base rxdf200000, size 1048576, enabled
    bar   [18] = type Prefetchable Memory, range 64, base rxdf800000, size 8388608, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 03[48] = VPD
    cap 11[9c] = MSI-X supports 128 messages, enabled
                 Table in map 0x10[0x7c000], PBA in map 0x10[0x7d000]
    cap 10[60] = PCI-Express 2 endpoint max data 128(256) FLR
                 link x8(x8) speed 5.0(5.0) ASPM disabled(L0s)
    ecap 000e[100] = ARI 1
    ecap 0003[148] = Serial 1 0002c90300529b42

However, the interface never appears in ifconfig, and is therefore unusable.  If it will help I can give devs a remote shell account to a machine with the card in it.
Comment 1 Ian 2019-09-17 00:57:09 UTC 
After much hassle, I've gotten the card to work under 12.0-RELEASE.  The issue is that this card requires "options INET6" in the kernel, but this is *not* documented anywhere in MLX4EN(4)!  This really needs to be documented for a few reasons:

1. Other cards that have similar IPv6 only options do not display this behavior when IPv6 is removed from the kernel.  For example, look at the overlap in options between this card and an igb interface:

igb options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
mlxen options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>

They are nearly identical, yet the igb interface will continue to work without IPv6 while the mlxen will not.

2. There is no hint anywhere that I can see, in dmesg, ifconfig, or even Mellanox's own software (mst) that this is the cause when your interface doesn't appear.

3. Some of us still live in the one country that hates IPv6 and thus aren't in the habit of keeping it in our kernels.

I'll reassign the PR to documentation.