Bug 195234

Summary: unable to utilize bandwidth of all lacp lagg links using multiple clients and proper hashing algo
Product: Base System Reporter: freebsd
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: New ---    
Severity: Affects Only Me CC: freebsd
Priority: ---    
Version: 9.2-STABLE   
Hardware: amd64   
OS: Any   

Description freebsd 2014-11-20 23:35:45 UTC
I'm using FreeNAS 9.2.1.8 distro for storage usacases. Storage wors great, the lagg is giving me headache :) Issue is that I've 4 1G links connected between lacp enabled switch ports and the FreeBSD. I've multiple other hosts connected to the same switch and when opening the connection from all the hosts to the FreeBSD target, all lacp links are used, but the total used performance/capacity is equal to the capacity of single link. Meaning that even though the traffic is splitted between all 4 links, the overall performance is identical as if I'd be using one link only :) I've connected RHEL box to the switch using the same 4 links with connected bonding in mode=4(lacp) and it works as expected there. I've googled a lot to check what I'm doing wrong and found one post describing my situation. http://serverfault.com/questions/619574/freebsd-link-aggregation-no-faster-than-single-link  I've also tried to configure (undocumented) lagghash option to l4 only, but no luck. I've tried to configure the switch part to active/passive mode, disabling LAN HW offload, etc... My last network configuration (with disabled HW offload)is here:
[root@vmstore] ~# ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 00:24:1d:db:1b:16
        inet 10.50.27.99 netmask 0xfffffe00 broadcast 10.50.27.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:11:0a:5d:ec:48
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:11:0a:5d:ec:48
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:11:0a:5d:ec:48
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:11:0a:5d:ec:48
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xe
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:11:0a:5d:ec:48
        inet 172.16.0.100 netmask 0xffffff00 broadcast 172.16.0.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
Comment 1 freebsd 2014-11-20 23:36:13 UTC
The same applies to FreeBSD 9.3
Comment 2 freebsd 2014-11-21 11:33:56 UTC
More info for case when FreeNAs and RHEl are connected via single LACP configured switch:

BSD(em0-3 in the lagg):
                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   ||

      Interface           Traffic               Peak                Total
          lagg0  in     95.525 MB/s         95.874 MB/s           21.107 GB
                 out     4.317 MB/s          4.345 MB/s         1003.498 MB

            lo0  in      0.000 KB/s          0.455 KB/s           10.353 MB
                 out     0.000 KB/s          0.455 KB/s           10.353 MB

            em3  in     45.499 MB/s         49.629 MB/s           38.391 GB
                 out     2.073 MB/s          2.260 MB/s            1.786 GB

            em2  in     50.026 MB/s         56.275 MB/s           10.922 GB
                 out     1.226 MB/s          1.371 MB/s          136.151 MB

            em1  in      0.111 KB/s          0.366 KB/s          218.477 MB
                 out     0.075 KB/s          0.103 KB/s            1.340 GB

            em0  in      0.127 KB/s          0.357 KB/s           27.590 GB
                 out     1.017 MB/s          2.525 MB/s          323.180 MB

            re0  in      1.256 KB/s          6.423 KB/s           62.622 MB
                 out     0.217 KB/s         26.639 KB/s           62.129 MB



RHEL(eth1-2 in the lagg):
KB/s in  KB/s out   KB/s in  KB/s out   KB/s in  KB/s out   KB/s in  KB/s out
    0.56      0.19   2167.10  18433.90   2268.08  80626.50   4435.18  99060.40
    0.78      0.45   2183.97  18892.50   2256.93  80111.36   4440.89  99003.87
    0.66      0.19   2120.19  22640.73   2348.60  76848.66   4468.79  99489.39
    0.47      0.19   2119.32  25776.27   2283.31  72791.63   4402.63  98567.91
    0.41      0.19   2161.53  18827.42   2282.04  80256.05   4443.57  99083.47
    1.05      0.19   2088.83  27862.58   2323.93  70801.49   4412.77  98664.07
    0.36      0.19   2061.01  28573.16   2407.07  70757.54   4468.09  99330.69
    0.73      0.19   2198.07  14530.13   2294.56  85392.74   4492.69  99924.35