I'm using FreeNAS 9.2.1.8 distro for storage usacases. Storage wors great, the lagg is giving me headache :) Issue is that I've 4 1G links connected between lacp enabled switch ports and the FreeBSD. I've multiple other hosts connected to the same switch and when opening the connection from all the hosts to the FreeBSD target, all lacp links are used, but the total used performance/capacity is equal to the capacity of single link. Meaning that even though the traffic is splitted between all 4 links, the overall performance is identical as if I'd be using one link only :) I've connected RHEL box to the switch using the same 4 links with connected bonding in mode=4(lacp) and it works as expected there. I've googled a lot to check what I'm doing wrong and found one post describing my situation. http://serverfault.com/questions/619574/freebsd-link-aggregation-no-faster-than-single-link I've also tried to configure (undocumented) lagghash option to l4 only, but no luck. I've tried to configure the switch part to active/passive mode, disabling LAN HW offload, etc... My last network configuration (with disabled HW offload)is here: [root@vmstore] ~# ifconfig re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE> ether 00:24:1d:db:1b:16 inet 10.50.27.99 netmask 0xfffffe00 broadcast 10.50.27.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 00:11:0a:5d:ec:48 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether 00:11:0a:5d:ec:48 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:11:0a:5d:ec:48 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:11:0a:5d:ec:48 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect (1000baseT <full-duplex>) status: active ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536 nd6 options=9<PERFORMNUD,IFDISABLED> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xe inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:11:0a:5d:ec:48 inet 172.16.0.100 netmask 0xffffff00 broadcast 172.16.0.255 nd6 options=9<PERFORMNUD,IFDISABLED> media: Ethernet autoselect status: active laggproto lacp lagghash l2,l3,l4 laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
The same applies to FreeBSD 9.3
More info for case when FreeNAs and RHEl are connected via single LACP configured switch: BSD(em0-3 in the lagg): /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 Load Average || Interface Traffic Peak Total lagg0 in 95.525 MB/s 95.874 MB/s 21.107 GB out 4.317 MB/s 4.345 MB/s 1003.498 MB lo0 in 0.000 KB/s 0.455 KB/s 10.353 MB out 0.000 KB/s 0.455 KB/s 10.353 MB em3 in 45.499 MB/s 49.629 MB/s 38.391 GB out 2.073 MB/s 2.260 MB/s 1.786 GB em2 in 50.026 MB/s 56.275 MB/s 10.922 GB out 1.226 MB/s 1.371 MB/s 136.151 MB em1 in 0.111 KB/s 0.366 KB/s 218.477 MB out 0.075 KB/s 0.103 KB/s 1.340 GB em0 in 0.127 KB/s 0.357 KB/s 27.590 GB out 1.017 MB/s 2.525 MB/s 323.180 MB re0 in 1.256 KB/s 6.423 KB/s 62.622 MB out 0.217 KB/s 26.639 KB/s 62.129 MB RHEL(eth1-2 in the lagg): KB/s in KB/s out KB/s in KB/s out KB/s in KB/s out KB/s in KB/s out 0.56 0.19 2167.10 18433.90 2268.08 80626.50 4435.18 99060.40 0.78 0.45 2183.97 18892.50 2256.93 80111.36 4440.89 99003.87 0.66 0.19 2120.19 22640.73 2348.60 76848.66 4468.79 99489.39 0.47 0.19 2119.32 25776.27 2283.31 72791.63 4402.63 98567.91 0.41 0.19 2161.53 18827.42 2282.04 80256.05 4443.57 99083.47 1.05 0.19 2088.83 27862.58 2323.93 70801.49 4412.77 98664.07 0.36 0.19 2061.01 28573.16 2407.07 70757.54 4468.09 99330.69 0.73 0.19 2198.07 14530.13 2294.56 85392.74 4492.69 99924.35