| Summary: | FreeBSD 12.0 ix driver TX TSO vlan-tagged network regression from 11.2 | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Nick <freebsd> |
| Component: | misc | Assignee: | freebsd-net (Nobody) <net> |
| Status: | New --- | ||
| Severity: | Affects Only Me | Keywords: | regression |
| Priority: | --- | ||
| Version: | 12.0-RELEASE | ||
| Hardware: | amd64 | ||
| OS: | Any | ||
Hi, I've done some additional testing: -Failing over to the igb interfaces doesn't show any problems with TSO enabled or disabled, so this is probably specific to ix -Dropping the MTU to 1500 has no effect -Removing an ix interfaces from its lagg and configuring an untagged interface directly on it doesn't shows significantly but improved TX performance, however this is seen with TSO enabled and disabled. I haven't managed to rule out the switch configuration as the cause of this yet, as I don't normally have any untagged ports. I'll update in a bit when I've done some more testing with the untagged interface configuration. Kind regards, Nick (In reply to Nick from comment #1) Sorry, I snafu'd that last comment. Should have read: -Removing an ix interface from its lagg and configuring an untagged interface directly on it shows significantly improved, but still poor, TX performance - however this is seen both with TSO enabled and disabled. I haven't managed to rule out the switch configuration as the cause of this yet, as I don't normally have any untagged ports. Hi, One further point of data: ipv6 testing shows the same results as ipv4, both for tagged and untagged interfaces. I've also discovered something odd: once TSO has been disabled on an interface, or if it was disabled at boot, I'm unable to subsequently trigger the issue on a tagged interface. Either turning TSO on after boot for some reason doesn't cause the problem, or my attempts to turn TSO on after boot are failing. I'm not really sure how to check for definite, but I do seee that the tsx_tx counters don't increase from 0 in this case, so I suspect TSO hasn't really been re-enabled for some reason: Booted with TSO disabled on ix0: 19:09:27 root@filer:/root/ sysctl dev.ix.0 | grep -i tso dev.ix.0.queue3.tso_tx: 0 dev.ix.0.queue2.tso_tx: 0 dev.ix.0.queue1.tso_tx: 0 dev.ix.0.queue0.tso_tx: 0 Enable TSO: 19:15:11 root@filer:/root/ ifconfig ix0 tso TX test shows good behaviour: 19:18:00 root@filer:/root/ iperf3 -c 10.21.10.4 Connecting to host 10.21.10.4, port 5201 [ 5] local 10.21.10.2 port 54437 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 1.00-2.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.09 MBytes [ 5] 2.00-3.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 4.00-5.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 7.00-8.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.09 MBytes [ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec receiver Re-check TSO counters, discover they haven't increased: 19:18:11 root@filer:/root/ sysctl dev.ix.0 | grep -i tso dev.ix.0.queue3.tso_tx: 0 dev.ix.0.queue2.tso_tx: 0 dev.ix.0.queue1.tso_tx: 0 dev.ix.0.queue0.tso_tx: 0 I'm pretty much out of ideas as to how to proceeed with diagnosis at this point. I'm happy to perform any further testing you may need, but disabling TSO seems to adequately solve the problem for me, so I'll just do that. If you want to attempt to reproduce, I think our minimal config to reproduce is thus: FreeBSD 12.0-Release (amd64) running with an X520-DA2 card installed, with one port in a failover lagg, configured with a vlan-tagged interface on top, with TSO enabled on ix<whichever> in rc.conf. In case the platform is relevant, this machine is a Xeon E3-1245v3 (with HT disabled in BIOS), running on an ASRock C226WS with 4 x DDR3 ECC DIMMs installed. Kind regards, Nick |
Hi, Immediately after upgrading from FreeBSD 11.2 to 12.0, network TX performance dropped significantly. Seems to be related to TSO, or at least disabling it resolves the problem for me. Setup is a physical FreeBSD 12 host connected to a mixture of physical and virtual Debian and Ubuntu machines via a 10gbit switch. 23:58:18 root@filer:/root/ uname -a FreeBSD filer.<censored> 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64 iperf3 tests from physical FreeBSD to virtual Debian: 23:32:39 root@filer:/root/ iperf3 -c 10.21.10.4 Connecting to host 10.21.10.4, port 5201 [ 5] local 10.21.10.2 port 16129 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 70.0 KBytes 572 Kbits/sec 8 17.5 KBytes [ 5] 1.00-2.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 2.00-3.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 3.00-4.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes [ 5] 4.00-5.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 5.00-6.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 6.00-7.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes [ 5] 7.00-8.00 sec 35.0 KBytes 286 Kbits/sec 8 17.5 KBytes [ 5] 8.00-9.00 sec 35.0 KBytes 287 Kbits/sec 8 17.5 KBytes [ 5] 9.00-10.00 sec 43.7 KBytes 358 Kbits/sec 10 17.5 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 411 KBytes 336 Kbits/sec 86 sender [ 5] 0.00-10.00 sec 376 KBytes 308 Kbits/sec receiver iperf Done. 23:32:51 root@filer:/root/ iperf3 -c 10.21.10.4 -R Connecting to host 10.21.10.4, port 5201 Reverse mode, remote host 10.21.10.4 is sending [ 5] local 10.21.10.2 port 63402 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.15 GBytes 9.84 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.86 Gbits/sec [ 5] 3.00-4.00 sec 1.13 GBytes 9.71 Gbits/sec [ 5] 4.00-5.00 sec 1.13 GBytes 9.73 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.85 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.84 Gbits/sec receiver iperf Done. Turning off TSO: 23:33:17 root@filer:/root/ ifconfig ix0 -tso Re-running iperf3: 23:33:57 root@filer:/root/ iperf3 -c 10.21.10.4 Connecting to host 10.21.10.4, port 5201 [ 5] local 10.21.10.2 port 60137 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.15 GBytes 9.88 Gbits/sec 0 1.09 MBytes [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 2.00-3.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 3.00-4.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec 0 1.09 MBytes [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.09 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec receiver iperf Done. 23:34:12 root@filer:/root/ iperf3 -c 10.21.10.4 -R Connecting to host 10.21.10.4, port 5201 Reverse mode, remote host 10.21.10.4 is sending [ 5] local 10.21.10.2 port 31621 connected to 10.21.10.4 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.14 GBytes 9.81 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.87 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 3.00-4.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 4.00-5.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.84 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.87 Gbits/sec receiver iperf Done. Network configuration is not entirely trivial. From /etc/rc.conf (ipv6 global addresses censored): ipv6_activate_all_interfaces="YES" ifconfig_igb0="up mtu 9000" ifconfig_igb1="up mtu 9000" ifconfig_ix0="up mtu 9000" ifconfig_ix1="up mtu 9000" cloned_interfaces="lagg0 lagg1 vlan10 vlan11 vlan30 vlan34 vlan35" ifconfig_lagg0="laggproto failover laggport ix0 laggport igb0" ifconfig_lagg1="laggproto failover laggport ix1 laggport igb1" ifconfig_vlan10="vlan 10 vlandev lagg0 mtu 9000" ifconfig_vlan10_alias0="inet 10.21.10.2/24" ifconfig_vlan10_ipv6="inet6 fd9c:bee6:9d33:10::2 prefixlen 64" ifconfig_vlan11="vlan 11 vlandev lagg1 mtu 9000" ifconfig_vlan11_alias0="inet 10.21.11.2/24" ifconfig_vlan11_ipv6="inet6 fd9c:bee6:9d33:11::2 prefixlen 64" ifconfig_vlan30="vlan 30 vlandev lagg0 mtu 9000" ifconfig_vlan30_alias0="inet 10.21.30.3/24" ifconfig_vlan30_ipv6="inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64" ifconfig_vlan34="vlan 34 vlandev lagg1 mtu 9000" ifconfig_vlan34_alias0="inet 10.21.34.2/24" ifconfig_vlan34_ipv6="inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64" ifconfig_vlan35="vlan 35 vlandev lagg1 mtu 1500" ifconfig_vlan35_alias0="inet 10.21.35.3/24" ifconfig_vlan35_ipv6="inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64" defaultrouter="10.21.34.1" ipv6_defaultrouter="2a01:xxxx:xxxx:xxxx::1" 23:46:03 root@filer:/root/ ifconfig ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 hwaddr bc:5f:f4:c7:d5:58 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 hwaddr bc:5f:f4:c7:d5:56 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 inet6 fe80::a236:9fff:fe72:3594%lagg0 prefixlen 64 scopeid 0x6 laggproto failover lagghash l2,l3,l4 laggport: ix0 flags=5<MASTER,ACTIVE> laggport: igb0 flags=0<> groups: lagg media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 fe80::a236:9fff:fe72:3596%lagg1 prefixlen 64 scopeid 0x7 laggproto failover lagghash l2,l3,l4 laggport: ix1 flags=5<MASTER,ACTIVE> laggport: igb1 flags=0<> groups: lagg media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 inet6 fd9c:bee6:9d33:10::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3594%vlan10 prefixlen 64 scopeid 0x8 inet 10.21.10.2 netmask 0xffffff00 broadcast 10.21.10.255 groups: vlan vlan: 10 vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 fd9c:bee6:9d33:11::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan11 prefixlen 64 scopeid 0x9 inet 10.21.11.2 netmask 0xffffff00 broadcast 10.21.11.255 groups: vlan vlan: 11 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:94 inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64 inet6 fe80::a236:9fff:fe72:3594%vlan30 prefixlen 64 scopeid 0xa inet 10.21.30.3 netmask 0xffffff00 broadcast 10.21.30.255 groups: vlan vlan: 30 vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan34: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan34 prefixlen 64 scopeid 0xb inet 10.21.34.2 netmask 0xffffff00 broadcast 10.21.34.255 groups: vlan vlan: 34 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> vlan35: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:72:35:96 inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64 inet6 fe80::a236:9fff:fe72:3596%vlan35 prefixlen 64 scopeid 0xc inet 10.21.35.3 netmask 0xffffff00 broadcast 10.21.35.255 groups: vlan vlan: 35 vlanpcp: 0 parent interface: lagg1 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> 23:53:21 root@filer:/root/ netstat -I vlan10 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll vlan1 9000 <Link#8> a0:36:9f:72:35:94 1957078 0 0 2672967 6 0 vlan1 - fd9c:bee6:9d3 fd9c:bee6:9d33:10 451401 - - 372777 - - vlan1 - fe80::%vlan10 fe80::a236:9fff:f 0 - - 4 - - vlan1 - 10.21.10.0/24 10.21.10.2 1505540 - - 2301252 - - 23:53:25 root@filer:/root/ netstat -I lagg0 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll lagg0 9000 <Link#6> a0:36:9f:72:35:94 6043751 0 0 4156297 15 0 lagg0 - fe80::%lagg0/ fe80::a236:9fff:f 0 - - 3 - - 23:53:31 root@filer:/root/ netstat -I ix0 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ix0 9000 <Link#1> a0:36:9f:72:35:94 6043052 0 0 4156335 0 0 No seemingly relevant warnings or messages in dmesg or /var/log/messages. 23:56:26 root@filer:/root/ dmesg | grep "igb\|ix\|lagg" acpi0: Power Button (fixed) ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe020-0xe03f mem 0xf0500000-0xf05fffff,0xf0704000-0xf0707fff irq 16 at device 0.0 on pci1 ix0: using 2048 tx descriptors and 2048 rx descriptors ix0: msix_init qsets capped at 16 ix0: pxm cpus: 4 queue msgs: 63 admincnt: 1 ix0: using 4 rx queues 4 tx queues ix0: Using MSIX interrupts with 5 vectors ix0: allocated for 4 queues ix0: allocated for 4 rx queues ix0: Ethernet address: a0:36:9f:72:35:94 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix0: netmap queues/slots: TX 4/2048, RX 4/2048 ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe000-0xe01f mem 0xf0400000-0xf04fffff,0xf0700000-0xf0703fff irq 17 at device 0.1 on pci1 ix1: using 2048 tx descriptors and 2048 rx descriptors ix1: msix_init qsets capped at 16 ix1: pxm cpus: 4 queue msgs: 63 admincnt: 1 ix1: using 4 rx queues 4 tx queues ix1: Using MSIX interrupts with 5 vectors ix1: allocated for 4 queues ix1: allocated for 4 rx queues ix1: Ethernet address: a0:36:9f:72:35:96 ix1: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: netmap queues/slots: TX 4/2048, RX 4/2048 igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xb000-0xb01f mem 0xf0c00000-0xf0c7ffff,0xf0c80000-0xf0c83fff irq 18 at device 0.0 on pci5 igb0: attach_pre capping queues at 4 igb0: using 1024 tx descriptors and 1024 rx descriptors igb0: msix_init qsets capped at 4 igb0: pxm cpus: 4 queue msgs: 4 admincnt: 1 igb0: using 4 rx queues 4 tx queues igb0: Using MSIX interrupts with 5 vectors igb0: allocated for 4 tx_queues igb0: allocated for 4 rx_queues igb0: Ethernet address: bc:5f:f4:c7:d5:58 igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xa000-0xa01f mem 0xf0900000-0xf097ffff,0xf0980000-0xf0983fff irq 19 at device 0.0 on pci9 igb1: attach_pre capping queues at 4 igb1: using 1024 tx descriptors and 1024 rx descriptors igb1: msix_init qsets capped at 4 igb1: pxm cpus: 4 queue msgs: 4 admincnt: 1 igb1: using 4 rx queues 4 tx queues igb1: Using MSIX interrupts with 5 vectors igb1: allocated for 4 tx_queues igb1: allocated for 4 rx_queues igb1: Ethernet address: bc:5f:f4:c7:d5:56 igb1: netmap queues/slots: TX 4/1024, RX 4/1024 ix0: link state changed to UP lagg0: IPv6 addresses on ix0 have been removed before adding it as a member to prevent IPv6 address scope violation. lagg0: link state changed to UP lagg0: IPv6 addresses on igb0 have been removed before adding it as a member to prevent IPv6 address scope violation. ix1: link state changed to UP lagg1: IPv6 addresses on ix1 have been removed before adding it as a member to prevent IPv6 address scope violation. lagg1: link state changed to UP lagg1: IPv6 addresses on igb1 have been removed before adding it as a member to prevent IPv6 address scope violation. ix0: link state changed to DOWN lagg0: link state changed to DOWN ix1: link state changed to DOWN lagg1: link state changed to DOWN ix0: link state changed to UP lagg0: link state changed to UP ix1: link state changed to UP lagg1: link state changed to UP igb0: link state changed to UP igb1: link state changed to UP I'll perform some further tests over the weekend, see if failing over to the 1gbit igb interfaces or testing against a non-vlan-tagged interface makes any difference. Let me know if any further info could be useful. Kind regards, Nick