Bug 234624 - FreeBSD 12.0 ix driver TX TSO vlan-tagged network regression from 11.2
Summary: FreeBSD 12.0 ix driver TX TSO vlan-tagged network regression from 11.2
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-net mailing list
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2019-01-05 00:06 UTC by Nick
Modified: 2019-01-05 19:46 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nick 2019-01-05 00:06:25 UTC
Hi,

Immediately after upgrading from FreeBSD 11.2 to 12.0, network TX performance dropped significantly.  Seems to be related to TSO, or at least disabling it resolves the problem for me.  Setup is a physical FreeBSD 12 host connected to a mixture of physical and virtual Debian and Ubuntu machines via a 10gbit switch.

23:58:18 root@filer:/root/ uname -a
FreeBSD filer.<censored> 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC  amd64

iperf3 tests from physical FreeBSD to virtual Debian:

23:32:39 root@filer:/root/ iperf3 -c 10.21.10.4
Connecting to host 10.21.10.4, port 5201
[  5] local 10.21.10.2 port 16129 connected to 10.21.10.4 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  70.0 KBytes   572 Kbits/sec    8   17.5 KBytes
[  5]   1.00-2.00   sec  35.0 KBytes   286 Kbits/sec    8   17.5 KBytes
[  5]   2.00-3.00   sec  35.0 KBytes   287 Kbits/sec    8   17.5 KBytes
[  5]   3.00-4.00   sec  43.7 KBytes   358 Kbits/sec   10   17.5 KBytes
[  5]   4.00-5.00   sec  35.0 KBytes   286 Kbits/sec    8   17.5 KBytes
[  5]   5.00-6.00   sec  35.0 KBytes   287 Kbits/sec    8   17.5 KBytes
[  5]   6.00-7.00   sec  43.7 KBytes   358 Kbits/sec   10   17.5 KBytes
[  5]   7.00-8.00   sec  35.0 KBytes   286 Kbits/sec    8   17.5 KBytes
[  5]   8.00-9.00   sec  35.0 KBytes   287 Kbits/sec    8   17.5 KBytes
[  5]   9.00-10.00  sec  43.7 KBytes   358 Kbits/sec   10   17.5 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   411 KBytes   336 Kbits/sec   86             sender
[  5]   0.00-10.00  sec   376 KBytes   308 Kbits/sec                  receiver

iperf Done.
23:32:51 root@filer:/root/ iperf3 -c 10.21.10.4 -R
Connecting to host 10.21.10.4, port 5201
Reverse mode, remote host 10.21.10.4 is sending
[  5] local 10.21.10.2 port 63402 connected to 10.21.10.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.15 GBytes  9.84 Gbits/sec
[  5]   1.00-2.00   sec  1.15 GBytes  9.87 Gbits/sec
[  5]   2.00-3.00   sec  1.15 GBytes  9.86 Gbits/sec
[  5]   3.00-4.00   sec  1.13 GBytes  9.71 Gbits/sec
[  5]   4.00-5.00   sec  1.13 GBytes  9.73 Gbits/sec
[  5]   5.00-6.00   sec  1.15 GBytes  9.88 Gbits/sec
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   7.00-8.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   9.00-10.00  sec  1.15 GBytes  9.88 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.85 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.84 Gbits/sec                  receiver

iperf Done.

Turning off TSO:

23:33:17 root@filer:/root/ ifconfig ix0 -tso

Re-running iperf3:

23:33:57 root@filer:/root/ iperf3 -c 10.21.10.4
Connecting to host 10.21.10.4, port 5201
[  5] local 10.21.10.2 port 60137 connected to 10.21.10.4 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.88 Gbits/sec    0   1.09 MBytes
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.09 MBytes
[  5]   2.00-3.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.09 MBytes
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.09 MBytes
[  5]   4.00-5.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.09 MBytes
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.09 MBytes
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.09 MBytes
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.09 MBytes
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.09 MBytes
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec    0   1.09 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

iperf Done.
23:34:12 root@filer:/root/ iperf3 -c 10.21.10.4 -R
Connecting to host 10.21.10.4, port 5201
Reverse mode, remote host 10.21.10.4 is sending
[  5] local 10.21.10.2 port 31621 connected to 10.21.10.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.14 GBytes  9.81 Gbits/sec
[  5]   1.00-2.00   sec  1.15 GBytes  9.87 Gbits/sec
[  5]   2.00-3.00   sec  1.15 GBytes  9.88 Gbits/sec
[  5]   3.00-4.00   sec  1.15 GBytes  9.88 Gbits/sec
[  5]   4.00-5.00   sec  1.15 GBytes  9.88 Gbits/sec
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   7.00-8.00   sec  1.15 GBytes  9.84 Gbits/sec
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.87 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.87 Gbits/sec                  receiver

iperf Done.

Network configuration is not entirely trivial.  From /etc/rc.conf (ipv6 global addresses censored):

ipv6_activate_all_interfaces="YES"
ifconfig_igb0="up mtu 9000"
ifconfig_igb1="up mtu 9000"
ifconfig_ix0="up mtu 9000"
ifconfig_ix1="up mtu 9000"
cloned_interfaces="lagg0 lagg1 vlan10 vlan11 vlan30 vlan34 vlan35"
ifconfig_lagg0="laggproto failover laggport ix0 laggport igb0"
ifconfig_lagg1="laggproto failover laggport ix1 laggport igb1"
ifconfig_vlan10="vlan 10 vlandev lagg0 mtu 9000"
ifconfig_vlan10_alias0="inet 10.21.10.2/24"
ifconfig_vlan10_ipv6="inet6 fd9c:bee6:9d33:10::2 prefixlen 64"
ifconfig_vlan11="vlan 11 vlandev lagg1 mtu 9000"
ifconfig_vlan11_alias0="inet 10.21.11.2/24"
ifconfig_vlan11_ipv6="inet6 fd9c:bee6:9d33:11::2 prefixlen 64"
ifconfig_vlan30="vlan 30 vlandev lagg0 mtu 9000"
ifconfig_vlan30_alias0="inet 10.21.30.3/24"
ifconfig_vlan30_ipv6="inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64"
ifconfig_vlan34="vlan 34 vlandev lagg1 mtu 9000"
ifconfig_vlan34_alias0="inet 10.21.34.2/24"
ifconfig_vlan34_ipv6="inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64"
ifconfig_vlan35="vlan 35 vlandev lagg1 mtu 1500"
ifconfig_vlan35_alias0="inet 10.21.35.3/24"
ifconfig_vlan35_ipv6="inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64"
defaultrouter="10.21.34.1"
ipv6_defaultrouter="2a01:xxxx:xxxx:xxxx::1"

23:46:03 root@filer:/root/ ifconfig
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:94
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:96
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:94
        hwaddr bc:5f:f4:c7:d5:58
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:96
        hwaddr bc:5f:f4:c7:d5:56
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:94
        inet6 fe80::a236:9fff:fe72:3594%lagg0 prefixlen 64 scopeid 0x6
        laggproto failover lagghash l2,l3,l4
        laggport: ix0 flags=5<MASTER,ACTIVE>
        laggport: igb0 flags=0<>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:96
        inet6 fe80::a236:9fff:fe72:3596%lagg1 prefixlen 64 scopeid 0x7
        laggproto failover lagghash l2,l3,l4
        laggport: ix1 flags=5<MASTER,ACTIVE>
        laggport: igb1 flags=0<>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:94
        inet6 fd9c:bee6:9d33:10::2 prefixlen 64
        inet6 fe80::a236:9fff:fe72:3594%vlan10 prefixlen 64 scopeid 0x8
        inet 10.21.10.2 netmask 0xffffff00 broadcast 10.21.10.255
        groups: vlan
        vlan: 10 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:96
        inet6 fd9c:bee6:9d33:11::2 prefixlen 64
        inet6 fe80::a236:9fff:fe72:3596%vlan11 prefixlen 64 scopeid 0x9
        inet 10.21.11.2 netmask 0xffffff00 broadcast 10.21.11.255
        groups: vlan
        vlan: 11 vlanpcp: 0 parent interface: lagg1
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan30: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:94
        inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64
        inet6 fe80::a236:9fff:fe72:3594%vlan30 prefixlen 64 scopeid 0xa
        inet 10.21.30.3 netmask 0xffffff00 broadcast 10.21.30.255
        groups: vlan
        vlan: 30 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan34: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:96
        inet6 2a01:xxxx:xxxx:xxxx::2 prefixlen 64
        inet6 fe80::a236:9fff:fe72:3596%vlan34 prefixlen 64 scopeid 0xb
        inet 10.21.34.2 netmask 0xffffff00 broadcast 10.21.34.255
        groups: vlan
        vlan: 34 vlanpcp: 0 parent interface: lagg1
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vlan35: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=600403<RXCSUM,TXCSUM,LRO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether a0:36:9f:72:35:96
        inet6 2a01:xxxx:xxxx:xxxx::3 prefixlen 64
        inet6 fe80::a236:9fff:fe72:3596%vlan35 prefixlen 64 scopeid 0xc
        inet 10.21.35.3 netmask 0xffffff00 broadcast 10.21.35.255
        groups: vlan
        vlan: 35 vlanpcp: 0 parent interface: lagg1
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

23:53:21 root@filer:/root/ netstat -I vlan10
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
vlan1  9000 <Link#8>      a0:36:9f:72:35:94  1957078     0     0  2672967     6     0
vlan1     - fd9c:bee6:9d3 fd9c:bee6:9d33:10   451401     -     -   372777     -     -
vlan1     - fe80::%vlan10 fe80::a236:9fff:f        0     -     -        4     -     -
vlan1     - 10.21.10.0/24 10.21.10.2         1505540     -     -  2301252     -     -
23:53:25 root@filer:/root/ netstat -I lagg0 
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
lagg0  9000 <Link#6>      a0:36:9f:72:35:94  6043751     0     0  4156297    15     0
lagg0     - fe80::%lagg0/ fe80::a236:9fff:f        0     -     -        3     -     -
23:53:31 root@filer:/root/ netstat -I ix0  
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ix0    9000 <Link#1>      a0:36:9f:72:35:94  6043052     0     0  4156335     0     0


No seemingly relevant warnings or messages in dmesg or /var/log/messages.

23:56:26 root@filer:/root/ dmesg | grep "igb\|ix\|lagg"
acpi0: Power Button (fixed)
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe020-0xe03f mem 0xf0500000-0xf05fffff,0xf0704000-0xf0707fff irq 16 at device 0.0 on pci1
ix0: using 2048 tx descriptors and 2048 rx descriptors
ix0: msix_init qsets capped at 16
ix0: pxm cpus: 4 queue msgs: 63 admincnt: 1
ix0: using 4 rx queues 4 tx queues
ix0: Using MSIX interrupts with 5 vectors
ix0: allocated for 4 queues
ix0: allocated for 4 rx queues
ix0: Ethernet address: a0:36:9f:72:35:94
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xe000-0xe01f mem 0xf0400000-0xf04fffff,0xf0700000-0xf0703fff irq 17 at device 0.1 on pci1
ix1: using 2048 tx descriptors and 2048 rx descriptors
ix1: msix_init qsets capped at 16
ix1: pxm cpus: 4 queue msgs: 63 admincnt: 1
ix1: using 4 rx queues 4 tx queues
ix1: Using MSIX interrupts with 5 vectors
ix1: allocated for 4 queues
ix1: allocated for 4 rx queues
ix1: Ethernet address: a0:36:9f:72:35:96
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xb000-0xb01f mem 0xf0c00000-0xf0c7ffff,0xf0c80000-0xf0c83fff irq 18 at device 0.0 on pci5
igb0: attach_pre capping queues at 4
igb0: using 1024 tx descriptors and 1024 rx descriptors
igb0: msix_init qsets capped at 4
igb0: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb0: using 4 rx queues 4 tx queues
igb0: Using MSIX interrupts with 5 vectors
igb0: allocated for 4 tx_queues
igb0: allocated for 4 rx_queues
igb0: Ethernet address: bc:5f:f4:c7:d5:58
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xa000-0xa01f mem 0xf0900000-0xf097ffff,0xf0980000-0xf0983fff irq 19 at device 0.0 on pci9
igb1: attach_pre capping queues at 4
igb1: using 1024 tx descriptors and 1024 rx descriptors
igb1: msix_init qsets capped at 4
igb1: pxm cpus: 4 queue msgs: 4 admincnt: 1
igb1: using 4 rx queues 4 tx queues
igb1: Using MSIX interrupts with 5 vectors
igb1: allocated for 4 tx_queues
igb1: allocated for 4 rx_queues
igb1: Ethernet address: bc:5f:f4:c7:d5:56
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
ix0: link state changed to UP
lagg0: IPv6 addresses on ix0 have been removed before adding it as a member to prevent IPv6 address scope violation.
lagg0: link state changed to UP
lagg0: IPv6 addresses on igb0 have been removed before adding it as a member to prevent IPv6 address scope violation.
ix1: link state changed to UP
lagg1: IPv6 addresses on ix1 have been removed before adding it as a member to prevent IPv6 address scope violation.
lagg1: link state changed to UP
lagg1: IPv6 addresses on igb1 have been removed before adding it as a member to prevent IPv6 address scope violation.
ix0: link state changed to DOWN
lagg0: link state changed to DOWN
ix1: link state changed to DOWN
lagg1: link state changed to DOWN
ix0: link state changed to UP
lagg0: link state changed to UP
ix1: link state changed to UP
lagg1: link state changed to UP
igb0: link state changed to UP
igb1: link state changed to UP

I'll perform some further tests over the weekend, see if failing over to the 1gbit igb interfaces or testing against a non-vlan-tagged interface makes any difference.  Let me know if any further info could be useful.

Kind regards,

Nick
Comment 1 Nick 2019-01-05 15:55:19 UTC
Hi,

I've done some additional testing:

-Failing over to the igb interfaces doesn't show any problems with TSO enabled or disabled, so this is probably specific to ix

-Dropping the MTU to 1500 has no effect

-Removing an ix interfaces from its lagg and configuring an untagged interface directly on it doesn't shows significantly but improved TX performance, however this is seen with TSO enabled and disabled.  I haven't managed to rule out the switch configuration as the cause of this yet, as I don't normally have any untagged ports.

I'll update in a bit when I've done some more testing with the untagged interface configuration.

Kind regards,

Nick
Comment 2 Nick 2019-01-05 16:00:24 UTC
(In reply to Nick from comment #1)

Sorry, I snafu'd that last comment.  Should have read:

-Removing an ix interface from its lagg and configuring an untagged interface directly on it shows significantly improved, but still poor, TX performance - however this is seen both with TSO enabled and disabled.  I haven't managed to rule out the switch configuration as the cause of this yet, as I don't normally have any untagged ports.
Comment 3 Nick 2019-01-05 19:46:57 UTC
Hi,

One further point of data: ipv6 testing shows the same results as ipv4, both for tagged and untagged interfaces.

I've also discovered something odd: once TSO has been disabled on an interface, or if it was disabled at boot, I'm unable to subsequently trigger the issue on a tagged interface.  Either turning TSO on after boot for some reason doesn't cause the problem, or my attempts to turn TSO on after boot are failing.  I'm not really sure how to check for definite, but I do seee that the tsx_tx counters don't increase from 0 in this case, so I suspect TSO hasn't really been re-enabled for some reason:

Booted with TSO disabled on ix0:

19:09:27 root@filer:/root/ sysctl dev.ix.0 | grep -i tso
dev.ix.0.queue3.tso_tx: 0
dev.ix.0.queue2.tso_tx: 0
dev.ix.0.queue1.tso_tx: 0
dev.ix.0.queue0.tso_tx: 0

Enable TSO:

19:15:11 root@filer:/root/ ifconfig ix0 tso

TX test shows good behaviour:

19:18:00 root@filer:/root/ iperf3 -c 10.21.10.4
Connecting to host 10.21.10.4, port 5201
[  5] local 10.21.10.2 port 54437 connected to 10.21.10.4 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.09 MBytes       
[  5]   1.00-2.00   sec  1.15 GBytes  9.88 Gbits/sec    0   1.09 MBytes       
[  5]   2.00-3.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.09 MBytes       
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.09 MBytes       
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.09 MBytes       
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.09 MBytes       
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec    0   1.09 MBytes       
[  5]   7.00-8.00   sec  1.15 GBytes  9.88 Gbits/sec    0   1.09 MBytes       
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.09 MBytes       
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.09 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

Re-check TSO counters, discover they haven't increased:

19:18:11 root@filer:/root/ sysctl dev.ix.0 | grep -i tso
dev.ix.0.queue3.tso_tx: 0
dev.ix.0.queue2.tso_tx: 0
dev.ix.0.queue1.tso_tx: 0
dev.ix.0.queue0.tso_tx: 0

I'm pretty much out of ideas as to how to proceeed with diagnosis at this point.  I'm happy to perform any further testing you may need, but disabling TSO seems to adequately solve the problem for me, so I'll just do that.  If you want to attempt to reproduce, I think our minimal config to reproduce is thus:

FreeBSD 12.0-Release (amd64) running with an X520-DA2 card installed, with one port in a failover lagg, configured with a vlan-tagged interface on top, with TSO enabled on ix<whichever> in rc.conf.

In case the platform is relevant, this machine is a Xeon E3-1245v3 (with HT disabled in BIOS), running on an ASRock C226WS with 4 x DDR3 ECC DIMMs installed.

Kind regards,

Nick