Comment 1 Zhenlei Huang 2024-07-23 08:20:58 UTC

(In reply to pascal.guitierrez from comment #0)
The if_bridge(4) promote underlaying errors from bridge members,

```
2108                 if ((err = dst_ifp->if_transmit(dst_ifp, m))) {
2109                         int n;
2110 
2111                         for (m = m0, n = 1; m != NULL; m = m0, n++) {
2112                                 m0 = m->m_nextpkt;
2113                                 m_freem(m);
2114                         }
2115                         if_inc_counter(sc->sc_ifp, IFCOUNTER_OERRORS, n);
2116                         break;
2117                 }
```

may you please also share the statistics for output dropped packets, aka `netstat -di` ?

(In reply to Zhenlei Huang from comment #1)

Thanks for your response.

Just ran the tests again.
There are no dropped packets detected even though the Oerr count is increasing, see below for netstat -di output:

Name      Mtu Network         Address                Ipkts Ierrs Idrop      Opkts Oerrs  Coll  Drop
igb0     1500 <Link#1>        5c:ed:8c:e9:c2:48   91053412     0     0   78158209     0     0     0 
igb1*    1500 <Link#2>        5c:ed:8c:e9:c2:49          0     0     0          0     0     0     0 
igb2*    1500 <Link#3>        5c:ed:8c:e9:c2:4a          0     0     0          0     0     0     0 
igb3*    1500 <Link#4>        5c:ed:8c:e9:c2:4b          0     0     0          0     0     0     0 
lo0     16384 <Link#5>        lo0                    62786     0     0      62786     0     0     0 
lo0         - localhost       localhost                  0     -     -          0     -     -     - 
lo0         - fe80::%lo0/64   fe80::1%lo0                0     -     -          0     -     -     - 
lo0         - your-net        localhost              62786     -     -      62786     -     -     - 
bridge0  1500 <Link#6>        58:9c:fc:00:07:00  168029633     0     0  168394069 269681     0     0 
bridge0     - 192.168.0.0/24  192.168.0.150         719881     -     -    1307016     -     -     - 
ue0      1500 <Link#7>        72:84:d1:bf:ad:2f       8907     0     0       8908     0     0     0 
ue0         - 16.1.15.0/30    16.1.15.2               8208     -     -       8208     -     -     -


Here is the relevant ifconfig:

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 5c:ed:8c:e9:c2:48
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:00:07:00
        inet 192.168.0.150 netmask 0xffffff00 broadcast 192.168.0.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>

Comment 3 Zhenlei Huang 2024-07-25 04:34:07 UTC

(In reply to pascal.guitierrez from comment #2)
Emm, igb(4) has been converted to use iflib(4). It seems that the iflib implementation does not report statistics of output dropped packets ( IFCOUNTER_OQDROPS ).

May you please have a look at the driver statistics report from sysctl ? I do not have igb(4) devices but have em(4) which use the same driver e1000 with igb(4). For em(4) the statistics report for dropped packets are 
```
# sysctl dev.em.0.iflib | grep r_drops
dev.em.0.iflib.txq0.r_drops: 8402780
```

For igb(4) I guess they should be
```
# sysctl dev.igb.0.iflib | grep r_drops
```

(In reply to Zhenlei Huang from comment #3)

Here's the output from sysctl dev.igb.0.iflib | grep r_drops

dev.igb.0.iflib.txq3.r_drops: 37
dev.igb.0.iflib.txq2.r_drops: 37
dev.igb.0.iflib.txq1.r_drops: 0
dev.igb.0.iflib.txq0.r_drops: 4184

I just set up the same test the best I can (albeit a different IP address assigned to the bridge) and using an igb interface (Intel I350).

I can not reproduce the problem on 14.0-RELEASE-p6. I can try again with 14.1 later this week but I can not bring this system I used for testing offline until then.


After performing the same iperf3 test with the identical ifconfig parameters, connected to a bridge with the iperf client being the FreeBSD machine AND just for testing purposes the other way around:
root@hv-kryptos:~ # sysctl dev.igb.0.iflib | grep r_drops
dev.igb.0.iflib.txq7.r_drops: 0
dev.igb.0.iflib.txq6.r_drops: 0
dev.igb.0.iflib.txq5.r_drops: 0
dev.igb.0.iflib.txq4.r_drops: 0
dev.igb.0.iflib.txq3.r_drops: 0
dev.igb.0.iflib.txq2.r_drops: 0
dev.igb.0.iflib.txq1.r_drops: 0
dev.igb.0.iflib.txq0.r_drops: 0

I might need to push the system harder to get some error generation perhaps.

What happens if you disable checksum offloading and / or segmentation offloading?
Also, what sort of iperf stats are you getting? I am not quite reaching the full gigabit link rates on my interface, so I'd just like a reference in case this only happens when getting closer to the full link bandwidth.

It also looks like our dmesg output differs a bit (particularly the option roms are different):

igb0: <Intel(R) I350 (Copper)> port 0x6020-0x603f mem 0xc7a00000-0xc7afffff,0xc7c04000-0xc7c07fff irq 26 at device 0.0 numa-domain 0 on pci3
igb0: EEPROM V1.63-0 Option ROM V0-b385-p144 eTrack 0x80000e74
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 8 RX queues 8 TX queues
igb0: Using MSI-X interrupts with 9 vectors
igb0: Ethernet address: 70:df:2f:ae:58:94
igb0: netmap queues/slots: TX 8/1024, RX 8/1024

(In reply to fatalnix from comment #5)
As a fix for the previous message where I asked what happens if you disable offloading, I meant to say, "What happens if you do not disable offloading".

(In reply to fatalnix from comment #6)

it's my understanding that adding the device to the bridge switches off the NIC offloading features automatically.

i'm getting 942Mb/sec and interestingly the dropped packets are limited to a single txq, which changes between each test run:

dev.igb.0.iflib.txq3.r_drops: 188
dev.igb.0.iflib.txq2.r_drops: 130
dev.igb.0.iflib.txq1.r_drops: 69 <-- only one of these increments during each test 
dev.igb.0.iflib.txq0.r_drops: 4636

Comment 8 Mark Johnston 2024-07-26 17:53:04 UTC

(In reply to pascal.guitierrez from comment #7)
> i'm getting 942Mb/sec and interestingly the dropped packets are limited to a single txq, which changes between each test run:

If you're testing a single flow (i.e., the default for iperf3) then all packets would be steered to a single queue.  Otherwise you'd end up with packet reordering.  If you have multiple flows, then that's more surprising.

(In reply to Mark Johnston from comment #8)

Thanks Mark, i'm using iperf3 -s on the server and iperf3 -P4 -c 192.168.0.5 -t 60 on the client, which i believe does create multiple flows:

tcp4       0 1303796 192.168.0.150.53387    192.168.0.5.5201       ESTABLISHED
tcp4       0 1310084 192.168.0.150.17413    192.168.0.5.5201       ESTABLISHED
tcp4       0 1311532 192.168.0.150.56093    192.168.0.5.5201       ESTABLISHED
tcp4       0 1295148 192.168.0.150.54089    192.168.0.5.5201       ESTABLISHED


Here are the counters before the test run:
dev.igb.0.iflib.txq3.r_drops: 570
dev.igb.0.iflib.txq2.r_drops: 130
dev.igb.0.iflib.txq1.r_drops: 168
dev.igb.0.iflib.txq0.r_drops: 6250


Here are the counters after the 60 second test:
dev.igb.0.iflib.txq3.r_drops: 570
dev.igb.0.iflib.txq2.r_drops: 130
dev.igb.0.iflib.txq1.r_drops: 168
dev.igb.0.iflib.txq0.r_drops: 6360 <-- only this value increased


Is that expected behaviour?

I have upgraded to 14.1-release-p2 and I am now able to reproduce this issue using the same tests.

I did a few different tests, ensuring a solid 950~ Mbit/s link during my iperf3 tests. Each of them raised error when using the FreeBSD box as the iperf3 client.

For 60 seconds each as you did, I tested though with

iperf3 -P4:
bridge123  1500 <Link#18>        58:9c:fc:10:ff:92   624390     0     0  4886859    13     0

iperf -P8
bridge123  1500 <Link#18>        58:9c:fc:10:ff:92  1262380     0     0  9777020   187     0
bridge123     - 192.168.88.0/24  192.168.88.3       1216769     -     -  9777104     -     -

and iperf3 -P40, which is as many threads as I have physically:
bridge123  1500 <Link#18>        58:9c:fc:10:ff:92   2179910     0     0  14681352  1756     0
bridge123     - 192.168.88.0/24  192.168.88.3        2134298     -     -  14683001     -     -

The Oerrs field increases in error significantly, but if you notice, different P values do not seem to change the rate of errors generated either.

Checking with sysctl after:
# sysctl dev.igb.0.iflib | grep r_drops
dev.igb.0.iflib.txq7.r_drops: 94
dev.igb.0.iflib.txq6.r_drops: 291
dev.igb.0.iflib.txq5.r_drops: 52
dev.igb.0.iflib.txq4.r_drops: 356
dev.igb.0.iflib.txq3.r_drops: 232
dev.igb.0.iflib.txq2.r_drops: 252
dev.igb.0.iflib.txq1.r_drops: 311
dev.igb.0.iflib.txq0.r_drops: 157


This stayed at 0 yesterday when I was still running on 14.0 release p6.

(In reply to fatalnix from comment #10)

Ugh. Actually, it does increase with the thread increase. I was looking at the wrong number heheh. So, this might be sometghing that's load related somehow.

Could you try adding the following flags : -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso -vlanhwfilter

I use this :

ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso -vlanhwfilter"

I have a supermicro with an intel card with a hardware bug. This causes extremely erratic but consistent slow connections. Took me ages to troubleshoot. Had the same behavior under openindiana. Using the flags from above I would get consistent linespeed.

igb0@pci0:0:20:0:       class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1f41 subvendor=0x15d9 subdevice=0x1f41
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I354'
    class      = network
    subclass   = ethernet

(In reply to erwin.freebsd-bugzilla from comment #12)

thanks for your reply.

can you run the same iperf3 tests without and with a bridge involved? In my testing, there are no dropped packets when using the igb0 device, it's just when it's part of a bridge that the packet drops occur.

(In reply to pascal.guitierrez from comment #13)

.. and yes I repeated the tests with those flags and i'm still seeing packet drops

Comment 15 Kevin Bowling 2024-08-05 18:19:49 UTC

What does top -psh look like when running the test?  It is possible you are running out of CPU, especially if any filtering is enabled.

(In reply to Kevin Bowling from comment #15)

Thanks Kevin

no bridge filtering is enabled (sysctl net.link.bridge):

net.link.bridge.ipfw: 0
net.link.bridge.log_mac_flap: 1
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 1

and system does not appear to be CPU constrained, here's top -PSH below during test run using iperf -P4:

last pid: 25350;  load averages:  0.19,  0.25,  0.28                                                                                   up 4+19:09:14  11:32:05
321 threads:   5 running, 301 sleeping, 15 waiting
CPU 0:  0.0% user,  0.0% nice,  5.9% system,  0.0% interrupt, 94.1% idle
CPU 1:  0.0% user,  0.0% nice,  4.3% system,  0.0% interrupt, 95.7% idle
CPU 2:  0.4% user,  0.0% nice,  7.5% system,  0.0% interrupt, 92.1% idle
CPU 3:  0.0% user,  0.0% nice,  6.7% system,  0.0% interrupt, 93.3% idle
Mem: 16M Active, 24M Inact, 1744K Laundry, 6786M Wired, 8656M Free
ARC: 5248M Total, 854M MFU, 3834M MRU, 537M Header, 22M Other
     4644M Compressed, 6394M Uncompressed, 1.38:1 Ratio

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        187 ki31     0B    64K CPU0     0 108.9H  93.56% idle{idle: cpu0}
   11 root        187 ki31     0B    64K CPU1     1 107.8H  93.47% idle{idle: cpu1}
   11 root        187 ki31     0B    64K CPU3     3 108.0H  91.74% idle{idle: cpu3}
   11 root        187 ki31     0B    64K RUN      2 107.7H  91.54% idle{idle: cpu2}
25350 root         24    0    20M  7800K sbwait   3   0:00   7.80% iperf3{iperf3}
25350 root         23    0    20M  7800K sbwait   0   0:00   7.31% iperf3{iperf3}
    0 root        -60    -     0B  2096K -        1   1:20   5.95% kernel{if_io_tqg_1}
    0 root        -60    -     0B  2096K -        0   3:59   5.83% kernel{if_io_tqg_0}
25350 root         21    0    20M  7800K sbwait   3   0:00   1.29% iperf3{iperf3}
25350 root         21    0    20M  7800K sbwait   3   0:00   1.18% iperf3{iperf3}

drops are only seen when using if_bridge:

dev.igb.0.iflib.txq3.r_drops: 4594 <-- this value is increasing during the above test run
dev.igb.0.iflib.txq2.r_drops: 563
dev.igb.0.iflib.txq1.r_drops: 2551
dev.igb.0.iflib.txq0.r_drops: 2625

The value for "Oerrs" on the bridge0 device appears to increase in tandem with the r_drop value:

Name      Mtu Network         Address                Ipkts Ierrs Idrop      Opkts Oerrs  Coll  Drop
igb0     1500 <Link#1>        5c:ed:8c:e9:c2:48   81252823     0     0  131980027     0     0     0 
igb1*    1500 <Link#2>        5c:ed:8c:e9:c2:49          0     0     0          0     0     0     0 
igb2*    1500 <Link#3>        5c:ed:8c:e9:c2:4a          0     0     0          0     0     0     0 
igb3*    1500 <Link#4>        5c:ed:8c:e9:c2:4b          0     0     0          0     0     0     0 
lo0     16384 <Link#5>        lo0                    62021     0     0      62021     0     0     0 
lo0         - localhost       localhost                  0     -     -          0     -     -     - 
lo0         - fe80::%lo0/64   fe80::1%lo0                0     -     -          0     -     -     - 
lo0         - your-net        localhost              62021     -     -      62021     -     -     - 
bridge0  1500 <Link#6>        58:9c:fc:00:07:00  122336886     0     0  199687836 287872     0     0 
bridge0     - 192.168.0.0/24  192.168.0.150       13289878     -     -   90898935     -     -     -

Comment 17 Aleksandr Fedorov 2024-08-07 14:36:51 UTC

Can you show the value of sysctl dev.igb.0.fc?

And try your test with dev.igb.0.fc = 0

(In reply to Aleksandr Fedorov from comment #17)

currently at default of dev.igb.0.fc=3

changing it has no apparent effect.

however what i notice is that re-running the test results in some tests having 0 errors occur and others consistent errors, see below output from iperf3 with only 2 parallel tests (system has 4 cores):

this test had no errors (see the "Retr" field):

iperf3 -P2 -c 192.168.0.5 -t 600
Connecting to host 192.168.0.5, port 5201
[  5] local 192.168.0.150 port 47836 connected to 192.168.0.5 port 5201
[  7] local 192.168.0.150 port 12391 connected to 192.168.0.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  57.8 MBytes   484 Mbits/sec    0   2.00 MBytes       
[  7]   0.00-1.00   sec  57.9 MBytes   485 Mbits/sec    0   2.00 MBytes       
[SUM]   0.00-1.00   sec   116 MBytes   969 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  56.1 MBytes   471 Mbits/sec    0   2.00 MBytes       
[  7]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec    0   2.00 MBytes       
[SUM]   1.00-2.00   sec   112 MBytes   941 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  56.1 MBytes   471 Mbits/sec    0   2.00 MBytes       
[  7]   2.00-3.00   sec  56.2 MBytes   472 Mbits/sec    0   2.00 MBytes       
[SUM]   2.00-3.00   sec   112 MBytes   943 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0   2.00 MBytes       
[  7]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0   2.00 MBytes       
[SUM]   3.00-4.00   sec   112 MBytes   940 Mbits/sec    0             
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-4.55   sec  30.6 MBytes   469 Mbits/sec    0   2.00 MBytes       
[  7]   4.00-4.55   sec  30.8 MBytes   471 Mbits/sec    0   2.00 MBytes       
[SUM]   4.00-4.55   sec  61.4 MBytes   941 Mbits/sec    0             


then if run again, it may experience errors, and if it does, the errors are consistently seen for the rest of the test run:

iperf3 -P2 -c 192.168.0.5 -t 600
Connecting to host 192.168.0.5, port 5201
[  5] local 192.168.0.150 port 60974 connected to 192.168.0.5 port 5201
[  7] local 192.168.0.150 port 30555 connected to 192.168.0.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  48.5 MBytes   406 Mbits/sec    6   2.00 MBytes       
[  7]   0.00-1.00   sec  67.0 MBytes   560 Mbits/sec    3   2.00 MBytes       
[SUM]   0.00-1.00   sec   116 MBytes   966 Mbits/sec    9             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  52.5 MBytes   441 Mbits/sec   12   1.41 MBytes       
[  7]   1.00-2.00   sec  59.5 MBytes   500 Mbits/sec    4   1.44 MBytes       
[SUM]   1.00-2.00   sec   112 MBytes   941 Mbits/sec   16             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  58.5 MBytes   490 Mbits/sec    2   1011 KBytes       
[  7]   2.00-3.00   sec  54.2 MBytes   454 Mbits/sec    3    510 KBytes       
[SUM]   2.00-3.00   sec   113 MBytes   944 Mbits/sec    5             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  53.0 MBytes   445 Mbits/sec    3   2.00 MBytes       
[  7]   3.00-4.00   sec  59.2 MBytes   497 Mbits/sec    3    168 KBytes       
[SUM]   3.00-4.00   sec   112 MBytes   942 Mbits/sec    6             
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-4.64   sec  42.5 MBytes   559 Mbits/sec    2    338 KBytes       
[  7]   4.00-4.64   sec  29.0 MBytes   381 Mbits/sec    1   2.00 MBytes       
[SUM]   4.00-4.64   sec  71.5 MBytes   941 Mbits/sec    3             

the behaviour occurs independently of the dev.igb.0.fc value (3 or 0)

Comment 19 Kevin Bowling 2024-09-25 05:50:07 UTC

(In reply to pascal.guitierrez from comment #18)
if you watch 'systat -vmstat', does the zero drop case happen when all the igb interrupts are on one core, or two cores?  If you rerun the -P2 test several times you should see both cases as RSS hashing differs.

(In reply to Kevin Bowling from comment #19)


Thanks Kevin, yes you are right, repeating the test gets to a state where interrupts are processed on a single core and packet drops occur:

  13 igb0:rxq0
7972 igb0:rxq1
  97 igb0:rxq2
   1 igb0:rxq3

On a good run there are zero drops and interrupts being processed on two cores:

8041 igb0:rxq0
   1 igb0:rxq1
8038 igb0:rxq2
     igb0:rxq3


Do you know if this is expected behaviour or is it a problem?

Comment 21 Kevin Bowling 2024-10-21 19:59:37 UTC

(In reply to pascal.guitierrez from comment #20)
> Do you know if this is expected behaviour or is it a problem?
Caveat it has been a long time since I have been line-rating large numbers of FreeBSD systems so I may not be privy to all the developments in the network stack.. but I would say more /known/ than /expected/.  With the if_bridge in play maybe you are more likely to see this since that may imply TSO was masked off and you have a more expensive transmit path.

I think https://reviews.freebsd.org/D4295 would be one way to fix the issue but I am not sure it is the only or best way.

In principle there isn't a problem because dropping packets is an expected behavior of overfull queues at any point of the network and TCP or other protocols should be dealing with it.  But it is obviously suboptimal if we have enough context to do something smarter.

Comment 22 Kevin Bowling 2024-10-21 20:10:25 UTC

(In reply to Kevin Bowling from comment #21)
Thinking back a bit harder one thing that might help workaround this is to increase the number of transmit descriptors.. try something like this in /boot/loader.conf: dev.igb.0.iflib.override_ntxqs=4096 and let us know how it does.

(In reply to Kevin Bowling from comment #21)
(In reply to Kevin Bowling from comment #22)

Interesting, thanks for that.

I tried changing dev.igb.0.iflib.override_ntxqs=4096 but to no effect.

What did make a difference was to use the rack tcp stack net.inet.tcp.functions_default=rack

using rack there was next to no packet loss when interrupts were scheduled onto a single core, obviously not fixing the root cause but is somewhat of a workaround.

Comment 24 Kevin Bowling 2024-10-22 17:57:03 UTC

(In reply to pascal.guitierrez from comment #23)
Thanks for reporting back.  That is very interesting.

There are two possibilities that come to mind 1) the RACK stack is correctly identifying the loss as feedback for it's flow/congestion control or 2) the RACK stack has some feature that is causing it to not overflow the lower layers.

I'll see if I can find someone with more recent TCP stack experience to glance at this and provide any additional insight or suggestions for the base stack.

Comment 25 Cheng Cui 2024-10-22 20:00:53 UTC

(In reply to Kevin Bowling from comment #22)
(In reply to pascal.guitierrez from comment #23)

I had a similar experience last year when I was debugging a ENOBUFS error returned to TCP on using bce NICs. But I am not sure if you can find a similar solution.

<snap from 2023>
Turns out the root cause is the default NIC send queue length is too small. The enobufs error came from the _IF_QFULL check in ifq.h.

However, tuning "sysctl net.link.ifqmaxlen" directly does not work. There is a per NIC interface setup in the driver to setup device tx/rx queues. I have to increase the tx queue "ifq_maxlen" from the device sysctl "hw.bce.tx_pages". After tuning that, I can achieve a stable 1Gbps x 100ms delay BDP.
</snap from 2023>

Talking about review D4295, it reminds me of `Linux has some work like TCP small queue at the sender side.`

Talking about workaround, you may also test the following two patches I prepared in stable/14 branch as a workaround of improving TCP performance in congestion control.

https://reviews.freebsd.org/D47218     << apply this patch firstly
https://reviews.freebsd.org/D47213     << apply this patch secondly

Comment 26 Andrew Gallatin 2024-10-23 00:29:22 UTC

(In reply to Kevin Bowling from comment #24)

Indeed, when it gets an ENOBUFS back, RACK will start pacing the connection. Search for this comment:

                /*
                 * Failures do not advance the seq counter above. For the
                 * case of ENOBUFS we will fall out and retry in 1ms with
                 * the hpts. Everything else will just have to retransmit
                 * with the timer.
                 *
                 * In any case, we do not want to loop around for another
                 * send without a good reason.
                 */