Comment 1 Zhenlei Huang 2024-07-23 08:20:58 UTC

(In reply to pascal.guitierrez from comment #0)
The if_bridge(4) promote underlaying errors from bridge members,

```
2108                 if ((err = dst_ifp->if_transmit(dst_ifp, m))) {
2109                         int n;
2110 
2111                         for (m = m0, n = 1; m != NULL; m = m0, n++) {
2112                                 m0 = m->m_nextpkt;
2113                                 m_freem(m);
2114                         }
2115                         if_inc_counter(sc->sc_ifp, IFCOUNTER_OERRORS, n);
2116                         break;
2117                 }
```

may you please also share the statistics for output dropped packets, aka `netstat -di` ?

(In reply to Zhenlei Huang from comment #1)

Thanks for your response.

Just ran the tests again.
There are no dropped packets detected even though the Oerr count is increasing, see below for netstat -di output:

Name      Mtu Network         Address                Ipkts Ierrs Idrop      Opkts Oerrs  Coll  Drop
igb0     1500 <Link#1>        5c:ed:8c:e9:c2:48   91053412     0     0   78158209     0     0     0 
igb1*    1500 <Link#2>        5c:ed:8c:e9:c2:49          0     0     0          0     0     0     0 
igb2*    1500 <Link#3>        5c:ed:8c:e9:c2:4a          0     0     0          0     0     0     0 
igb3*    1500 <Link#4>        5c:ed:8c:e9:c2:4b          0     0     0          0     0     0     0 
lo0     16384 <Link#5>        lo0                    62786     0     0      62786     0     0     0 
lo0         - localhost       localhost                  0     -     -          0     -     -     - 
lo0         - fe80::%lo0/64   fe80::1%lo0                0     -     -          0     -     -     - 
lo0         - your-net        localhost              62786     -     -      62786     -     -     - 
bridge0  1500 <Link#6>        58:9c:fc:00:07:00  168029633     0     0  168394069 269681     0     0 
bridge0     - 192.168.0.0/24  192.168.0.150         719881     -     -    1307016     -     -     - 
ue0      1500 <Link#7>        72:84:d1:bf:ad:2f       8907     0     0       8908     0     0     0 
ue0         - 16.1.15.0/30    16.1.15.2               8208     -     -       8208     -     -     -


Here is the relevant ifconfig:

igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 5c:ed:8c:e9:c2:48
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:00:07:00
        inet 192.168.0.150 netmask 0xffffff00 broadcast 192.168.0.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 55
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>

Comment 3 Zhenlei Huang 2024-07-25 04:34:07 UTC

(In reply to pascal.guitierrez from comment #2)
Emm, igb(4) has been converted to use iflib(4). It seems that the iflib implementation does not report statistics of output dropped packets ( IFCOUNTER_OQDROPS ).

May you please have a look at the driver statistics report from sysctl ? I do not have igb(4) devices but have em(4) which use the same driver e1000 with igb(4). For em(4) the statistics report for dropped packets are 
```
# sysctl dev.em.0.iflib | grep r_drops
dev.em.0.iflib.txq0.r_drops: 8402780
```

For igb(4) I guess they should be
```
# sysctl dev.igb.0.iflib | grep r_drops
```

(In reply to Zhenlei Huang from comment #3)

Here's the output from sysctl dev.igb.0.iflib | grep r_drops

dev.igb.0.iflib.txq3.r_drops: 37
dev.igb.0.iflib.txq2.r_drops: 37
dev.igb.0.iflib.txq1.r_drops: 0
dev.igb.0.iflib.txq0.r_drops: 4184

I just set up the same test the best I can (albeit a different IP address assigned to the bridge) and using an igb interface (Intel I350).

I can not reproduce the problem on 14.0-RELEASE-p6. I can try again with 14.1 later this week but I can not bring this system I used for testing offline until then.


After performing the same iperf3 test with the identical ifconfig parameters, connected to a bridge with the iperf client being the FreeBSD machine AND just for testing purposes the other way around:
root@hv-kryptos:~ # sysctl dev.igb.0.iflib | grep r_drops
dev.igb.0.iflib.txq7.r_drops: 0
dev.igb.0.iflib.txq6.r_drops: 0
dev.igb.0.iflib.txq5.r_drops: 0
dev.igb.0.iflib.txq4.r_drops: 0
dev.igb.0.iflib.txq3.r_drops: 0
dev.igb.0.iflib.txq2.r_drops: 0
dev.igb.0.iflib.txq1.r_drops: 0
dev.igb.0.iflib.txq0.r_drops: 0

I might need to push the system harder to get some error generation perhaps.

What happens if you disable checksum offloading and / or segmentation offloading?
Also, what sort of iperf stats are you getting? I am not quite reaching the full gigabit link rates on my interface, so I'd just like a reference in case this only happens when getting closer to the full link bandwidth.

It also looks like our dmesg output differs a bit (particularly the option roms are different):

igb0: <Intel(R) I350 (Copper)> port 0x6020-0x603f mem 0xc7a00000-0xc7afffff,0xc7c04000-0xc7c07fff irq 26 at device 0.0 numa-domain 0 on pci3
igb0: EEPROM V1.63-0 Option ROM V0-b385-p144 eTrack 0x80000e74
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 8 RX queues 8 TX queues
igb0: Using MSI-X interrupts with 9 vectors
igb0: Ethernet address: 70:df:2f:ae:58:94
igb0: netmap queues/slots: TX 8/1024, RX 8/1024

(In reply to fatalnix from comment #5)
As a fix for the previous message where I asked what happens if you disable offloading, I meant to say, "What happens if you do not disable offloading".

(In reply to fatalnix from comment #6)

it's my understanding that adding the device to the bridge switches off the NIC offloading features automatically.

i'm getting 942Mb/sec and interestingly the dropped packets are limited to a single txq, which changes between each test run:

dev.igb.0.iflib.txq3.r_drops: 188
dev.igb.0.iflib.txq2.r_drops: 130
dev.igb.0.iflib.txq1.r_drops: 69 <-- only one of these increments during each test 
dev.igb.0.iflib.txq0.r_drops: 4636

Comment 8 Mark Johnston 2024-07-26 17:53:04 UTC

(In reply to pascal.guitierrez from comment #7)
> i'm getting 942Mb/sec and interestingly the dropped packets are limited to a single txq, which changes between each test run:

If you're testing a single flow (i.e., the default for iperf3) then all packets would be steered to a single queue.  Otherwise you'd end up with packet reordering.  If you have multiple flows, then that's more surprising.

(In reply to Mark Johnston from comment #8)

Thanks Mark, i'm using iperf3 -s on the server and iperf3 -P4 -c 192.168.0.5 -t 60 on the client, which i believe does create multiple flows:

tcp4       0 1303796 192.168.0.150.53387    192.168.0.5.5201       ESTABLISHED
tcp4       0 1310084 192.168.0.150.17413    192.168.0.5.5201       ESTABLISHED
tcp4       0 1311532 192.168.0.150.56093    192.168.0.5.5201       ESTABLISHED
tcp4       0 1295148 192.168.0.150.54089    192.168.0.5.5201       ESTABLISHED


Here are the counters before the test run:
dev.igb.0.iflib.txq3.r_drops: 570
dev.igb.0.iflib.txq2.r_drops: 130
dev.igb.0.iflib.txq1.r_drops: 168
dev.igb.0.iflib.txq0.r_drops: 6250


Here are the counters after the 60 second test:
dev.igb.0.iflib.txq3.r_drops: 570
dev.igb.0.iflib.txq2.r_drops: 130
dev.igb.0.iflib.txq1.r_drops: 168
dev.igb.0.iflib.txq0.r_drops: 6360 <-- only this value increased


Is that expected behaviour?

I have upgraded to 14.1-release-p2 and I am now able to reproduce this issue using the same tests.

I did a few different tests, ensuring a solid 950~ Mbit/s link during my iperf3 tests. Each of them raised error when using the FreeBSD box as the iperf3 client.

For 60 seconds each as you did, I tested though with

iperf3 -P4:
bridge123  1500 <Link#18>        58:9c:fc:10:ff:92   624390     0     0  4886859    13     0

iperf -P8
bridge123  1500 <Link#18>        58:9c:fc:10:ff:92  1262380     0     0  9777020   187     0
bridge123     - 192.168.88.0/24  192.168.88.3       1216769     -     -  9777104     -     -

and iperf3 -P40, which is as many threads as I have physically:
bridge123  1500 <Link#18>        58:9c:fc:10:ff:92   2179910     0     0  14681352  1756     0
bridge123     - 192.168.88.0/24  192.168.88.3        2134298     -     -  14683001     -     -

The Oerrs field increases in error significantly, but if you notice, different P values do not seem to change the rate of errors generated either.

Checking with sysctl after:
# sysctl dev.igb.0.iflib | grep r_drops
dev.igb.0.iflib.txq7.r_drops: 94
dev.igb.0.iflib.txq6.r_drops: 291
dev.igb.0.iflib.txq5.r_drops: 52
dev.igb.0.iflib.txq4.r_drops: 356
dev.igb.0.iflib.txq3.r_drops: 232
dev.igb.0.iflib.txq2.r_drops: 252
dev.igb.0.iflib.txq1.r_drops: 311
dev.igb.0.iflib.txq0.r_drops: 157


This stayed at 0 yesterday when I was still running on 14.0 release p6.

(In reply to fatalnix from comment #10)

Ugh. Actually, it does increase with the thread increase. I was looking at the wrong number heheh. So, this might be sometghing that's load related somehow.

Could you try adding the following flags : -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso -vlanhwfilter

I use this :

ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso -vlanhwfilter"

I have a supermicro with an intel card with a hardware bug. This causes extremely erratic but consistent slow connections. Took me ages to troubleshoot. Had the same behavior under openindiana. Using the flags from above I would get consistent linespeed.

igb0@pci0:0:20:0:       class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1f41 subvendor=0x15d9 subdevice=0x1f41
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I354'
    class      = network
    subclass   = ethernet

(In reply to erwin.freebsd-bugzilla from comment #12)

thanks for your reply.

can you run the same iperf3 tests without and with a bridge involved? In my testing, there are no dropped packets when using the igb0 device, it's just when it's part of a bridge that the packet drops occur.

(In reply to pascal.guitierrez from comment #13)

.. and yes I repeated the tests with those flags and i'm still seeing packet drops

Comment 15 Kevin Bowling 2024-08-05 18:19:49 UTC

What does top -psh look like when running the test?  It is possible you are running out of CPU, especially if any filtering is enabled.

(In reply to Kevin Bowling from comment #15)

Thanks Kevin

no bridge filtering is enabled (sysctl net.link.bridge):

net.link.bridge.ipfw: 0
net.link.bridge.log_mac_flap: 1
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 1

and system does not appear to be CPU constrained, here's top -PSH below during test run using iperf -P4:

last pid: 25350;  load averages:  0.19,  0.25,  0.28                                                                                   up 4+19:09:14  11:32:05
321 threads:   5 running, 301 sleeping, 15 waiting
CPU 0:  0.0% user,  0.0% nice,  5.9% system,  0.0% interrupt, 94.1% idle
CPU 1:  0.0% user,  0.0% nice,  4.3% system,  0.0% interrupt, 95.7% idle
CPU 2:  0.4% user,  0.0% nice,  7.5% system,  0.0% interrupt, 92.1% idle
CPU 3:  0.0% user,  0.0% nice,  6.7% system,  0.0% interrupt, 93.3% idle
Mem: 16M Active, 24M Inact, 1744K Laundry, 6786M Wired, 8656M Free
ARC: 5248M Total, 854M MFU, 3834M MRU, 537M Header, 22M Other
     4644M Compressed, 6394M Uncompressed, 1.38:1 Ratio

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        187 ki31     0B    64K CPU0     0 108.9H  93.56% idle{idle: cpu0}
   11 root        187 ki31     0B    64K CPU1     1 107.8H  93.47% idle{idle: cpu1}
   11 root        187 ki31     0B    64K CPU3     3 108.0H  91.74% idle{idle: cpu3}
   11 root        187 ki31     0B    64K RUN      2 107.7H  91.54% idle{idle: cpu2}
25350 root         24    0    20M  7800K sbwait   3   0:00   7.80% iperf3{iperf3}
25350 root         23    0    20M  7800K sbwait   0   0:00   7.31% iperf3{iperf3}
    0 root        -60    -     0B  2096K -        1   1:20   5.95% kernel{if_io_tqg_1}
    0 root        -60    -     0B  2096K -        0   3:59   5.83% kernel{if_io_tqg_0}
25350 root         21    0    20M  7800K sbwait   3   0:00   1.29% iperf3{iperf3}
25350 root         21    0    20M  7800K sbwait   3   0:00   1.18% iperf3{iperf3}

drops are only seen when using if_bridge:

dev.igb.0.iflib.txq3.r_drops: 4594 <-- this value is increasing during the above test run
dev.igb.0.iflib.txq2.r_drops: 563
dev.igb.0.iflib.txq1.r_drops: 2551
dev.igb.0.iflib.txq0.r_drops: 2625

The value for "Oerrs" on the bridge0 device appears to increase in tandem with the r_drop value:

Name      Mtu Network         Address                Ipkts Ierrs Idrop      Opkts Oerrs  Coll  Drop
igb0     1500 <Link#1>        5c:ed:8c:e9:c2:48   81252823     0     0  131980027     0     0     0 
igb1*    1500 <Link#2>        5c:ed:8c:e9:c2:49          0     0     0          0     0     0     0 
igb2*    1500 <Link#3>        5c:ed:8c:e9:c2:4a          0     0     0          0     0     0     0 
igb3*    1500 <Link#4>        5c:ed:8c:e9:c2:4b          0     0     0          0     0     0     0 
lo0     16384 <Link#5>        lo0                    62021     0     0      62021     0     0     0 
lo0         - localhost       localhost                  0     -     -          0     -     -     - 
lo0         - fe80::%lo0/64   fe80::1%lo0                0     -     -          0     -     -     - 
lo0         - your-net        localhost              62021     -     -      62021     -     -     - 
bridge0  1500 <Link#6>        58:9c:fc:00:07:00  122336886     0     0  199687836 287872     0     0 
bridge0     - 192.168.0.0/24  192.168.0.150       13289878     -     -   90898935     -     -     -

Comment 17 Aleksandr Fedorov 2024-08-07 14:36:51 UTC

Can you show the value of sysctl dev.igb.0.fc?

And try your test with dev.igb.0.fc = 0

(In reply to Aleksandr Fedorov from comment #17)

currently at default of dev.igb.0.fc=3

changing it has no apparent effect.

however what i notice is that re-running the test results in some tests having 0 errors occur and others consistent errors, see below output from iperf3 with only 2 parallel tests (system has 4 cores):

this test had no errors (see the "Retr" field):

iperf3 -P2 -c 192.168.0.5 -t 600
Connecting to host 192.168.0.5, port 5201
[  5] local 192.168.0.150 port 47836 connected to 192.168.0.5 port 5201
[  7] local 192.168.0.150 port 12391 connected to 192.168.0.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  57.8 MBytes   484 Mbits/sec    0   2.00 MBytes       
[  7]   0.00-1.00   sec  57.9 MBytes   485 Mbits/sec    0   2.00 MBytes       
[SUM]   0.00-1.00   sec   116 MBytes   969 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  56.1 MBytes   471 Mbits/sec    0   2.00 MBytes       
[  7]   1.00-2.00   sec  56.0 MBytes   470 Mbits/sec    0   2.00 MBytes       
[SUM]   1.00-2.00   sec   112 MBytes   941 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  56.1 MBytes   471 Mbits/sec    0   2.00 MBytes       
[  7]   2.00-3.00   sec  56.2 MBytes   472 Mbits/sec    0   2.00 MBytes       
[SUM]   2.00-3.00   sec   112 MBytes   943 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0   2.00 MBytes       
[  7]   3.00-4.00   sec  56.0 MBytes   470 Mbits/sec    0   2.00 MBytes       
[SUM]   3.00-4.00   sec   112 MBytes   940 Mbits/sec    0             
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-4.55   sec  30.6 MBytes   469 Mbits/sec    0   2.00 MBytes       
[  7]   4.00-4.55   sec  30.8 MBytes   471 Mbits/sec    0   2.00 MBytes       
[SUM]   4.00-4.55   sec  61.4 MBytes   941 Mbits/sec    0             


then if run again, it may experience errors, and if it does, the errors are consistently seen for the rest of the test run:

iperf3 -P2 -c 192.168.0.5 -t 600
Connecting to host 192.168.0.5, port 5201
[  5] local 192.168.0.150 port 60974 connected to 192.168.0.5 port 5201
[  7] local 192.168.0.150 port 30555 connected to 192.168.0.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  48.5 MBytes   406 Mbits/sec    6   2.00 MBytes       
[  7]   0.00-1.00   sec  67.0 MBytes   560 Mbits/sec    3   2.00 MBytes       
[SUM]   0.00-1.00   sec   116 MBytes   966 Mbits/sec    9             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  52.5 MBytes   441 Mbits/sec   12   1.41 MBytes       
[  7]   1.00-2.00   sec  59.5 MBytes   500 Mbits/sec    4   1.44 MBytes       
[SUM]   1.00-2.00   sec   112 MBytes   941 Mbits/sec   16             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  58.5 MBytes   490 Mbits/sec    2   1011 KBytes       
[  7]   2.00-3.00   sec  54.2 MBytes   454 Mbits/sec    3    510 KBytes       
[SUM]   2.00-3.00   sec   113 MBytes   944 Mbits/sec    5             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  53.0 MBytes   445 Mbits/sec    3   2.00 MBytes       
[  7]   3.00-4.00   sec  59.2 MBytes   497 Mbits/sec    3    168 KBytes       
[SUM]   3.00-4.00   sec   112 MBytes   942 Mbits/sec    6             
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-4.64   sec  42.5 MBytes   559 Mbits/sec    2    338 KBytes       
[  7]   4.00-4.64   sec  29.0 MBytes   381 Mbits/sec    1   2.00 MBytes       
[SUM]   4.00-4.64   sec  71.5 MBytes   941 Mbits/sec    3             

the behaviour occurs independently of the dev.igb.0.fc value (3 or 0)

Comment 19 Kevin Bowling 2024-09-25 05:50:07 UTC

(In reply to pascal.guitierrez from comment #18)
if you watch 'systat -vmstat', does the zero drop case happen when all the igb interrupts are on one core, or two cores?  If you rerun the -P2 test several times you should see both cases as RSS hashing differs.