Bug 263229 - virtio vtnet driver with rxcsum errors
Summary: virtio vtnet driver with rxcsum errors
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.1-STABLE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-04-11 22:58 UTC by mike
Modified: 2025-04-26 20:09 UTC (History)
5 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description mike 2022-04-11 22:58:02 UTC 
I was setting up a VM pf firewall and noticed I was not able to nat out for some reason. Looking at the pcap, it seems when the vm is in forwarding mode, I get tcp checksum errors. If I do a

ifconfig vtnet1 -rxcsum

ifconfig vtnet0 -rxcsum

nat then seems to work fine

The setup is a simple VM with the hypervisor libvirt/KVM ubuntu 20 LTS.  Guest is RELENG_13 from Apr 11/2022



e.g. vtnet1 is facing another internal VM. If I try and connect to the internal NIC vtnet1 (192.168.199.7) I dont get any errors. But if I try and get out via nat, it fails. If I switch both nics to Intel em NICs, it works without modification
public_internet <---> vtnet0-vtnet1 <---> another guest vm

vtnet0 = public IP
vtnet1 = 192.168.199.7/24
guest = 192.168.199.100


 tcpdump -s0 -vnei vtnet1 port 443
tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:35:05.937364 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 64547, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.100.56996 > 172.217.1.4.443: Flags [S], cksum 0x3619 (incorrect -> 0x4755), seq 2994289493, win 64240, options [mss 1460,sackOK,TS val 1606656477 ecr 0,nop,wscale 7], length 0
18:35:06.939305 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 64548, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.100.56996 > 172.217.1.4.443: Flags [S], cksum 0x3619 (incorrect -> 0x436b), seq 2994289493, win 64240, options [mss 1460,sackOK,TS val 1606657479 ecr 0,nop,wscale 7], length 0
18:35:21.040936 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 45298, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.100.59876 > 192.168.199.7.443: Flags [S], cksum 0x0fec (incorrect -> 0xf8f6), seq 1122263205, win 64240, options [mss 1460,sackOK,TS val 2124691085 ecr 0,nop,wscale 7], length 0
18:35:21.040993 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.7.443 > 192.168.199.100.59876: Flags [S.], cksum 0x0fec (incorrect -> 0x5be5), seq 3948792593, ack 1122263206, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2270714410 ecr 2124691085], length 0
18:35:21.041224 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 45299, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.100.59876 > 192.168.199.7.443: Flags [.], cksum 0x0fe4 (incorrect -> 0x88b9), ack 1, win 502, options [nop,nop,TS val 2124691086 ecr 2270714410], length 0
18:35:22.843871 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 45300, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.100.59876 > 192.168.199.7.443: Flags [F.], cksum 0x0fe4 (incorrect -> 0x81ae), seq 1, ack 1, win 502, options [nop,nop,TS val 2124692888 ecr 2270714410], length 0
18:35:22.843910 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.7.443 > 192.168.199.100.59876: Flags [.], cksum 0x0fe4 (incorrect -> 0x7895), ack 2, win 1027, options [nop,nop,TS val 2270716214 ecr 2124692888], length 0
18:35:22.843980 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.7.443 > 192.168.199.100.59876: Flags [F.], cksum 0x0fe4 (incorrect -> 0x7894), seq 1, ack 2, win 1027, options [nop,nop,TS val 2270716214 ecr 2124692888], length 0
18:35:22.844159 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.100.59876 > 192.168.199.7.443: Flags [.], cksum 0x7aa0 (correct), ack 2, win 502, options [nop,nop,TS val 2124692889 ecr 2270716214], length 0


e.g. after disabling rxcsum (ifconfig vtnet1 -rxcsum ifconfig vtnet0 -rxcsum)

tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:39:57.425940 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 53398, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.101.55810 > 142.251.32.68.443: Flags [S], cksum 0x5467 (correct), seq 2136333436, win 64240, options [mss 1460,sackOK,TS val 2746738194 ecr 0,nop,wscale 7], length 0
18:39:57.432020 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 123, id 58394, offset 0, flags [none], proto TCP (6), length 60)
    142.251.32.68.443 > 192.168.199.101.55810: Flags [S.], cksum 0xa929 (correct), seq 292800283, ack 2136333437, win 65535, options [mss 1430,sackOK,TS val 460829236 ecr 2746738194,nop,wscale 8], length 0
18:39:57.432265 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 53399, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.101.55810 > 142.251.32.68.443: Flags [.], cksum 0xd5dc (correct), ack 1, win 502, options [nop,nop,TS val 2746738200 ecr 460829236], length 0
18:39:58.605990 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 53400, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.101.55810 > 142.251.32.68.443: Flags [F.], cksum 0xd145 (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 2746739374 ecr 460829236], length 0
18:39:58.612128 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 123, id 58984, offset 0, flags [none], proto TCP (6), length 52)
    142.251.32.68.443 > 192.168.199.101.55810: Flags [F.], cksum 0xcd9e (correct), seq 1, ack 2, win 256, options [nop,nop,TS val 460830416 ecr 2746739374], length 0
18:39:58.612402 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 53401, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.101.55810 > 142.251.32.68.443: Flags [.], cksum 0xcca2 (correct), ack 2, win 502, options [nop,nop,TS val 2746739380 ecr 460830416], length 0


I am guessing txcsum is fine as I can connect out from the FreeBSD VM, so packets originating on the firewall's vtnet0 interface is ok.
Comment 1 mike 2022-04-11 22:58:58 UTC 
Actually this looks related to this long standing bug :( 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059
Comment 2 Danilo Egea Gondolfo freebsd_committer freebsd_triage 2025-02-27 11:21:07 UTC 
I'm having the same issue on FreeBSD 15 running on LXD (qemu/kvm).

In my case DHCP fails with the error below. Disabling RXCSUM makes it work immediately.

vtnet0: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether 00:16:3e:e2:a7:60
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@:/mnt/root # dhclient vtnet0
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 4
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 6
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 9
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 9
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 13
5 bad udp checksums in 5 packets
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 13
^C
root@:/mnt/root # ifconfig vtnet0 -rxcsum
root@:/mnt/root # dhclient vtnet0
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 3
DHCPOFFER from 10.86.126.1
DHCPREQUEST on vtnet0 to 255.255.255.255 port 67
DHCPACK from 10.86.126.1
bound to 10.86.126.96 -- renewal in 1800 seconds.