I was setting up a VM pf firewall and noticed I was not able to nat out for some reason. Looking at the pcap, it seems when the vm is in forwarding mode, I get tcp checksum errors. If I do a

ifconfig vtnet1 -rxcsum

ifconfig vtnet0 -rxcsum

nat then seems to work fine

The setup is a simple VM with the hypervisor libvirt/KVM ubuntu 20 LTS.  Guest is RELENG_13 from Apr 11/2022



e.g. vtnet1 is facing another internal VM. If I try and connect to the internal NIC vtnet1 (192.168.199.7) I dont get any errors. But if I try and get out via nat, it fails. If I switch both nics to Intel em NICs, it works without modification
public_internet <---> vtnet0-vtnet1 <---> another guest vm

vtnet0 = public IP
vtnet1 = 192.168.199.7/24
guest = 192.168.199.100


 tcpdump -s0 -vnei vtnet1 port 443
tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:35:05.937364 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 64547, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.100.56996 > 172.217.1.4.443: Flags [S], cksum 0x3619 (incorrect -> 0x4755), seq 2994289493, win 64240, options [mss 1460,sackOK,TS val 1606656477 ecr 0,nop,wscale 7], length 0
18:35:06.939305 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 64548, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.100.56996 > 172.217.1.4.443: Flags [S], cksum 0x3619 (incorrect -> 0x436b), seq 2994289493, win 64240, options [mss 1460,sackOK,TS val 1606657479 ecr 0,nop,wscale 7], length 0
18:35:21.040936 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 45298, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.100.59876 > 192.168.199.7.443: Flags [S], cksum 0x0fec (incorrect -> 0xf8f6), seq 1122263205, win 64240, options [mss 1460,sackOK,TS val 2124691085 ecr 0,nop,wscale 7], length 0
18:35:21.040993 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.7.443 > 192.168.199.100.59876: Flags [S.], cksum 0x0fec (incorrect -> 0x5be5), seq 3948792593, ack 1122263206, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2270714410 ecr 2124691085], length 0
18:35:21.041224 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 45299, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.100.59876 > 192.168.199.7.443: Flags [.], cksum 0x0fe4 (incorrect -> 0x88b9), ack 1, win 502, options [nop,nop,TS val 2124691086 ecr 2270714410], length 0
18:35:22.843871 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 45300, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.100.59876 > 192.168.199.7.443: Flags [F.], cksum 0x0fe4 (incorrect -> 0x81ae), seq 1, ack 1, win 502, options [nop,nop,TS val 2124692888 ecr 2270714410], length 0
18:35:22.843910 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.7.443 > 192.168.199.100.59876: Flags [.], cksum 0x0fe4 (incorrect -> 0x7895), ack 2, win 1027, options [nop,nop,TS val 2270716214 ecr 2124692888], length 0
18:35:22.843980 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.7.443 > 192.168.199.100.59876: Flags [F.], cksum 0x0fe4 (incorrect -> 0x7894), seq 1, ack 2, win 1027, options [nop,nop,TS val 2270716214 ecr 2124692888], length 0
18:35:22.844159 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.100.59876 > 192.168.199.7.443: Flags [.], cksum 0x7aa0 (correct), ack 2, win 502, options [nop,nop,TS val 2124692889 ecr 2270716214], length 0


e.g. after disabling rxcsum (ifconfig vtnet1 -rxcsum ifconfig vtnet0 -rxcsum)

tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:39:57.425940 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 53398, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.199.101.55810 > 142.251.32.68.443: Flags [S], cksum 0x5467 (correct), seq 2136333436, win 64240, options [mss 1460,sackOK,TS val 2746738194 ecr 0,nop,wscale 7], length 0
18:39:57.432020 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 123, id 58394, offset 0, flags [none], proto TCP (6), length 60)
    142.251.32.68.443 > 192.168.199.101.55810: Flags [S.], cksum 0xa929 (correct), seq 292800283, ack 2136333437, win 65535, options [mss 1430,sackOK,TS val 460829236 ecr 2746738194,nop,wscale 8], length 0
18:39:57.432265 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 53399, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.101.55810 > 142.251.32.68.443: Flags [.], cksum 0xd5dc (correct), ack 1, win 502, options [nop,nop,TS val 2746738200 ecr 460829236], length 0
18:39:58.605990 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 53400, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.101.55810 > 142.251.32.68.443: Flags [F.], cksum 0xd145 (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 2746739374 ecr 460829236], length 0
18:39:58.612128 52:54:00:09:6e:82 > 52:54:00:b6:ae:7b, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 123, id 58984, offset 0, flags [none], proto TCP (6), length 52)
    142.251.32.68.443 > 192.168.199.101.55810: Flags [F.], cksum 0xcd9e (correct), seq 1, ack 2, win 256, options [nop,nop,TS val 460830416 ecr 2746739374], length 0
18:39:58.612402 52:54:00:b6:ae:7b > 52:54:00:09:6e:82, ethertype IPv4 (0x0800), length 66: (tos 0x10, ttl 64, id 53401, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.199.101.55810 > 142.251.32.68.443: Flags [.], cksum 0xcca2 (correct), ack 2, win 502, options [nop,nop,TS val 2746739380 ecr 460830416], length 0


I am guessing txcsum is fine as I can connect out from the FreeBSD VM, so packets originating on the firewall's vtnet0 interface is ok.