I use a raspberry 3 as DSL router. builtin interface for lan and external usb interface for pppoe. all works fine, ipv4 and ipv6. The exact same setup with a raspberry pi 4 does not work for ipv6. I use dhcpcd to setup the ipv6 addresses. I captured a telnet connection to www.kame.org on both interfaces: freebsd@raspirouter:~ % sudo tcpdump -vvv -s0 -n -i intern0 host 2001:200:dff:fff1:216:3eff:feb1:44d7 tcpdump: listening on intern0, link-type EN10MB (Ethernet), capture size 262144 bytes 11:30:44.013698 IP6 (flowlabel 0xb0700, hlim 64, next-header TCP (6) payload length: 44) 2003:c1:df10:9801:3cd9:d041:c580:584b.63658 > 2001:200:dff:fff1:216:3eff:feb1:44d7.80: Flags [SEW], cksum 0x3f10 (correct), seq 3119817516, win 65535, options [mss 1440,nop,wscale 6,nop,nop,TS val 874444215 ecr 0,sackOK,eol], length 0 11:30:44.288079 IP6 (flowlabel 0x8c63f, hlim 54, next-header TCP (6) payload length: 40) 2001:200:dff:fff1:216:3eff:feb1:44d7.80 > 2003:c1:df10:9801:3cd9:d041:c580:584b.63658: Flags [S.E], cksum 0xb33e (correct), seq 2308490815, ack 3119817517, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 2506143756 ecr 874444215], length 0 11:30:44.288412 IP6 (flowlabel 0xb0700, hlim 64, next-header TCP (6) payload length: 32) 2003:c1:df10:9801:3cd9:d041:c580:584b.63658 > 2001:200:dff:fff1:216:3eff:feb1:44d7.80: Flags [.], cksum 0xd91f (correct), seq 1, ack 1, win 2052, options [nop,nop,TS val 874444489 ecr 2506143756], length 0 11:30:49.718541 IP6 (class 0x02, flowlabel 0xb0700, hlim 64, next-header TCP (6) payload length: 48) 2003:c1:df10:9801:3cd9:d041:c580:584b.63658 > 2001:200:dff:fff1:216:3eff:feb1:44d7.80: Flags [P.], cksum 0xff80 (correct), seq 1:17, ack 1, win 2052, options [nop,nop,TS val 874449917 ecr 2506143756], length 16: HTTP, length: 16 GET index.html 11:30:49.992868 IP6 (flowlabel 0x8c63f, hlim 54, next-header TCP (6) payload length: 32) 2001:200:dff:fff1:216:3eff:feb1:44d7.80 > 2003:c1:df10:9801:3cd9:d041:c580:584b.63658: Flags [F.], cksum 0xb0a8 (correct), seq 227, ack 17, win 1035, options [nop,nop,TS val 2506149461 ecr 874449917], length 0 11:30:49.993273 IP6 (flowlabel 0xb0700, hlim 64, next-header TCP (6) payload length: 44) 2003:c1:df10:9801:3cd9:d041:c580:584b.63658 > 2001:200:dff:fff1:216:3eff:feb1:44d7.80: Flags [.], cksum 0xeb3a (correct), seq 17, ack 1, win 2052, options [nop,nop,TS val 874450191 ecr 2506143756,nop,nop,sack 1 {227:228}], length 0 ^C 6 packets captured 2525 packets received by filter 0 packets dropped by kernel freebsd@raspirouter:~ % sudo tcpdump -vvv -s0 -n -i ng0 host 2001:200:dff:fff1:216:3eff:feb1:44d7 tcpdump: listening on ng0, link-type NULL (BSD loopback), capture size 262144 bytes 11:30:44.013777 IP6 (flowlabel 0xb0700, hlim 63, next-header TCP (6) payload length: 44) 2003:c1:df10:9801:3cd9:d041:c580:584b.63658 > 2001:200:dff:fff1:216:3eff:feb1:44d7.80: Flags [SEW], cksum 0x3f10 (correct), seq 3119817516, win 65535, options [mss 1440,nop,wscale 6,nop,nop,TS val 874444215 ecr 0,sackOK,eol], length 0 11:30:44.287428 IP6 (flowlabel 0x8c63f, hlim 55, next-header TCP (6) payload length: 40) 2001:200:dff:fff1:216:3eff:feb1:44d7.80 > 2003:c1:df10:9801:3cd9:d041:c580:584b.63658: Flags [S.E], cksum 0xb33e (correct), seq 2308490815, ack 3119817517, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 2506143756 ecr 874444215], length 0 11:30:44.288446 IP6 (flowlabel 0xb0700, hlim 63, next-header TCP (6) payload length: 32) 2003:c1:df10:9801:3cd9:d041:c580:584b.63658 > 2001:200:dff:fff1:216:3eff:feb1:44d7.80: Flags [.], cksum 0xd91f (correct), seq 1, ack 1, win 2052, options [nop,nop,TS val 874444489 ecr 2506143756], length 0 11:30:49.718607 IP6 (class 0x02, flowlabel 0xb0700, hlim 63, next-header TCP (6) payload length: 48) 2003:c1:df10:9801:3cd9:d041:c580:584b.63658 > 2001:200:dff:fff1:216:3eff:feb1:44d7.80: Flags [P.], cksum 0xff80 (correct), seq 1:17, ack 1, win 2052, options [nop,nop,TS val 874449917 ecr 2506143756], length 16: HTTP, length: 16 GET index.html 11:30:49.992764 IP6 (class 0x02, flowlabel 0x8c63f, hlim 55, next-header TCP (6) payload length: 258) 2001:200:dff:fff1:216:3eff:feb1:44d7.80 > 2003:c1:df10:9801:3cd9:d041:c580:584b.63658: Flags [P.], cksum 0xbb69 (correct), seq 1:227, ack 17, win 1035, options [nop,nop,TS val 2506149461 ecr 874449917], length 226: HTTP 11:30:49.992844 IP6 (flowlabel 0x8c63f, hlim 55, next-header TCP (6) payload length: 32) 2001:200:dff:fff1:216:3eff:feb1:44d7.80 > 2003:c1:df10:9801:3cd9:d041:c580:584b.63658: Flags [F.], cksum 0xb0a8 (correct), seq 227, ack 17, win 1035, options [nop,nop,TS val 2506149461 ecr 874449917], length 0 11:30:49.993307 IP6 (flowlabel 0xb0700, hlim 63, next-header TCP (6) payload length: 44) 2003:c1:df10:9801:3cd9:d041:c580:584b.63658 > 2001:200:dff:fff1:216:3eff:feb1:44d7.80: Flags [.], cksum 0xeb3a (correct), seq 17, ack 1, win 2052, options [nop,nop,TS val 874450191 ecr 2506143756,nop,nop,sack 1 {227:228}], length 0 11:30:51.070753 IP6 (flowlabel 0x8c63f, hlim 55, next-header TCP (6) payload length: 258) 2001:200:dff:fff1:216:3eff:feb1:44d7.80 > 2003:c1:df10:9801:3cd9:d041:c580:584b.63658: Flags [FP.], cksum 0xb620 (correct), seq 1:227, ack 17, win 1035, options [nop,nop,TS val 2506150539 ecr 874450191], length 226: HTTP 11:30:52.974578 IP6 (flowlabel 0x8c63f, hlim 55, next-header TCP (6) payload length: 258) 2001:200:dff:fff1:216:3eff:feb1:44d7.80 > 2003:c1:df10:9801:3cd9:d041:c580:584b.63658: Flags [FP.], cksum 0xaeb0 (correct), seq 1:227, ack 17, win 1035, options [nop,nop,TS val 2506152443 ecr 874450191], length 226: HTTP Here you can see, that the http answer arrives in ng0 but leaves intern0 somehow broken. All packets before have the same checksum in and out of the raspberry until the HTTP answer arrives. It seems the payload is then missing.
Could you try disabling transmit checksum offoad (ifconfig genet0 -txcsum -txcsum6)? Both v4 and v6 need to be disabled to be effective on genet. Thanks.
yes, tried that (and verified now again) and it does not change a thing. some additional notes: I can initiate connections to and from the raspberry pi 4 itself. just forwarding seems not to work. in the tcp dump output above intern0 is genet0.
Thanks, that eliminates some of the trickier code. It is a bit of a long shot, but you could try setting the debug flag (ifconfig intern0 debug) and watching for console messages, or check dmesg or /var/log/messages. Are you able to build a kernel if I send a patch?
One other thing to check when testing: see if any of the error counters are increasing in the output of "netstat -sp ip6". It is possible the packets are not making it to the genet driver.
I tried debug and there is nothing in the logs or dmesg. I did not check the error counter because it definitely enters and leaves genet0 because I see it in the tcpdump output. the valid packet enters the system on ng0 and then leaves genet0 without payload but a correct checksum. also the flag is changed. so something is messing with the packet. this packet enters ng0: 11:30:49.992764 IP6 (class 0x02, flowlabel 0x8c63f, hlim 55, next-header TCP (6) payload length: 258) 2001:200:dff:fff1:216:3eff:feb1:44d7.80 > 2003:c1:df10:9801:3cd9:d041:c580:584b.63658: Flags [P.], cksum 0xbb69 (correct), seq 1:227, ack 17, win 1035, options [nop,nop,TS val 2506149461 ecr 874449917], length 226: HTTP this packet leaves genet0: 11:30:49.992868 IP6 (flowlabel 0x8c63f, hlim 54, next-header TCP (6) payload length: 32) 2001:200:dff:fff1:216:3eff:feb1:44d7.80 > 2003:c1:df10:9801:3cd9:d041:c580:584b.63658: Flags [F.], cksum 0xb0a8 (correct), seq 227, ack 17, win 1035, options [nop,nop,TS val 2506149461 ecr 874449917], length 0
The 2 packets you listed in the last comment are not the same packet. The packet with the length of 226 is the HTTP response. The packet with the FIN (F flag) is the following packet, and appears in both traces. The FIN packet is identical in the two traces except the timestamp and the hop limit, as it should be. The response packet is missing from the internal trace. That is why I suggested checking the error counters, although I suspect they are not increasing. I tried to reproduce this with traffic coming and going via genet0, and everything has been working. I suspect it has something to do with the packet layout in buffers, which may be different with the other interface. I am putting together a diagnostic patch. Can you build a kernel, or should I put one somewhere for download?
sorry, missed the difference in the packets. would be nice if you could put the kernel somewhere to download for me
I sent email about a diagnostic kernel. Did you receive it?
(In reply to Mike Karels from comment #8) yes, I replied to the sender, but that address did not work for a week. I replied 7min ago.
anyhow, your kernel is running fine and I got this in the logs with debug mode on: Mar 8 13:02:32 raspirouter kernel: genet0: v6 TCP w/only ether Mar 8 13:02:32 raspirouter syslogd: last message repeated 3507 times Mar 8 13:03:02 raspirouter kernel: y ether Mar 8 13:03:02 raspirouter kernel: genet0: v6 TCP w/only ether Mar 8 13:03:02 raspirouter syslogd: last message repeated 3538 times Mar 8 13:03:32 raspirouter kernel: y ether Mar 8 13:03:32 raspirouter kernel: genet0: v6 TCP w/only ether Mar 8 13:03:49 raspirouter syslogd: last message repeated 3898 times Mar 8 13:05:32 raspirouter syslogd: last message repeated 4490 times Mar 8 13:07:56 raspirouter syslogd: last message repeated 373 times
Thanks, I'll clean up the changes and get them ready. In the meantime, ifconfig -debug on the interface will make the extra log messages stop.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=2bdcf6237744b2d9d9707d623660d33931daeb52 commit 2bdcf6237744b2d9d9707d623660d33931daeb52 Author: Mike Karels <karels@FreeBSD.org> AuthorDate: 2021-03-18 00:19:24 +0000 Commit: Mike Karels <karels@FreeBSD.org> CommitDate: 2021-03-18 00:25:43 +0000 genet: Fix problem with forwarding some TCP/IPv6 packets TCP/IPv6 packets to be forwarded can be laid out with only the Ethernet header in the first mbuf, and these packets are lost. There was a previous hack to pullup ICMPv6 packets with such a layout for the same reason. Generalize, and pullup any IPv6 packets with only the Ethernet header in the first mbuf. Possibly this should also include IPv4, but that situation has not been observed to fail. PR: 254060 Reported by: denis at h3q.com MFC after: 3 days sys/arm64/broadcom/genet/if_genet.c | 46 +++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 25 deletions(-)
mark this fixed/closed?
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=35e7b6bff0b39aa7c536dfbe3dd783abc983ea97 commit 35e7b6bff0b39aa7c536dfbe3dd783abc983ea97 Author: Mike Karels <karels@FreeBSD.org> AuthorDate: 2021-03-18 00:19:24 +0000 Commit: Mike Karels <karels@FreeBSD.org> CommitDate: 2021-03-21 18:46:32 +0000 genet: Fix problem with forwarding some TCP/IPv6 packets TCP/IPv6 packets to be forwarded can be laid out with only the Ethernet header in the first mbuf, and these packets are lost. There was a previous hack to pullup ICMPv6 packets with such a layout for the same reason. Generalize, and pullup any IPv6 packets with only the Ethernet header in the first mbuf. Possibly this should also include IPv4, but that situation has not been observed to fail. PR: 254060 Reported by: denis at h3q.com MFC after: 3 days (cherry picked from commit 2bdcf6237744b2d9d9707d623660d33931daeb52) sys/arm64/broadcom/genet/if_genet.c | 46 +++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 25 deletions(-)