Overview -------- The genet interface is not able transmit/receive any packets after the genet interface is marked "down" firstly then marked "up". The reason is that the implementation genet driver code will reset the genet interface but not change the runtime flag of genet driver. Therefore, when "ifconfig genet up" command is used, the genet driver cannot be re-initialized again. Moreover, the initialization of genet driver is to enbale the driver's DMA and interrupts, which is not enbale the genet driver being reset when genet is marked down. Steps to Reproduce ------------------ 1. Config a static IP address to both of the host's interface and the genet interface In the DUT(Device Under Test), with genet interface and Freebsd running, side root@generic:~ # ifconfig genet0 192.168.3.129 netmask 255.255.255.0 root@generic:~ # ifconfig genet0 genet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=68000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether dc:a6:32:bf:f4:e6 inet 192.168.3.129 netmask 0xffffff00 broadcast 192.168.3.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> In the host computer with direct Ethernet cable connection to the DUT $ sudo ifconfig enp0s31f6 192.168.3.2 $ ifconfig enp0s31f6 enp0s31f6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.2 netmask 255.255.255.0 broadcast 192.168.3.255 ether 8c:8c:aa:c1:2b:c3 txqueuelen 1000 (Ethernet) RX packets 384 bytes 37406 (37.4 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 5674 bytes 1175798 (1.1 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 16 memory 0xae380000-ae3a0000 2. Host's interface and genet interface can transmit/receive packet to/from each other In the DUT root@generic:~ # ping 192.168.3.2 PING 192.168.3.2 (192.168.3.2): 56 data bytes 64 bytes from 192.168.3.2: icmp_seq=0 ttl=64 time=0.445 ms 64 bytes from 192.168.3.2: icmp_seq=1 ttl=64 time=0.321 ms 64 bytes from 192.168.3.2: icmp_seq=2 ttl=64 time=0.361 ms ^C --- 192.168.3.2 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.321/0.376/0.445/0.051 ms In the host $ ping 192.168.3.129 PING 192.168.3.129 (192.168.3.129) 56(84) bytes of data. 64 bytes from 192.168.3.129: icmp_seq=1 ttl=64 time=0.242 ms 64 bytes from 192.168.3.129: icmp_seq=2 ttl=64 time=0.233 ms 64 bytes from 192.168.3.129: icmp_seq=3 ttl=64 time=0.180 ms ^C --- 192.168.3.129 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2049ms rtt min/avg/max/mdev = 0.180/0.218/0.242/0.027 ms 3. Mark the genet interface "down", then mark the interface "up" root@generic:~ # ifconfig genet0 down root@generic:~ # ifconfig genet0 genet0: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=68000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether dc:a6:32:bf:f4:e6 inet 192.168.3.129 netmask 0xffffff00 broadcast 192.168.3.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@generic:~ # ifconfig genet0 up root@generic:~ # ifconfig genet0 genet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=68000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether dc:a6:32:bf:f4:e6 inet 192.168.3.129 netmask 0xffffff00 broadcast 192.168.3.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> Actual Results -------------- 1. Genet interface is not able to transmit/receive any packet In the DUT root@generic:~ # ping 192.168.3.2 PING 192.168.3.2 (192.168.3.2): 56 data bytes ^C --- 192.168.3.2 ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet loss In the host $ ping 192.168.3.129 PING 192.168.3.129 (192.168.3.129) 56(84) bytes of data. From 192.168.3.2 icmp_seq=9 Destination Host Unreachable From 192.168.3.2 icmp_seq=10 Destination Host Unreachable From 192.168.3.2 icmp_seq=11 Destination Host Unreachable ^C --- 192.168.3.129 ping statistics --- 13 packets transmitted, 0 received, +3 errors, 100% packet loss, time 12280ms pipe 4 Expected Results ---------------- The "ifconfig genet0 up" can re-enable the genet inerface after the genet interface is marked as "down". The data communication between the genet and the host should be recovered when the interface is re-enbale. Build Date & Hardware --------------------- root@generic:~ # uname -a FreeBSD generic 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n254105-d53927b0bae: Thu Mar 31 09:26:32 UTC 2022 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64
Thanks for the report. I have reproduced this, and am looking into it.
Created attachment 233010 [details] preliminary patch Can you try this patch to src/sys/arm64/broadcom/genet/if_genet.c? It seems to work.
Created attachment 233077 [details] Bug fix batch of genet interface down A copy of the bug fix patch which referred the implementation of Openbsd code.
(In reply to Mike Karels from comment #2) Hi Mike, Thanks for the patch. I tried your patch in my board. It seems like it is not working in my rpi4. Iam wondering if it is working in your board. Hoping I am not missing any things. I have also tried fixed this bug before. My patch is also attached, which is referred from the implementation from the OpenBsd. It is working in my board. I attach a copy of the bug fix patch from our internal repo. Jiahao
Created attachment 233100 [details] hybrid patch Thanks for your patch. My earlier patch worked on my RPi4, which is an original model with 4 GB. Your patch is more complete, which would also be useful if someone turns this into a module. I hybridized the two, as I wanted the full reset done by gen_reset; some of the other code may not be needed with that, but I left it. Could you try this one? Thanks.
(In reply to Mike Karels from comment #5) Hi Mike, Sorry for the late reply. The new hybrid patch works well in my rpi4 model B 8G RAM board. Thanks, Jiahao
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8f45652b6bccc258eb58f8721dea10184f1aaa17 commit 8f45652b6bccc258eb58f8721dea10184f1aaa17 Author: Mike Karels <karels@FreeBSD.org> AuthorDate: 2022-04-11 19:44:49 +0000 Commit: Mike Karels <karels@FreeBSD.org> CommitDate: 2022-04-14 19:10:13 +0000 genet: fix problems with interface down/up The genet interface did not resume operation correctly after doing ifconfig down then up. The down/reset procedure did not clear the RUNNING flag, and did not reset enough of the hardware state. This patch is modeled on OpenBSD code, with a call to gen_reset added to reset the controller completely. Regularize the parameter to gen_dma_disable() while here. PR: 263091 Submitted by: jiahali@blackberry.com sys/arm64/broadcom/genet/if_genet.c | 86 ++++++++++++++++++++++++++++++++----- 1 file changed, 75 insertions(+), 11 deletions(-)
Any chance this can be MFCd before 13.2?
(In reply to Daniel O'Connor from comment #8) The change has been on stable/13 since April, so will automatically be included in 13.2.
Thanks! (Sorry for the noise, I checked the wrong branch..)