Created attachment 233772 [details] Preliminary fix patch Overview -------- After adding a new mac address to the dwc0 interface, the dwc0 interface is not able to receive packets. However, "ifconfig dwc0" shows that the interface is UP. Actual Results -------------- 1. Set up In dwc driver's Freebsd terminal root@generic:~ # uname -a FreeBSD generic 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n254961-b91a48693a5: Thu Apr 21 09:35:51 UTC 2022 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 root@generic:~ # ifconfig dwc0 dwc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE> ether fa:97:92:f6:f1:09 inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@generic:~ # ping -c 1 192.168.3.2 PING 192.168.3.2 (192.168.3.2): 56 data bytes 64 bytes from 192.168.3.2: icmp_seq=0 ttl=64 time=0.509 ms --- 192.168.3.2 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.509/0.509/0.509/0.000 ms Host Terminal 1 Check interface config $ ifconfig enp0s31f6 enp0s31f6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.3.2 netmask 255.255.255.0 broadcast 192.168.3.255 inet6 fe80::f897:92ff:fef6:f102 prefixlen 64 scopeid 0x20<link> ether 8c:8c:aa:c1:2b:c3 txqueuelen 1000 (Ethernet) RX packets 1662 bytes 516194 (516.1 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 10885 bytes 1346113 (1.3 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 16 memory 0xae380000-ae3a0000 Run the tcpdump on interface 'enp0s31f6" sudo tcpdump -U -pi enp0s31f6 -en 2. Test Results In dwc driver's Freebsd terminal root@generic:~ # ifconfig dwc0 ether 22:33:44:55:66:77 root@generic:~ # ifconfig dwc0 dwc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE> ether 22:33:44:55:66:77 hwaddr fa:97:92:f6:f1:09 inet 192.168.3.129 netmask 0xffffff00 broadcast 192.168.3.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@generic:~ # netstat -n -I dwc0 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll dwc0 1500 <Link#1> 22:33:44:55:66:77 52 0 0 196 0 0 dwc0 - 192.168.3.0/2 192.168.3.129 16 - - 10 - - root@generic:~ # ping -c 1 192.168.3.2 PING 192.168.3.2 (192.168.3.2): 56 data bytes --- 192.168.3.2 ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss root@generic:~ # netstat -n -I dwc0 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll dwc0 1500 <Link#1> 22:33:44:55:66:77 52 0 0 198 0 0 dwc0 - 192.168.3.0/2 192.168.3.129 16 - - 11 - - Packets captured by the tcpdump at Host Terminal 1 $ sudo tcpdump -U -pi enp0s31f6 -en tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp0s31f6, link-type EN10MB (Ethernet), capture size 262144 bytes 11:13:17.395542 22:33:44:55:66:77 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.3.2 tell 192.168.3.129, length 46 11:13:17.395584 8c:8c:aa:c1:2b:c3 > 22:33:44:55:66:77, ethertype ARP (0x0806), length 42: Reply 192.168.3.2 is-at 8c:8c:aa:c1:2b:c3, length 28 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel Analysis -------------- According to the changes in the Ipkts section from the netstat results, the ARP response packet is not received. the "ifconfig dwc0" shows that the dwc0 interface is UP. For the Opkts section, the ARP request packet is counted twice, which I have reported in Bug 263817. However, if I use the "ifconfig dwc0 192.168.3.129/24" command to configure the interface after setting the new mac address. The interface can work properly. root@generic:~ # ifconfig dwc0 192.168.3.129/24 root@generic:~ # ifconfig dwc0 dwc0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE> ether 22:33:44:55:66:77 hwaddr fa:97:92:f6:f1:09 inet 192.168.3.129 netmask 0xffffff00 broadcast 192.168.3.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@generic:~ # ping -c 1 192.168.3.2 PING 192.168.3.2 (192.168.3.2): 56 data bytes 64 bytes from 192.168.3.2: icmp_seq=0 ttl=64 time=0.565 ms --- 192.168.3.2 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.565/0.565/0.565/0.000 ms I also tried to fix this bug in my development environment. I found that the interface can tx/rx packets after adding new mac address if I add the "dwc_setup_rxfilter()" function at the end of the "dwc_init_locked()". I decomposed the "dwc_setup_rxfilter()" function continuously and found that it is the setting of the mac address making the interface work again. My patch is attached. Could you help to analyze why we should add the mac address to the driver twice to make the driver run when setting a new MAC address? Because the mac address is configured into the driver once at the "dwc_setup_rxfilter()" at the beginning of the "dwc_init_locked()".
(In reply to Jiahao LI from comment #0) Hi, I agree it is unintuitive why your patch seems to fix the issue (or why the issue exists). Can you narrow it down further: is it the second call to dwc_setup_macaddr() or its placement at the end of dwc_init_locked() that makes the difference? Having looked through a few other NIC drivers, they follow much the same structure for setting the MAC address. I will have to see if I can reproduce the bug on my particular hardware.
(In reply to Mitchell Horne from comment #1) Hi, Thanks. I have tried to put the second call of "dwc_setup_rxfilter()" at different locations of the "dwc_init_locked()", such as right after the "dwc_init_dma()", after the "if_setdrvflagbits()" and after the "callout_reset()". It is the placement at the end of dwc_init_locked() that makes the difference, which will be equivalent to call the "ifconfig dwc0 IP" command after adding new mac address. Then, I decompose the "dwc_setup_rxfilter()" into the "dwc_setup_macaddr()". My device is RockPro64 LPDDR4:4G.
Created attachment 234029 [details] Another approach to fix the problem I referred to other open-source software. In OpenBSD, they will configure the media, mii_mediachg, firstly before setting the dwc driver's registers. (https://github.com/openbsd/src/blob/master/sys/dev/fdt/if_dwge.c line 1080) I have also tried this in my hardware, RockPro64, and attached the patch. The patch can fix the problem. Intuitively, it looks like the setting of dwc driver should be after the setting of media based on the two attached patches.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6501fcdc0a97faa3c59f6ece314bf7754303db6f commit 6501fcdc0a97faa3c59f6ece314bf7754303db6f Author: Jiahao LI <jiahali@blackberry.com> AuthorDate: 2022-09-08 15:50:37 +0000 Commit: Mitchell Horne <mhorne@FreeBSD.org> CommitDate: 2022-09-09 15:31:26 +0000 if_dwc: fix reinitialization with changed MAC It was observed that on RockPro64 hardware, the dwc interface is unable to receive packets after being assigned a new MAC address. The fix is simply to call mii_mediachg() before touching any device registers in dwc_init_locked(). This is consistent with what the OpenBSD driver does. PR: 263820 MFC after: 1 week sys/dev/dwc/if_dwc.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=9465af5ee2de0374b143802ea32823c45f3d73be commit 9465af5ee2de0374b143802ea32823c45f3d73be Author: Jiahao LI <jiahali@blackberry.com> AuthorDate: 2022-09-08 15:50:37 +0000 Commit: Mitchell Horne <mhorne@FreeBSD.org> CommitDate: 2022-09-16 17:34:48 +0000 if_dwc: fix reinitialization with changed MAC It was observed that on RockPro64 hardware, the dwc interface is unable to receive packets after being assigned a new MAC address. The fix is simply to call mii_mediachg() before touching any device registers in dwc_init_locked(). This is consistent with what the OpenBSD driver does. PR: 263820 MFC after: 1 week (cherry picked from commit 6501fcdc0a97faa3c59f6ece314bf7754303db6f) sys/dev/dwc/if_dwc.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
A commit in branch stable/12 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=c1cec62c0db0cb15d2d09aa5dfc5bd3727bbf3f6 commit c1cec62c0db0cb15d2d09aa5dfc5bd3727bbf3f6 Author: Jiahao LI <jiahali@blackberry.com> AuthorDate: 2022-09-08 15:50:37 +0000 Commit: Mitchell Horne <mhorne@FreeBSD.org> CommitDate: 2022-09-16 17:41:54 +0000 if_dwc: fix reinitialization with changed MAC It was observed that on RockPro64 hardware, the dwc interface is unable to receive packets after being assigned a new MAC address. The fix is simply to call mii_mediachg() before touching any device registers in dwc_init_locked(). This is consistent with what the OpenBSD driver does. PR: 263820 MFC after: 1 week (cherry picked from commit 6501fcdc0a97faa3c59f6ece314bf7754303db6f) sys/dev/dwc/if_dwc.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
Thanks for submitting the fix :) we should be all done here. Are there any remaining open bugs that I missed?