Summary: When pinging the ip address 255.255.255.255, the MAC address should be set to the broadcast address. Description: I narrowed this down to sys/netinet/ip_output.c where it looks like the decision about whether to set M_BCAST is happening on line 505 for this case: isbroadcast = in_ifaddr_broadcast(gw->sin_addr, ia); It looks like in_ifaddr_broadcast is only checking whether the address is the same as the interface broadcast (e.g. 192.168.122.255), but not 255.255.255.255. If I ping 192.168.122.255, then I get the expected MAC broadcast address. Actual results: -------------- tcpdump -vvvv -xx -ni em0 icmp & ping -nc 1 255.255.255.255 root@bsd-vbox:/home/jcaplan # tcpdump -vvvv -xx -ni em0 icmp & [1] 5429 root@bsd-vbox:/home/jcaplan # tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes root@bsd-vbox:/home/jcaplan # ping -nc 1 255.255.255.255 PING 255.255.255.255 (255.255.255.255): 56 data bytes 02:01:55.895257 IP (tos 0x0, ttl 64, id 46687, offset 0, flags [none], proto ICMP (1), length 84) 192.168.1.32 > 255.255.255.255: ICMP echo request, id 13845, seq 0, length 64 0x0000: 3c37 862c 1f63 0800 27b8 57c0 0800 4500 Note the first six bytes are not 0xff. Expected Results: ---------------- root@bsd-vbox:/home/jcaplan # tcpdump -vvvv -xx -ni em0 icmp & [1] 5429 root@bsd-vbox:/home/jcaplan # tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes root@bsd-vbox:/home/jcaplan # ping -nc 1 255.255.255.255 PING 255.255.255.255 (255.255.255.255): 56 data bytes 02:01:55.895257 IP (tos 0x0, ttl 64, id 46687, offset 0, flags [none], proto ICMP (1), length 84) 192.168.1.32 > 255.255.255.255: ICMP echo request, id 13845, seq 0, length 64 0x0000: ffff ffff ffff 0800 27b8 57c0 0800 4500 Note the first 6 bytes are 0xff.
Created attachment 221503 [details] proposed patch
Note that FreeBSD 13.0 added a swap of the dst to the gw just before this check for broadcast. That swap needs to be moved to after the check for broadcast.
Hi, our application using UDP broadcast encounters the same issue. Like in the original description, it is due to the unset broadcast flag, and then the wrong physical address (usually is of the gateway) instead of the broadcasting physical address is used. This prevents the packet from broadcasting. I investigated the origin of this issue, which seems to be introduced in D7266 (commit 90cc51a1ab4be2388560ee1d543d3fddc8d2c6db), affecting release/13.1.0, release/13.0.0, release/12.3.0, release/12.2.0, release/12.1.0 and release/12.0.0. However, the patch does not work for me because in my configuration there is a router, so another branch is used. I propose a different patch after all the branches: by testing whether the IP address is a broadcast one (255.255.255.255) or an anycast one.
Created attachment 235278 [details] Check IP address for broadcast after all branches
There is a documented way to broadcast using INADDR_BROADCAST (all ones), and it explicitly specifies the outgoing interface. See IP_SENDONES in ip(4). In short, you set the IP_SENDONES option, then send to the desired interface's configured broadcast address (e.g. 192.168.122.255).
Sorry, that's the IP_ONESBCAST option.
(In reply to Mike Karels from comment #5) Yes, I saw the option in the kernel code, there is. But as far as I can see, none of the applications/frameworks seems to have set the flag explicitly (at least the ping, and the Qt stuff). They worked before because in_broadcast() does check the INET_BROADCAST address. But the mentioned commit changes the callee to in_ifaddr_broadcast(), which does not contain such processing anymore. I checked the FreeBSD part in the XNU (macOS kernel), which is an old release. The ping and our UDP broadcast keeps working on macOS. But they are broken on current FreeBSD kernel. For the compatibility, should the flag be fixed in the kernel instead of explicitly being set by the application, when the address is set to 255.255.255.255 (all ones)?
I have mixed feelings about restoring the checks for INADDR_BROADCAST and INADDR_ANY. The problem with a send to 255.255.255.255 for a broadcast is that it doesn't allow any way to select the outgoing interface. The in(4) man page is out of date on this; the interface selection is now done via routing. This means that a send to 255.255.255.255 would often follow the default route, which may or may not be right. A more explicit route can be added, although that seems like a poor way to control an application, and only allows one interface to be used. The application already needs to set the SO_BROADCAST option; setting IP_ONESBCAST and providing a way to specify the destination address doesn't seem like a big increment. Granted, ping will not have a way to do this, but that doesn't seem like a very important use case. If the change you cited was made recently, I might be more worried about compatibility. But it was made almost 6 years ago.
(In reply to Mike Karels from comment #8) Thanks very much for the comments ! Yes, a simple UDP broadcast using 255.255.255.255 with default route returns "Network is unreachable" because of no route. Our application wants to use a UDP broadcast through 255.255.255.255 to discover a certain service. In the previous kernels, including the FreeBSD part that macOS is using, it works fine. But we found that it does not work any more on FreeBSD 13 (although we have not tested it for long time). Finally, our workaround is to iterate all the non-local interfaces and send to their broadcast addresses. This should be coherent with the current design in the FreeBSD kernel. Thanks again !
(In reply to Inoki from comment #9) > Yes, a simple UDP broadcast using 255.255.255.255 with default route returns "Network is unreachable" because of no route. Did you mean "no default route"? If there is a default route, there is a route to anything (although it could be a reject route).
(In reply to Inoki from comment #9) Hi. >Finally, our workaround is to iterate all the non-local interfaces and send to their >broadcast addresses. This should be coherent with the current design in the FreeBSD >kernel. I see the same issue on my FreeBSD 14-CURRENT kernel and did the same workaround just like you did. I had a chance to talk with hrs@ about this issue. He gave me an advise that adding '255.255.255.255' route entry via specific interface also solves this issue without source code change. For example, I have 'em0' interface for IPv4. Set a route entry as 'route add 255.255.255.255 -iface em0'. Then 'ping 255.255.255.255' works as I expected. When my 'em0' has a broadcast address '192.168.1.255/24', `route add 255.255.255.255 192.168.1.255' also works for me.
(In reply to jcaplan from comment #1) Please consider that FreeBSD obeys RFC 3021 "Using 31-Bit Prefixes on IPv4 Point-to-Point Links". In the case of /31 addressing, the broadcast address for such a link is 255.255.255.255. It's implemented and working fine. Will the proposed patch respect and not break the RFC 3021 requirements?
(In reply to Yuichiro NAITO from comment #11) > For example, I have 'em0' interface for IPv4. > Set a route entry as 'route add 255.255.255.255 -iface em0'. > Then 'ping 255.255.255.255' works as I expected. Found an interesting bug, the ICMP request is duplicated (and hence the duplicated ICMP reply). ``` # route add 255.255.255.255 -iface em0 # ping -c1 255.255.255.255 ``` The tcpdump session: ``` # tcpdump -nvei em0 'icmp' 11:16:47.097435 00:0c:29:73:4f:98 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 8264, offset 0, flags [none], proto ICMP (1), length 84) 192.168.117.154 > 255.255.255.255: ICMP echo request, id 50203, seq 0, length 64 11:16:47.097476 00:0c:29:73:4f:98 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 8264, offset 0, flags [none], proto ICMP (1), length 84) 192.168.117.154 > 255.255.255.255: ICMP echo request, id 50203, seq 0, length 64 11:16:47.097688 00:50:56:c0:00:08 > 00:0c:29:73:4f:98, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 61857, offset 0, flags [none], proto ICMP (1), length 84) 192.168.117.1 > 192.168.117.154: ICMP echo reply, id 50203, seq 0, length 64 11:16:47.097692 00:50:56:f2:7d:1e > 00:0c:29:73:4f:98, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 128, id 2139, offset 0, flags [none], proto ICMP (1), length 84) 192.168.117.2 > 192.168.117.154: ICMP echo reply, id 50203, seq 0, length 64 ```
(In reply to Zhenlei Huang from comment #13) > Found an interesting bug, the ICMP request is duplicated (and hence the duplicated ICMP reply). I don't think it is a bug of protocol stack. The first ICMP request is really sent to the ethernet, the second one is received packet that is broadcasted. The sending interface is also in the broadcast domain. If you make a bridge interface that includes 'em0' and use the bridge interface for 255.255.255.255. You can see just sending packets on 'em0'. Example commands are here. ``` ifconfig bridge0 create ifconfig bridge0 addm em0 ifconfig bridge0 inet <some address>/<netmask> route add 255.255.255.255 -iface bridge0 tcpdump -nvei em0 icmp ``` One packet can be seen on 'em0' while `ping -c1 255.255.255.255`. And, two received ICMP responses are different in source address. They are '192.168.117.1' and '192.168.117.2'. Two hosts responded to your broadcast packet.