For quite a while (I think it started with 10.x) I have a CARP problem with a pair of opnSense routers. No one could help over there, so I'm posting here. The router pair was running for some years correctly with ~20 vifs on 4 interfaces. After an update (incl kernel), things went wrong: the backup machine turned to CARP master on the wan interface ix4, other interfaces remained backup, with the master machine having all interfaces as carp master. The result wasn't healthy, no surprise. Debugging using tcpdump, I found that the backup router didn't receive any carp announcements on the wan interface, until I disabled the firewall (pfctl -d) on the master (!) completely. Obviously, pf is filtering the outgoing carp traffic on the master, but I couldn't find any configuration that would do so. Even a "pass out quick on ix4 proto carp from {any} to {any}" as first filter rule didn't help. Any helpful hints?
Hi, 1. Do you have a link to the github issue? 2. Did you also test with plain FreeBSD? 3. Do you use IP Alias in OPNsene? They are not synced and you'll get this behavior when an alias only exists on one unit. 4. Which version update triggered the error? There was a long ticket regarding faulty/unsupported SFP's. Michael
I posted on this topic on the opnsense forum: https://forum.opnsense.org/index.php?topic=6228.0 and https://forum.opnsense.org/index.php?topic=6290.0 I'm not using IP aliases, thus the ~20 virtual IPs. The problem started with an update of opnsense from 17.1.x to 17.7, both using FreeBSD11. In the meantime, I have duplicated those routers in a virtual environment (Xen), with firewall and virtual IP settings copied, and do NOT see that problem there. The NIC on the problematic interface is an on-board 1GB. After a mainboard failure, the NIC driver changed from igb to ix, the problem persisted. The bad behaving interface doesn't use vlan or lagg, while well-behaving do have lagg and/or vlans defined.
I left you a note in the forums, it might be a wrong config option migrated from every old version to the current one in your config.xml.
After debugging a while, I found out that outgoing CARP packets are sent to the expected MAC address 01:00:5e:00:00:12 with pf disabled, on some interfaces (those with gateways) instead of coding the lower 23bits of 224.0.0.18 into the address some gateway on that local network is used. This is not conforming to RFC1112 6.4, and apparently some IGMP aware switches will filter those malformed packets. In some case, flipping some IGMP instructions on switches affected may fix CARP functionality, but the basic problem is in the freebsd kernel.
Sorry, did post too early: with pf disabled, destination mac address is correct, with pf enabled it may be wrong on some interfaces.
I don't know of an obvious way for pf to affect Ethernet destination MAC addresses. That's not to say this is impossible, but it is unlikely. Can you post your full pf ruleset, configuration and network captures demonstrating the problem? It would also be useful if you could look at what the wrong MAC address is, and what IP address that would correspond to. There may be a hint there.
Created attachment 204244 [details] Ruleset tested Before applying the test ruleset, I always pfctl -F flushed the firewall.
I can't post the original ruleset, but I shrinked it to a minimum and found the line causing the problem on the WAN interface ix4: pass out log route-to ( ix4 ##.##.##.## ) from {ix4} to {!(ix4:network)} keep state allow-opts ##.##.##.## is the WAN upstream default gateway. Without this line present in the ruleset, the destination MAC will be 01:00:5e:00:00:12 as expected by the IP multicast address 224.0.0.18, but with this rule present the destination mac is derived from the default gw ip address instead. NB: I'm observing wrong dst addresses on another interface as well, with more complicated symptoms. There's no stdgw present there, but two VIFs. One VIF will announce CARP using the partial IP of the other VIF, and the other VIF uses IP of a gateway on the same network. Didn't try to examine this further.
(In reply to Andreas Pflug from comment #8) Right, so the issue is likely that the route-to rule isn't working/matching as expected. I have a vague memory of there being an issue with '!' (in {!(ix4:network)}), but I can't immediately find either a relevant commit or bug for it. It might be worth trying to rephrase that rule without '!', just to confirm/rule out.
AFAICS the rule actually does as advised, since naively speaking 224.0.0.18 isn't in the ix4 network and so anything should be routed to the stdgw. But it's IP multicast, and so route-to shouldn't be applied in any case on 224.0.0.0/4, according to RFC1112. I added "pass out quick from {ix4} to 224.0.0.0/4" before that route-to rule, preventing the misshaped multicast packet. So actually pf does as advised to straight forward, but this will break multicast specifications. IMHO pf should implement the special behaviour on multicast packets implicitely, i.e. excepting them from routing, because this issue has been around for years without being diagnosed correctly (probably blaming some innocent switch IGMP implementations), and users probably don't expect a standard default gateway routing command breaking local multicast traffic.
I'm not sure that's a good idea for a change. As you say, you can solve your problem by telling pf to not route-to for the multicast subnet, but if we'd add the (also somewhat unexpected) implicit behaviour of not doing that for multicast ranges there'd be no way to configure what you just ran into. You might not want to have route-to rewrite the MAC address, but there's certainly cases where you'd want it to route out of a different interface (which is possible if you don't specify an IP address with the route-to). I also don't think we should make that impossible.
Well applying routing to a non-routable protocol (in the sense of normal IP routing) seems not too sensible in any case. Maybe omitting 224.0.0.0/4 from routing unless it's specified explicitely might be a solution. Should be probably discussed elsewhere. In my very first post, I mentioned an early pass out quick wouldn't help either: I tested again, and now it works, maybe not flushing the rules was the problem back then. pass out quick carp from {(self)} to 224.0.0.18 is the fix for misformed CARP packets.
(In reply to Andreas Pflug from comment #12) I would like to strongly discourage any special treatment of 224/4 by pf itself, the special treatment should be encoded in rules, not the kernel code. Putting these special cases in the kernel code is implementing policy, not methods. If it must be done be absolutely sure to use sys/netinet/in.h IN_MULTICAST as the source of truth as to what is a multicast packet some of us are running with alternate ideas as a test bed.
(In reply to Andreas Pflug from comment #12) Having thought about this some more I think even more strongly that changing the behaviour would be wrong. Multicast is a routable protocol, even if the Ethernet address resolution is different from unicast traffic. There may also be use cases for the current behaviour (and route-to *does* do exactly what it's designed and documented to do). Moreover, the problem you ran into is specific to Ethernet, so introducing changes in a layer 3 firewall for the sake of one layer 2 protocol is also the wrong strategy. Rodney is also correct that this is a policy question which you can express in your rules.