I discovered this doing some testing with the latest OPNsense. However, they insisted this was upstream and sure enough I was able to replicate the behavior in FreeBSD 11.3. Here is the mentioned RFC: https://tools.ietf.org/html/rfc1122#page-47 Please note that in section 3.5 INTERNET LAYER REQUIREMENTS SUMMARY, "Use address mask in local/remote decision" is marked as "MUST". Here is the bug report with OPNsense: https://github.com/opnsense/core/issues/3952 And the discussion on their forum: https://forum.opnsense.org/index.php?topic=15900.0 This was my testing rule set when confirming the FreeBSD/pf behavior. The commented out line (at the bottom) is what breaks my SSH connection from the local subnet 192.168.169.0/24. Connection from 192.168.169.200 to 192.168.169.197(FreeBSD). scrub on lo0 all fragment reassemble scrub on vtnet0 all fragment reassemble block drop in log inet all label "02f4bab031b57d1e30553ce08e0ec131" block drop in log quick inet proto tcp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5" block drop in log quick inet proto udp from any port = 0 to any label "7b5bdc64d7ae74be1932f6764a591da5" block drop in log quick inet proto tcp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5" block drop in log quick inet proto udp from any to any port = 0 label "ae69f581dc429e3484a65f8ecd63baa5" pass in log on vtnet0 proto udp from any port = bootps to any port = bootpc keep state label "613fb331c903de9502461c121104e092" pass out log on vtnet0 proto udp from any port = bootpc to any port = bootps keep state label "b8e1da9ac60ce8edb8e5a84bc5cec53e" pass in log quick on lo0 all flags S/SA keep state label "59162224cde3be673a9b295d6e24dcea" pass out log all flags S/SA keep state allow-opts label "fae559338f65e11c53669fc3642c93c2" pass in quick on vtnet0 inet proto icmp from (vtnet0:network) to (vtnet0) keep state label "b16c302604774ef7a3969da93953d4da" pass in log quick on vtnet0 inet proto tcp from (vtnet0:network) to (vtnet0) port = ssh flags S/SA keep state label "ssh" #pass in log quick on vtnet0 reply-to (vtnet0 192.168.169.254) inet proto tcp from (vtnet0:network) to (vtnet0) port = ssh flags S/SA keep state label "ssh"
(In reply to ctminime from comment #0) I forgot to mention that switch the commenting out between the 2 SSH rules. I don't have them both enabled at the same time.
I'm sorry, but this is pf behaving exactly as documented and expected. pf has been configured to send replies via gateway 192.168.169.254 on vtnet0, and that's what it does. The administrator has defined policy for those packets, and that's that pf is for: enforcing network administrator policy. One could similarly argue that every block drop rule also violates RFCs (in that we don't send an error message for closed ports). Administrator policy trumps the RFC. If you don't want pf to send certain reply packets via gateway 192.168.169.254 on vtnet0 that can be configured.
(In reply to Kristof Provost from comment #2) As far as I can tell, you are wrong about dropping packets violates RFC. I couldn't find anything of the sorts in RFC 793 or RFC 1122. What I did find, is some commentary in RFC 3360 under "The history of TCP resets". Go ahead and read it yourself. However, if you you can find that in an RFC, I would be interested to read it. I can't think of a single way that making "reply-to" RFC compliant (by default) would break any setup. It would only fix the couple of use case that it breaks. It is my personal opinion that the default behavior of a function should be RFC compliant. Then it could be up to the administrator if he chooses to violate RFC to accomplish whatever he wants. It could be done like this: "reply-to" sends all traffic, except local subnet traffic, to the gateway. (FYI, this is how pfSense behaves.) But if an administrator wanted to, for some unfathomable reason, there could be an option like "reply-to absolute" that would be specifically for violating RFC and send all traffic to the gateway. Just because this is the way it has worked for a decade+, doesn't mean it is right or should stay that way.
(In reply to Kristof Provost from comment #2) I am re-opening this for a couple of reasons: 1. I don't think my previous reply was seen by Kristof (who closed this). 2. It was closed so fast it left NO room for others in the community to comment 3. One person should not be able to dictate weather something gets looked at. I ask that this be left open for at least a week for commenters. While, the feature may currently work as intended and documented, this could still be an improvement on the reply-to feature. And it is my personal opinion (and backed by RFC) the the behavior I outlined is wrong. If others think this should be addressed, great. Then it should stay open. If others think nothing needs/should be done. I will accept the "closed - works as intended".
I have indeed seen your response. There was no need to reopen the issue for that. As I'm currently the only maintainer of pf in FreeBSD I don't expect much follow-up, but I'm happy to leave this open for another week. I will not work on it.
(In reply to Kristof Provost from comment #5) I did not know there was only 1 maintainer of pf. So, thank you for entertaining my impertinence. Would you mind pointing out where the reply-to feature is in the source code? I am not a developer, but just in case someone else feels inclined to take a look.
(In reply to ctminime from comment #6) There is no one place to look for this. Rules are parsed (at configuration time) by the code in sbin/pfctl, then passed to the kernel. The code in sys/netpfil/pf parses packets and decides what actions to take based on those rules. Relevant functions are pf_test() and pf_route(), PF_REPLYTO is a useful constant to look at. This is nontrivial code with subtle edge cases.
I agree with the earlier comment by Kristof Provost. This is not a FreeBSD bug. pf is being told to route all reply packets back through a certain gateway, and that is in fact what it's doing. If the way the administrator configures FreeBSD violates an RFC, that's on the administrator. There are many ways to configure firewall rules that go counter to what's written in an RFC, if that's what you want to do. It is also conversely possible to configure PF rules that do not cause this behaviour, if that's what you want to do. If a project or system administrator that uses pf generates pf rules that end up violating an RFC, it's on whoever's or whatever writing the rules to write them differently. In this case the firewall rule is working exactly as intended. You might be able to argue that it would be useful for pf to have a feature that would route packets down a certain interface, as opposed to specifically through a specific gateway, but that would mean talking about introducing a new feature, rather than changing behaviour of an old one. I think it might be a good idea, but if you really want pf to do that, you can already do that by writing rules that handle same-subnet traffic differently to cross-subnet traffic, although it'd end up a bit messy. Incidentally, I agree with ctminime's core problem description. The way OPNsense and pfSense use this feature is bogus. But it has nothing to do with FreeBSD or pf. It's doing what it's being told, and pf should not second guess what the administrator is telling it to do. There may be good reasons to configure your firewall in that way.