Bug 212331

Summary: pfil processing order
Product: Base System Reporter: srijannandi <srijan.nandi>
Component: kernAssignee: freebsd-net (Nobody) <net>
Status: New ---    
Severity: Affects Some People CC: julian, kp
Priority: ---    
Version: 10.3-STABLE   
Hardware: i386   
OS: Any   
Description Flags
Packet Capture none

Description srijannandi 2016-09-02 12:30:02 UTC
Created attachment 174315 [details]
Packet Capture

Hello Everyone,

I am running FreeBSD 10.3-RELEASE-p7.

The setup that I have is, that I have 2 WAN interfaces and 1 LAN interface. I have configure both IPFW (for Traffic Shaping and Captive Portal) and PF for all other filtering and nat rules.

em0 - LAN
em1 - WAN1
em2 - WAN2

My scenario is that as soon as a connection is made to a website, IPFW catches this port 80 traffic and redirects it to a Captive Portal listening on port 9000 and then after authentication is successful via Captive Portal, the traffic is then passed onto PF for further processing.

Here are the relevant IPFW rules:
1. For Captive Portal
add 5000 fwd,8000 tcp from any to any dst-port 443 in via em0
add 5000 allow ip from any to any dst-port 443 via em0
add 5000 fwd,9000 tcp from any to any dst-port 80 in via em0
add 5000 allow ip from any to any dst-port 80 via em0
2. Allow authenticated traffic to PF
add 65533 pass ip from any to any
PF upon receiving these packets makes a connection to the website and it opens up.

As can be see from the rule below:
pass in quick on em0 inet from to any flags S/SA keep state
Everything works pretty well.

However, as soon as I apply a route-to rule in PF, the processing order breaks. So now, no longer does IPFW get the port 80 traffic to be redirected to port 9000. It simply passes it outside.

PF rule with rout-to set:
pass in quick on em0 route-to (pppoe0 X.X.X.X) inet from to any flags S/SA keep state.
I require the processing to be such that for incoming traffic, IPFW should process the packets first and then pass it on to PF. This works when route-to is not set.

Attached is a tcpdump, explaining the same.

Any help will be highly appreciated. I have been banging my head around this issue for days.
Comment 1 Kristof Provost freebsd_committer 2016-09-02 12:38:28 UTC
Packets that hit the route-to rule are sent directly to the output interface, they no longer pass through any firewall code. pf calls ipf->if_output() directly. Doing anything else would risk having the packets loop endlessly.

I'm not sure it's possible to get that to work.
Comment 2 srijannandi 2016-09-05 10:37:07 UTC
Any workaround to get this working?

I really need this running.
Comment 3 Kristof Provost freebsd_committer 2016-09-05 11:04:51 UTC
(In reply to srijannandi from comment #2)
What are you trying to accomplish with the route-to rule?
Comment 4 srijannandi 2016-09-05 11:57:55 UTC
With route-to, I can have a multi gateway rule. This multi-gateway acts for load balancing or failover. So, in case a gateway fails, the connections will be handled by the active gateway. Or else, it would load balance traffic using the route-to rule.
Comment 5 srijannandi 2016-09-05 11:59:55 UTC
According to this post,


I have ipfw process packets first and then pf. As soon as 'route-to' is added to pf, it breaks this order and ipfw no longer comes to picture.
Comment 6 srijannandi 2016-09-05 14:41:44 UTC
I am facing the exact issue faced by Alexey. This was reported way back in 2010, do we have a fix or a workaround for this yet.

Comment 7 Julian Elischer freebsd_committer 2016-09-12 02:56:49 UTC
Since I don't exactly understand pf, havgn never used it in production, I  don't really understand what you are trying to do but it would surprise me if there was no way to do it just in ipfw. The only pf feature it definately doesn't have that I know of is the fingerprinting.
Comment 8 srijannandi 2016-09-12 09:59:06 UTC
ipfw is doing it's job perfectly well. It's just that I use ipfw for bandwidth shaping and also have captive portal running on ipfw. Rest all firewall'ing and nat'ting features are used in pf.

So, I have ipfw processing packets destined to port 80. As soon as it receives a packet for destination port 80, it throws the user a captive portal login page. The user than logs in using his/her username and password. After which the packet is marked as authenticated and passed on to pf for further processing. pf then allows or denies the traffic as per the rule configured.

This works fine, without issues.

As soon as I enable a route-to rule in pf to pass this traffic via a specific gateway, then when a packet is received by FreeBSD for destination port 80, ipfw no longer comes into picture and pf shorts this packet and start processing it. Therefore, in this scenario, I no longer get the captive portal page.