Bug 253164

Comment 1 Kristof Provost 2021-02-01 15:13:52 UTC

Please post full ruleset(s) and explain what 'does not work' mean in this context.
What packets are (not) sent and what packets are expected to (not) be sent?

"It's broken" is not generally a useful bug report without a lot more detail.

pass in on $ext_if_1 reply-to ($ext_if_1 $gw_1) inet proto tcp to ($ext_if_1) port { $tcp_svc } tag EXT_IF_A
pass in on $ext_if_1 inet proto tcp from ($ext_if_1:network) to ($ext_if_1) port { $tcp_svc } tag EXT_IF_A
# Lan4ever
pass in on $ext_if_2 reply-to ($ext_if_2 $gw_2) inet proto tcp to ($ext_if_2) port { $tcp_svc } tag EXT_IF_B
pass in on $ext_if_2 inet proto tcp from ($ext_if_2:network) to ($ext_if_2) port { $tcp_svc } tag EXT_IF_B

pass in quick from ($ext_if_1:network) tagged EXT_IF_A keep state
pass in quick reply-to ($ext_if_1 $gw_1) tagged EXT_IF_A keep state

pass in quick from ($ext_if_2:network) tagged EXT_IF_B keep state
pass in quick reply-to ($ext_if_2 $gw_2) tagged EXT_IF_B keep state

pass out route-to ($ext_if_1 $gw_1) inet from ($ext_if_1) keep state
pass out route-to ($ext_if_2 $gw_2) inet from ($ext_if_2) keep state

This rule set for TCP, but the same behaviour and for UDP too. When I use tcpdump, reply packets appear on the interface, which point to the default GW.

On the FreeBSD 12.1 reply packets appears on the interface which they came from.

Do you need more details?

Comment 3 Kristof Provost 2021-02-01 15:32:36 UTC

(In reply to skeletor from comment #2)
Yes.

(a) **FULL** rule sets
(b) a description of what happens and what's supposed to happen.

"It's broken" is not an actionable bug report.

ext_if_2="igb0"
ext_if_1="bge0"
int_if="vlan12"
vlan1920_net="192.168.0.0/24"

lo="lo0"
int_net="10.11.12.0/24"

gw_2="BB.BB.BB.YY"
gw_1="AA.AA.AA.YY"

# services
tcp_svc="ssh, 53, 110,143,993,995,25,465"
udp_svc="53"

#skip iface
set skip on lo0
set block-policy drop
set limit states 300000
set limit frags 300000
set limit src-nodes 300000

# scrub
scrub in on $ext_if_1 all
scrub in on $ext_if_2 all
scrub in on $int_if all fragment reassemble max-mss 1496

# NAT

nat on $ext_if_1 inet from {$int_net} -> $ext_if_1:0
nat on $ext_if_2 inet from {$int_net} -> $ext_if_2:0

########## FIREWALL
block in quick from any os NMAP

block quick on $ext_if_2 proto udp from any to $ext_if_2 port 161
block quick on $ext_if_2 proto tcp from any to $ext_if_2 port { 199, 179 }

pass in

pass quick from <whitelist> to any

#bruteforce
block drop quick from <badhosts> to any label "ssh bruteforce"
block drop quick from any to <badhosts> label "ssh bruteforce"

# ICMP
pass in on $ext_if_1 reply-to ($ext_if_1 $gw_1) inet proto icmp to ($ext_if_1) tag EXT_IF_A icmp-type echoreq code 0
pass in on $ext_if_1 inet proto icmp from ($ext_if_1:network) to ($ext_if_1) icmp-type echoreq code 0
pass in on $ext_if_2 reply-to ($ext_if_2 $gw_2) inet proto icmp to ($ext_if_2) tag EXT_IF_B icmp-type echoreq code 0
pass in on $ext_if_2 inet proto icmp from ($ext_if_2:network) to ($ext_if_2) icmp-type echoreq code 0

# allow tcp ports
pass in on $ext_if_1 reply-to ($ext_if_1 $gw_1) inet proto tcp to ($ext_if_1) port { $tcp_svc } tag EXT_IF_A
pass in on $ext_if_1 inet proto tcp from ($ext_if_1:network) to ($ext_if_1) port { $tcp_svc } tag EXT_IF_A
pass in on $ext_if_2 reply-to ($ext_if_2 $gw_2) inet proto tcp to ($ext_if_2) port { $tcp_svc } tag EXT_IF_B
pass in on $ext_if_2 inet proto tcp from ($ext_if_2:network) to ($ext_if_2) port { $tcp_svc } tag EXT_IF_B

# allow udp ports
pass in on $ext_if_1 reply-to ($ext_if_1 $gw_1) inet proto udp to ($ext_if_1) port { $udp_svc } tag EXT_IF_A
pass in on $ext_if_1 inet proto udp from ($ext_if_1:network) to ($ext_if_1) port { $udp_svc } tag EXT_IF_A
pass in on $ext_if_2 reply-to ($ext_if_2 $gw_2) inet proto udp to ($ext_if_2) port { $udp_svc } tag EXT_IF_B
pass in on $ext_if_2 inet proto udp from ($ext_if_2:network) to ($ext_if_2) port { $udp_svc } tag EXT_IF_B

pass in quick from ($ext_if_1:network) tagged EXT_IF_A keep state
pass in quick reply-to ($ext_if_1 $gw_1) tagged EXT_IF_A keep state

pass in quick from ($ext_if_2:network) tagged EXT_IF_B keep state
pass in quick reply-to ($ext_if_2 $gw_2) tagged EXT_IF_B keep state

pass quick from <full_access> to any
pass quick from any to <full_access>

pass in quick from <whitelist> to any
pass out quick from any to <whitelist>

## ssh access
pass in quick proto tcp from any to any port ssh flags S/SA keep state \
        (max-src-conn 3, max-src-conn-rate 3/5, overload <badhosts> flush global)

#snmp
pass quick proto udp from $monitoring to any port 161
pass quick proto tcp from $monitoring to any port 199
pass quick proto icmp from $monitoring to any
block quick proto udp from any to any port 161
block quick proto tcp from any to any port 199

pass out inet from (self:network)
pass in inet proto icmp to (self:network)
pass in inet proto tcp from $int_net to (self:network)
pass in inet proto udp from $int_net to (self:network)

# LOCAL NETWORK
pass quick on $int_if

# OUTGOING ROUTE
pass out route-to ($ext_if_1 $gw_1) inet from ($ext_if_1) keep state
pass out route-to ($ext_if_2 $gw_2) inet from ($ext_if_2) keep state

pass out inet from { $ext_if_1 $ext_if_2 } to (self:network)

Here query on ext_if2:

17:40:42.342576 IP XX.XX.XX.XX.60318 > BB.BB.BB.BB.22: Flags [S], seq 2980536276, win 64240, options [mss 1460,sackOK,TS val 824407439 ecr 0,nop,wscale 7], length 0
17:40:43.345460 IP XX.XX.XX.XX.60318 > BB.BB.BB.BB.22: Flags [S], seq 2980536276, win 64240, options [mss 1460,sackOK,TS val 824408442 ecr 0,nop,wscale 7], length 0

Here reply on ext_if1:

17:40:42.342717 IP BB.BB.BB.BB.22 > XX.XX.XX.XX.60318: Flags [S.], seq 2922590039, ack 2980536277, win 64395, options [mss 1460,nop,wscale 6,nop,nop,TS val 1012050357 ecr 824407439], length 0
17:40:43.342422 IP BB.BB.BB.BB.22 > XX.XX.XX.XX.60318: Flags [S.], seq 2922590039, ack 2980536277, win 64395, options [mss 1460,nop,wscale 6,nop,nop,TS val 1012051357 ecr 824407439], length 0

Where XX.XX.XX.XX - host, from which i try to connect to BB.BB.BB.BB (ip on interface ext_if2)

ext_if1 point to default GW. ext_if2 - secondary ISP.

If not enough - please, let me know.

Comment 5 Kristof Provost 2021-02-01 15:56:38 UTC

(In reply to skeletor from comment #4)
Again: explain what happens and what you expect to have happen instead.

I've looked at the 12.1 and 12.2 code and there are no obvious differences in the reply-to code, but I still have no idea at all what your problem is.

I expect, that, if I send packets to secondary (ext_if_2) interface (not to point default GW) I receive it from ext_if_2. But really I see it on ext_if_1 and don't see on ext_if_2 (but should see).

Comment 7 Kristof Provost 2021-02-01 16:03:05 UTC

(In reply to skeletor from comment #6)
Send packets from where? From the machine? From the internet?

Yes, from outside (some host in internet).

Comment 9 Kristof Provost 2021-02-01 16:23:55 UTC

(In reply to skeletor from comment #8)
Have you confirmed, with pflog, that the route-to rule is still getting hit?

Thank you for right direction to check via pflog! Without "quick" keyword rule set with reply-to doesn't work. Now, everything is working.

Sorry, for my mistakes.