Bug 253164 - reply-to in PF brokens after upgrade from 12.1 to 12.2
Summary: reply-to in PF brokens after upgrade from 12.1 to 12.2
Status: Closed Not A Bug
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.2-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-pf (Nobody)
URL:
Keywords: regression
Depends on:
Blocks:
 
Reported: 2021-02-01 15:02 UTC by forumforeign
Modified: 2021-02-02 15:21 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description forumforeign 2021-02-01 15:02:12 UTC
After upgrading FreeBSD from 12.1 to 12.2 "reply-to" stopped working. I have the same on 2 different machines. Both of them had working "reply-to" on 12.1 and stopped working after upgrading to 12.2

Packets reply via route map and doesn't consider firewall rules with "reply-to/route-to"

Rule set doesn't matter, because 'reply-to' doesn't work on 2 different machines with 2 different rule set.

I use only IPv4, so, i can't check it for IPv6.

$ uname -a
FreeBSD host1.domain.com 12.2-RELEASE-p2 FreeBSD 12.2-RELEASE-p2 r368704 GW5  amd64

$ uname -a
FreeBSD host2.domain.com 12.2-RELEASE FreeBSD 12.2-RELEASE r367267 kern2  amd64

kernel config host1:

include GENERIC
ident GW5

options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ
options ALTQ_NOPCC
options ALTQ_CDNR
options ALTQ_DEBUG

device pf
device pflog

device                coretemp

maxusers        512
options         NBUF=4096

options ROUTETABLES=15


kernel config host2:

include         GENERIC
ident           kern2

options ALTQ
options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ
options ALTQ_NOPCC
options ALTQ_CDNR
options ALTQ_DEBUG

device pf
device pflog

maxusers        512

device                coretemp

options         ROUTETABLES=15
Comment 1 Kristof Provost freebsd_committer freebsd_triage 2021-02-01 15:13:52 UTC
Please post full ruleset(s) and explain what 'does not work' mean in this context.
What packets are (not) sent and what packets are expected to (not) be sent?

"It's broken" is not generally a useful bug report without a lot more detail.
Comment 2 forumforeign 2021-02-01 15:31:22 UTC
pass in on $ext_if_1 reply-to ($ext_if_1 $gw_1) inet proto tcp to ($ext_if_1) port { $tcp_svc } tag EXT_IF_A
pass in on $ext_if_1 inet proto tcp from ($ext_if_1:network) to ($ext_if_1) port { $tcp_svc } tag EXT_IF_A
# Lan4ever
pass in on $ext_if_2 reply-to ($ext_if_2 $gw_2) inet proto tcp to ($ext_if_2) port { $tcp_svc } tag EXT_IF_B
pass in on $ext_if_2 inet proto tcp from ($ext_if_2:network) to ($ext_if_2) port { $tcp_svc } tag EXT_IF_B

pass in quick from ($ext_if_1:network) tagged EXT_IF_A keep state
pass in quick reply-to ($ext_if_1 $gw_1) tagged EXT_IF_A keep state

pass in quick from ($ext_if_2:network) tagged EXT_IF_B keep state
pass in quick reply-to ($ext_if_2 $gw_2) tagged EXT_IF_B keep state

pass out route-to ($ext_if_1 $gw_1) inet from ($ext_if_1) keep state
pass out route-to ($ext_if_2 $gw_2) inet from ($ext_if_2) keep state

This rule set for TCP, but the same behaviour and for UDP too. When I use tcpdump, reply packets appear on the interface, which point to the default GW.

On the FreeBSD 12.1 reply packets appears on the interface which they came from.

Do you need more details?
Comment 3 Kristof Provost freebsd_committer freebsd_triage 2021-02-01 15:32:36 UTC
(In reply to skeletor from comment #2)
Yes.

(a) **FULL** rule sets
(b) a description of what happens and what's supposed to happen.

"It's broken" is not an actionable bug report.
Comment 4 forumforeign 2021-02-01 15:53:31 UTC
ext_if_2="igb0"
ext_if_1="bge0"
int_if="vlan12"
vlan1920_net="192.168.0.0/24"

lo="lo0"
int_net="10.11.12.0/24"

gw_2="BB.BB.BB.YY"
gw_1="AA.AA.AA.YY"

# services
tcp_svc="ssh, 53, 110,143,993,995,25,465"
udp_svc="53"

#skip iface
set skip on lo0
set block-policy drop
set limit states 300000
set limit frags 300000
set limit src-nodes 300000

# scrub
scrub in on $ext_if_1 all
scrub in on $ext_if_2 all
scrub in on $int_if all fragment reassemble max-mss 1496

# NAT

nat on $ext_if_1 inet from {$int_net} -> $ext_if_1:0
nat on $ext_if_2 inet from {$int_net} -> $ext_if_2:0

########## FIREWALL
block in quick from any os NMAP

block quick on $ext_if_2 proto udp from any to $ext_if_2 port 161
block quick on $ext_if_2 proto tcp from any to $ext_if_2 port { 199, 179 }

pass in

pass quick from <whitelist> to any

#bruteforce
block drop quick from <badhosts> to any label "ssh bruteforce"
block drop quick from any to <badhosts> label "ssh bruteforce"

# ICMP
pass in on $ext_if_1 reply-to ($ext_if_1 $gw_1) inet proto icmp to ($ext_if_1) tag EXT_IF_A icmp-type echoreq code 0
pass in on $ext_if_1 inet proto icmp from ($ext_if_1:network) to ($ext_if_1) icmp-type echoreq code 0
pass in on $ext_if_2 reply-to ($ext_if_2 $gw_2) inet proto icmp to ($ext_if_2) tag EXT_IF_B icmp-type echoreq code 0
pass in on $ext_if_2 inet proto icmp from ($ext_if_2:network) to ($ext_if_2) icmp-type echoreq code 0

# allow tcp ports
pass in on $ext_if_1 reply-to ($ext_if_1 $gw_1) inet proto tcp to ($ext_if_1) port { $tcp_svc } tag EXT_IF_A
pass in on $ext_if_1 inet proto tcp from ($ext_if_1:network) to ($ext_if_1) port { $tcp_svc } tag EXT_IF_A
pass in on $ext_if_2 reply-to ($ext_if_2 $gw_2) inet proto tcp to ($ext_if_2) port { $tcp_svc } tag EXT_IF_B
pass in on $ext_if_2 inet proto tcp from ($ext_if_2:network) to ($ext_if_2) port { $tcp_svc } tag EXT_IF_B

# allow udp ports
pass in on $ext_if_1 reply-to ($ext_if_1 $gw_1) inet proto udp to ($ext_if_1) port { $udp_svc } tag EXT_IF_A
pass in on $ext_if_1 inet proto udp from ($ext_if_1:network) to ($ext_if_1) port { $udp_svc } tag EXT_IF_A
pass in on $ext_if_2 reply-to ($ext_if_2 $gw_2) inet proto udp to ($ext_if_2) port { $udp_svc } tag EXT_IF_B
pass in on $ext_if_2 inet proto udp from ($ext_if_2:network) to ($ext_if_2) port { $udp_svc } tag EXT_IF_B

pass in quick from ($ext_if_1:network) tagged EXT_IF_A keep state
pass in quick reply-to ($ext_if_1 $gw_1) tagged EXT_IF_A keep state

pass in quick from ($ext_if_2:network) tagged EXT_IF_B keep state
pass in quick reply-to ($ext_if_2 $gw_2) tagged EXT_IF_B keep state

pass quick from <full_access> to any
pass quick from any to <full_access>

pass in quick from <whitelist> to any
pass out quick from any to <whitelist>

## ssh access
pass in quick proto tcp from any to any port ssh flags S/SA keep state \
        (max-src-conn 3, max-src-conn-rate 3/5, overload <badhosts> flush global)

#snmp
pass quick proto udp from $monitoring to any port 161
pass quick proto tcp from $monitoring to any port 199
pass quick proto icmp from $monitoring to any
block quick proto udp from any to any port 161
block quick proto tcp from any to any port 199

pass out inet from (self:network)
pass in inet proto icmp to (self:network)
pass in inet proto tcp from $int_net to (self:network)
pass in inet proto udp from $int_net to (self:network)

# LOCAL NETWORK
pass quick on $int_if

# OUTGOING ROUTE
pass out route-to ($ext_if_1 $gw_1) inet from ($ext_if_1) keep state
pass out route-to ($ext_if_2 $gw_2) inet from ($ext_if_2) keep state

pass out inet from { $ext_if_1 $ext_if_2 } to (self:network)

Here query on ext_if2:

17:40:42.342576 IP XX.XX.XX.XX.60318 > BB.BB.BB.BB.22: Flags [S], seq 2980536276, win 64240, options [mss 1460,sackOK,TS val 824407439 ecr 0,nop,wscale 7], length 0
17:40:43.345460 IP XX.XX.XX.XX.60318 > BB.BB.BB.BB.22: Flags [S], seq 2980536276, win 64240, options [mss 1460,sackOK,TS val 824408442 ecr 0,nop,wscale 7], length 0

Here reply on ext_if1:

17:40:42.342717 IP BB.BB.BB.BB.22 > XX.XX.XX.XX.60318: Flags [S.], seq 2922590039, ack 2980536277, win 64395, options [mss 1460,nop,wscale 6,nop,nop,TS val 1012050357 ecr 824407439], length 0
17:40:43.342422 IP BB.BB.BB.BB.22 > XX.XX.XX.XX.60318: Flags [S.], seq 2922590039, ack 2980536277, win 64395, options [mss 1460,nop,wscale 6,nop,nop,TS val 1012051357 ecr 824407439], length 0

Where XX.XX.XX.XX - host, from which i try to connect to BB.BB.BB.BB (ip on interface ext_if2)

ext_if1 point to default GW. ext_if2 - secondary ISP.

If not enough - please, let me know.
Comment 5 Kristof Provost freebsd_committer freebsd_triage 2021-02-01 15:56:38 UTC
(In reply to skeletor from comment #4)
Again: explain what happens and what you expect to have happen instead.

I've looked at the 12.1 and 12.2 code and there are no obvious differences in the reply-to code, but I still have no idea at all what your problem is.
Comment 6 forumforeign 2021-02-01 16:02:01 UTC
I expect, that, if I send packets to secondary (ext_if_2) interface (not to point default GW) I receive it from ext_if_2. But really I see it on ext_if_1 and don't see on ext_if_2 (but should see).
Comment 7 Kristof Provost freebsd_committer freebsd_triage 2021-02-01 16:03:05 UTC
(In reply to skeletor from comment #6)
Send packets from where? From the machine? From the internet?
Comment 8 forumforeign 2021-02-01 16:12:05 UTC
Yes, from outside (some host in internet).
Comment 9 Kristof Provost freebsd_committer freebsd_triage 2021-02-01 16:23:55 UTC
(In reply to skeletor from comment #8)
Have you confirmed, with pflog, that the route-to rule is still getting hit?
Comment 10 forumforeign 2021-02-02 15:21:50 UTC
Thank you for right direction to check via pflog! Without "quick" keyword rule set with reply-to doesn't work. Now, everything is working.

Sorry, for my mistakes.