Bug 233581 - Bugg in PF or in PF man-page?
Summary: Bugg in PF or in PF man-page?
Status: Closed Works As Intended
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.2-STABLE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-pf mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-11-27 21:30 UTC by peos42
Modified: 2018-12-04 22:00 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description peos42 2018-11-27 21:30:45 UTC
Bugg in PF or in PF man-page? I vote for bug in PF itself…

Man page says that “set skip on lo0” should ignore all traffic over lo0. This is not true. It just ignores 127* traffic.


Let us assume the FreeBSD 11.2 host has IP 1.2.3.4. I created a jail and installed DNS/bind in it. The jail uses share IP with host (i.e no vnet recompiled kernel)… As there is no 127.0.0.1 I had to reconfigure rndc to listen and use 1.2.3.4 instead of 127.0.0.1 in the jail. I then noted that rndc did not work.

In the FreeBSD main host pf.conf I had to explicit add a pass rule to allow 1.2.3.4 to 1.2.3.4 on lo0. Using tcpdump listening on lo0 I could also see the rndc traffic from 1.2.3.4 to 1.2.3.4 was going over lo0.

So “set skip lo0” does not work as man page says which is…

—snip—
    set skip on <ifspec>
           List interfaces for which packets should not be filtered.  Packets
           passing in or out on such interfaces are passed as if pf was
           disabled, i.e. pf does not process them in any way.  This can be
           useful on loopback and other virtual interfaces, when packet
           filtering is not desired and can have unexpected effects.  For
           example:

                 set skip on lo0
—snip—


Now… I have not used FreeBSD that much. Especially not with jails. Have I missed something obvious and is too quick to log this? Otherwise, please enlighten me :)
Comment 1 Kristof Provost freebsd_committer 2018-11-28 11:39:28 UTC
Please include a full pf.conf, and ifconfig output.

There are some issues around set skip handling in 11.2 (although I believe those are limited to set skip <group>). It's worth re-testing this on head.
Comment 2 peos42 2018-11-28 12:09:14 UTC
vtnet0 has one IPv4 and and one IPv6 address on it. All jails use shared IP stack. No vnet...

This is a snippet from the pf.conf in the main host. As this is a new host, I have not had the time to add variables into it...

Let us assume the vtnet0 IPv4 IP is 1.2.3.4

--snip--
set skip on lo0
block all
pass out quick on { lo0 vtnet0 } inet proto {tcp gre esp udp icmp ipv6} all keep state
pass out quick on { lo0 vtnet0 } inet6  proto {tcp gre esp udp icmp6} all keep state
pass out quick on { lo0 vtnet0 } inet6 all keep state
-- REST OF PASS RULES CUT OUT --
pass in quick on lo0 inet proto tcp from 1.2.3.4 to 1.2.3.4 port 953 flags S/SAFR keep state
--snip--


In the "DNS" jail, rndc refuse to work unless the last rule within the snippet above is added. So named.conf in the jail has controls to the 1.2.3.4 IP and rndc.conf has default listener conf set to 1.2.3.4 instead of 127.0.0.1


/Peo
Comment 3 Kristof Provost freebsd_committer 2018-12-01 15:36:57 UTC
'set skip' merely means do not run pf on this interface. It is not an 'allow all traffic from this interface' rule.

In this case your 'set skip' rule means that traffic from lo0 is not blocked, but your block all rule is stopping it from being accepted on *vtnet0*, where your IP address is assigned. You do need the rule to actually accept traffic.
Comment 4 peos42 2018-12-01 18:20:13 UTC
Hmmm

From man page regarding "set skip"....
--snip--
List interfaces for which packets should not be filtered.  Packets
           passing in or out on such interfaces are passed as if pf was
           disabled, i.e. pf does not process them in any way.
--snip--

I think the text is clear in the man page... Packets are passed as if PF was disabled. It also states that PF should not process them in any way. 

Two comments on this..

# 1
If a PF default block rule blocks traffic on lo0 for me when "set skip on lo0" on active. Then PF *IS* processing packages which the man page clearly say it should not!

# 2
Lets assume you are right... Then the default block rule should also block 127.0.0.1 over lo0 so that as well have to be explicitly allowed. But it does not!


So you comment.... Well... I do not agree. But please tell me if I misinterpret something.

/Peo
Comment 5 peos42 2018-12-01 18:24:30 UTC
Seems I cannot edit previous post. So here is an addition..

You say..

--snip--
but your block all rule is stopping it from being accepted on *vtnet0*, where your IP address is assigned. You do need the rule to actually accept traffic.
--snip--

If you read my post I had to add a **pass in quick on lo0**  an NOT vtnet0

Peo
Comment 6 Kristof Provost freebsd_committer 2018-12-01 18:34:45 UTC
Your packet goes out lo0 and in vtnet0. So without the pass rule it’s blocked.
Comment 7 peos42 2018-12-01 18:39:48 UTC
I added this --ONLY-- to get it working...

pass in quick on lo0 inet proto tcp from 1.2.3.4 to 1.2.3.4 port 953 flags S/SAFR keep state

That is...
Pass *IN* on lo0 and *NOT* vtnet0



So please explain your last comment so I understand.

Regards
Peo
Comment 8 peos42 2018-12-02 23:43:28 UTC
I have now tested on my primary firewall that is OpenBSD 6.4. 

There I have "set skip on lo0". And I can in the firewall ping with the source IP of one of my interfaces and the target IP the set to the same. And I do not have to add a pass rule from that IP, to that IP, on lo0.

On FreeBSD as stated I have to add a pass rule for incoming traffic on lo0 for this to work even though "set skip on lo0" is in the ruleset.



So it works on OpenBSD... I guess that supports what I am saying.

/Peo
Comment 9 Kristof Provost freebsd_committer 2018-12-03 06:54:03 UTC
(In reply to peos42 from comment #8)
Please include full pf.conf, ifconfig output and command line.
Have you tested this on head as well?
Comment 10 peos42 2018-12-04 21:32:19 UTC
Have not tested on head. Is something fixed regarding this?


Config posted below as requested. Note that IPv4 and IPv6 addresses are substituted to fake. 


#######################
### FROM MAIN HOST ####
#######################
22:09:30 huey:~ # ifconfig -a
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:16:3c:7f:67:0e
	hwaddr 00:16:3c:7f:67:0e
	inet 1.2.3.4 netmask 0xffffff00 broadcast 1.2.3.255 
	inet6 fe80::216:3cff:fe7f:670e%vtnet0 prefixlen 64 scopeid 0x1 
	inet6 2222:3333:6:6df::1111 prefixlen 48 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
	groups: pflog 
22:09:32 huey:~ # 


Note that the PF below will be rebuilt from scratch with variables and tagging etc. But for this case it doesn't matter....

22:10:21 huey:~ # more /etc/pf.conf |grep -v ^#|sed '/^$/d'
set skip on lo0
block all
pass out quick on { lo0 vtnet0 } inet proto {tcp gre esp udp icmp ipv6} all keep state
pass out quick on { lo0 vtnet0 } inet6  proto {tcp gre esp udp icmp6} all keep state
pass out quick on { lo0 vtnet0 } inet6 all keep state
antispoof quick for vtnet0
pass in log quick on vtnet0 inet proto icmp from any to vtnet0 icmp-type { 8 code 0 , 3 code 3 , 11 code 0  } keep state
pass in quick on vtnet0 inet6 proto { ipv6-icmp } from any to any keep state
block in log quick on vtnet0 proto tcp from <bad_hosts_SSH_MAIN_HOST> to  vtnet0 port { 22 }
pass in log quick on vtnet0 inet proto tcp  from any to vtnet0 port { 22 } flags S/SAFR keep state (max 100, max-src-nodes 30, max-src-states 30, max-src-conn 20, max-src-conn-rate 3/30, overload <bad_hosts_SSH_MAIN_HOST> flush global)
pass in log quick on vtnet0 inet6 proto tcp  from any to vtnet0 port { 22 } flags S/SAFR keep state (max 100, max-src-nodes 30, max-src-states 30, max-src-conn 20, max-src-conn-rate 3/30, overload <bad_hosts_SSH_MAIN_HOST> flush global)
block in log quick on vtnet0 proto tcp from <bad_hosts_SSH_DNS_HOST> to  vtnet0 port { 10022 }
pass in log quick on vtnet0 inet proto tcp  from any to vtnet0 port { 10022 } flags S/SAFR keep state (max 100, max-src-nodes 30, max-src-states 30, max-src-conn 20, max-src-conn-rate 3/30, overload <bad_hosts_SSH_DNS_HOST> flush global)
pass in log quick on vtnet0 inet6 proto tcp  from any to  vtnet0 port { 10022 } flags S/SAFR keep state (max 100, max-src-nodes 30, max-src-states 30, max-src-conn 20, max-src-conn-rate 3/30, overload <bad_hosts_SSH_DNS_HOST> flush global)
pass in quick on vtnet0 inet proto tcp  from any to vtnet0 port { 53 } flags S/SAFR keep state
pass in quick on vtnet0 inet6 proto tcp  from any to  vtnet0 port { 53 } flags S/SAFR keep state
pass in quick on vtnet0 inet proto udp  from any to vtnet0 port { 53 }  keep state
pass in quick on vtnet0 inet6 proto udp  from any to  vtnet0  port { 53 }  keep state
pass in quick on lo0 inet proto tcp from 1.2.3.4 to 1.2.3.4 port 953 flags S/SAFR keep state
block in log quick on vtnet0 proto tcp from <bad_hosts_SSH_MAIL_HOST> to  vtnet0 port { 20022 }
pass in log quick on vtnet0 inet proto tcp  from any to vtnet0 port { 20022 } flags S/SAFR keep state (max 100, max-src-nodes 30, max-src-states 30, max-src-conn 20, max-src-conn-rate 3/30, overload <bad_hosts_SSH_MAIL_HOST> flush global)
pass in log quick on vtnet0 inet6 proto tcp  from any to  vtnet0 port { 20022 } flags S/SAFR keep state (max 100, max-src-nodes 30, max-src-states 30, max-src-conn 20, max-src-conn-rate 3/30, overload <bad_hosts_SSH_MAIL_HOST> flush global)
pass in log quick on vtnet0 inet proto tcp  from any to vtnet0 port { 25 465 587 } flags S/SAFR keep state
pass in log quick on vtnet0 inet6 proto tcp  from any to  vtnet0 port { 25 465 587 } flags S/SAFR keep state
block in log quick on vtnet0 proto tcp from <bad_hosts_SSH_WEB_HOST> to  vtnet0 port { 30022 }
pass in log quick on vtnet0 inet proto tcp  from any to vtnet0 port { 30022 } flags S/SAFR keep state (max 100, max-src-nodes 30, max-src-states 30, max-src-conn 20, max-src-conn-rate 3/30, overload <bad_hosts_SSH_WEB_HOST> flush global)
pass in log quick on vtnet0 inet6 proto tcp  from any to  vtnet0 port { 30022 } flags S/SAFR keep state (max 100, max-src-nodes 30, max-src-states 30, max-src-conn 20, max-src-conn-rate 3/30, overload <bad_hosts_SSH_WEB_HOST> flush global)
pass in log quick on vtnet0 inet proto tcp  from any to vtnet0 port { 80 443 } flags S/SAFR keep state
pass in log quick on vtnet0 inet6 proto tcp  from any to  vtnet0 port { 80 443 } flags S/SAFR keep state
22:10:24 huey:~ # 



###########################
### FROM DNS JAIL HOST ####
############################


"rndc reload" does NOT work in this jail if the following pf.conf row is removed from the main host...

pass in quick on lo0 inet proto tcp from 1.2.3.4 to 1.2.3.4 port 953 flags S/SAFR keep state

On OpenBSD this is not needed as "set skip on lo0" works... But all this I have already written in earlier posts.



22:11:25 DNS:~ # ifconfig -a
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 00:16:3c:7f:67:0e
	hwaddr 00:16:3c:7f:67:0e
	inet 1.2.3.4 netmask 0xffffff00 broadcast 1.2.3.255 
	inet6 2222:3333:6:6df::1111 prefixlen 48 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
	groups: pflog 
22:11:27 DNS:~ # 


22:13:24 DNS:~ # more /usr/local/etc/namedb/rndc.conf |grep default-server
	default-server 1.2.3.4;
22:13:25 DNS:~ # 

22:13:26 DNS:~ # more /usr/local/etc/namedb/named.conf |grep 953
 	inet 1.2.3.4 port 953 allow { 1.2.3.4; 127.0.0.1; 2222:3333:5:6df::1111; } keys { "rndc-key"; };
22:13:31 DNS:~ #
Comment 11 Kristof Provost freebsd_committer 2018-12-04 21:35:13 UTC
(In reply to peos42 from comment #10)
Yes, there have been changes around set skip handling (mostly for groups). See comment #1.

Do you have a smaller test case?
Comment 12 peos42 2018-12-04 22:00:26 UTC
Hi Kristof

This is small :)

However... To try to give you something smaller I started by removing the 
pass in quick on lo0 inet proto tcp from 1.2.3.4 to 1.2.3.4 port 953 flags S/SAFR keep state

from the main host pf.conf and reloaded PF. The weird thing is that "rndc relosd" still works in the jail. I restarted the jail and it still works. As it should!!! The ONLY thing I have done since my initial post where it didn't work is to upgrade host and jail from 11.2-p4 to 11.2p5. ?????????????