Bug 212013 - 11.0-RC1: vimage jail with pf not working
Summary: 11.0-RC1: vimage jail with pf not working
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-RC1
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-08-20 14:14 UTC by Joe Barbish
Modified: 2018-10-19 23:32 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joe Barbish 2016-08-20 14:14:51 UTC
Tested on 11.0-RC1 with only vimage compiled into the kernel.
Tested pf in vnet jail and no firewall on host.
Tested pf in vnet jail and on the host.

Vnet jail used this /etc/devfs.rules rule

[devfsrules_vjail_pf=70]
add include $devfsrules_jail
add path pf     unhide
add path pfsync unhide
add path pflog  unhide


Testing no pf firewall running on host, just in vnet jail.

When starting vnet jail with pf, I check if pf kernel modules are
loaded, if not them load them. Auto loading of modules does not happen.

Issuing "ps ax" command on the host show this for pf,
 925  -  DL    0:00.35 [pf purge]

Issuing "ifconfig -a" command from the started vnet jail console shows

pflog0: flags=0<> metric 0 mtu 33184
        groups: pflog

The pf log file is allocated but not enabled in the vnet jail.

Issuing the pf command "pfctl -sr -vv" from the started vnet jail
console show this

No ALTQ support in kernel
ALTQ related functions disabled
@0 block drop in quick on epair23b inet proto tcp from any to any port = nicname
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 1301 State Creations: 0     ]
@1 pass log (all) quick on epair23b all flags S/SA keep state
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 1301 State Creations: 0     ]

This at lease seems to indicate pf is running in the vnet jail.

Issuing the "ping" command from the started vnet jail console works.

Issuing the "whois" command from the started vnet jail console works also,
but should not work because of the above block rule on port 43.

This indicates that the pf rules in a vnet jail are not functioning.
No pf log messages are posted in the vnet jail and no log messages
are posted in the hosts pf log.


Testing pf firewall running on host and vnet jail.

After booting host, Issuing "ps ax" command on the host show this for pf,
393  -  DL   0:00.07 [pf purge]
406  -  Is   0:00.01 pflogd: [priv] (pflogd)
409  -  S    0:00.00 pflogd: [running] -s 116 -i pflog0 -f /var/log/pflog (pflo

Issuing "ifconfig -a" command from the started vnet jail console shows

pflog0: flags=0<> metric 0 mtu 33184
        groups: pflog

The pf log file is allocated but not enabled in the vnet jail.

Issuing the pf command "pfctl -sr -vv" from the started vnet jail
console shows this

No ALTQ support in kernel
ALTQ related functions disabled
@0 block drop in quick on epair23b inet proto tcp from any to any port = nicname
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 1165 State Creations: 0     ]
@1 pass log (all) quick on epair23b all flags S/SA keep state
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 1165 State Creations: 0     ]

Issuing the pf command "pfctl -sr -vv" from the host console shows this
No ALTQ support in kernel
ALTQ related functions disabled
@0 pass log (all) quick on fxp0 all flags S/SA keep state
  [ Evaluations: 24        Packets: 9         Bytes: 740         States: 0     ]
  [ Inserted: uid 0 pid 411 State Creations: 3     ]


The vnet jail results are the same as above.

But the hosts pf log, logs this on vnet jail startup. 

pass out on fxp0: :: > ff02::16: HBH ICMP6, multicast listener report v2[|icmp6]
pass out on epair23a: :: > ff02::16: HBH ICMP6, multicast listener report v2[|ic
pass out on fxp0: :: > ff02::1:ff00:60b: ICMP6, neighbor solicitation[|icmp6]
pass out on bridge0: :: > ff02::16: HBH ICMP6, multicast listener report v2[|icm
pass out on fxp0: :: > ff02::16: HBH ICMP6, multicast listener report v2[|icmp6]
pass out on epair23a: :: > ff02::16: HBH ICMP6, multicast listener report v2[|ic
pass out on bridge0: :: > ff02::16: HBH ICMP6, multicast listener report v2[|icm
pass out on fxp0: :: > ff02::16: HBH ICMP6, multicast listener report v2[|icmp6]
pass out on fxp0: fe80::c1:ff:fe00:60b > ff02::16: HBH ICMP6, multicast listener




Issuing the "ping" command from the started vnet jail console works and the
hosts pf log shows this

pass out on fxp0: 10.23.0.2 > 8.8.8.8: ICMP echo request
pass in on fxp0: 8.8.8.8 > 10.23.0.2: ICMP echo reply
pass in on bridge0: 8.8.8.8 > 10.23.0.2: ICMP echo reply
pass out on fxp0: 10.23.0.2 > 8.8.8.8: ICMP echo request
pass in on fxp0: 8.8.8.8 > 10.23.0.2: ICMP echo reply
pass in on bridge0: 8.8.8.8 > 10.23.0.2: ICMP echo reply
pass out on fxp0: 10.23.0.2 > 8.8.8.8: ICMP echo request
pass in on fxp0: 8.8.8.8 > 10.23.0.2: ICMP echo reply
pass in on bridge0: 8.8.8.8 > 10.23.0.2: ICMP echo reply
pass out on fxp0: 10.23.0.2 > 8.8.8.8: ICMP echo request
pass in on fxp0: 8.8.8.8 > 10.23.0.2: ICMP echo reply
pass in on bridge0: 8.8.8.8 > 10.23.0.2: ICMP echo reply

The hosts pf firewall is passing and logging all the traffic just as 
the host firewall single rule says to do.

Issuing the "whois" command from the started vnet jail console works 
and it should not work because the vnet jail pf rules says to
block all outbound traffic going to post 43. The hosts pf log shows this

pass out on fxp0: 10.23.0.2.19180 > 199.212.0.46.43:
pass in on fxp0: 199.212.0.46.43 > 10.23.0.2.19180:
pass in on bridge0: 199.212.0.46.43 > 10.23.0.2.19180:
pass out on fxp0: 10.23.0.2.19180 > 199.212.0.46.43:
pass out on fxp0: 10.23.0.2.19180 > 199.212.0.46.43:
pass in on fxp0: 199.212.0.46.43 > 10.23.0.2.19180:
pass in on bridge0: 199.212.0.46.43 > 10.23.0.2.19180:
pass in on fxp0: 199.212.0.46.43 > 10.23.0.2.19180:
pass in on bridge0: 199.212.0.46.43 > 10.23.0.2.19180:
pass in on fxp0: 199.212.0.46.43 > 10.23.0.2.19180:
pass in on bridge0: 199.212.0.46.43 > 10.23.0.2.19180:
pass out on fxp0: 10.23.0.2.19180 > 199.212.0.46.43:
pass in on fxp0: 199.212.0.46.43 > 10.23.0.2.19180:
pass in on bridge0: 199.212.0.46.43 > 10.23.0.2.19180:
pass in on fxp0: 199.212.0.46.43 > 10.23.0.2.19180:
pass in on bridge0: 199.212.0.46.43 > 10.23.0.2.19180:

Issuing the pf command "pfctl -sr -vv" from the started vnet jail
console shows that the counts have not increased. Another indicator that
the pf firewall in the vnet jail is not functioning.


Problems.
1. Why is the vnet jail issuing all that ipv6 traffic? It should only be
   generated if the vnet jail interface has a ipv6 ip address coded.
2. Why is ipv4 & ipv6 traffic making it to the host and NOT showing up on
   the epair23b interface? 
   This is a problem if the host has a few vnet jails running at same time.
   IE: how am I going to control traffic on host to target correct vnet jail.
   In my case I use the epair number [23] in the ip address of the vnet jail.
3. Why are the pf rules in the vnet jail not being enforced?
4. Why does pf in the vnet jail not work when pf is not run on host?
5. Why does pf in the vnet jail not log to a log file in the vnet jails
   /var/log directory?
Comment 1 Bjoern A. Zeeb freebsd_committer 2016-08-21 00:14:34 UTC
Just in reply to #3 as you say yourself in your description, it's outgoing packets, but your rule inside the jail specifies "in":

0 block drop in quick on epair23b inet proto tcp from any to any port = nicname

Can you change that to "out" and see if it starts working?

Currently on your "in" directions whois packets would originate from src port 43 and thus don't match the dest port 43.
Comment 2 Joe Barbish 2016-08-21 13:36:46 UTC
I changed "in" to "out" in the vnet jail pf rules file. Here is the rules from inside of the vnet jail

pfctl -sr -vv
No ALTQ support in kernel
ALTQ related functions disabled
@0 block drop out quick on epair23b inet proto tcp from any to any port = nicnam
e
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 1171 State Creations: 0     ]
@1 pass log (all) quick on epair23b all flags S/SA keep state
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 1171 State Creations: 0     ]

With pf on the host and in the vnet jail issuing the "whois" command from within the vnet jail still worked, and it should have not worked. The vnet pf firewall rules are not being enforced.

Here is a snip it from the host pf log.

pass out on fxp0: 10.23.0.2.29486 > 192.0.32.59.43:
pass in on fxp0: 192.0.32.59.43 > 10.23.0.2.29486:
pass in on bridge0: 192.0.32.59.43 > 10.23.0.2.29486:
pass out on fxp0: 10.23.0.2.29486 > 192.0.32.59.43:
pass in on fxp0: 192.0.32.59.43 > 10.23.0.2.29486:
pass in on bridge0: 192.0.32.59.43 > 10.23.0.2.29486:
pass out on fxp0: 10.23.0.2.29486 > 192.0.32.59.43:
pass out on fxp0: 10.23.0.2.29486 > 192.0.32.59.43:
pass in on fxp0: 192.0.32.59.43 > 10.23.0.2.29486:
pass in on bridge0: 192.0.32.59.43 > 10.23.0.2.29486:
pass in on fxp0: 192.0.32.59.43 > 10.23.0.2.29486:
pass in on bridge0: 192.0.32.59.43 > 10.23.0.2.29486:

In a net shell nothing changed from the first post.

Those ipv6 packets are still being generated. The following is info for maybe debugging this problem.

This is how I create the epair setup
ifconfig ${nicname} alias 10.${vnetid}.0.1
ifconfig epair${vnetid} create 
ifconfig bridge0 addm epair${vnetid}a
ifconfig epair${vnetid}a up

This is the output of ifconfig -a command on the host after the vnet jail has started.
/root >ifconfig -a
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 15
00
        options=2009<RXCSUM,VLAN_MTU,WOL_MAGIC>
        ether 00:0c:f1:cd:55:ea
        inet 10.0.10.12 netmask 0xfffffff0 broadcast 10.0.10.15
        inet 10.23.0.1 netmask 0xff000000 broadcast 10.255.255.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33184
        groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:8f:94:84:0c:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair23a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: fxp0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
epair23a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mt
u 1500
        options=8<VLAN_MTU>
        ether 02:c1:00:00:05:0a
        inet6 fe80::c1:ff:fe00:50a%epair23a prefixlen 64 scopeid 0x5
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

Here is the output of ifconfig -a command issued from within the started vnet jail.

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
pflog0: flags=0<> metric 0 mtu 33184
        groups: pflog
epair23b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:c1:00:00:06:0b
        inet 10.23.0.2 netmask 0xff000000 broadcast 10.255.255.255
        inet6 fe80::c1:ff:fe00:60b%epair23b prefixlen 64 scopeid 0x3
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
Comment 3 Kristof Provost freebsd_committer 2018-10-19 23:32:37 UTC
vnet is supported in 12, and pf works both on host and in vnet jails there.