Bug 212000 - 11.0-RC1: vimage jail with ipfilter not working
Summary: 11.0-RC1: vimage jail with ipfilter not working
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 11.0-RC1
Hardware: Any Any
: --- Affects Only Me
Assignee: Cy Schubert
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-08-19 17:19 UTC by Joe Barbish
Modified: 2018-11-24 18:37 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joe Barbish 2016-08-19 17:19:54 UTC
Tested on 11.0-RC1 with only vimage compiled into the kernel.
Tested ipfilter in vnet jail and no firewall on host.
Tested ipfilter in vnet jail and on the host.

Vnet jail used this /etc/devfs.rules rule

[devfsrules_vjail_ipf=60]
add include $devfsrules_jail
add path ipl     unhide
add path ipl0    unhide
add path ipf     unhide
add path ipauth  unhide
add path ipnat   unhide
add path ipstate unhide
# used by ipstate
#add path kmem    unhide
#add path kernel  unhide


Testing no ipfilter firewall running on host, just in vnet jail.

When starting vnet jail with ipfilter, I check if ipfilter kernel modules are
loaded, if not them loads them. Auto loading of modules does not happen.

Issuing the ipfilter command "ipfstat -hnoi" from the started vnet jail 
console show this

0 @1 pass out quick on lo0 all
0 @2 block out log quick on epair17b proto tcp from any to any port = nicname
0 @3 pass out log quick on epair17b all
0 @1 pass in quick on lo0 all
0 @2 pass in log quick on epair17b all

There are 0 counts because the ipstate command is restricted from accessing
kmem & kernel. 
But this at lease seems to prove ipfilter is running in the vnet jail.

Issuing the "ping" command from the started vnet jail console works.

Issuing the "whois" command from the started vnet jail console works also,
but should not work because of the above block rule on port 43.

This indicates that the ipfilter rules in a vnet jail are not functioning.
No ipfilter log messages are posted in the vnat jail and no log messages
are posted in the hosts log.


Testing ipfilter firewall running on host and vnet jail.
Issuing the ipfilter command "ipfstat -hnoi" from the host console show this

0 @1 pass out quick on lo0 all
0 @2 pass out log quick on fxp0 all
0 @1 pass in quick on lo0 all
1 @2 pass in log quick on fxp0 all

The vnet jail results are the same as above.
But the hosts ipfilter log, logs this on vnet jail startup and keeps repeating
it for the whole time the vnet jail is running. 

fxp0 @0:2 p 10.0.10.2,67 -> 10.0.10.12,68 PR udp len 20 328 IN
fxp0 @0:2 p :: -> ff02::1:ff00:50b PR icmpv6 len 40 72 icmpv6 neighborsolicit/0 OUT multicast
fxp0 @0:2 p :: -> ff02::16 PR icmpv6 len 48 76 icmpv6 icmpv6type(143)/0 OUT multicast
fxp0 @0:2 p :: -> ff02::16 PR icmpv6 len 48 96 icmpv6 icmpv6type(143)/0 OUT multicast
fxp0 @0:2 p :: -> ff02::16 PR icmpv6 len 48 76 icmpv6 icmpv6type(143)/0 OUT multicast
fxp0 @0:2 p fe80::c1:ff:fe00:50b -> ff02::16 PR icmpv6 len 48 96 icmpv6 icmpv6type(143)/0 OUT multicast
fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast
fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast
fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast
fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast
fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast
fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast
fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast
fxp0 @0:2 p 10.0.10.7,68 -> 255.255.255.255,67 PR udp len 20 328 IN broadcast

Issuing the "ping" command from the started vnet jail console works and the
hosts ipfilter log shows this

fxp0 @0:2 p 10.0.10.7,68 -> 255.255.255.255,67 PR udp len 20 328 IN broadcast
fxp0 @0:2 p 10.11.0.2 -> 8.8.8.8 PR icmp len 20 84 icmp echo/0 OUT
fxp0 @0:2 p 8.8.8.8 -> 10.11.0.2 PR icmp len 20 84 icmp echoreply/0 IN
fxp0 @0:2 p 10.11.0.2 -> 8.8.8.8 PR icmp len 20 84 icmp echo/0 OUT
fxp0 @0:2 p 8.8.8.8 -> 10.11.0.2 PR icmp len 20 84 icmp echoreply/0 IN
fxp0 @0:2 p 10.11.0.2 -> 8.8.8.8 PR icmp len 20 84 icmp echo/0 OUT
fxp0 @0:2 p 8.8.8.8 -> 10.11.0.2 PR icmp len 20 84 icmp echoreply/0 IN
fxp0 @0:2 p 10.11.0.2 -> 8.8.8.8 PR icmp len 20 84 icmp echo/0 OUT
fxp0 @0:2 p 8.8.8.8 -> 10.11.0.2 PR icmp len 20 84 icmp echoreply/0 IN

The hosts ipfilter firewall is logging the traffic on the fxp0 interface,
this is normal and expected.

I see 4 things that are strange.
1. Why is the vnet jail issuing all that ipv6 traffic? It should only be
   generated if the vnet jail interface has a ipv6 ip address coded.
2. Why is ipv4 & ipv6 traffic making it to the host and NOT showing up on
   the epair11b interface? 
   This is a problem if the host has a few vnet jails running at same time.
   IE: how am I going to control traffic on host to target correct vnet jail.
   In my case I use the epair number [11] in the ip address of the vnet jail.
3. Why are the ipfilter rules in the vnet jail not being enforced?
4. Why in the case of no firewall on the host and ipfilter in the vnet jail has
   no logging any place?
Comment 1 Joe Barbish 2018-11-20 16:25:08 UTC
12.0-RC1 and still have this problem.
Comment 2 Cy Schubert freebsd_committer 2018-11-24 06:23:34 UTC
I've set up an environment like yours on my testbed. Confirmed the problem exists locally.
Comment 3 Joe Barbish 2018-11-24 17:28:05 UTC
This is the network setup test system config I used to verify the 11.0-RC1 reported bug still exists in 12.0-RC1. Its good to know you confirmed it.

This is configuration of my test system. It's running 12.0-RC1 i386 and is on LAN behind my production gateway host which is running 11.2 amd64 with ipf firewall.

Just reran this test configuration on the test system. 
NO firewall on test system host. kldload ipl.ko from host console. Vnet jail using bridge/epair with ipf configured using normal parameters in vnet jail's rc.conf. "Local0  /var/log/ipf.log" line in vnet jails /etc/syslog.conf to allow ipf logging. Using same /etc/devfs.rules file as posted previously.

Rules file contains only 2 lines
block out log quick all
pass in log all

From the vnet jail console "ipfstat -hnoi" shows the 2 rules with zero count. Then issue ping -c2 8.8.8.8 and receive the normal reply that all packets returned. 
"ipfstat -hnoi" shows the 2 rules still have zero count.

If ipf running in the vnet jail was functional, the ping would have been blocked and logged which did not happen. 

It's important that ipf logging populate it's log file in the vnet jail ipf is running in. 

Thanks for working on this bug.
Comment 4 Cy Schubert freebsd_committer 2018-11-24 18:37:00 UTC
No problem. Your setup is similar to mine. My testbed has an unused interface to my DMZ for this purpose. Rather than port 43 I verified using port 22 to my prod firewall in the DMZ. I suspect a global variable wasn't converted to V_ in 2016 by r302298 & friends.

Logging is another thing that needs to be looked at.