Tested on 11.0-RC1 with only vimage compiled into the kernel. Tested ipfilter in vnet jail and no firewall on host. Tested ipfilter in vnet jail and on the host. Vnet jail used this /etc/devfs.rules rule [devfsrules_vjail_ipf=60] add include $devfsrules_jail add path ipl unhide add path ipl0 unhide add path ipf unhide add path ipauth unhide add path ipnat unhide add path ipstate unhide # used by ipstate #add path kmem unhide #add path kernel unhide Testing no ipfilter firewall running on host, just in vnet jail. When starting vnet jail with ipfilter, I check if ipfilter kernel modules are loaded, if not them loads them. Auto loading of modules does not happen. Issuing the ipfilter command "ipfstat -hnoi" from the started vnet jail console show this 0 @1 pass out quick on lo0 all 0 @2 block out log quick on epair17b proto tcp from any to any port = nicname 0 @3 pass out log quick on epair17b all 0 @1 pass in quick on lo0 all 0 @2 pass in log quick on epair17b all There are 0 counts because the ipstate command is restricted from accessing kmem & kernel. But this at lease seems to prove ipfilter is running in the vnet jail. Issuing the "ping" command from the started vnet jail console works. Issuing the "whois" command from the started vnet jail console works also, but should not work because of the above block rule on port 43. This indicates that the ipfilter rules in a vnet jail are not functioning. No ipfilter log messages are posted in the vnat jail and no log messages are posted in the hosts log. Testing ipfilter firewall running on host and vnet jail. Issuing the ipfilter command "ipfstat -hnoi" from the host console show this 0 @1 pass out quick on lo0 all 0 @2 pass out log quick on fxp0 all 0 @1 pass in quick on lo0 all 1 @2 pass in log quick on fxp0 all The vnet jail results are the same as above. But the hosts ipfilter log, logs this on vnet jail startup and keeps repeating it for the whole time the vnet jail is running. fxp0 @0:2 p 10.0.10.2,67 -> 10.0.10.12,68 PR udp len 20 328 IN fxp0 @0:2 p :: -> ff02::1:ff00:50b PR icmpv6 len 40 72 icmpv6 neighborsolicit/0 OUT multicast fxp0 @0:2 p :: -> ff02::16 PR icmpv6 len 48 76 icmpv6 icmpv6type(143)/0 OUT multicast fxp0 @0:2 p :: -> ff02::16 PR icmpv6 len 48 96 icmpv6 icmpv6type(143)/0 OUT multicast fxp0 @0:2 p :: -> ff02::16 PR icmpv6 len 48 76 icmpv6 icmpv6type(143)/0 OUT multicast fxp0 @0:2 p fe80::c1:ff:fe00:50b -> ff02::16 PR icmpv6 len 48 96 icmpv6 icmpv6type(143)/0 OUT multicast fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast fxp0 @0:2 p fe80::d950:d6dc:db92:f20d,546 -> ff02::1:2,547 PR udp len 40 137 IN low-ttl multicast fxp0 @0:2 p 10.0.10.7,68 -> 255.255.255.255,67 PR udp len 20 328 IN broadcast Issuing the "ping" command from the started vnet jail console works and the hosts ipfilter log shows this fxp0 @0:2 p 10.0.10.7,68 -> 255.255.255.255,67 PR udp len 20 328 IN broadcast fxp0 @0:2 p 10.11.0.2 -> 8.8.8.8 PR icmp len 20 84 icmp echo/0 OUT fxp0 @0:2 p 8.8.8.8 -> 10.11.0.2 PR icmp len 20 84 icmp echoreply/0 IN fxp0 @0:2 p 10.11.0.2 -> 8.8.8.8 PR icmp len 20 84 icmp echo/0 OUT fxp0 @0:2 p 8.8.8.8 -> 10.11.0.2 PR icmp len 20 84 icmp echoreply/0 IN fxp0 @0:2 p 10.11.0.2 -> 8.8.8.8 PR icmp len 20 84 icmp echo/0 OUT fxp0 @0:2 p 8.8.8.8 -> 10.11.0.2 PR icmp len 20 84 icmp echoreply/0 IN fxp0 @0:2 p 10.11.0.2 -> 8.8.8.8 PR icmp len 20 84 icmp echo/0 OUT fxp0 @0:2 p 8.8.8.8 -> 10.11.0.2 PR icmp len 20 84 icmp echoreply/0 IN The hosts ipfilter firewall is logging the traffic on the fxp0 interface, this is normal and expected. I see 4 things that are strange. 1. Why is the vnet jail issuing all that ipv6 traffic? It should only be generated if the vnet jail interface has a ipv6 ip address coded. 2. Why is ipv4 & ipv6 traffic making it to the host and NOT showing up on the epair11b interface? This is a problem if the host has a few vnet jails running at same time. IE: how am I going to control traffic on host to target correct vnet jail. In my case I use the epair number [11] in the ip address of the vnet jail. 3. Why are the ipfilter rules in the vnet jail not being enforced? 4. Why in the case of no firewall on the host and ipfilter in the vnet jail has no logging any place?
12.0-RC1 and still have this problem.
I've set up an environment like yours on my testbed. Confirmed the problem exists locally.
This is the network setup test system config I used to verify the 11.0-RC1 reported bug still exists in 12.0-RC1. Its good to know you confirmed it. This is configuration of my test system. It's running 12.0-RC1 i386 and is on LAN behind my production gateway host which is running 11.2 amd64 with ipf firewall. Just reran this test configuration on the test system. NO firewall on test system host. kldload ipl.ko from host console. Vnet jail using bridge/epair with ipf configured using normal parameters in vnet jail's rc.conf. "Local0 /var/log/ipf.log" line in vnet jails /etc/syslog.conf to allow ipf logging. Using same /etc/devfs.rules file as posted previously. Rules file contains only 2 lines block out log quick all pass in log all From the vnet jail console "ipfstat -hnoi" shows the 2 rules with zero count. Then issue ping -c2 8.8.8.8 and receive the normal reply that all packets returned. "ipfstat -hnoi" shows the 2 rules still have zero count. If ipf running in the vnet jail was functional, the ping would have been blocked and logged which did not happen. It's important that ipf logging populate it's log file in the vnet jail ipf is running in. Thanks for working on this bug.
No problem. Your setup is similar to mine. My testbed has an unused interface to my DMZ for this purpose. Rather than port 43 I verified using port 22 to my prod firewall in the DMZ. I suspect a global variable wasn't converted to V_ in 2016 by r302298 & friends. Logging is another thing that needs to be looked at.
Created attachment 204898 [details] ipfilter vnet enabler This patch should address your issue. Can you give it a spin?
A commit references this bug: Author: cy Date: Wed Jun 12 11:06:55 UTC 2019 New revision: 348986 URL: https://svnweb.freebsd.org/changeset/base/348986 Log: Register pfil hooks when VNET != vnet0. r302298, which virtualized ipf, assumed the pfil hook registration performed in ipf_modload() would take care of this. However ipf_modload() is only called when the ipl kld is loaded or when ipfilter is first called when it is statically linked into the kernel at build time. Prior to this, even though r302298 has been in the tree for a while, it has never been used. So, r302298 in reality begins now. PR: 212000 Reported by: ahsanb@ MFC after: 1 month Changes: head/sys/contrib/ipfilter/netinet/mlfk_ipl.c
Fixed.
A commit references this bug: Author: cy Date: Fri Jul 12 00:35:44 UTC 2019 New revision: 349926 URL: https://svnweb.freebsd.org/changeset/base/349926 Log: MFC r348986: Register pfil hooks when VNET != vnet0. r302298, which virtualized ipf, assumed the pfil hook registration performed in ipf_modload() would take are of this. However ipf_modload() is only called when the ipl kld is loaded or when ipfilter is first called when it is statically linked into the kernel at build time. Prior to this, even though r302298 has been in the tree for a while, it has never been used. So, r302298 in reality begins now. PR: 212000 Reported by: ahsanb@ Changes: _U stable/12/ stable/12/sys/contrib/ipfilter/netinet/mlfk_ipl.c