Bug 234021
| Summary: | 12.0 gateway host with vnet jail running pf firewall & NAT has no internet access | ||
|---|---|---|---|
| Product: | Base System | Reporter: | Joe Barbish <qjail1> |
| Component: | kern | Assignee: | freebsd-bugs (Nobody) <bugs> |
| Status: | Closed Not A Bug | ||
| Severity: | Affects Only Me | CC: | kp |
| Priority: | --- | ||
| Version: | 12.0-STABLE | ||
| Hardware: | Any | ||
| OS: | Any | ||
Why are you trying to do NAT in the jail when the host has the public IP? If you want to NAT in the jail and bridge you'll need to assign the public IP inside the jail, not on the host. Or even simpler, just assign vge0 to the jail. Or even simpler, do NAT on the host, because that's what your configuration suggests right now. Your jail has a private IP address, and a single interface. Your host has both WAN (vge0) and LAN (em0) interfaces. (In reply to Kristof Provost from comment #1) I am having a real hard time trying to understand your comments. Its my understanding that because vnet jails have their own ip stack that's outside of the hosts ip stack, that they act like individual computers. This is the only difference between non-vnet jails and vnet jails. For network connectivity vnet jails use the bridge/epair or netgraph methods. Non-vnet jails uses the host network stack. This fact is well know by people who have read any of the vnet jail documentation. The whole reason for changing ipfw and pf firewalls was because vnet jails on gateway hosts need a vnet aware firewall to filter and NAT their traffic. Based on this information, I can not get a so configured vnet jail running on a gateway host to access the public internet. To verify this problems exist is the purpose of this bug report. See /usr/share/examples/jails for details and who wrote the content of the files. From your comments you seem to be implying this is untrue. Please point me to vnet jail documentation that supports your position. I'm always ready to learn new things about vnet jails. A example of a working vnet jail setup environment would enable me to replicate it here. Vnet jails do indeed have their own network stack. As such you'd expect a NAT jail to have the WAN interface and IP address assigned to it. After all, how can a machine (i.e. the jail) perform the NAT role if it doesn't have a WAN and LAN interface? Your bug report shows that's not the case: The host has the WAN IP (65.xxx.62.234 and interface vge0). The jail has a private IP (10.0.110.25 and an epair epair15b). Note also that you can assign any interface into a vnet jail, not just epairs. (In reply to Kristof Provost from comment #3) Your comment does not add any clarity to this discussion. You say there is something missing from my vnet setup. But you do not state where or what needs to be fixed. Where is the public documentation that you think demonstrates a working vnet jail running on a gateway host? Did you review the contents of /usr/share/examples/jails? Are you saying Devin Teske got vnet jail communication all wrong? Devin runs many vnet jails at his place of work, but all of then are nodes on LANs. Maybe he already discovered this bug and that is why his vnet jails are LAN bound as a simple work a round. Looking for proof that this problem of no internet access from a bridge/epair attached vnet jail does not work running from a gateway host. It's my contention that vimage or pf nat is broken. This will take a real field test case to find out if it really works or not. Are you in a position to perform this field test case? Your setup is misconfigured for what you stay you're trying to do. You cannot do NAT in the jail unless the jail has the public IP address. I don't know how else I can explain this. As shown in the original post the nat statement in the pf rules file does have the public ip address. Look at pip. #vnet jails pf rules file oif=epair15b jip=10.0.110.25 pip=65.xxx.62.234 set block-policy drop set fail-policy drop set state-policy if-bound scrub in on $oif all set skip on lo0 nat on $oif from $jip to any -> $pip block out log quick on $oif inet proto tcp from any to any port 43 pass out log (all) quick on $oif from any to any pass in log (all) quick on $oif from any to any This bug is not closed until I have a solution. (In reply to Joe Barbish from comment #6) How do you expect pf in the jail to NAT to that address if it's not assigned to the jail? From your original report: Issued from the host with the vnet jail running /root >ifconfig -a ... vge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3899<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST, WOL_MCAST,WOL_MAGIC> ether 10:00:60:21:00:93 inet 65.xxx.62.234 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT <full-duplex,master>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>\ You cannot NAT on an address that's not available on that jail. Your configuration is wrong, this won't work. This is not a bug, this is a configuration problem. Bugzilla is not the appropriate venue for tech support. 65.xxx.62.234 is available to the jail through the bridge. The bridge has 2 members, the vge0 public interface and the "A" part of the epair pair. The "B" epair is assigned to the jail as shown in the jail.conf file. The path between the vnet jail and the public internet should be open. This method is documented every where. Its not a configuration problem like you say. All those other people who wrote how-to could not all be wrong. You have to provide proof of a working configuration or tell me what to change to get my configuration to work. I need proof that this is a configuration problem. Only a live test can verify this is not a bug. You saying this is a configuration problem is just your personal opinion and your reluctance to backup what you say is disappointing. Lets just exchange information and work to verify one way or the other if this is a bug based on actual usage. (In reply to Joe Barbish from comment #9) No, the address is not available to the jail. It's not assigned to the jail. It's assigned on the host. The jail needs the WAN IP assigned to it (on an interface that belongs to the jail, not bridged to it!). In addition to that, you cannot have two machines (think of vnet jails as different machines) in two different subnets communicate directly, and that's exactly what you're trying to do here. Your configuration is wrong. Write down how you would configure this with two different machines, then try to replicate that with the jail and host. (Basically move vge0 into the jail, have it obtain the WAN IP. Add a second interface, an epair, with an IP in the LAN. On the host bridge the other end of that epair to your em0 LAN interface.) Do not reopen this bug. This is a configuration problem, not a bug. I will not comment further. If you still cannot get it to work seek help on freebsd-questions@freebsd.org or the freebsd forums. This is not the correct venue for debugging configuration problems. |
Trying to get a vnet jail to access the public internet. Issuing "ping -c 2 8.8.8.8" returns 100.0% packet loss message. The host running the vnet jail is a gateway host ie: connected directly to my ISP. The pf firewall is running on the host and in the vnet jail. The host and the lan behind it are functioning normally. The pf rules in the vnet jail are doing NAT. The pflog in the vnet jail shows outbound packets only, never a inbound reply. gateway_enable is in the vnet jails rc.conf plus the normal pf enable statements. Not using the "service jail" command for starting or stopping the vnet jail. I start and stop the vnet jail using the native jail(8) jail command. Using bridge/epair method for vnet jail networking. Tried a second variation where I ran ipfilter on the host and pf in the vnet jail with the same out come. Running this same setup on a LAN host works. IE; the vnet jail can ping the public internet. Reviewing google search results shows all the vnet jail examples are vnet jails on lan hosts. Have suspicion that gateway vnet jails have never worked because I have tested it my self in 10.x and 11.x. Never posted a bug report because thought it was a vimage problem due to its experimental nature. Now that vimage is included in the base kernel time for a bug report. Need someone from the vimage kernel project or the pf vimage aware project to perform their own test of vnet on a gateway host to verify if it works or not. Also have same results if ipfw is the vnet jail firewall. Below is some info about my setup that may help or may not. /root >cat /etc/jail.vnetpf1.conf vnetpf1 { host.hostname = "vnetpf1"; path = "/usr/jails/vnetpf1"; exec.consolelog = "/var/log/jail.vnetpf1.console.log"; mount.devfs; devfs_ruleset = "70"; vnet = "new"; vnet.interface = "epair15b"; exec.start = "ifconfig epair15b 10.0.110.25/24"; exec.start += "route add default 10.0.110.2"; exec.start += "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; } Issued from the host console >netstat -nr4 Routing tables Internet: Destination Gateway Flags Netif Expire default 65.xxx.48.1 UGS vge0 10.0.0.0/8 link#1 U em0 10.0.10.2 link#1 UHS lo0 65.xxx.48.0/20 link#2 U vge0 65.xxx.62.234 link#2 UHS lo0 127.0.0.1 link#3 UH lo0 Issued from the vnet jails console vnetpf1 /root >netstat -nr4 Routing tables Internet: Destination Gateway Flags Netif Expire default 10.0.110.2 UGS epair15b 10.0.110.0/24 link#3 U epair15b 10.0.110.25 link#3 UHS lo0 127.0.0.1 link#1 UH lo0 # devfsrules for pf to function in a vnet jail. [vnet_pf=70] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add include $devfsrules_jail add path 'bpf*' unhide add path pf unhide add path pflog unhide add path pfsync unhide Issued from the host with the vnet jail running /root >ifconfig -a em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO, WOL_MAGIC,VLAN_HWFILTER> ether d0:50:99:93:75:98 inet 10.0.10.2 netmask 0xff000000 broadcast 10.255.255.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> vge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=3899<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST, WOL_MCAST,WOL_MAGIC> ether 10:00:60:21:00:93 inet 65.xxx.62.234 netmask 0xfffff000 broadcast 255.255.255.255 media: Ethernet autoselect (1000baseT <full-duplex,master>) status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160 groups: pflog bridge10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:3a:f8:d2:63:0a id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: epair15a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6 priority 128 path cost 2000 member: vge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 2 priority 128 path cost 20000 groups: bridge nd6 options=1<PERFORMNUD> epair15a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU> ether 02:9b:6a:d0:c6:0a inet6 fe80::9b:6aff:fed0:c60a%epair15a prefixlen 64 scopeid 0x6 groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> #vnet jails pf rules file oif=epair15b jip=10.0.110.25 pip=65.xxx.62.234 set block-policy drop set fail-policy drop set state-policy if-bound scrub in on $oif all set skip on lo0 nat on $oif from $jip to any -> $pip block out log quick on $oif inet proto tcp from any to any port 43 pass out log (all) quick on $oif from any to any pass in log (all) quick on $oif from any to any