Bug 251624

Summary: [jls] is unable to list IP(s) of vnet jails.
Product: Base System Reporter: Chris Watson <chris>
Component: binAssignee: freebsd-jail (Nobody) <jail>
Status: Open ---    
Severity: Affects Some People CC: bz, emaste, melifaro, otis, pat
Priority: --- Keywords: feature
Version: CURRENT   
Hardware: Any   
OS: Any   

Description Chris Watson 2020-12-05 22:19:12 UTC
Im quite certain the authors and those who have had their hands in the VNET code are aware of this issue. I just could not find any discussion about this or any explanation of what the issue is. So my hope is this winds up as a PR, but also hopefully someone can chime in and post a link to a discussion about this or an explanation of the issue so I can understand the rabbit hole here.

When creating a jail with vnet only and with vnet plus DHCP (using BastilleBSD), on 12.2-RELEASE r366954, the jail's get created fine. But jls or jls -N do not display the IP of said created jails.

This is an issue when you spin up a dozen, hundreds or a thousand vnet jails and do a 'jls' or 'jls -N' and get nothing returned for the IP. It makes using jls and tooling built around it like BastilleBSD... difficult. 

I looked at how SmartOS does the output of a 'list' command and with their 'vmadm list' you simply get back the primary or first interface IP of the zone. With a 'vmadm get' you get a JSON array returned with every interface and IP attached to it from each zone. 

I looked at jls.c and didn't see anything regarding this issue. And again, I couldn't find any discussion about this or a blog entry anywhere. This has to be pretty widely known, I would just like to understand what the issue is so hopefully someone familiar with the code can chime in and explain the issue. Either way I have at least filed a PR for the record.

Thank you for any explanation of the issue provided.
Comment 1 Juraj Lutter freebsd_committer freebsd_triage 2020-12-05 22:34:23 UTC
FWIW, SmartOS keeps track of IPs in zone xml files (under /etc/zones). Then its behavior depends on whether you have IP and/or MAC spoofing turned on, etc. So you might or might not end up with the actual list of IPs within a guest.

vminfod is responsible for returning this information back to the client and management tools.
Comment 2 Juraj Lutter freebsd_committer freebsd_triage 2020-12-05 22:40:06 UTC
I don't know much about BastilleBSD, but, for example, if you happen to manage jails using `iocage', you might also use `iocage list' like:

# iocage list
+------+------+-------+---------+-------------+
| JID  | NAME | STATE | RELEASE |     IP4     |
+======+======+=======+=========+=============+
| 2530 | jb12 | up    | EMPTY   | 100.64.3.12 |
+------+------+-------+---------+-------------+
Comment 3 Chris Watson 2020-12-05 23:04:11 UTC
(In reply to Juraj Lutter from comment #1)

Thanks for that! That provides a little more insight into how they handle things.
Comment 4 Chris Watson 2020-12-05 23:41:03 UTC
(In reply to Juraj Lutter from comment #2)

Just installed the latest iocage, not the iocage-devel pkg.
So for VNET and DHCP they just display DHCP as the IP address.
But they do get the IP address of a vnet only jail. So im off to take a look at that.
Comment 5 Bjoern A. Zeeb freebsd_committer freebsd_triage 2020-12-06 12:16:32 UTC
I cannot comment on what/how other management solution on top of jails do, but jls does list IP addresses for classic jail (where IP addresses are tied to jails).

With VNET jails the jail has its own network stack and potentially any kind of addresses (and address families) and as super user inside that jail you can configure at random what you she wants in terms of IPv4/v6 addresses.

jls is not the tool to show or list them as jails do not care about addresses in the vnet case.

Works as expected is the answer.

If other management solutions show DHCP or RA or whatever there I would find that at least mindly confusing and a limited mindset.  Whatever runs inside the vnet jail is a matter of the rc.conf (or other mechanisms) and provisioning of the network, not of the jail.  Printing "VNET" would probably be best.


As a bit more of explanation:

depending on how you configure vnet jails (bridge or routed) you want to use the normal network admin controls on bridges (switches) or routers you are using (which could include the base system or another vnet jail or another box).  Your DHCP config, DHCP server, L2 tables (ARP / v6 neighbour tables), routes, MAC addresses, ...  (whatever else you'd do or use there).   But that might also be dependent on the connectivity of a vnet jail.
A vnet jail could be a router or firewall itself and have 5 different network connections.  It might have an external and an internal interface, might have VLANs on top of one, ...

If you really want to list the addresses of a jail, you'd need to run ifconfig/netstat in the jail context.  But then you'd still only know the addresses and not necessarily the broadcast domain context.
Comment 6 Chris Watson 2020-12-07 05:20:40 UTC
(In reply to Bjoern A. Zeeb from comment #5)

Thanks for taking a second to chime in bz.

So I understand what you are saying. And from the engineering pov, you are correct. But from a user pov this makes it incredibly difficult to manage more than a handful of vnet jails since you have no idea at a glance how to reach them. 

My first thought was to, as you note, just log in to the jail and run 'ifconfig'.
But that is notoriously slow and painful. Authenticating. Issuing the ifconfig. That would be painful on all but a handful or fewer vnet jails. 

So then I had the idea of upon creating the jail run an 'ifconfig -a > /tmp/$FILE'.
And setting up a cronjob as part of your post creation config to run that command every n seconds. So that you have a reasonably fresh and update picture of the interfaces.

That would at least get you the configuration for the vnet jails interfaces. And $FILE could be read from the host and massaged into whatever tooling you are using to manage your jails.

Thanks again for taking a moment to shed insight into the behavior.

This should be changed from a bug to a feature request.
Comment 7 Bjoern A. Zeeb freebsd_committer freebsd_triage 2020-12-07 12:20:35 UTC
(In reply to Chris Watson from comment #6)

But you are assuming that root inside the jail couldn't disable the cronjob or write into that file whatever they want.  In short, you cannot realistically rely on that data (unless you actually control all the jails as well yourself and have no 3rd party).

I'd assume that using libkvm to get all vnets and then iterate over them and list all interfaces and their addresses that way from base system would probably be a better approach though that would require some C hacking but in theory everything needed should be (or once was) in place for that and just needs to be glued together.
Comment 8 Chris Watson 2020-12-08 23:04:18 UTC
(In reply to Bjoern A. Zeeb from comment #7)

That's correct. I am thinking of it from the system administrators POV, where she/he has control of all jails and there is no third party. 


It seems ill need to dig into libkvm as you suggest to obtain the info I seek.
Thank's for the pointer!