Bug 252334

Summary: Regression for running jails with fibs
Product: Base System Reporter: Tom <freebsd>
Component: miscAssignee: freebsd-jail (Nobody) <jail>
Status: New ---    
Severity: Affects Some People CC: arne, dch, rob2g2-freebsd
Priority: --- Keywords: regression
Version: CURRENT   
Hardware: amd64   
OS: Any   

Description Tom 2021-01-01 18:02:33 UTC
I've noticed that there appears to be a regression in FreeBSD 13 when trying to run a jail with a fib. I've created a minimal example script: https://gist.github.com/tomalexander/3ecde8933e482794ef808a37047b07a0

If you take that script and run it in a FreeBSD virtual machine with a clean install of FreeBSD with git installed then it will update itself to a revision you specify at the top of the script and build a jail using the same revision. I strongly advise everyone to only run the script inside a VM because it will overwrite system config files.

This process will involve multiple reboots, but the script installs itself as a service to auto-launch itself on reboots so it should fully set itself up without intervention. Regardless, the script is quite small and readable so you should be able to follow the steps manually if you'd like (explanation at bottom of email).

If you set REVISION="stable/12" at the top of the script then after it goes through its full setup you can run "service jail onestart test" and you can read /var/log/jail.log and see that no errors related to the fib are printed. The jail itself won't have network connectivity because this setup is missing other pieces like a firewall to do NAT since I wanted to cut this down to a minimal example.

If instead you set REVISION="main" at the top of the script then if you had KERNCONF="GENERIC" at the top of the script you will get dropped into a debugger upon running "service jail onestart test": https://user-images.githubusercontent.com/458272/103318144-75636800-49fb-11eb-92f8-66351c65ca7d.png . If instead you set KERNCONF="GENERIC-NODEBUG" then you can read /var/log/jail.log and see messages that say "route: fiboptlist_csv failed."

Through some printf debugging, I have found that in FreeBSD 12 the numfibs in route.c inside the jail is 2 but in FreeBSD 13 the numfibs in route.c inside the jail is 1. Thats as far as my debugging has reached so far. I've attempted to git bisect my way there, but I've been unsuccessful (hitting a lot of revision that fail to build between stable/12 and main)

Finally, a quick overview of what the script does for people who want to recreate this on their own. I'll leave out the minutia of the script injecting itself as a service and cleaning itself up:

1. Creates devfs rule for vnet
2. sets net.fibs=2 in the host /boot/loader.conf
3. sets net.add_addr_allfibs=1 in the host /etc/sysctl.conf
4. sets up a jail.conf using the example from /usr/share/examples/jails for if_bridge but set the fib for the jail to 1
5. Checks out the revision of freebsd specified in the script. Builds and installs that version to the host.
6. Also install that built version to /jail to be used as a jail
7. Configure the jail's rc.conf to use a static local ip address
Comment 1 Tom 2021-03-13 21:25:00 UTC
This no longer happens on the latest FreeBSD HEAD. numfibs is now 2 inside the jail and there is no longer a kernel panic. Marking it as CLOSED - FIXED.
Comment 2 Tom 2021-03-26 15:31:06 UTC
I'm sorry, I was mistaken. This is still happening on the latest HEAD.
Comment 3 Arne Steinkamm 2021-06-24 14:14:25 UTC
Same here with 13.0-RELEASE-p2 FreeBSD
Comment 4 Arne Steinkamm 2021-06-26 08:16:41 UTC
The problem is that you now have to set net.fibs inside the (vnet) jail to at least the number of the fib the jail is using.

Without doing this all network commands working on the on the host side configured fib for the jail will fail because there is only one fib (0) visible inside the jail.

I'm not too happy that this has to be configured now inside the jail and that the number of fibs is now known inside the jail from a security point of view.