Bug 252334

I've noticed that there appears to be a regression in FreeBSD 13 when trying to run a jail with a fib. I've created a minimal example script: https://gist.github.com/tomalexander/3ecde8933e482794ef808a37047b07a0

If you take that script and run it in a FreeBSD virtual machine with a clean install of FreeBSD with git installed then it will update itself to a revision you specify at the top of the script and build a jail using the same revision. I strongly advise everyone to only run the script inside a VM because it will overwrite system config files.

This process will involve multiple reboots, but the script installs itself as a service to auto-launch itself on reboots so it should fully set itself up without intervention. Regardless, the script is quite small and readable so you should be able to follow the steps manually if you'd like (explanation at bottom of email).

If you set REVISION="stable/12" at the top of the script then after it goes through its full setup you can run "service jail onestart test" and you can read /var/log/jail.log and see that no errors related to the fib are printed. The jail itself won't have network connectivity because this setup is missing other pieces like a firewall to do NAT since I wanted to cut this down to a minimal example.

If instead you set REVISION="main" at the top of the script then if you had KERNCONF="GENERIC" at the top of the script you will get dropped into a debugger upon running "service jail onestart test": https://user-images.githubusercontent.com/458272/103318144-75636800-49fb-11eb-92f8-66351c65ca7d.png . If instead you set KERNCONF="GENERIC-NODEBUG" then you can read /var/log/jail.log and see messages that say "route: fiboptlist_csv failed."

Through some printf debugging, I have found that in FreeBSD 12 the numfibs in route.c inside the jail is 2 but in FreeBSD 13 the numfibs in route.c inside the jail is 1. Thats as far as my debugging has reached so far. I've attempted to git bisect my way there, but I've been unsuccessful (hitting a lot of revision that fail to build between stable/12 and main)

Finally, a quick overview of what the script does for people who want to recreate this on their own. I'll leave out the minutia of the script injecting itself as a service and cleaning itself up:

1. Creates devfs rule for vnet
2. sets net.fibs=2 in the host /boot/loader.conf
3. sets net.add_addr_allfibs=1 in the host /etc/sysctl.conf
4. sets up a jail.conf using the example from /usr/share/examples/jails for if_bridge but set the fib for the jail to 1
5. Checks out the revision of freebsd specified in the script. Builds and installs that version to the host.
6. Also install that built version to /jail to be used as a jail
7. Configure the jail's rc.conf to use a static local ip address