Bug 228535

Summary: emulators/virtualbox-ose-kmod: 11.2-BETA3 - kldload vboxdrv leads to panic
Product: Documentation Reporter: Ka Ho Ng <khng300>
Component: DocumentationAssignee: freebsd-doc mailing list <doc>
Status: Closed FIXED    
Severity: Affects Many People CC: admin, alc, dclarke, debdrup, ict, khng300, kib, konstantin.ashaev, m.fujimoto, markj, me, re, rgrimes, rkoberman, stable, vangyzen
Priority: ---    
Version: Latest   
Hardware: amd64   
OS: Any   
Description Flags
Dmesg of core dump
pkg info virtualbox-ose-kmod none

Description Ka Ho Ng 2018-05-27 08:12:28 UTC
Suspected KBI breakage happens in 11.1-BETA series.
Comment 1 Ka Ho Ng 2018-05-27 08:13:24 UTC
Created attachment 193752 [details]
Dmesg of core dump
Comment 2 Ka Ho Ng 2018-05-27 08:14:00 UTC
Created attachment 193753 [details]
pkg info virtualbox-ose-kmod
Comment 3 Ka Ho Ng 2018-05-27 09:42:57 UTC
(In reply to Ka Ho Ng from comment #0)
Typo. Should be 11.2-BETA
Comment 4 Ka Ho Ng 2018-05-28 04:30:00 UTC
After rebuilding the kmod package, `kldload vboxdrv` does not cause kernel panic anymore. I am thinking there is a high chance of KBI incompatibilities between FreeBSD 11.1-RELEASE-p9 and FreeBSD 11.2-BETA3 .
Comment 5 Ka Ho Ng 2018-06-01 08:54:53 UTC
Below is one of the suspected change that causes KBI breakage, as vm_map_min/max/pmap are inline helpers before being made non-inline by https://svnweb.freebsd.org/base?view=revision&revision=332091 :

Comment 6 Rodney W. Grimes freebsd_committer 2018-06-01 19:30:38 UTC
virtual box's module is a "well known" problem with respect to ABI issues, the best way to fix this is to compile the kmod on the specific release you are running Virtual box on.

This module just knows way to much about the insides of the kernel to easily maintain.
Comment 7 Eric van Gyzen freebsd_committer 2018-06-01 20:13:04 UTC
At the very least, this needs special mention in the release notes.

Maybe the release notes could suggest that users change their configuration to avoid loading these modules until their packages have been updated.  Otherwise, the system could get into a panic-on-boot loop, which is...shall we say...unfriendly.
Comment 8 rkoberman 2018-06-01 22:39:13 UTC
This is not something new. While I have seen the assertion that the KBI/KPI are supposed to be stable between major releases as the system ABI/API are, I am not sure that this is official policy and, even it is, it seems to be frequently ignored.

In any case, it is because of this that the PORTS_MODULES variable and the support for it in makekernel came into being. If you build your own kernel, that takes care of it. If you use freebsd-update it would be a good thing if the script printed a reminder to re-build ports after an upgrade operation. It probably could  use pkg to find all kmod ports installed and offer to re-build them.
Comment 9 me 2018-06-26 09:25:28 UTC
I have hit this problem, too.

For me, it is different than previous problems with kernel modules from ports after minor FreeBSD version updates: There were modules that would not load, but very rarely an old module would lead to a panic.

I have vboxdrv_load="YES" in /boot/loader.conf and I could not immediately tell what caused the panic. Obviously, single user mode would not work, either, but loading the old kernel did.

Having freebsd-update print a warning in the end to update ports with kernel modules would not help in this case, either: The panic already happens after installing the kernel and rebooting and you cannot even install the userland. If there was such a warning, it would have to be with the first prompt for a reboot. Rebuilding ports at that time would not be the best idea, either, since the userland is still old. The message would have to tell you to disable everything from ports in /boot/loader.conf, /etc/rc.conf|kld_list, and similar places.

Besides emulators/virtualbox-ose-kmod, I also had to rebuild x11/nvidia-driver, but it was nicer: There was no crash, xorg simply would not start.
Comment 10 Kubilay Kocak freebsd_committer freebsd_triage 2018-08-22 05:24:24 UTC
Issue was documented (with workaround) in the 11.2-RELEASE Errata notes [1]:

[2018-06-26] An issue had been discovered late in the release cycle where a system crash could occur after installing emulators/virtualbox-ose from upstream package mirrors via pkg(8).

Building emulators/virtualbox-ose from the ports(7) collection has been observed to work around the crash.

[1] https://www.freebsd.org/releases/11.2R/errata.html
Comment 11 Dennis Clarke 2018-09-04 16:26:50 UTC
The build process for virtualbox-ose is fraught with problems. At least from 'ports' where multiple little bugs seem to exist.
Comment 12 rkoberman 2018-09-04 18:19:57 UTC
(In reply to Kubilay Kocak from comment #10)
No, virtualbox-ose is not the problem and there is no reason to build it. It is virtualbox-ose-kmod that requires a build against an 11.2-RELEASE kernel. It is small, takes seconds to build and install.

NOTE! You must have sources for 11.2-RELEASE on you system when you build it. The version of the kmod must match the version of virtualbox-ose exactly.

If you install the package of virtualbox-ose and then build and install virtualbax-ose-kmod of the same version, you will be ready to go.
Comment 13 Kubilay Kocak freebsd_committer freebsd_triage 2018-09-09 10:49:57 UTC
(In reply to rkoberman from comment #12)

The text from comment 10 was copied verbatim from the published 11.2-RELEASE Errata notes. If the instructions are incorrect (or incomplete), please re-open this issue, describing the exact correction to be made
Comment 14 rkoberman 2018-09-09 15:50:37 UTC
(In reply to Kubilay Kocak from comment #13)
I will open a bug report on this. I will admit that I never read the Errata item. I just made foolish assumptions. Sigh.

Looking forward to three weeks from now when 11.1 goes EOL and a functioning virtualbox-ose-kmod package for 11.2 will be available. (Well, the first package build after that date.)
Comment 15 Dennis Clarke 2018-09-09 20:34:04 UTC
(In reply to rkoberman from comment #12)

virtualbox-ose has been a real pain to deal with for the last few weeks
here.  At least for me and my team of sysadmins who are doing experiments
with FreeBSD RELEASE 11.2. I can report that I have seen the panic-on-boot
loop "feature" and it is fun. 

Well the package for virtualbox-ose is broken on the RELEASE version 
of 11.2 and so yes it is a problem and will remain a problem until some
future date. Near future we hope.  One may perform the experiment, as I
have, repeatedly, and see a kernel panic repeatedly.

One must build the virtualbox-ose-kmod bits from ports and then overwrite 
the existing pkg which was installed via dependencies for virtualbox-ose.

The entirely repeatable process :

1) install RELEASE 11.2 

2) at first boot you may su - and as root do very very few things :

# /usr/sbin/pkg query -e '%a = 0' '%o'
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.10.5_1...
Extracting pkg-1.10.5_1: 100%

Get on with the process of an update to the ports data

# /usr/sbin/portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching snapshot tag from your-org.portsnap.freebsd.org... done.

Then do the portsnap extract.

3) Full stop right here and we have a fresh RELEASE 11.2 system with not much on it : 

# /usr/sbin/pkg query -e '%a = 0' '%o'

4)  install virtualbox-ose BEFORE attempting to build virtualbox-ose-kmod

# pkg install virtualbox-ose 
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 125 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        virtualbox-ose: 5.2.12_1
        qt5-x11extras: 5.10.1
        qt5-gui: 5.10.1
.  a lot of stuff gets dragged in and that includes broken virtualbox-ose-kmod: 5.2.12


5) manually build virtualbox-ose-kmod in ports to replace the broken 5.2.12 pkg 

# cd /usr/ports/emulators/virtualbox-ose-kmod/
# make install clean  

  a few items to watch for : 

       Be sure to select libsigsegv for GNU m4 
       Disable DTRACE Build with DTrace probes
       gmp-6.1.2  enable CPU optimizations 

6) at some point in the reasonably near future you will have
   two virtualbox-ose version pkgs thus : 

virtualbox-ose-5.2.12_1        General-purpose full virtualizer for x86 hardware
virtualbox-ose-kmod-5.2.18     VirtualBox kernel module for FreeBSD

Note the different versions. 

After a minor edit to /boot/loader.conf and /etc/rc.conf one may reboot and 
have a workable mixed version virtualbox-ose.  Sort of.  The kernel module version
isn't really a perfect match but seems to work thus far.   I have attempted to
build all of virtualbox-ose from ports, repeatedly, over and over there are little
bugs in the process that stop the build.  One gets fixed and another pops up.

I don't think your average user or sysadmin out there will be able to easily get
around this mess.  So when is the next big release ?  Because I don't see how
this can be marked as "fixed".  At best there is a hack workaround.
Comment 16 rkoberman 2018-09-09 23:01:54 UTC
(In reply to Dennis Clarke from comment #15)
Now I am completely baffled. I just checked the archive and there was never a 5.2.12_1. I see no way that the package virtualbox-ose-5.2.12_1 could exist.

That said, you need to get the kmod port for 5.2.12. To do that:
$ svn update -r 469570 /usr/ports/emulators/virtualbox-ose-kmod

That will downgrade the port to a version that matches the package for virtualbox-ose. (PORTREVISION should not matter.) Then you can build the port. If you need it on multiple system, do a "make package" so that you will have a package you can just copy to other systems and install with "pkg add /usr/ports/packages/All/virtualbox-ose-kmod-5.2.12.txz". This assumes that you have not set any environmental variable that overrides the default location.

This is a fairly ugly mess that should never have happened. When it did, I think the proper action would have been to build a package of 5.2.12 on an 11.2 system an put it somewhere on the ftp server. (The same applies to the nVidia driver which has a similar issue.) This would have been documented in the errata.

But I have no say in this and what you see is what ports-manager chose to do.

I can make a package you need available, but, if I was you, I would not trust a driver from an unknown source. Let me know if you want it, though.
Comment 17 rkoberman 2018-09-09 23:55:58 UTC
(In reply to Dennis Clarke from comment #15)
just tried this and I need to slightly modify the instructions:
$ svnlite update -r 469570 /usr/ports/emulators/virtualbox-ose-kmod
$ svn update -r 469570 /usr/ports/emulators/virtualbox-ose
$ make -C /usr/ports/emulators/virtualbox-ose-kmod package
$ pkg add /usr/ports/packages/All/virtualbox-ose-kmod-5.2.12.txz

That file may be copied to other system and installed with "pkg add". Be sure to delete any installed version of virtualbox-ose-kmod already installed with:
# pkg delete -f virtualbox-ose-kmod
before adding the newly built package.
Comment 18 Daniel Ebdrup Jensen freebsd_committer 2018-11-06 19:49:49 UTC
To clarify, this issue is fixed as of the EOL date of 11.1, since the 11.2 packages have now been built on 11.2.
Comment 19 Jan Jurkus 2018-11-06 23:33:15 UTC
Well, I just (this evening) upgraded a server to 11.2-RELEASE and still got a kernel panic.

Starting the old kernel, commenting out the virtualbox stuff in /boot/loader.conf and /etc/rc.conf solved the issue.
Then I could start the new kernel and update the pacakges to the latest versions.

Well of course it all went tits up again because of the VNC problem with virtualbox-ose-nox11-5.2.20, that will teach me to not migrate stuff to bhyve ;-)
Comment 20 Rodney W. Grimes freebsd_committer 2018-11-07 05:42:09 UTC
(In reply to D. Ebdrup from comment #18)
And has a 90+% chance of breaking again when 11.3 is release, this is a general issue being address by some other means that are not yet in place.  Either special "per release" kmod packages, or "kmods install source and require kernel sources to rebuild them" are currently being looked at as possible long term solutions to this KABI breakage issue.