Bug 264114 - net/wireguard-kmod: Crashes on 13.1-RELEASE: panic: vm_fault_lookup: fault on nofault entry, addr: 0 (wireguard)
Summary: net/wireguard-kmod: Crashes on 13.1-RELEASE: panic: vm_fault_lookup: fault on...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: i386 Any
: --- Affects Some People
Assignee: Bernhard Froehlich
URL:
Keywords: crash, needs-qa
Depends on:
Blocks: 264030
  Show dependency treegraph
 
Reported: 2022-05-21 06:09 UTC by Nick Briggs
Modified: 2022-06-17 23:30 UTC (History)
8 users (show)

See Also:
koobs: maintainer-feedback? (decke)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Briggs 2022-05-21 06:09:05 UTC
Upgraded to 13.1-RELEASE from 13.0-RELEASE-p(whatever was current).

System is configured with wireguard, and when I attempted to bring up the tunnel, found that it had died:

  Version String: FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC
  Panic String: vm_fault_lookup: fault on nofault entry, addr: 0

Tried to bring up wireguard again with the same result.

% kldstat
Id Refs Address        Size Name
 1   38   0x800000  1831444 kernel
 2    1 0x19800000     5000 fdescfs.ko
 3    1 0x19805000   179000 radeonkms.ko
 4    2 0x1997e000    89000 drm.ko
 5    4 0x19a07000    28000 linuxkpi.ko
 6    2 0x19a2f000     4000 backlight.ko
 7    3 0x19a33000    10000 linuxkpi_gplv2.ko
 8    1 0x19a43000    13000 ttm.ko
 9    1 0x19a56000     3000 radeon_R100_cp_bin.ko
10    1 0x19a59000     a000 mqueuefs.ko
11    1 0x19a63000     6000 acpi_wmi.ko
12    1 0x19a69000    2b000 if_wg.ko
13    1 0x19a94000     3000 mac_ntpd.ko

# pkg provides if_wg.ko
Name    : wireguard-kmod-0.0.20211105_1
Desc    : WireGuard implementation for the FreeBSD kernel
Repo    : FreeBSD
Filename: boot/modules/if_wg.ko

# pkg info wireguard-kmod-0.0.20211105_1
wireguard-kmod-0.0.20211105_1
Name           : wireguard-kmod
Version        : 0.0.20211105_1
Installed on   : Sun May  8 14:05:11 2022 PDT
Origin         : net/wireguard-kmod
Architecture   : FreeBSD:13:i386
Prefix         : /usr/local
Categories     : net-vpn net kld
Licenses       : MIT
Maintainer     : decke@FreeBSD.org
WWW            : https://git.zx2c4.com/wireguard-freebsd/
Comment        : WireGuard implementation for the FreeBSD kernel
Annotations    :
	FreeBSD_version: 1300139
	repo_type      : binary
	repository     : FreeBSD
Flat size      : 114KiB
Description    :
Kernel module for FreeBSD to support Wireguard.

At this time this code is new, unvetted, possibly buggy, and should be
considered "experimental". It might contain security issues. We gladly
welcome your testing and bug reports, but do keep in mind that this code
is new, so some caution should be exercised at the moment for using it
in mission critical environments.

WWW: https://git.zx2c4.com/wireguard-freebsd/

It was running without issue on 13.0-RELEASE -- perhaps this version should be tagged as not compatible with 13.1?  This seems to be the latest version of wireguard available for the FreeBSD kernel.
Comment 1 Nick Briggs 2022-05-22 00:14:11 UTC
Looks as though it may just take a recompile on a 13.1-RELEASE system -- I rebuilt from the package sources, reinstalled, and it appears to be working without causing a panic.

Could the package maintainer tag the builds as specific to the kernels they were built on so that the appropriate one gets installed?
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2022-05-22 01:38:39 UTC
^Triage: Assign to maintainer for coordination
Comment 3 Nick Briggs 2022-05-22 03:28:43 UTC
It would be a much better UX if the wireguard kmod if_wg.ko were tagged such that the kernel module loader noticed that it was not compatible with the new kernel.

The virtualbox module seems to be coded so that the module loader detects that situation, reports an error, and refuses to load.  I much prefer that over having to deal with a system that panics each time someone tries establish a tunnel.
Comment 4 Bernhard Froehlich freebsd_committer freebsd_triage 2022-05-22 06:25:13 UTC
Okay I keep this PR open to mark all future PRs as duplicates.

The reason for the panic is that all packages are build on 13.0 which works fine for userland packages because they are compatible with 13.1. For kernel modules there is no such compatibility guarantee so you have to build kernel modules for exactly that kernel or you get panics like these.

This has been the situation for ages. Nothing new. Virtual box has had such bug reports for years until all users were aware (removing module before reboot etc.). With vbox it was even more critical because we traditionally loaded it very early via loader which could result in a bootloop of panics. In a DevSummit in 2012 we talked about that problem in a session. Finally about two weeks ago jhb@ did send patches for virtualbox-kmod to address this.

https://cgit.freebsd.org/ports/commit/?id=bc6d5725ed6c7b6538da70328d89afe901736a90

It will not take another 10 years to fix it for wireguard.

For the moment if anyone experiences panics after an update (or fresh install from 13.1) please don't use the binary package but install from ports using "make install" to make sure it matches your kernel.
Comment 5 lbfoo 2022-05-22 09:03:03 UTC
(In reply to Bernhard Froehlich from comment #4)

Thanks. But it may takes a lot of time and space when doing port compiling on embedded system.

It would be great if someone manages to resolve related package system drawbacks. Or someone adds some description in the Freebsd Handbook.
Comment 6 Marek Zarychta 2022-05-22 09:09:46 UTC
Maybe it's high time to change the default policy of the Project? Instead of building all packages on approaching EoL minor version (13.0) in this case, the packages might be built on the upcoming minor version with some advance, let's say entering the RC phase could be the turning point.
Such an approach definitely would improve the user experience.
Comment 7 Bernhard Froehlich freebsd_committer freebsd_triage 2022-05-22 11:32:17 UTC
(In reply to Marek Zarychta from comment #6)
No that does not work. Compatibility only works in one direction. Packages are build with the oldest supported version on that branch which is correct. A possibility would be to have a small additional repository for all kmod ports and build that per release.

I will drag this to portmgr@ to make them aware of the problem. I will let you all know if there are any news.
Comment 8 Kyle Evans freebsd_committer freebsd_triage 2022-05-22 23:18:50 UTC
I wish we had known this was an issue back in neta/rc... we put time into diagnosing an issue with binary compatible with the drm ports this cycle, we would have done the same for wireguard.
Comment 9 Bernhard Froehlich freebsd_committer freebsd_triage 2022-05-28 11:07:49 UTC
So far no response from portmgr@ but I have created a patch similar to what was done in virtualbox-ose-kmod to avoid that the kernel module can be loaded on a kernel with a different minor version.

https://reviews.freebsd.org/D35341