Bug 255882 - vxlan(4): kernel panic when unloading module if vxlan interface in VNET jails not shutdown before jail shutdown
Summary: vxlan(4): kernel panic when unloading module if vxlan interface in VNET jails...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-net (Nobody)
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2021-05-14 15:42 UTC by Joshua Kinard
Modified: 2022-10-12 00:50 UTC (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joshua Kinard 2021-05-14 15:42:23 UTC
Found a corner case bug in the if_vxlan module where, if a VNET jail with an active vxlan interface is shutdown before the interface is destroyed, and the if_vxlan module is then unloaded on the host, the kernel will panic.

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x30
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d1bce3
stack pointer	        = 0x28:0xfffffe00c7c2c880
frame pointer	        = 0x28:0xfffffe00c7c2c8c0
code segment		= base rx0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 83973 (kldunload)
trap number		= 12
panic: page fault
cpuid = 0
time = 1621005409
KDB: stack backtrace:
#0 0xffffffff80c57345 at kdb_backtrace+0x65
#1 0xffffffff80c09d21 at vpanic+0x181
#2 0xffffffff80c09b93 at panic+0x43
#3 0xffffffff8108b187 at trap_fatal+0x387
#4 0xffffffff8108b1df at trap_pfault+0x4f
#5 0xffffffff8108a83d at trap+0x27d
#6 0xffffffff810617a8 at calltrap+0x8
#7 0xffffffff80d1ae6e at if_detach_internal+0xbe
#8 0xffffffff80d1abdb at if_detach+0x5b
#9 0xffffffff82923d53 at vxlan_clone_destroy+0x83
#10 0xffffffff80d21fa5 at if_clone_destroyif+0x1b5
#11 0xffffffff80d227b8 at if_clone_detach+0xb8
#12 0xffffffff829230b4 at vxlan_modevent+0xb4
#13 0xffffffff80be7058 at module_unload+0x38
#14 0xffffffff80bd8daa at linker_file_unload+0x1ea
#15 0xffffffff80bda0e0 at kern_kldunload+0xe0
#16 0xffffffff8108ba8c at amd64_syscall+0x10c
#17 0xffffffff810620ce at fast_syscall_common+0xf8
Uptime: 21m51s
Dumping 681 out of 7128 MB:..3%..12%..22%..31%..43%..52%..62%..71%..83%..92%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55	/usr/src/sys/amd64/include/pcpu_aux.h: No such file or directory.
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:399
#2  0xffffffff80c09916 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:486
#3  0xffffffff80c09d90 in vpanic (fmt=<optimized out>, ap=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:919
#4  0xffffffff80c09b93 in panic (fmt=<unavailable>)
    at /usr/src/sys/kern/kern_shutdown.c:843
#5  0xffffffff8108b187 in trap_fatal (frame=0xfffffe00c7c2c7c0, eva=48)
    at /usr/src/sys/amd64/amd64/trap.c:915
#6  0xffffffff8108b1df in trap_pfault (frame=frame@entry=0xfffffe00c7c2c7c0, 
    usermode=false, signo=<optimized out>, signo@entry=0x0, 
    ucode=<optimized out>, ucode@entry=0x0)
    at /usr/src/sys/amd64/amd64/trap.c:732
#7  0xffffffff8108a83d in trap (frame=0xfffffe00c7c2c7c0)
    at /usr/src/sys/amd64/amd64/trap.c:398
#8  <signal handler called>
#9  0xffffffff80d1bce3 in _if_delgroup_locked (
    ifp=ifp@entry=0xfffff8000e7d8000, ifgl=ifgl@entry=0xfffff8012e5e2b20, 
    groupname=groupname@entry=0xfffffe00c7c2c8e0 "all")
    at /usr/src/sys/net/if.c:1587
#10 0xffffffff80d1ae6e in if_delgroups (ifp=0xfffff8000e7d8000)
    at /usr/src/sys/net/if.c:1640
#11 if_detach_internal (ifp=ifp@entry=0xfffff8000e7d8000, 
    vmove=vmove@entry=0, ifcp=ifcp@entry=0x0) at /usr/src/sys/net/if.c:1174
#12 0xffffffff80d1abdb in if_detach (ifp=0xfffff8000e7d8000, ifp@entry=0x0)
    at /usr/src/sys/net/if.c:1127
#13 0xffffffff80d2419a in ether_ifdetach (ifp=ifp@entry=0xfffff8000e7d8000)
    at /usr/src/sys/net/if_ethersubr.c:1034
#14 0xffffffff82923d53 in vxlan_clone_destroy (ifp=0xfffff8000e7d8000)
    at /usr/src/sys/net/if_vxlan.c:3233
#15 0xffffffff80d21fa5 in ifc_simple_destroy (ifc=0xfffff80110143900, 
    ifp=0xfffff8000e7d8000) at /usr/src/sys/net/if_clone.c:740
#16 if_clone_destroyif (ifc=ifc@entry=0xfffff80110143900, 
    ifp=0xfffff8000e7d8000) at /usr/src/sys/net/if_clone.c:335
#17 0xffffffff80d227b8 in if_clone_detach (ifc=0xfffff80110143900)
    at /usr/src/sys/net/if_clone.c:458
#18 0xffffffff829230b4 in vxlan_unload () at /usr/src/sys/net/if_vxlan.c:3606
#19 vxlan_modevent (mod=<optimized out>, type=<optimized out>, 
    unused=<optimized out>) at /usr/src/sys/net/if_vxlan.c:3623
#20 0xffffffff80be7058 in module_unload (mod=mod@entry=0xfffff8010ffef400)
    at /usr/src/sys/kern/kern_module.c:261
#21 0xffffffff80bd8daa in linker_file_unload (
    file=file@entry=0xfffff801124c1000, flags=flags@entry=0)
    at /usr/src/sys/kern/kern_linker.c:697
#22 0xffffffff80bda0e0 in kern_kldunload (td=<optimized out>, 
    fileid=<optimized out>, flags=0) at /usr/src/sys/kern/kern_linker.c:1150
#23 0xffffffff8108ba8c in syscallenter (td=0xfffffe00c85fe700)
    at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189
#24 amd64_syscall (td=0xfffffe00c85fe700, traced=0)
    at /usr/src/sys/amd64/amd64/trap.c:1156
#25 <signal handler called>
#26 0x00000008003803ea in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffd248
(kgdb) 

Steps to reproduce:

1. Install FreeBSD
2. Create minimal /etc/jail.conf on host
3. Create two minimal VNET jails, 'j1' and 'j2'
4. Create a new epair interface set on host.
5. Assign epair0a to jail j1 and epair0b to jail j2 and start the jails
6. On the host, kldload if_vxlan
7. In each jail, assign a /31 point-to-point IP to the epair0x interfaces
8. In each jail, ping other jail to verify epair tunnel is up
9. In each jail, create a new vxlan interface:
   * vxlanid: any
   * vxlanlocal: j1 is epair0a IP, j2 is epair0b IP
   * vxlanremote: j1 is j2's epair0b IP, j2 is j1's epair0a IP
10. In each jail, assign new IPv4/IPv6 addresses to the vxlan interfaces
11. In each jail, ping the other jail across the vxlan tunnel to verify connectivity.
12. On the host, shutdown both jails *without* shutting down the vxlan or epair interfaces
13. On the host, kldunload if_vxlan --> kernel panic
Comment 1 Aleksandr Fedorov freebsd_committer freebsd_triage 2021-05-17 09:33:18 UTC
The problem is easily reproducible:

# kldload if_vxlan
# jail -c name=j1 persist vnet
# jexec j1 ifconfig vxlan create
vxlan0
# jail -r j1
# jail -c name=j1 persist vnet
# jexec j1 ifconfig vxlan create
vxlan1
# jail -r j1
# kldunload if_vxlan
panic!!!

Unfortunately if_vxlan (4) does not support VNET.
There was an attempt, but not finished: https://reviews.freebsd.org/D2868?id=6328#change-57yzNcgb46N6