Bug 231038 - memguard is broken with options NUMA
Summary: memguard is broken with options NUMA
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Mark Johnston
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-08-30 21:11 UTC by Mark Johnston
Modified: 2018-09-20 15:47 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mark Johnston freebsd_committer freebsd_triage 2018-08-30 21:11:43 UTC
With INVARIANTS enabled I get:

# sysctl vm.memguard.desc=mbuf
panic: vm_reserv_extend: Domain mismatch from reservation.                                                                                                   
cpuid = 4                                                                                      
time = 1535660704                                                                                                                                            
Uptime: 44s                                                                                    
Dumping 1295 out of 32617 MB:..2%..12%..22%..31%..41%..51%..61%..71%..81%..91%                                                                               
                                                                                                                                                             
__curthread () at ./machine/pcpu.h:230                                       
230             __asm("movq %%gs:%1,%0" : "=r" (td)                                                                                                          
(kgdb) bt                                                                                                                                                    
#0  __curthread () at ./machine/pcpu.h:230                                                                                                                   
#1  doadump (textdump=1) at /usr/home/markj/src/freebsd-dev/sys/kern/kern_shutdown.c:366                                                                     
#2  0xffffffff80768ab0 in kern_reboot (howto=260) at /usr/home/markj/src/freebsd-dev/sys/kern/kern_shutdown.c:446                                            
#3  0xffffffff80768f03 in vpanic (fmt=<optimized out>, ap=0xfffffe0f812a1570) at /usr/home/markj/src/freebsd-dev/sys/kern/kern_shutdown.c:872                
#4  0xffffffff80768c63 in panic (fmt=<unavailable>) at /usr/home/markj/src/freebsd-dev/sys/kern/kern_shutdown.c:799                                          
#5  0xffffffff80ad729c in vm_reserv_extend (req=546, object=0xffffffff81312c78 <kernel_object_store>, pindex=2053, domain=0, mpred=<optimized out>)          
    at /usr/home/markj/src/freebsd-dev/sys/vm/vm_reserv.c:879                            
#6  0xffffffff80ac86fe in vm_page_alloc_domain_after (object=<optimized out>, pindex=2053, domain=<optimized out>, req=546, mpred=0xfffff8086998d9e8)
    at /usr/home/markj/src/freebsd-dev/sys/vm/vm_page.c:1835                                                    
#7  0xffffffff80ab5958 in kmem_back_domain (domain=0, object=<optimized out>, addr=<optimized out>, size=4096, flags=<optimized out>)                        
    at /usr/home/markj/src/freebsd-dev/sys/vm/vm_kern.c:440                                                     
#8  0xffffffff80ab5c49 in kmem_back (object=<optimized out>, addr=18446741874694705152, size=4096, flags=<unavailable>)
    at /usr/home/markj/src/freebsd-dev/sys/vm/vm_kern.c:487                                 
#9  0xffffffff80aafe72 in memguard_alloc (req_size=256, flags=1) at /usr/home/markj/src/freebsd-dev/sys/vm/memguard.c:351
#10 0xffffffff80aaa4be in uma_zalloc_arg (zone=0xfffffe101af84000, udata=0xfffffe0f812a1870, flags=1)
    at /usr/home/markj/src/freebsd-dev/sys/vm/uma_core.c:2385                                                                                                
#11 0xffffffff80890d58 in m_gethdr (how=1, type=255) at /usr/home/markj/src/freebsd-dev/sys/sys/mbuf.h:790                                                   
#12 _iflib_fl_refill (ctx=0xfffff8000b676800, fl=<optimized out>, count=<optimized out>) at /usr/home/markj/src/freebsd-dev/sys/net/iflib.c:1974             
#13 0xffffffff8089054b in __iflib_fl_refill_lt (ctx=<optimized out>, max=24, fl=<optimized out>) at /usr/home/markj/src/freebsd-dev/sys/net/iflib.c:2071
#14 iflib_rxeof (rxq=<optimized out>, budget=24) at /usr/home/markj/src/freebsd-dev/sys/net/iflib.c:2723
#15 0xffffffff8088c0f9 in _task_fn_rx (context=0xfffff80480637580) at /usr/home/markj/src/freebsd-dev/sys/net/iflib.c:3834
#16 0xffffffff807af459 in gtaskqueue_run_locked (queue=0xfffff8000b1dca00) at /usr/home/markj/src/freebsd-dev/sys/kern/subr_gtaskqueue.c:332
#17 0xffffffff807af218 in gtaskqueue_thread_loop (arg=<optimized out>) at /usr/home/markj/src/freebsd-dev/sys/kern/subr_gtaskqueue.c:507
#18 0xffffffff80728e24 in fork_exit (callout=0xffffffff807af190 <gtaskqueue_thread_loop>, arg=0xfffffe101af9a068, frame=0xfffffe0f812a1ac0)
    at /usr/home/markj/src/freebsd-dev/sys/kern/kern_fork.c:1057
#19 <signal handler called>

memguard is the sole remaining consumer of the kmem_back() API, which takes a KVA range and backs it with pages from a domain selected using an iterator.  For normal allocations with kmem_malloc(), the iterator is used to select the vmem arena as well.  The per-domain vmem arenas have a quantum equal to the superpage size, which ensures that we consistently use the same physical memory domain to back regions in a superpage-sized KVA range.  memguard just uses a single global arena with a 4KB quantum, and basically picks a random domain which may not have the same "colour" as the 4KB KVA range.  This mismatch triggers the panic.
Comment 1 Mark Johnston freebsd_committer freebsd_triage 2018-09-14 21:00:13 UTC
https://reviews.freebsd.org/D17175
Comment 2 commit-hook freebsd_committer freebsd_triage 2018-09-20 15:45:28 UTC
A commit references this bug:

Author: markj
Date: Thu Sep 20 15:45:13 UTC 2018
New revision: 338830
URL: https://svnweb.freebsd.org/changeset/base/338830

Log:
  Change the domain selection policy in kmem_back().

  Ensure that pages backing the same virtual large page come from the
  same physical domain, as kmem_malloc_domain() does.

  PR:		231038
  Reviewed by:	alc, kib
  Approved by:	re (gjb)
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D17248

Changes:
  head/sys/vm/vm_kern.c