Bug 252541 - Early Panic during boot (Too many early devmatch mappings)
Summary: Early Panic during boot (Too many early devmatch mappings)
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: arm (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: Mitchell Horne
URL:
Keywords: panic
Depends on:
Blocks:
 
Reported: 2021-01-09 09:57 UTC by Gordon Bergling
Modified: 2021-01-18 14:09 UTC (History)
3 users (show)

See Also:


Attachments
arm64 boot panic message (610.68 KB, image/jpeg)
2021-01-09 09:57 UTC, Gordon Bergling
no flags Details
fix pmap_mapdev assert (507 bytes, patch)
2021-01-13 17:36 UTC, Mitchell Horne
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Gordon Bergling freebsd_committer 2021-01-09 09:57:56 UTC
Created attachment 221415 [details]
arm64 boot panic message

On a recent build of -CURRENT on arm64 I get this panic during boot:

"Too many early devmatch mappings".
Comment 1 Gordon Bergling freebsd_committer 2021-01-10 11:18:34 UTC
This panic happens on GENERIC and on GENERIC-MMCCAM.
Comment 2 Gordon Bergling freebsd_committer 2021-01-11 20:38:55 UTC
After some time bisecting commits, I identified,

-------------------------------------------------------------------------------
commit e83fdf8bb391579fa422d34663cd8c1f82a00dc0
Author:     Chuck Tuffli <chuck@FreeBSD.org>
AuthorDate: 2021-01-08 22:36:37 +0000
Commit:     Chuck Tuffli <chuck@FreeBSD.org>
CommitDate: 2021-01-08 22:41:45 +0000

   fix big-endian platforms after 6733401935f8
-------------------------------------------------------------------------------

as the offending commit that leads towards the kernel panic, at least on
the RPi4B.

The following commit boots just fine.
-------------------------------------------------------------------------------
commit a82f07fc2e2456b0ddc847d611f56a3152a6f940
Author:     Dimitry Andric <dim@FreeBSD.org>
AuthorDate: 2021-01-08 22:38:21 +0000
Commit:     Dimitry Andric <dim@FreeBSD.org>
CommitDate: 2021-01-08 22:38:30 +0000

   Fix 32-bit build post 6733401935f83754b4b2744bc3d33ef84b1271e0
-------------------------------------------------------------------------------


KERNCONF is the following:
-------------------------------------------------------------------------------
include		GENERIC

# Extented TCP Congestion Control Methods
options		RATELIMIT
options		TCPHPTS

# Scalable Route Multipath
options		ROUTE_MPATH

# Modular FIB Lookup Framework
options		FIB_ALGO

# IN Kernel TLS -> Disabled PR: 247945
#options		KERN_TLS

# New FENESTRASX Random Number Generator
options		RANDOM_FENESTRASX

# Add CAMDEBUG stuff
#options 	CAMDEBUG
#options 	CAM_DEBUG_FLAGS=(CAM_DEBUG_INFO|CAM_DEBUG_PROBE|CAM_DEBUG_PERIPH)

# pass(4) device
device		pass
options		MMCCAM

nodevice	mmc
nodevice	mmcsd
options IOMMU
-------------------------------------------------------------------------------
Comment 3 Mark Millard 2021-01-12 12:19:26 UTC
Skipping earlier steps in the investigation, the below from
the lists (slightly adjusted) indicates that the failures in
my testing are tied to having a monitor connected . . .

NEAR QUOTE
Going back to my 19cca0b9613d based debug kernel build that
has the printf's reporting the values used in the test, but
with no monitor attached, it boots fine and reports:

pmap_mapdev early_boot: akva_devmap_vaddr: ffff007ffffff000 size: 1000
pmap_mapdev early_boot: va: ffff007fffffe000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000

That compares to the previously reported failure figures from
having the monitor attached for that debug kernel:

pmap_mapdev early_boot: akva_devmap_vaddr: ffff007fff816000 size: 1000
pmap_mapdev early_boot: va: ffff007fff815000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
panic: Too many early devmap mappings

where the code does:

               KASSERT(va >= VM_MAX_KERNEL_ADDRESS - L2_SIZE,
                   ("Too many early devmap mappings"));

Looks like akva_devmap_vaddr gets smaller to make room in high kernel
memory for monitor related data and so va can end up being too small
by the criteria of this test.

I've no clue who would be appropriate for dealing with this.
END NEAR QUOTE
Comment 4 Mark Millard 2021-01-13 06:49:06 UTC
(In reply to Mark Millard from comment #3)

Turns out that this "too much high kernel memory in use" issue happens for
a combination of 2 things being true at the same time:

A) Monitor attached (sufficiently large pixel count?)
B) GDB enabled, per bbfa199cbc16 .

A git bisect with a 1029x1080 monitor attach ended up with:

# git bisect good
bbfa199cbc1698631a0e932848e62dd76559d4d7 is the first bad commit
commit bbfa199cbc1698631a0e932848e62dd76559d4d7
Author: mhorne <mhorne@FreeBSD.org>
Date:   Wed Dec 9 16:38:42 2020 -0400

   arm64: gdb(4) machine-dependent bits

   Everything required for remote kernel debugging over a serial
   connection. For FDT-based systems, a debug port can be specified by
   setting hw.fdt.dbgport to the desired device tree node in loader.conf.
   For example, hw.fdt.dbgport="uart1", or
   hw.fdt.dbgport="serial@ff1a0000".

   Looks good:     emaste
   Tested by:      rwatson
   MFC after:      2 weeks
   Sponsored by:   The FreeBSD Foundation
   Differential Revision:  https://reviews.freebsd.org/D27727

sys/arm64/arm64/gdb_machdep.c   | 112 ++++++++++++++++++++++++++++++++++++++++
sys/arm64/conf/GENERIC          |   2 +-
sys/arm64/include/gdb_machdep.h |  81 +++++++++++++++++++++++++++++
sys/conf/files.arm64            |   1 +
4 files changed, 195 insertions(+), 1 deletion(-)
create mode 100644 sys/arm64/arm64/gdb_machdep.c
create mode 100644 sys/arm64/include/gdb_machdep.h

That, in turn, involves:

-#options 	GDB			# Support remote GDB.
+options 	GDB			# Support remote GDB.
Comment 5 Mitchell Horne freebsd_committer 2021-01-13 17:36:04 UTC
Created attachment 221535 [details]
fix pmap_mapdev assert

Hi Mark,

Thank you describing how to reproduce the issue, I was able to trigger the same issue on my rockpro64 with the HDMI cable attached and GDB enabled. Please try the attached patch and let me know if it fixes the issue for you.

Indeed it appears to be an interesting combination of factors. The crux of the issue is that the KASSERT in pmap_mapdev() assumes a static devmap size of 2MB, when really it has grown to 16MB to accommodate large things like framebuffers. You'll note that pmap_mapdev_attr() does not have this issue.

In my case, the mapped size of the framebuffer (1920x1080 bytes) is slightly smaller than 2MB, meaning it won't trigger the KASSERT if it is the first thing added to the early devmap. The addition of GDB means the uart gets an entry in the static as well, pushing the framebuffer va just below that assertion threshold. Of course, removing the HDMI means the framebuffer isn't mapped, so the issue isn't triggered without it.
Comment 6 Emmanuel Vadot freebsd_committer 2021-01-13 18:52:50 UTC
Patch works for me on pine64 with HDMI connected too.
Thanks as I probably was the one who forgot to change this when I bumped the early mapping size for efifb.
Comment 7 Mark Millard 2021-01-13 19:21:54 UTC
(In reply to Mitchell Horne from comment #5)

The patch worked fine on the 8 GiByte RPi4B: booted like normal and the
monitor is displaying the console like it should.

Details for the test:

# uname -apKU
FreeBSD RPi4B 13.0-CURRENT FreeBSD 13.0-CURRENT #8 pure-src-c255871-g1a816c756003-dirty: Wed Jan 13 11:02:21 PST 2021     root@FBSDFHUGE:/usr/obj/aarch64dbg_default_config_clang/arm64.aarch64/usr/fbsd/pure-src/arm64.aarch64/sys/GENERIC  arm64 aarch64 1300134 1300134

# ~/fbsd-based-on-what-freebsd-main.sh pure-src
1a816c75600335c4482070df055add6f96885d54
CommitDate: 2021-01-11 11:16:42 -0800

# git diff
diff --git a/sys/kern/subr_devmap.c b/sys/kern/subr_devmap.c
index 581e85086f0f..8e07199b7f73 100644
--- a/sys/kern/subr_devmap.c
+++ b/sys/kern/subr_devmap.c
@@ -275,7 +275,7 @@ pmap_mapdev(vm_offset_t pa, vm_size_t size)
        if (early_boot) {
                akva_devmap_vaddr = trunc_page(akva_devmap_vaddr - size);
                va = akva_devmap_vaddr;
-               KASSERT(va >= VM_MAX_KERNEL_ADDRESS - L2_SIZE,
+               KASSERT(va >= VM_MAX_KERNEL_ADDRESS - PMAP_MAPDEV_EARLY_SIZE,
                    ("Too many early devmap mappings"));
        } else
 #endif

And my "1029x1080" was a typing problem: really 1920x1080.
Comment 8 Mitchell Horne freebsd_committer 2021-01-13 21:40:10 UTC
This should be fixed by the following commit. Unfortunately, I incorrectly tagged the PR number. Thanks everyone for your reports and testing.

commit 818390ce0ca539300dd15d7a817784f1e3f7a9b8
Author:     Mitchell Horne <mhorne@FreeBSD.org>
AuthorDate: 2021-01-13 18:30:50 +0000
Commit:     Mitchell Horne <mhorne@FreeBSD.org>
CommitDate: 2021-01-13 21:27:44 +0000

    arm64: fix early devmap assertion

    The purpose of this KASSERT is to ensure that we do not run out of space
    in the early devmap. However, the devmap grew beyond its initial size of
    2MB in r336519, and this assertion did not grow with it.

    A devmap mapping of a 1080p framebuffer requires 1920x1080 bytes, or
    1.977 MB, so it is just barely able to fit without triggering the
    assertion, provided no other devices are mapped before it. With the
    addition of `options GDB` in GENERIC by bbfa199cbc16, the uart is now
    mapped for the purposes of a debug port, before mapping the framebuffer.
    The presence of both these conditions pushes the selected virtual
    address just below the threshold, triggering the assertion.

    To fix this, use the correct size of the devmap, defined by
    PMAP_MAPDEV_EARLY_SIZE. Since this code is shared with RISC-V, define
    it for that platform as well (although it is a different size).

    PR:             25241
    Reported by:    gbe
    MFC after:      3 days
    Sponsored by:   The FreeBSD Foundation
Comment 9 Gordon Bergling freebsd_committer 2021-01-18 14:09:23 UTC
@mhorne, thanks for the quick bugfix!