Bug 253051 - panic: mips_intrcnt_setname: NULL counter ; on qemu-system-mips64 (regression)
Summary: panic: mips_intrcnt_setname: NULL counter ; on qemu-system-mips64 (regression)
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: mips Any
: --- Affects Only Me
Assignee: Oleksandr Tymoshenko
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-01-27 20:33 UTC by Michał Górny
Modified: 2021-02-03 18:37 UTC (History)
3 users (show)

See Also:


Attachments
delay call to cpu_init_interrupts (722 bytes, patch)
2021-01-27 22:34 UTC, Mitchell Horne
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny 2021-01-27 20:33:29 UTC
I'm hitting a kernel panic when trying to boot recent versions of FreeBSD (both main and stable/13) on qemu-system-mips64.  I've been able to establish that this is a regression caused by:

248f0cabca75 make maximum interrupt number tunable on ARM, ARM64, MIPS, and RISC-V


I've been following the build instructions from (to the letter):
https://wiki.freebsd.org/QemuRecipes#mips64

When booting the kernel, I got the following panic:

```
entry: platform_start()
cmd line: kernel  
envp:
	memsize = 268435456
	ememsize = 2147483648
	modetty0 = 38400n8r
memsize = 268435456 (0x10000000)
ememsize = 2147483648
Cache info:
  icache is virtual
  picache_stride    = 8192
  picache_loopcount = 4
  pdcache_stride    = 4096
  pdcache_loopcount = 8
  max line size     = 32
Physical memory chunk(s):
0x951000 - 0xfffffff, 258666496 bytes (63151 pages)
0x90000000 - 0xffffffff, 1879048192 bytes (458752 pages)
Maxmem is 0x100000000
---<<BOOT>>---
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #3 master-bae4112-dirty: Wed Jan 27 20:24:56 UTC 2021
    root@fb:/usr/obj/usr/src/mips.mips64/sys/MALTA64 mips
FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)
Preloaded elf kernel "kernel" at 0xffffffff80949970.
CPU: MIPS Technologies processor v160.130
  MMU: Standard TLB, 48 entries (4K 16K 64K 256K 1M 16M 64M 256M pg sizes)
  L1 i-cache: 4 ways of 256 sets, 32 bytes per line
  L1 d-cache: 4 ways of 256 sets, 32 bytes per line
  L2 cache: disabled
  Config1=0xdea3519b<PerfCount,WatchRegs,EJTAG,FPU>
  FPU ID=0xf8200<S,D,PS,3D,W>
  Config2=0x80000000
real memory  = 2147483648 (2097152K bytes)
Physical memory chunk(s):
0x00a6e000 - 0x0fffffff, 257499136 bytes (62866 pages)
0x90000000 - 0xfcd7bfff, 1826078720 bytes (445820 pages)
avail memory = 2077196288 (1980MB)
panic: mips_intrcnt_setname: NULL counter
time = 1
KDB: enter: panic
[ thread pid 0 tid 0 ]
Stopped at      0x4
```


If I revert the following commit, the kernel boots fine:

commit 248f0cabca75f421553633184b5bf2ffa459d3de
Author: Oleksandr Tymoshenko <gonzo@FreeBSD.org>
Date:   2020-12-30 07:59:03 +0100

    make maximum interrupt number tunable on ARM, ARM64, MIPS, and RISC-V
    
    Use a machdep.nirq tunable intead of compile-time constant NIRQ
    as a value for maximum number of interrupts. It allows keep a system
    footprint small by default with an option to increase the limit
    for large systems like server-grade ARM64
    
    Reviewd by:     mhorne
    Differential Revision:  https://reviews.freebsd.org/D27844
    Submitted by:   Klara, Inc.
    Sponsored by:   Ampere Computing


I don't have any real MIPS hardware to know if it's specific to QEMU or a generic problem.
Comment 1 Mitchell Horne freebsd_committer freebsd_triage 2021-01-27 21:19:01 UTC
Hi, do you have a backtrace from the panic?

By code inspection, I suspect the problem comes from the fact that cpu_init_interrupts() is called before the intrcnt array is allocated.
Comment 2 Michał Górny 2021-01-27 21:58:43 UTC
I'm afraid not:

db> bt
Tracing pid 0 tid 0 td 0xffffffff80900240
--- swapped out

Not even if I swapped kernel.full in place of kernel.
Comment 3 Mitchell Horne freebsd_committer freebsd_triage 2021-01-27 22:34:50 UTC
Created attachment 221970 [details]
delay call to cpu_init_interrupts

(In reply to Michał Górny from comment #2)

That's unfortunate. Can you see if the attached patch fixes it? I have not compiled or tried it myself yet.
Comment 4 Michał Górny 2021-01-27 22:54:41 UTC
It still fails but I can get some backtrace now ;-).

---
entry: platform_start()
cmd line: kernel  
envp:
	memsize = 134217728
	ememsize = 134217728
	modetty0 = 38400n8r
memsize = 134217728 (0x08000000)
ememsize = 134217728
Cache info:
  icache is virtual
  picache_stride    = 8192
  picache_loopcount = 4
  pdcache_stride    = 4096
  pdcache_loopcount = 8
  max line size     = 32
Physical memory chunk(s):
0x94e000 - 0x7ffffff, 124461056 bytes (30386 pages)
Maxmem is 0x8000000
---<<BOOT>>---
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-ALPHA2 #8 master-bae4112-dirty: Wed Jan 27 22:47:19 UTC 2021
    root@fb:/usr/obj/usr/src/mips.mips64/sys/MALTA64 mips
FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)
Preloaded elf kernel "kernel" at 0xffffffff80947930.
CPU: MIPS Technologies processor v160.130
  MMU: Standard TLB, 48 entries (4K 16K 64K 256K 1M 16M 64M 256M pg sizes)
  L1 i-cache: 4 ways of 256 sets, 32 bytes per line
  L1 d-cache: 4 ways of 256 sets, 32 bytes per line
  L2 cache: disabled
  Config1=0xdea3519b<PerfCount,WatchRegs,EJTAG,FPU>
  FPU ID=0xf8200<S,D,PS,3D,W>
  Config2=0x80000000
real memory  = 134217728 (131072K bytes)
Physical memory chunk(s):
0x00a6b000 - 0x07d14fff, 120233984 bytes (29354 pages)
avail memory = 119300096 (113MB)
random: no preloaded entropy cache
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
hostuuid: using 00000000-0000-0000-0000-000000000000
panic: mips_intrcnt_setname: NULL counter
time = 1
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at      0x4
db> bt
Tracing pid 0 tid 100000 td 0xffffffff80900240
kdb_enter+0x94 (?,?,?,?) ra ffffffff8046e474 sp 9800000000967b70 sz 32
vpanic+0x23c (?,?,?,?) ra ffffffff8046e0a8 sp 9800000000967b90 sz 64
panic+0x30 (?,?,?,?) ra ffffffff80772288 sp 9800000000967bd0 sz 80
cpu_init_interrupts+0x318 (?,?,?,?) ra ffffffff80755720 sp 9800000000967c20 sz 128
is_cacheable_mem+0x2d8 (?,?,?,?) ra ffffffff803d4e68 sp 9800000000967ca0 sz 16
mi_startup+0x280 (?,?,?,?) ra ffffffff80100404 sp 9800000000967cb0 sz 96
btext+0xdc (?,?,?,?) ra 0 sp 9800000000967d10 sz 0
Comment 5 Oleksandr Tymoshenko freebsd_committer freebsd_triage 2021-01-28 06:30:08 UTC
Ooops, sorry for the breakage. Mitchell, are you looking into this or should I take this PR?
Comment 6 Mitchell Horne freebsd_committer freebsd_triage 2021-01-28 13:45:21 UTC
(In reply to Oleksandr Tymoshenko from comment #5)

Please take it. I will not have a chance to debug any further this week.
Comment 7 Oleksandr Tymoshenko freebsd_committer freebsd_triage 2021-01-30 06:23:34 UTC
This should fix the problem: https://reviews.freebsd.org/D28424
Let me know if the patch works for you.
Comment 8 commit-hook freebsd_committer freebsd_triage 2021-01-31 21:45:06 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=e0a0a3efcb09a10ad1de29aca622ea580b2663d2

commit e0a0a3efcb09a10ad1de29aca622ea580b2663d2
Author:     Oleksandr Tymoshenko <gonzo@FreeBSD.org>
AuthorDate: 2021-01-30 06:11:24 +0000
Commit:     Oleksandr Tymoshenko <gonzo@FreeBSD.org>
CommitDate: 2021-01-31 21:44:45 +0000

    mips: fix early kernel panic when setting up interrupt counters

    Commit 248f0ca converted intrcnt and intrnames from u_long[]
    and char[] to u_long* and char* respectively, but for non-INTRNG mips
    these symbols were defined in .S file as a pre-allocated static arrays,
    so the problem wasn't cought at compile time. Conversion from an array
    to a pointer requires pointer initialization and it wasn't done
    for MIPS, so whatever happenned to be in the begginning of intcnt[]
    array was used as a pointer value.

    Move intrcnt/intrnames to C code and allocate them dynamically
    although with a fixed size at the moment.

    Reviewed by:    emaste
    PR:             253051
    Differential Revision:  https://reviews.freebsd.org/D28424
    MFC after:      1 day

 sys/mips/mips/exception.S    | 36 ------------------------------------
 sys/mips/mips/intr_machdep.c | 22 ++++++++++++++++++++++
 2 files changed, 22 insertions(+), 36 deletions(-)
Comment 9 commit-hook freebsd_committer freebsd_triage 2021-02-01 07:59:36 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=d6f9c5a6d2f87865f9714d2b8dfd1a9f3080c71e

commit d6f9c5a6d2f87865f9714d2b8dfd1a9f3080c71e
Author:     Oleksandr Tymoshenko <gonzo@FreeBSD.org>
AuthorDate: 2021-02-01 07:56:22 +0000
Commit:     Oleksandr Tymoshenko <gonzo@FreeBSD.org>
CommitDate: 2021-02-01 07:56:22 +0000

    mips: fix NLM platforms breakage caused by e0a0a3ef

    NetLogic platforms have their own implementation of cpu_init_interrupts.
    Apply the same logic to it as to intr_machdep.c.

    PR:     253051

 sys/mips/nlm/intr_machdep.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
Comment 10 commit-hook freebsd_committer freebsd_triage 2021-02-03 05:07:36 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=10625515ad47b52f38c13f4f1d8f8c848e9df4ee

commit 10625515ad47b52f38c13f4f1d8f8c848e9df4ee
Author:     Oleksandr Tymoshenko <gonzo@FreeBSD.org>
AuthorDate: 2021-02-01 07:56:22 +0000
Commit:     Oleksandr Tymoshenko <gonzo@FreeBSD.org>
CommitDate: 2021-02-03 05:06:37 +0000

    MFC: mips: fix early kernel panic when setting up interrupt counters

    Commit 248f0ca converted intrcnt and intrnames from u_long[]
    and char[] to u_long* and char* respectively, but for non-INTRNG mips
    these symbols were defined in .S file as a pre-allocated static arrays,
    so the problem wasn't cought at compile time. Conversion from an array
    to a pointer requires pointer initialization and it wasn't done
    for MIPS, so whatever happenned to be in the begginning of intcnt[]
    array was used as a pointer value.

    Move intrcnt/intrnames to C code and allocate them dynamically
    although with a fixed size at the moment.

    Reviewed by:    emaste
    PR:             253051
    Differential Revision:  https://reviews.freebsd.org/D28424
    MFC after:      1 day

    (cherry picked from commit e0a0a3efcb09a10ad1de29aca622ea580b2663d2)

    mips: fix NLM platforms breakage caused by e0a0a3ef

    NetLogic platforms have their own implementation of cpu_init_interrupts.
    Apply the same logic to it as to intr_machdep.c.

    PR:     253051

    (cherry picked from commit d6f9c5a6d2f87865f9714d2b8dfd1a9f3080c71e)

 sys/mips/mips/exception.S    | 36 ------------------------------------
 sys/mips/mips/intr_machdep.c | 22 ++++++++++++++++++++++
 sys/mips/nlm/intr_machdep.c  | 18 ++++++++++++++++++
 3 files changed, 40 insertions(+), 36 deletions(-)
Comment 11 commit-hook freebsd_committer freebsd_triage 2021-02-03 05:07:37 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=10625515ad47b52f38c13f4f1d8f8c848e9df4ee

commit 10625515ad47b52f38c13f4f1d8f8c848e9df4ee
Author:     Oleksandr Tymoshenko <gonzo@FreeBSD.org>
AuthorDate: 2021-02-01 07:56:22 +0000
Commit:     Oleksandr Tymoshenko <gonzo@FreeBSD.org>
CommitDate: 2021-02-03 05:06:37 +0000

    MFC: mips: fix early kernel panic when setting up interrupt counters

    Commit 248f0ca converted intrcnt and intrnames from u_long[]
    and char[] to u_long* and char* respectively, but for non-INTRNG mips
    these symbols were defined in .S file as a pre-allocated static arrays,
    so the problem wasn't cought at compile time. Conversion from an array
    to a pointer requires pointer initialization and it wasn't done
    for MIPS, so whatever happenned to be in the begginning of intcnt[]
    array was used as a pointer value.

    Move intrcnt/intrnames to C code and allocate them dynamically
    although with a fixed size at the moment.

    Reviewed by:    emaste
    PR:             253051
    Differential Revision:  https://reviews.freebsd.org/D28424
    MFC after:      1 day

    (cherry picked from commit e0a0a3efcb09a10ad1de29aca622ea580b2663d2)

    mips: fix NLM platforms breakage caused by e0a0a3ef

    NetLogic platforms have their own implementation of cpu_init_interrupts.
    Apply the same logic to it as to intr_machdep.c.

    PR:     253051

    (cherry picked from commit d6f9c5a6d2f87865f9714d2b8dfd1a9f3080c71e)

 sys/mips/mips/exception.S    | 36 ------------------------------------
 sys/mips/mips/intr_machdep.c | 22 ++++++++++++++++++++++
 sys/mips/nlm/intr_machdep.c  | 18 ++++++++++++++++++
 3 files changed, 40 insertions(+), 36 deletions(-)