Bug 275830

Comment 1 Kristof Provost 2023-12-18 20:08:05 UTC

You're going to have to attach a serial port (via the microUSB port) and investigate from there. There's absolutely no way to debug this based on the information provided.

Thanks, I tried to connect with my macbook, but didn't get any signals after it froze. Will try with a freebsd box next time. 

Do you think I need to keep the serial console attached until it freezes, or hooking it up after it froze will be sufficient?

Comment 3 Kristof Provost 2023-12-18 20:25:09 UTC

(In reply to Espen from comment #2)
I have no idea.

Make sure your console works while it's running at least.

I cought a crash in /var/crash this night. But it didn't freeze it cycled the machine.

I can provide an archive from the /var/crash directory of needed.

First time I started the debugger, so this might not be helpful, but here goes:

# kgdb /boot/kernel/kernel -n 0
GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:
spin lock 0xffffffff82896440 (sched lock 0) held by 0xfffffe0008c9c3a0 (tid 100003) too long
timeout stopping cpus
panic: spin lock held too long
cpuid = 1
time = 1703112000
KDB: stack backtrace:
#0 0xffffffff80b9009d at kdb_backtrace+0x5d
#1 0xffffffff80b431a2 at vpanic+0x132
#2 0xffffffff80b43063 at panic+0x43
#3 0xffffffff80b20067 at _mtx_lock_indefinite_check+0x67
#4 0xffffffff80b1fc35 at _mtx_lock_spin_cookie+0xd5
#5 0xffffffff80b7640f at sched_add+0x27f
#6 0xffffffff80b60f42 at callout_process+0x2c2
#7 0xffffffff80ad0d40 at handleevents+0x180
#8 0xffffffff80ad26fa at cpu_activeclock+0x6a
#9 0xffffffff80fd1286 at cpu_idle+0xa6
#10 0xffffffff80b78116 at sched_idletd+0x576
#11 0xffffffff80afdb7f at fork_exit+0x7f
#12 0xffffffff80fe4a9e at fork_trampoline+0xe
Uptime: 2d3h34m28s
Dumping 513 out of 4038 MB:..4%..13%..22%..32%..41%..53%..63%..72%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57
57	/usr/src/sys/amd64/include/pcpu_aux.h: No such file or directory.
(kgdb)

I was able to get a freeze with the serial console attached. There was no relevant information displayed. I can share a screenshot if needed.

Meanwhile I've done a few things that I thought might help, but didn't:
- Disabled igmpproxy
- Updated the BIOS: https://pcengines.github.io/

I also looked for similar issues on pfsense with no luck. I know a lot of those APU boards are used there and they've been on 14 for a while:
2023-12-07 14.0-CURRENT@0c783a37d5d5 via https://docs.netgate.com/pfsense/en/latest/releases/versions.html

I'll run a memtest the next days and if that's all good, I'll most likely revert back to 13.x or is there anything else I can do?

Best, Espen

Hi

Meanwhile I did a re-install from scratch on 14.0-RELEASE. That did not help either, so I decided to roll back to 13.2 and the router is stable and running without freezes since 16 days.

There's something with 14 that crashes this particular board it seems.

I started poking around in the git commits and realized that pfsense branched off much earlier than 14-RELEASE:
14.0-CURRENT@0c783a37d5d5

commit 0c783a37d5d52af8cd706eb462e73801e77c6b35
Author: Justin Hibbits <jhibbits@FreeBSD.org>
Date:   Wed Aug 2 10:05:21 2023 -0400

That could explain why there's not more reports with this board. There's a bigger community on pfsense running these boards I assume than FreeBSD.

I have a replacement board that I can test lying here. But as long as it's stable on 13.2 I don't see a reason.

Replaced the board after another freeze. And it has been running stable on 13.2 since 17 days. 

Many thanks to pcengines for sending a replacement board so fast!

Closing the gug-report.