Bug 276405 - panic in sockbuf code on TCP socket close(2)
Summary: panic in sockbuf code on TCP socket close(2)
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 13.2-RELEASE
Hardware: powerpc Any
: --- Affects Only Me
Assignee: freebsd-ppc (Nobody)
URL:
Keywords: crash
Depends on:
Blocks:
 
Reported: 2024-01-17 15:49 UTC by Luciano Mannucci
Modified: 2024-02-23 10:24 UTC (History)
6 users (show)

See Also:

Attachments
The whole core3.txt, for reference. (149.96 KB, text/plain)
2024-01-25 13:30 UTC, Luciano Mannucci 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Luciano Mannucci 2024-01-17 15:49:02 UTC 
On heavy io, kernel panics.

From info.0:

Dump header from device: /dev/da1p1
  Architecture: powerpc64
  Architecture Version: 1
  Dump Length: 1728684032
  Blocksize: 512
  Compression: none
  Dumptime: 2024-01-17 14:30:06 +0100
  Hostname: ekaguaro
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 13.2-RELEASE-p9 releng/13.2-n254652-c78c31d2ef40 GENERIC
  Panic String: data storage interrupt trap
  Dump Parity: 816171299
  Bounds: 0
  Dump Status: good

core0.txt:
Unable to find a kernel debugger.
Please install the devel/gdb port or gdb package.

There is no gdb installed on this machine. I'll install it.

On another sibling I get:

info.1:

Dump header from device: /dev/vtbd0s2b
  Architecture: powerpc64
  Architecture Version: 1
  Dump Length: 903544832
  Blocksize: 512
  Compression: none
  Dumptime: 2024-01-16 11:36:57 +0100
  Hostname: numeron
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 13.2-RELEASE-p9 releng/13.2-n254652-c78c31d2ef40 GENERIC
  Panic String: data storage interrupt trap
  Dump Parity: 1018450777
  Bounds: 1
  Dump Status: good

core.txt.1:

umeron dumped core - see /var/crash/vmcore.1

Tue Jan 16 11:38:42 CET 2024

FreeBSD numeron 13.2-RELEASE-p9 FreeBSD 13.2-RELEASE-p9 releng/13.2-n254652-c78c31d2ef40 GENERIC  powerpc

panic: data storage interrupt trap

GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64-portbld-freebsd13.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
Failed to open vmcore: invalid corefile
(kgdb) 

Fatal signal: Segmentation fault
----- Backtrace -----
0x107c795f ???
0x10931e0f ???
0x10932a8b ???
0x812d05187 handle_signal
	/usr/src/lib/libthr/thread/thr_sig.c:303
0x812d04343 thr_sighandler
	/usr/src/lib/libthr/thread/thr_sig.c:246
---------------------
A fatal error internal to GDB has been detected, further
debugging is not possible.  GDB will now terminate.

This is a bug, please report it.  For instructions, see:
<https://www.gnu.org/software/gdb/bugs/>.

Segmentation fault (core dumped)


------------------------------------------------------------------------
ps -axlww

ps: invalid corefile

------------------------------------------------------------------------
vmstat -s

vmstat: kvm_openfiles: invalid corefile

------------------------------------------------------------------------
vmstat -m

vmstat: kvm_openfiles: invalid corefile

------------------------------------------------------------------------
vmstat -z

vmstat: kvm_openfiles: invalid corefile

------------------------------------------------------------------------
vmstat -i

vmstat: kvm_openfiles: invalid corefile

------------------------------------------------------------------------
pstat -T

pstat: kvm_openfiles: invalid corefile

------------------------------------------------------------------------
pstat -s

pstat: kvm_openfiles: invalid corefile

------------------------------------------------------------------------
iostat

iostat: kvm_openfiles: invalid corefile

------------------------------------------------------------------------
ipcs -a

ipcs: kvm_openfiles: invalid corefile

------------------------------------------------------------------------
ipcs -T

ipcs: kvm_openfiles: invalid corefile

------------------------------------------------------------------------
netstat -s

netstat: kvm not available: invalid corefile

------------------------------------------------------------------------
netstat -m

netstat: kvm not available: invalid corefile
netstat: kvm not available: invalid corefile

------------------------------------------------------------------------
netstat -anA

netstat: kvm not available: invalid corefile

------------------------------------------------------------------------
netstat -aL

netstat: kvm not available: invalid corefile

------------------------------------------------------------------------
fstat

fstat: kvm_openfiles(): invalid corefile
fstat: procstat_open()

------------------------------------------------------------------------
dmesg

dmesg: invalid corefile

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	GENERIC
machine	powerpc
cpu	AIM
makeoptions	WITH_CTF=1
makeoptions	DEBUG=-g
options	HID_DEBUG
options	EVDEV_SUPPORT
options	ATH_ENABLE_11N
options	AH_AR5416_INTERRUPT_MITIGATION
options	IEEE80211_DEBUG
options	IEEE80211_SUPPORT_MESH
options	KBD_INSTALL_CDEV
options	USB_DEBUG
options	AHC_ALLOW_MEMIO
options	NVME_USE_NVD=0
options	PCI_HP
options	SMP
options	NETDUMP
options	DEBUGNET
options	ZSTDIO
options	GZIO
options	EKCD
options	KDB_TRACE
options	KDB
options	RCTL
options	RACCT_DEFAULT_TO_DISABLED
options	RACCT
options	INCLUDE_CONFIG_FILE
options	DDB_CTF
options	KDTRACE_HOOKS
options	MAC
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	HWPMC_HOOKS
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD12
options	COMPAT_FREEBSD11
options	COMPAT_FREEBSD10
options	COMPAT_FREEBSD9
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD32
options	GEOM_LABEL
options	GEOM_PART_GPT
options	GEOM_PART_APM
options	TMPFS
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	MD_ROOT_MEM
options	MD_ROOT
options	QUOTA
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP_SUPPORT
options	TCP_RFC7413
options	TCP_HHOOK
options	TCP_BLACKBOX
options	TCP_OFFLOAD
options	ROUTE_MPATH
options	IPSEC_SUPPORT
options	INET6
options	INET
options	VIMAGE
options	PREEMPTION
options	NUMA
options	SCHED_ULE
options	FDT
options	POWERNV
options	PSERIES
options	QEMU
options	MAMBO
options	PS3
options	POWERMAC
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_BSD
device	mem
device	uart_ns8250
device	cpufreq
device	pci
device	agp
device	ahci
device	ata
device	mvs
device	siis
device	nvme
device	nvd
device	aacraid
device	ahc
device	isp
device	ispfw
device	mrsas
device	mpt
device	mps
device	mpr
device	sym
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	vt
device	kbdmux
device	scc
device	uart
device	uart_z8530
device	iflib
device	em
device	ix
device	ixl
device	ixv
device	glc
device	llan
device	cxgbe
device	miibus
device	bge
device	gem
device	dc
device	fxp
device	re
device	rl
device	crypto
device	loop
device	ether
device	vlan
device	tuntap
device	md
device	ofwd
device	gif
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	xhci
device	usb
device	uhid
device	ukbd
device	umass
device	ums
device	aue
device	axe
device	cdce
device	cue
device	kue
device	wlan
device	wlan_wep
device	wlan_ccmp
device	wlan_tkip
device	wlan_amrr
device	ath
device	ath_pci
device	ath_hal
device	ath_rate_sample
device	firewire
device	sbp
device	fwe
device	iicbus
device	iic
device	kiic
device	ad7417
device	ds1631
device	ds1775
device	fcu
device	max6690
device	powermac_nvram
device	smu
device	atibl
device	nvbl
device	opalflash
device	adb
device	pmu
device	sound
device	snd_ai2s
device	snd_hda
device	snd_uaudio
device	netmap
device	evdev
device	uinput
device	virtio
device	virtio_pci
device	vtnet
device	virtio_blk
device	virtio_scsi
device	virtio_balloon
device	hid

------------------------------------------------------------------------
ddb capture buffer

ddb: ddb_capture: kvm_openfiles: invalid corefile
Comment 1 Luciano Mannucci 2024-01-17 15:53:33 UTC 
Of course vmcore are a bit big; I can upload them somewhere if needed.
Comment 2 Mark Johnston freebsd_committer freebsd_triage 2024-01-19 17:08:14 UTC 
We at least need a backtrace to see where the kernel crashed.

> Reading symbols from /boot/kernel/kernel...
> Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
> Failed to open vmcore: invalid corefile

Perhaps there's a problem with kernel dumps on powerpc?
Comment 3 Robert Wing freebsd_committer freebsd_triage 2024-01-20 19:06:19 UTC 
can you try the following patch: https://reviews.freebsd.org/D43526
Comment 4 Robert Wing freebsd_committer freebsd_triage 2024-01-20 19:08:04 UTC 
(In reply to Robert Wing from comment #3)

That patch isn't to fix the panic...but to access the corefile.
Comment 5 Luciano Mannucci 2024-01-21 11:55:16 UTC 
(In reply to Robert Wing from comment #4)
I guess I don't need to rebuild the kernel, I'm rebuilding just
the library...

What do I need to run to make a working core.txt without waiting for a new panic?
Comment 6 Luciano Mannucci 2024-01-24 09:52:38 UTC 
(In reply to Robert Wing from comment #4)
Ok, I got a new crash last night. After the patch I have a reasonable debug output in the core.txt file:

less core.txt.3
numeron dumped core - see /var/crash/vmcore.3

Tue Jan 23 22:16:20 CET 2024

FreeBSD numeron 13.2-RELEASE-p9 FreeBSD 13.2-RELEASE-p9 #5 releng/13.2-n254652-c78c31d2ef40-dirty: Sun Jan 21 16:01:55 CET 2024     root@pippozzo:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64  powerpc

panic: data storage interrupt trap

GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64-portbld-freebsd13.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:

fatal kernel trap:

   exception       = 0x300 (data storage interrupt)
   virtual address = 0x8
   dsisr           = 0x40000000
   srr0            = 0xc00000000095e378 (0x95e378)
   srr1            = 0x8000000000009032
   current msr     = 0x8000000000009032
   lr              = 0xc00000000095e1d0 (0x95e1d0)
   frame           = 0xc008000064155e50
   curthread       = 0xc00800001dd52b00
          pid = 77305, comm = smtpd

panic: data storage interrupt trap
cpuid = 0
time = 1706030399
KDB: stack backtrace:
#0 0xc0000000008ef154 at kdb_backtrace+0x90
#1 0xc00000000087ead0 at vpanic+0x1b8
#2 0xc00000000087e904 at panic+0x44
#3 0xc000000000d5e9e0 at trap+0x304
#4 0xc000000000d52d84 at powerpc_interrupt+0x1b4
Uptime: 1d9h32m2s
Dumping 869 out of 4052 MB:..1%

0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
396     /usr/src/sys/kern/kern_shutdown.c: No such file or directory.
(kgdb) #0  0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
#1  0xc00000000087e424 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:484
#2  0xc00000000087eb8c in vpanic (fmt=0xc00000000119ed18 "%s trap", 
    ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:923
#3  0xc00000000087e908 in panic (fmt=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:847
#4  0xc000000000d5e9e4 in trap_fatal (frame=0xc008000064155e50)
    at /usr/src/sys/powerpc/powerpc/trap.c:534
#5  trap (frame=0xc008000064155e50) at /usr/src/sys/powerpc/powerpc/trap.c:494
#6  0xc000000000d52d88 in powerpc_interrupt (framep=0xc008000064155e50)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#7  <signal handler called>
#8  sbcut_internal (sb=0x9b8df8a8, len=20779031)
    at /usr/src/sys/kern/uipc_sockbuf.c:1497
#9  0xc00000000095e1d0 in sbflush_internal (sb=<optimized out>)
    at /usr/src/sys/kern/uipc_sockbuf.c:1437
#10 sbflush_locked (sb=<optimized out>)
    at /usr/src/sys/kern/uipc_sockbuf.c:1449
#11 sbflush (sb=0x9b8df8a8) at /usr/src/sys/kern/uipc_sockbuf.c:1457
#12 0xc000000000b15ce8 in tcp_disconnect (tp=0xc00800001dd0b0e0)
    at /usr/src/sys/netinet/tcp_usrreq.c:2585
#13 0xc000000000b0f5c8 in tcp_usr_disconnect (so=<optimized out>)
    at /usr/src/sys/netinet/tcp_usrreq.c:765
#14 0xc000000000961920 in sodisconnect (so=0x9b8df760)
    at /usr/src/sys/kern/uipc_socket.c:1370
#15 soclose (so=0x9b8df760) at /usr/src/sys/kern/uipc_socket.c:1192
#16 0xc00000000092dcb8 in soo_close (fp=<optimized out>, td=<optimized out>)
    at /usr/src/sys/kern/sys_socket.c:361
#17 0xc0000000007ef3f4 in fo_close (fp=0x96a75c80, td=<optimized out>)
    at /usr/src/sys/sys/file.h:384
#18 _fdrop (fp=0x96a75c80, td=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:3691
#19 0xc0000000007f4b58 in closef (fp=0x96a75c80, td=0xc00800001dd52b00)
    at /usr/src/sys/kern/kern_descrip.c:2937
#20 0xc0000000007faaa4 in closefp_impl (fdp=0xc00800001dd830c0, fd=15, 
    fp=0x96a75c80, td=0xc00800001dd52b00, audit=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:1307
#21 0xc0000000007f0e78 in closefp (fdp=0xc00800001dd830c0, fp=0x96a75c80, 
    td=0xc00800001dd52b00, holdleaders=true, audit=true, fd=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:1364
#22 kern_close (td=0xc00800001dd52b00, fd=15)
    at /usr/src/sys/kern/kern_descrip.c:1400
#23 0xc0000000007f0ca0 in sys_close (td=<optimized out>, uap=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:1381
#24 0xc000000000d5f3c0 in syscallenter (td=0xc00800001dd52b00)
    at /usr/src/sys/powerpc/powerpc/../../kern/subr_syscall.c:190
#25 syscall (frame=<optimized out>) at /usr/src/sys/powerpc/powerpc/trap.c:759
#26 0xc000000000d5ec70 in trap (frame=0xc008000064156810)
    at /usr/src/sys/powerpc/powerpc/trap.c:292
#27 0xc000000000d52d88 in powerpc_interrupt (framep=0xc008000064156810)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#28 <signal handler called>
#29 0x0000000810e5b1f0 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(kgdb) 

------------------------------------------------------------------------
I can attach the whole core.txt file if needed.

Luciano.
Comment 7 Luciano Mannucci 2024-01-25 13:30:57 UTC 
Created attachment 247948 [details]
The whole core3.txt, for reference.

Now I have several core.txt files, if needed.
Comment 8 Mark Johnston freebsd_committer freebsd_triage 2024-01-25 14:36:45 UTC 
(In reply to Luciano Mannucci from comment #7)
Do they all show the same panic message and backtrace?

cc'ing a few TCP folks, since the initial crash happened while shutting down a TCP socket.
Comment 9 Luciano Mannucci 2024-01-25 22:05:02 UTC 
(In reply to Mark Johnston from comment #8)
these are the backtraces (I can attach full core.txts):

core.txt.4
fatal kernel trap:

   exception       = 0x300 (data storage interrupt)
   virtual address = 0x10105ea05a8
   dsisr           = 0x40000000
   srr0            = 0xc000000000843e7c (0x843e7c)
   srr1            = 0x8000000000009032
   current msr     = 0x8000000000009032
   lr              = 0xc000000000842348 (0x842348)
   frame           = 0xc00800002694f1a0
   curthread       = 0xc008000002dcd100
          pid = 12, comm = irq4101: xhci0+

panic: data storage interrupt trap
cpuid = 2
time = 1706131844
KDB: stack backtrace:
#0 0xc0000000008ef154 at kdb_backtrace+0x90
#1 0xc00000000087ead0 at vpanic+0x1b8
#2 0xc00000000087e904 at panic+0x44
#3 0xc000000000d5e9e0 at trap+0x304
#4 0xc000000000d52d84 at powerpc_interrupt+0x1b4
Uptime: 1d0h14m54s
Dumping 848 out of 4052 MB:..1%

0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
396     /usr/src/sys/kern/kern_shutdown.c: No such file or directory.
(kgdb) #0  0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
#1  0xc00000000087e424 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:484
#2  0xc00000000087eb8c in vpanic (fmt=0xc00000000119ed18 "%s trap", 
    ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:923
#3  0xc00000000087e908 in panic (fmt=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:847
#4  0xc000000000d5e9e4 in trap_fatal (frame=0xc00800002694f1a0)
    at /usr/src/sys/powerpc/powerpc/trap.c:534
#5  trap (frame=0xc00800002694f1a0) at /usr/src/sys/powerpc/powerpc/trap.c:494
#6  0xc000000000d52d88 in powerpc_interrupt (framep=0xc00800002694f1a0)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#7  <signal handler called>
#8  mb_free_ext (m=0x1013de00) at /usr/src/sys/kern/kern_mbuf.c:1211
#9  0xc000000000842348 in m_free (m=0x1013de00)
    at /usr/src/sys/sys/mbuf.h:1477
#10 0xc000000000843c6c in m_freem (mb=0x1013de00)
    at /usr/src/sys/kern/kern_mbuf.c:1553
#11 0xc0000000006a3b68 in vtnet_rxq_merged_eof (rxq=0x42a3600, 
    m_head=0x1013de00, nbufs=<optimized out>)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:1966
#12 vtnet_rxq_eof (rxq=0x42a3600)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2110
#13 0xc0000000006a3324 in vtnet_rx_vq_process (rxq=0x42a3600, tries=-1)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2211
#14 0xc000000000698420 in virtqueue_intr (vq=<optimized out>)
    at /usr/src/sys/dev/virtio/virtqueue.c:472
#15 0xc00000000069ad78 in vtpci_intx_intr (xcn=0x4128308)
    at /usr/src/sys/dev/virtio/pci/virtio_pci.c:868
#16 0xc00000000081da44 in intr_event_execute_handlers (ie=0x432b300, 
    p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1169
#17 ithread_execute_handlers (ie=0x432b300, p=<optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1182
#18 ithread_loop (arg=0x43306e0) at /usr/src/sys/kern/kern_intr.c:1270
#19 0xc00000000081841c in fork_exit (
    callout=0xc00000000081d5ec <ithread_loop>, arg=0x43306e0, 
    frame=0xc00800002694f940) at /usr/src/sys/kern/kern_fork.c:1093
#20 0xc000000000d5dbbc in fork_trampoline ()
    at /usr/src/sys/powerpc/powerpc/swtch64.S:356
Backtrace stopped: frame did not save the PC
(kgdb) 

------------------------------------------------------------------------
core.txt.5
fatal kernel trap:

   exception       = 0x300 (data storage interrupt)
   virtual address = 0x8
   dsisr           = 0x40000000
   srr0            = 0xc00000000095e378 (0x95e378)
   srr1            = 0x8000000000009032
   current msr     = 0x8000000000009032
   lr              = 0xc00000000095e1d0 (0x95e1d0)
   frame           = 0xc008000065129e50
   curthread       = 0xc00800001dc0a000
          pid = 8211, comm = smtpd

panic: data storage interrupt trap
cpuid = 0
time = 1706155244
KDB: stack backtrace:
#0 0xc0000000008ef154 at kdb_backtrace+0x90
#1 0xc00000000087ead0 at vpanic+0x1b8
#2 0xc00000000087e904 at panic+0x44
#3 0xc000000000d5e9e0 at trap+0x304
#4 0xc000000000d52d84 at powerpc_interrupt+0x1b4
Uptime: 6h29m4s
Dumping 845 out of 4052 MB:..1%

0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
396     /usr/src/sys/kern/kern_shutdown.c: No such file or directory.
(kgdb) #0  0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
#1  0xc00000000087e424 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:484
#2  0xc00000000087eb8c in vpanic (fmt=0xc00000000119ed18 "%s trap", 
    ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:923
#3  0xc00000000087e908 in panic (fmt=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:847
#4  0xc000000000d5e9e4 in trap_fatal (frame=0xc008000065129e50)
    at /usr/src/sys/powerpc/powerpc/trap.c:534
#5  trap (frame=0xc008000065129e50) at /usr/src/sys/powerpc/powerpc/trap.c:494
#6  0xc000000000d52d88 in powerpc_interrupt (framep=0xc008000065129e50)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#7  <signal handler called>
#8  sbcut_internal (sb=0x4cf78a8, len=20779031)
    at /usr/src/sys/kern/uipc_sockbuf.c:1497
#9  0xc00000000095e1d0 in sbflush_internal (sb=<optimized out>)
    at /usr/src/sys/kern/uipc_sockbuf.c:1437
#10 sbflush_locked (sb=<optimized out>)
    at /usr/src/sys/kern/uipc_sockbuf.c:1449
#11 sbflush (sb=0x4cf78a8) at /usr/src/sys/kern/uipc_sockbuf.c:1457
#12 0xc000000000b15ce8 in tcp_disconnect (tp=0xc00800001dc83870)
    at /usr/src/sys/netinet/tcp_usrreq.c:2585
#13 0xc000000000b0f5c8 in tcp_usr_disconnect (so=<optimized out>)
    at /usr/src/sys/netinet/tcp_usrreq.c:765
#14 0xc000000000961920 in sodisconnect (so=0x4cf7760)
    at /usr/src/sys/kern/uipc_socket.c:1370
#15 soclose (so=0x4cf7760) at /usr/src/sys/kern/uipc_socket.c:1192
#16 0xc00000000092dcb8 in soo_close (fp=<optimized out>, td=<optimized out>)
    at /usr/src/sys/kern/sys_socket.c:361
#17 0xc0000000007ef3f4 in fo_close (fp=0x84819780, td=<optimized out>)
    at /usr/src/sys/sys/file.h:384
#18 _fdrop (fp=0x84819780, td=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:3691
#19 0xc0000000007f4b58 in closef (fp=0x84819780, td=0xc00800001dc0a000)
    at /usr/src/sys/kern/kern_descrip.c:2937
#20 0xc0000000007faaa4 in closefp_impl (fdp=0xc00800001d7ba4f0, fd=15, 
    fp=0x84819780, td=0xc00800001dc0a000, audit=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:1307
#21 0xc0000000007f0e78 in closefp (fdp=0xc00800001d7ba4f0, fp=0x84819780, 
    td=0xc00800001dc0a000, holdleaders=true, audit=true, fd=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:1364
#22 kern_close (td=0xc00800001dc0a000, fd=15)
    at /usr/src/sys/kern/kern_descrip.c:1400
#23 0xc0000000007f0ca0 in sys_close (td=<optimized out>, uap=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:1381
#24 0xc000000000d5f3c0 in syscallenter (td=0xc00800001dc0a000)
    at /usr/src/sys/powerpc/powerpc/../../kern/subr_syscall.c:190
#25 syscall (frame=<optimized out>) at /usr/src/sys/powerpc/powerpc/trap.c:759
#26 0xc000000000d5ec70 in trap (frame=0xc00800006512a810)
    at /usr/src/sys/powerpc/powerpc/trap.c:292
#27 0xc000000000d52d88 in powerpc_interrupt (framep=0xc00800006512a810)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#28 <signal handler called>
#29 0x0000000810e5b1f0 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(kgdb) 

------------------------------------------------------------------------
core.txt.6
fatal kernel trap:

   exception       = 0x300 (data storage interrupt)
   virtual address = 0x8
   dsisr           = 0x40000000
   srr0            = 0xc00000000095e378 (0x95e378)
   srr1            = 0x8000000000009032
   current msr     = 0x8000000000009032
   lr              = 0xc00000000095e394 (0x95e394)
   frame           = 0xc00800002694eaa0
   curthread       = 0xc008000002dcd100
          pid = 12, comm = irq4101: xhci0+

panic: data storage interrupt trap
cpuid = 1
time = 1706176901
KDB: stack backtrace:
#0 0xc0000000008ef154 at kdb_backtrace+0x90
#1 0xc00000000087ead0 at vpanic+0x1b8
#2 0xc00000000087e904 at panic+0x44
#3 0xc000000000d5e9e0 at trap+0x304
#4 0xc000000000d52d84 at powerpc_interrupt+0x1b4
Uptime: 2h20m42s
Dumping 769 out of 4052 MB:..1%

0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
396     /usr/src/sys/kern/kern_shutdown.c: No such file or directory.
(kgdb) #0  0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
#1  0xc00000000087e424 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:484
#2  0xc00000000087eb8c in vpanic (fmt=0xc00000000119ed18 "%s trap", 
    ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:923
#3  0xc00000000087e908 in panic (fmt=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:847
#4  0xc000000000d5e9e4 in trap_fatal (frame=0xc00800002694eaa0)
    at /usr/src/sys/powerpc/powerpc/trap.c:534
#5  trap (frame=0xc00800002694eaa0) at /usr/src/sys/powerpc/powerpc/trap.c:494
#6  0xc000000000d52d88 in powerpc_interrupt (framep=0xc00800002694eaa0)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#7  <signal handler called>
#8  sbcut_internal (sb=0x59d1a9c0, len=0)
    at /usr/src/sys/kern/uipc_sockbuf.c:1497
#9  0xc00000000095e5a4 in sbdrop (sb=0x59d1a9c0, len=5590)
    at /usr/src/sys/kern/uipc_sockbuf.c:1606
#10 0xc000000000aeba24 in tcp_do_segment (m=0x4fc5a00, th=0xa981002e, 
    so=0x59d1a760, tp=0xc00800001dc5a0e0, drop_hdrlen=<optimized out>, 
    tlen=<optimized out>, iptos=<optimized out>)
    at /usr/src/sys/netinet/tcp_input.c:1868
#11 0xc000000000ae7158 in tcp_input_with_port (mp=<optimized out>, 
    offp=<optimized out>, proto=<optimized out>, port=0)
    at /usr/src/sys/netinet/tcp_input.c:1413
#12 0xc000000000ae812c in tcp_input (mp=<optimized out>, 
    offp=<optimized out>, proto=<optimized out>)
    at /usr/src/sys/netinet/tcp_input.c:1509
#13 0xc000000000ad3e48 in ip_input (m=0x0)
    at /usr/src/sys/netinet/ip_input.c:840
#14 0xc000000000a35970 in netisr_dispatch_src (proto=1, source=0, m=0x4fc5a00)
    at /usr/src/sys/net/netisr.c:1143
#15 0xc000000000a35ed8 in netisr_dispatch (proto=<optimized out>, 
    m=<optimized out>) at /usr/src/sys/net/netisr.c:1234
#16 0xc000000000a0e978 in ether_demux (ifp=0x4115800, m=<optimized out>)
    at /usr/src/sys/net/if_ethersubr.c:921
#17 0xc000000000a104f8 in ether_input_internal (ifp=0x4115800, 
    m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:707
#18 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:737
#19 0xc000000000a35970 in netisr_dispatch_src (proto=5, source=0, m=0x4fc5a00)
    at /usr/src/sys/net/netisr.c:1143
#20 0xc000000000a35ed8 in netisr_dispatch (proto=<optimized out>, 
    m=<optimized out>) at /usr/src/sys/net/netisr.c:1234
#21 0xc000000000a0efc4 in ether_input (ifp=<optimized out>, m=0x4fc5a00)
    at /usr/src/sys/net/if_ethersubr.c:828
#22 0xc0000000006a3d04 in vtnet_rxq_input (rxq=0x42a3600, m=0x4fc5a00, 
    hdr=<optimized out>) at /usr/src/sys/dev/virtio/network/if_vtnet.c:2043
#23 vtnet_rxq_eof (rxq=0x42a3600)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2147
#24 0xc0000000006a3324 in vtnet_rx_vq_process (rxq=0x42a3600, tries=-1)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2211
#25 0xc000000000698420 in virtqueue_intr (vq=<optimized out>)
    at /usr/src/sys/dev/virtio/virtqueue.c:472
#26 0xc00000000069ad78 in vtpci_intx_intr (xcn=0x4128308)
    at /usr/src/sys/dev/virtio/pci/virtio_pci.c:868
#27 0xc00000000081da44 in intr_event_execute_handlers (ie=0x432b300, 
    p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1169
#28 ithread_execute_handlers (ie=0x432b300, p=<optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1182
#29 ithread_loop (arg=0x43306e0) at /usr/src/sys/kern/kern_intr.c:1270
#30 0xc00000000081841c in fork_exit (
    callout=0xc00000000081d5ec <ithread_loop>, arg=0x43306e0, 
    frame=0xc00800002694f940) at /usr/src/sys/kern/kern_fork.c:1093
#31 0xc000000000d5dbbc in fork_trampoline ()
    at /usr/src/sys/powerpc/powerpc/swtch64.S:356
Backtrace stopped: frame did not save the PC
(kgdb) 

Luciano.
Comment 10 Richard Scheffenegger freebsd_committer freebsd_triage 2024-01-25 22:16:52 UTC 
Comparing releng/13.2 with stable/13, this may be related (with just looking at the backtrace info).

https://reviews.freebsd.org/D2970

(incorrectly dealing with a transmission error of a FIN segment with data; ip_output errors are more likely on busy machines, and FIN segments are sent at the end of session).

So this is more from the symptomatic angle than anything - no guarantee it's this.

By the time of the panic, the offsets into the send buffer would be off by one, which becomes problematic for data consistency and crashes as the conclusion of the session, or when the application stops providing more data in time...
Comment 11 Richard Scheffenegger freebsd_committer freebsd_triage 2024-01-25 22:19:06 UTC 
Just saw that one backtrace also has a similar issue around the socket buffer during tcp_input processing. This is less likely to be related to the one I mentioned, so that core may be more interesting.
Comment 12 Luciano Mannucci 2024-01-26 00:16:15 UTC 
I've put the crash directory on-line at https://www.vespaperitivo.it/276405 (the
vmcore files are zipped).

Luciano.
Comment 13 Richard Scheffenegger freebsd_committer freebsd_triage 2024-01-26 01:58:53 UTC 
since you've recompiled the kernel, would it be possible to package the /boot/kernel directory too? Let me find the 13.2 powerpc binary for the first three panics...
Comment 14 Richard Scheffenegger freebsd_committer freebsd_triage 2024-01-26 02:52:21 UTC 
(In reply to Robert Wing from comment #3)

Is something similar preventing kgdb from detecting the vmcore.0 as valid core file (against the stock powerpc64 kernel from the iso file)?

[root@freebsd ~/276405]# kgdb /mnt/boot/kernel/kernel vmcore.0
GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd15.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /mnt/boot/kernel/kernel...
(No debugging symbols found in /mnt/boot/kernel/kernel)
Failed to open vmcore: invalid corefile
(kgdb) q
[root@freebsd ~/276405]# ls -als
total 1361100
     4 drwxr-xr-x   2 root wheel       512 Jan 26 03:49 .
     8 drwxr-x---  28 root wheel      8192 Jan 26 03:48 ..
859296 -rw-r--r--   1 root wheel 879648768 Jan 26 03:33 vmcore.0
242912 -rw-r--r--   1 root wheel 248640403 Jan 26 03:38 vmcore.1.gz
Comment 15 Robert Wing freebsd_committer freebsd_triage 2024-01-26 03:38:29 UTC 
(In reply to Richard Scheffenegger from comment #14)

Update libkvm with the following patch: https://reviews.freebsd.org/D43526

I'll commit that review soon, it looks like it worked for Luciano - let me know if it works for you.

I would have committed it already but, I want to write a better commit message that documents which MMU's are supported for powerpc dumps.
Comment 16 Luciano Mannucci 2024-01-26 07:48:03 UTC 
(In reply to Richard Scheffenegger from comment #13)
Here it is:

https://www.vespaperitivo.it/276405/kernel/
Comment 17 Richard Scheffenegger freebsd_committer freebsd_triage 2024-01-26 09:38:13 UTC 
(In reply to Robert Wing from comment #15)
Thanks. That allows kgdb to load the vmcore, but thereafter it stops - maybe having to do with me trying this on amd64 as cross-platform debugging session?

I'm afraid poking into these powerpc64 cores exceeds my capabilities - Mark, could you extract things like tp from the tcp_disconnect or tcp_do_segment frame? 

 kgdb kernel.new/kernel vmcore.3
GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd15.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kernel.new/kernel...
(No debugging symbols found in kernel.new/kernel)
/wrkdirs/usr/ports/devel/gdb/work-py39/gdb-13.2/gdb/thread.c:1337: internal-error: switch_to_thread: Assertion `thr != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
----- Backtrace -----
0x130cde1 ???
0x17a5926 ???
0x17a5788 ???
0x1c2e56e ???
0x176538f ???
0x14408f0 ???
0x174695e ???
0x1340095 ???
0x17731cf ???
0x15267a5 ???
0x152518b ???
0x1523a9b ???
0x1219263 ???
0x82d969c29 __libc_start1
        /root/freebsd-master/freebsd/lib/libc/csu/libc_start1.c:157
0x12187cf ???
---------------------
/wrkdirs/usr/ports/devel/gdb/work-py39/gdb-13.2/gdb/thread.c:1337: internal-error: switch_to_thread: Assertion `thr != NULL' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Comment 18 Richard Scheffenegger freebsd_committer freebsd_triage 2024-01-26 09:42:42 UTC 
(In reply to Luciano Mannucci from comment #16)

Would it be possible for you to compile a kernel with debug symbols?

Or did you already use the unmodified KERNCONF=GENERIC (or with the "makeoptions DEBUG=-g", in which case I don't understand why kgdb claims that no debug symbols are present...
Comment 19 Luciano Mannucci 2024-01-26 11:27:08 UTC 
(In reply to Richard Scheffenegger from comment #18)
I did use the default (which I guess is GENERIC).

I coul rebuild the kernel with other options, though the new one might not crash due to Heisenberg law...
Comment 20 Luciano Mannucci 2024-02-12 00:11:39 UTC 
(In reply to Richard Scheffenegger from comment #18)
Well, the problem seems to be due to /usr/src not available during the panic. I fixed that, and now I have a core.txt a bit more sensible (from core.txt.2):

Sun Feb 11 15:15:35 CET 2024

FreeBSD ekaguaro 13.2-RELEASE-p9 FreeBSD 13.2-RELEASE-p9 releng/13.2-n254652-c78c31d2ef40 GENERIC  powerpc

panic: data storage interrupt trap

GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64-portbld-freebsd13.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:

fatal kernel trap:

   exception       = 0x300 (data storage interrupt)
   virtual address = 0x0
   dsisr           = 0x42000000
   srr0            = 0xc000000000c6f450 (0xc6f450)
   srr1            = 0x8000000000009032
   current msr     = 0x8000000000009032
   lr              = 0xc000000000c762ec (0xc762ec)
   frame           = 0xc00800004ac9d100
   curthread       = 0xc008000003bb4080
          pid = 12, comm = irq4100: ++

panic: data storage interrupt trap
cpuid = 11
time = 1707660504
KDB: stack backtrace:
#0 0xc0000000008ef154 at kdb_backtrace+0x90
#1 0xc00000000087ead0 at vpanic+0x1b8
#2 0xc00000000087e904 at panic+0x44
#3 0xc000000000d5e9e0 at trap+0x304
#4 0xc000000000d52d84 at powerpc_interrupt+0x1b4
Uptime: 1d10h37m33s
Dumping 1684 out of 8404 MB:..1%

0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
396             dump_savectx();
(kgdb) #0  0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
#1  0xc00000000087e424 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:484
#2  0xc00000000087eb8c in vpanic (fmt=0xc00000000119ed18 "%s trap", 
    ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:923
#3  0xc00000000087e908 in panic (fmt=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:847
#4  0xc000000000d5e9e4 in trap_fatal (frame=0xc00800004ac9d100)
    at /usr/src/sys/powerpc/powerpc/trap.c:534
#5  trap (frame=0xc00800004ac9d100) at /usr/src/sys/powerpc/powerpc/trap.c:494
#6  0xc000000000d52d88 in powerpc_interrupt (framep=0xc00800004ac9d100)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#7  <signal handler called>
#8  0xc000000000c6f450 in slab_free_item (zone=0x533f000, slab=0x2864dfe0, 
    item=0x2864db00) at /usr/src/sys/vm/uma_core.c:4696
#9  zone_release (arg=0x533f000, bucket=0x12423d010, cnt=<optimized out>)
    at /usr/src/sys/vm/uma_core.c:4741
#10 0xc000000000c762ec in bucket_drain (zone=0x533f000, bucket=0x12423d000)
    at /usr/src/sys/vm/uma_core.c:1313
#11 0xc000000000c70e08 in bucket_free (zone=0x533f000, bucket=0x12423d000, 
    udata=0x0) at /usr/src/sys/vm/uma_core.c:520
#12 zone_put_bucket (zone=0x533f000, domain=<optimized out>, 
    bucket=0x12423d000, udata=0x0, ws=<optimized out>)
    at /usr/src/sys/vm/uma_core.c:849
#13 0xc000000000c710c8 in cache_free (zone=0x533f000, cache=<optimized out>, 
    udata=0x0, item=<optimized out>, itemdomain=0)
    at /usr/src/sys/vm/uma_core.c:4646
#14 0xc000000000c70198 in uma_zfree_arg (zone=0x533f000, item=0x2864db00, 
    udata=0x0) at /usr/src/sys/vm/uma_core.c:4449
#15 0xc000000000843fa4 in mb_free_ext (m=0x2864db00)
    at /usr/src/sys/kern/kern_mbuf.c:1232
#16 0xc000000000842348 in m_free (m=0x2864db00)
    at /usr/src/sys/sys/mbuf.h:1477
#17 0xc000000000843c6c in m_freem (mb=0x2864db00)
    at /usr/src/sys/kern/kern_mbuf.c:1553
#18 0xc0000000006a4824 in vtnet_txq_eof (txq=0x55a8700)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2835
#19 0xc0000000006a442c in vtnet_txq_mq_start_locked (txq=0x55a8700, 
    m=<optimized out>) at /usr/src/sys/dev/virtio/network/if_vtnet.c:2692
#20 0xc0000000006a5ef4 in vtnet_txq_mq_start (ifp=<optimized out>, 
    m=0x1d80b900) at /usr/src/sys/dev/virtio/network/if_vtnet.c:2745
#21 0xc000000000a0e798 in ether_output_frame (ifp=0x5603000, 
    m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:509
#22 0xc000000000a0e5c0 in ether_output (ifp=0x5603000, m=0x1d80b900, 
    dst=0x289e5b60, ro=0x289e5b40) at /usr/src/sys/net/if_ethersubr.c:436
#23 0xc000000000ad877c in ip_output_send (ifp=0x5603000, m=<optimized out>, 
    gw=0x289e5b60, ro=0x289e5b40, inp=<optimized out>, 
    stamp_tag=<optimized out>) at /usr/src/sys/netinet/ip_output.c:277
#24 ip_output (m=<optimized out>, opt=<optimized out>, ro=0x289e5b40, 
    flags=<optimized out>, imo=0x0, inp=0x289e59b0)
    at /usr/src/sys/netinet/ip_output.c:799
#25 0xc000000000af6ca4 in tcp_output (tp=0xc008000004e26870)
    at /usr/src/sys/netinet/tcp_output.c:1553
#26 0xc000000000aebe6c in tcp_do_segment (m=0x1d817500, th=0xc28e502e, 
    so=<optimized out>, tp=0xc008000004e26870, drop_hdrlen=<optimized out>, 
    tlen=<optimized out>, iptos=<optimized out>)
    at /usr/src/sys/netinet/tcp_input.c:2768
#27 0xc000000000ae7158 in tcp_input_with_port (mp=<optimized out>, 
    offp=<optimized out>, proto=<optimized out>, port=0)
    at /usr/src/sys/netinet/tcp_input.c:1413
#28 0xc000000000ae812c in tcp_input (mp=<optimized out>, 
    offp=<optimized out>, proto=<optimized out>)
    at /usr/src/sys/netinet/tcp_input.c:1509
#29 0xc000000000ad3e48 in ip_input (m=0x0)
    at /usr/src/sys/netinet/ip_input.c:840
#30 0xc000000000a35970 in netisr_dispatch_src (proto=1, source=0, 
    m=0x1d817500) at /usr/src/sys/net/netisr.c:1143
#31 0xc000000000a35ed8 in netisr_dispatch (proto=<optimized out>, 
    m=<optimized out>) at /usr/src/sys/net/netisr.c:1234
#32 0xc000000000a0e978 in ether_demux (ifp=0x5603000, m=<optimized out>)
    at /usr/src/sys/net/if_ethersubr.c:921
#33 0xc000000000a104f8 in ether_input_internal (ifp=0x5603000, 
    m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:707
#34 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:737
#35 0xc000000000a35970 in netisr_dispatch_src (proto=5, source=0, 
    m=0x1d817500) at /usr/src/sys/net/netisr.c:1143
#36 0xc000000000a35ed8 in netisr_dispatch (proto=<optimized out>, 
    m=<optimized out>) at /usr/src/sys/net/netisr.c:1234
#37 0xc000000000a0efc4 in ether_input (ifp=<optimized out>, m=0x1d817500)
    at /usr/src/sys/net/if_ethersubr.c:828
#38 0xc0000000006a3d04 in vtnet_rxq_input (rxq=0x55b2d80, m=0x1d817500, 
    hdr=<optimized out>) at /usr/src/sys/dev/virtio/network/if_vtnet.c:2043
#39 vtnet_rxq_eof (rxq=0x55b2d80)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2147
#40 0xc0000000006a3324 in vtnet_rx_vq_process (rxq=0x55b2d80, tries=-1)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2211
#41 0xc000000000698420 in virtqueue_intr (vq=<optimized out>)
    at /usr/src/sys/dev/virtio/virtqueue.c:472
#42 0xc00000000069ad78 in vtpci_intx_intr (xcn=0x55a8908)
    at /usr/src/sys/dev/virtio/pci/virtio_pci.c:868
#43 0xc00000000081da44 in intr_event_execute_handlers (ie=0x5393700, 
    p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1169
#44 ithread_execute_handlers (ie=0x5393700, p=<optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1182
#45 ithread_loop (arg=0x532e2e0) at /usr/src/sys/kern/kern_intr.c:1270
#46 0xc00000000081841c in fork_exit (
    callout=0xc00000000081d5ec <ithread_loop>, arg=0x532e2e0, 
    frame=0xc00800004ac9e940) at /usr/src/sys/kern/kern_fork.c:1093
#47 0xc000000000d5dbbc in fork_trampoline ()
    at /usr/src/sys/powerpc/powerpc/swtch64.S:356
Backtrace stopped: frame did not save the PC
(kgdb) 

------------------------------------------------------------------------

details @ http://www.vespaperitivo.it/276405/new/

Luciano.
Comment 21 Luciano Mannucci 2024-02-13 12:56:03 UTC 
(In reply to Luciano Mannucci from comment #20)
I have another one from another VM (same FreeBSD version) slighlty shorter but looks very similar. I can post the details if needed.

info.0:
Dump header from device: /dev/vtbd0s2b
  Architecture: powerpc64
  Architecture Version: 1
  Dump Length: 927391744
  Blocksize: 512
  Compression: none
  Dumptime: 2024-02-13 11:28:31 +0100
  Hostname: numeron
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 13.2-RELEASE-p9 #5 releng/13.2-n254652-c78c31d2ef40-dirty: Sun Jan 21 16:01:55 CET 2024
    root@pippozzo:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64
  Panic String: data storage interrupt trap
  Dump Parity: 1435306524
  Bounds: 0
  Dump Status: good

core.txt.0:
numeron dumped core - see /var/crash/vmcore.0

Tue Feb 13 12:13:02 CET 2024

FreeBSD numeron 13.2-RELEASE-p9 FreeBSD 13.2-RELEASE-p9 #5 releng/13.2-n254652-c78c31d2ef40-dirty: Sun Jan 21 16:01:55 CET 2024     root@pippozzo:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64  powerpc

panic: data storage interrupt trap

GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64-portbld-freebsd13.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:

fatal kernel trap:

   exception       = 0x300 (data storage interrupt)
   virtual address = 0x252540027
   dsisr           = 0x40000000
   srr0            = 0xc000000000954654 (0x954654)
   srr1            = 0x8000000000009032
   current msr     = 0x8000000000009032
   lr              = 0xc000000000842b10 (0x842b10)
   frame           = 0xc00800002694f120
   curthread       = 0xc008000002dcd100
          pid = 12, comm = irq4101: xhci0+

panic: data storage interrupt trap
cpuid = 1
time = 1707820111
KDB: stack backtrace:
#0 0xc0000000008ef154 at kdb_backtrace+0x90
#1 0xc00000000087ead0 at vpanic+0x1b8
#2 0xc00000000087e904 at panic+0x44
#3 0xc000000000d5e9e0 at trap+0x304
#4 0xc000000000d52d84 at powerpc_interrupt+0x1b4
Uptime: 8d1h48m58s
Dumping 884 out of 4052 MB:..1%

0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
396             dump_savectx();
(kgdb) #0  0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
#1  0xc00000000087e424 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:484
#2  0xc00000000087eb8c in vpanic (fmt=0xc00000000119ed18 "%s trap", 
    ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:923
#3  0xc00000000087e908 in panic (fmt=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:847
#4  0xc000000000d5e9e4 in trap_fatal (frame=0xc00800002694f120)
    at /usr/src/sys/powerpc/powerpc/trap.c:534
#5  trap (frame=0xc00800002694f120) at /usr/src/sys/powerpc/powerpc/trap.c:494
#6  0xc000000000d52d88 in powerpc_interrupt (framep=0xc00800002694f120)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#7  <signal handler called>
#8  m_tag_delete_chain (m=0x24936e00, t=<optimized out>)
    at /usr/src/sys/kern/uipc_mbuf2.c:355
#9  0xc000000000842b10 in mb_dtor_mbuf (mem=<optimized out>, 
    size=<optimized out>, arg=<optimized out>)
    at /usr/src/sys/kern/kern_mbuf.c:690
#10 0xc000000000c70244 in item_dtor (zone=0xc008000002cc6c00, 
    item=0x24936e00, udata=0x0, skip=SKIP_NONE, size=<optimized out>)
    at /usr/src/sys/vm/uma_core.c:3426
#11 uma_zfree_arg (zone=0xc008000002cc6c00, item=0x24936e00, udata=0x0)
    at /usr/src/sys/vm/uma_core.c:4396
#12 0xc0000000008423f0 in m_free (m=0x24936e00)
    at /usr/src/sys/kern/kern_mbuf.c:1301
#13 0xc000000000843c6c in m_freem (mb=0x24936e00)
    at /usr/src/sys/kern/kern_mbuf.c:1553
#14 0xc0000000006a3b68 in vtnet_rxq_merged_eof (rxq=0x42a3600, 
    m_head=0x24936e00, nbufs=<optimized out>)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:1966
#15 vtnet_rxq_eof (rxq=0x42a3600)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2110
#16 0xc0000000006a3324 in vtnet_rx_vq_process (rxq=0x42a3600, tries=-1)
    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2211
#17 0xc000000000698420 in virtqueue_intr (vq=<optimized out>)
    at /usr/src/sys/dev/virtio/virtqueue.c:472
#18 0xc00000000069ad78 in vtpci_intx_intr (xcn=0x4128308)
    at /usr/src/sys/dev/virtio/pci/virtio_pci.c:868
#19 0xc00000000081da44 in intr_event_execute_handlers (ie=0x432b300, 
    p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1169
#20 ithread_execute_handlers (ie=0x432b300, p=<optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1182
#21 ithread_loop (arg=0x43306e0) at /usr/src/sys/kern/kern_intr.c:1270
#22 0xc00000000081841c in fork_exit (
    callout=0xc00000000081d5ec <ithread_loop>, arg=0x43306e0, 
    frame=0xc00800002694f940) at /usr/src/sys/kern/kern_fork.c:1093
#23 0xc000000000d5dbbc in fork_trampoline ()
    at /usr/src/sys/powerpc/powerpc/swtch64.S:356
Backtrace stopped: frame did not save the PC
(kgdb)
Comment 22 Luciano Mannucci 2024-02-13 14:59:41 UTC 
(In reply to Richard Scheffenegger from comment #18)
Ok, recompiled with -g. The result is @ https://www.vespaperitivo.it/276405/kernel-g/,
waiting for the next panic...
Comment 23 Luciano Mannucci 2024-02-14 14:33:18 UTC 
(In reply to Luciano Mannucci from comment #22)
Ok, got it.

It doesn't seem that different from the previous ones. Here it is:

kaguaro dumped core - see /var/crash/vmcore.1

Wed Feb 14 15:20:45 CET 2024

FreeBSD ekaguaro 13.2-RELEASE-p9 FreeBSD 13.2-RELEASE-p9 #6 releng/13.2-n254652-c78c31d2ef40-dirty: Tue Feb 13 14:33:21 CET 2024     root@pippozzo:/usr/obj/usr/src/powerpc.powerpc64/sys/GENERIC64  powerpc

panic: data storage interrupt trap

GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD]
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64-portbld-freebsd13.2".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:
   frame           = 0xc00800008b03fc80
   curthread       = 0xc00800008a25c600
          pid = 11886, comm = rsync

panic: data storage interrupt trap
cpuid = 11
time = 1707919468
KDB: stack backtrace:
#0 0xc0000000008ef154 at kdb_backtrace+0x90
#1 0xc00000000087ead0 at vpanic+0x1b8
#2 0xc00000000087e904 at panic+0x44
#3 0xc000000000d5e9e0 at trap+0x304
#4 0xc000000000d52d84 at powerpc_interrupt+0x1b4
Uptime: 23h10m45s
Dumping 1609 out of 8404 MB:..1%

0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
396             dump_savectx();
(kgdb) #0  0xc00000000087e828 in doadump (textdump=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:396
#1  0xc00000000087e424 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:484
#2  0xc00000000087eb8c in vpanic (fmt=0xc00000000119ed18 "%s trap", 
    ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:923
#3  0xc00000000087e908 in panic (fmt=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:847
#4  0xc000000000d5e9e4 in trap_fatal (frame=0xc00800008b03fc80)
    at /usr/src/sys/powerpc/powerpc/trap.c:534
#5  trap (frame=0xc00800008b03fc80) at /usr/src/sys/powerpc/powerpc/trap.c:494
#6  0xc000000000d52d88 in powerpc_interrupt (framep=0xc00800008b03fc80)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#7  <signal handler called>
#8  sbcut_internal (sb=0x1d04098a8, len=0)
    at /usr/src/sys/kern/uipc_sockbuf.c:1497
#9  0xc00000000095e1d0 in sbflush_internal (sb=<optimized out>)
    at /usr/src/sys/kern/uipc_sockbuf.c:1437
#10 sbflush_locked (sb=<optimized out>)
    at /usr/src/sys/kern/uipc_sockbuf.c:1449
#11 sbflush (sb=0x1d04098a8) at /usr/src/sys/kern/uipc_sockbuf.c:1457
#12 0xc000000000b15ce8 in tcp_disconnect (tp=0xc00800008a3960e0)
    at /usr/src/sys/netinet/tcp_usrreq.c:2585
#13 0xc000000000b0f5c8 in tcp_usr_disconnect (so=<optimized out>)
    at /usr/src/sys/netinet/tcp_usrreq.c:765
#14 0xc000000000961920 in sodisconnect (so=0x1d0409760)
    at /usr/src/sys/kern/uipc_socket.c:1370
#15 soclose (so=0x1d0409760) at /usr/src/sys/kern/uipc_socket.c:1192
#16 0xc00000000092dcb8 in soo_close (fp=<optimized out>, td=<optimized out>)
    at /usr/src/sys/kern/sys_socket.c:361
#17 0xc0000000007ef3f4 in fo_close (fp=0xbdbb5a0, td=<optimized out>)
    at /usr/src/sys/sys/file.h:384
#18 _fdrop (fp=0xbdbb5a0, td=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:3691
#19 0xc0000000007f4b58 in closef (fp=0xbdbb5a0, td=0xc00800008a25c600)
    at /usr/src/sys/kern/kern_descrip.c:2937
#20 0xc0000000007f3fe0 in fdescfree_fds (td=0xc00800008a25c600, 
    fdp=0xc00800008a2824f0, needclose=<optimized out>)
    at /usr/src/sys/kern/kern_descrip.c:2644
#21 0xc0000000007f37a4 in fdescfree (td=0xc00800008a25c600)
    at /usr/src/sys/kern/kern_descrip.c:2690
#22 0xc00000000080ede0 in exit1 (td=0xc00800008a25c600, rval=<optimized out>, 
    signo=<optimized out>) at /usr/src/sys/kern/kern_exit.c:403
#23 0xc00000000080e558 in sys_sys_exit (td=<optimized out>, 
    uap=<optimized out>) at /usr/src/sys/kern/kern_exit.c:212
#24 0xc000000000d5f3c0 in syscallenter (td=0xc00800008a25c600)
    at /usr/src/sys/powerpc/powerpc/../../kern/subr_syscall.c:190
#25 syscall (frame=<optimized out>) at /usr/src/sys/powerpc/powerpc/trap.c:759
#26 0xc000000000d5ec70 in trap (frame=0xc00800008b040810)
    at /usr/src/sys/powerpc/powerpc/trap.c:292
#27 0xc000000000d52d88 in powerpc_interrupt (framep=0xc00800008b040810)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
    at /usr/src/sys/kern/kern_descrip.c:2644
#21 0xc0000000007f37a4 in fdescfree (td=0xc00800008a25c600)
    at /usr/src/sys/kern/kern_descrip.c:2690
#22 0xc00000000080ede0 in exit1 (td=0xc00800008a25c600, rval=<optimized out>, 
    signo=<optimized out>) at /usr/src/sys/kern/kern_exit.c:403
#23 0xc00000000080e558 in sys_sys_exit (td=<optimized out>, 
    uap=<optimized out>) at /usr/src/sys/kern/kern_exit.c:212
#24 0xc000000000d5f3c0 in syscallenter (td=0xc00800008a25c600)
    at /usr/src/sys/powerpc/powerpc/../../kern/subr_syscall.c:190
#25 syscall (frame=<optimized out>) at /usr/src/sys/powerpc/powerpc/trap.c:759
#26 0xc000000000d5ec70 in trap (frame=0xc00800008b040810)
    at /usr/src/sys/powerpc/powerpc/trap.c:292
#27 0xc000000000d52d88 in powerpc_interrupt (framep=0xc00800008b040810)
    at /usr/src/sys/powerpc/powerpc/interrupt.c:139
#28 <signal handler called>
#29 0x00000008108fad40 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(kgdb) 

------------------------------------------------------------------------
I can post the vmcore and the rest of details if needed...
Comment 24 Luciano Mannucci 2024-02-14 17:13:09 UTC 
Ok, I have a new one from the other VM, so I posted the details:

http://www.vespaperitivo.it/276405/new-g/

The VMs are ekaguaro and numeron, the vmcores are gzipped.
Comment 25 Luciano Mannucci 2024-02-19 14:52:37 UTC 
New panic today.
It seems to me it might be linked to the virtio ethernet driver. grep virtio core.txt.? gives:

core.txt.0:    at /usr/src/sys/dev/virtio/network/if_vtnet.c:1966
core.txt.0:    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2110
core.txt.0:    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2211
core.txt.0:    at /usr/src/sys/dev/virtio/virtqueue.c:472
core.txt.0:    at /usr/src/sys/dev/virtio/pci/virtio_pci.c:868

core.txt.1:    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2081
core.txt.1:    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2211
core.txt.1:    at /usr/src/sys/dev/virtio/virtqueue.c:472
core.txt.1:    at /usr/src/sys/dev/virtio/pci/virtio_pci.c:868

ore.txt.2:    hdr=<optimized out>) at /usr/src/sys/dev/virtio/network/if_vtnet.c:2043
core.txt.2:    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2147
core.txt.2:    at /usr/src/sys/dev/virtio/network/if_vtnet.c:2211
core.txt.2:    at /usr/src/sys/dev/virtio/virtqueue.c:472
core.txt.2:    at /usr/src/sys/dev/virtio/pci/virtio_pci.c:868

I'll move the ethernet interfaces to rtlXX or I82XX as a workaround, to see if it stops panics, as soon as my customers become still.
Comment 26 Luciano Mannucci 2024-02-21 13:26:52 UTC 
(In reply to Luciano Mannucci from comment #25)
I left a virtio (+realtek) to see if panics and it did. The new core.txt mentions virtio in the backtrace, so probably the problem is there. I've posted everything @:

http://www.vespaperitivo.it/276405/new-g/numeron/

the core.txt file is the dot 3.

I've now reconfigured the VM to use e1000 + rtl8139 as a workaround. We'll see if it crashes again.
Comment 27 Michael Tuexen freebsd_committer freebsd_triage 2024-02-22 18:13:04 UTC 
(In reply to Luciano Mannucci from comment #26)
Do you have a way to reproduce this? I do have a small Power9 machine, but it runs FreeBSD head, not 13.2...
Comment 28 Luciano Mannucci 2024-02-23 09:10:32 UTC 
(In reply to Michael Tuexen from comment #27)
This panic happens only if you run the freebsd machine as a guest, using
virtio as the driver of the thernet. To reproduce you need a fair amount of IP traffic; I use rsync to another machine on the same network to make it panic.
It panics also with "natural" heavy traffic (60 email users reading and writing messages).

Luciano.
Comment 29 Michael Tuexen freebsd_committer freebsd_triage 2024-02-23 10:24:44 UTC 
(In reply to Luciano Mannucci from comment #28)
Thanks for the information. I only have FreeBSD running on bare metal. So I don't have vtnet...