I know I am pushing things, I am running FreeBSD 14-BETA2 as a VM on TrueNAS Scale (I also had this happen on BETA1). But I thought you may want to look at it. I have three VMs running on the TrueNAS Scale hypervisor (same machine). All are not doing much yet at all (one just fresh install, another doing light DNS, third running DNS and running a very lightly used apache web server). So far only the web server is panicing, the other two hasnt panic yet. I can provide a core dump if needed, it is too large to upload to this ticket. ____ % uname -a FreeBSD venom-f5 14.0-BETA2 FreeBSD 14.0-BETA2 #1 releng/14.0-n265099-8d60ede293e: Sat Sep 16 11:18:03 CDT 2023 root@venom-f5:/usr/obj/usr/src/amd64.amd64/sys/VENOM-F5 amd64 _______ Sep 16 20:26:24 venom-f5 named[684]: clients-per-query decreased to 18 Sep 16 20:30:38 venom-f5 sshd[1647]: error: Fssh_kex_exchange_identification: Connection closed by remote host Sep 16 20:46:24 venom-f5 named[684]: clients-per-query decreased to 17 Sep 16 20:53:07 venom-f5 syslogd: last message repeated 1 times Sep 16 20:59:05 venom-f5 kernel: Sep 16 20:56:00 venom-f5 syslogd: kernel boot file is /boot/kernel/kernel Sep 16 20:56:00 venom-f5 kernel: Sep 16 20:56:00 venom-f5 kernel: Fatal trap 12: page fault while in kernel mode Sep 16 20:56:00 venom-f5 kernel: cpuid = 5; apic id = 05 Sep 16 20:56:00 venom-f5 kernel: fault virtual address = 0xb8 Sep 16 20:56:00 venom-f5 kernel: fault code = supervisor read data, page not present Sep 16 20:56:00 venom-f5 kernel: instruction pointer = 0x20:0xffffffff80d4e090 Sep 16 20:56:00 venom-f5 kernel: stack pointer = 0x28:0xfffffe01017a7c70 Sep 16 20:56:00 venom-f5 kernel: frame pointer = 0x28:0xfffffe01017a7cf0 Sep 16 20:56:00 venom-f5 kernel: code segment = base rx0, limit 0xfffff, type 0x1b Sep 16 20:56:00 venom-f5 kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 Sep 16 20:56:00 venom-f5 kernel: processor eflags = interrupt enabled, resume, IOPL = 0 Sep 16 20:56:00 venom-f5 kernel: current process = 684 (isc-net-0001) Sep 16 20:56:00 venom-f5 kernel: rdi: ffffffff816c79c0 rsi: ffffffff816c79c0 rdx: 0000000000010200 Sep 16 20:56:00 venom-f5 kernel: rcx: 0000000000000000 r8: fffff800073aa800 r9: 0000000000000000 Sep 16 20:56:00 venom-f5 kernel: rax: fffff801dc915e00 rbx: fffff801dc8f8c40 rbp: fffffe01017a7cf0 Sep 16 20:56:00 venom-f5 kernel: r10: 0000000000000000 r11: fffffe0102c14c60 r12: fffff801a252f220 Sep 16 20:56:00 venom-f5 kernel: r13: 00000000000013fe r14: 00000000000013fe r15: fffff800073aa800 Sep 16 20:56:00 venom-f5 kernel: trap number = 12 Sep 16 20:56:00 venom-f5 kernel: panic: page fault Sep 16 20:56:00 venom-f5 kernel: cpuid = 5 Sep 16 20:56:00 venom-f5 kernel: time = 1694915945 Sep 16 20:56:00 venom-f5 kernel: KDB: stack backtrace: Sep 16 20:56:00 venom-f5 kernel: #0 0xffffffff80b92cad at kdb_backtrace+0x5d Sep 16 20:56:00 venom-f5 kernel: #1 0xffffffff80b45db2 at vpanic+0x132 Sep 16 20:56:00 venom-f5 kernel: #2 0xffffffff80b45c73 at panic+0x43 Sep 16 20:56:00 venom-f5 kernel: #3 0xffffffff8103181c at trap_fatal+0x40c Sep 16 20:56:00 venom-f5 kernel: #4 0xffffffff8103186f at trap_pfault+0x4f Sep 16 20:56:00 venom-f5 kernel: #5 0xffffffff81008298 at calltrap+0x8 Sep 16 20:56:00 venom-f5 kernel: #6 0xffffffff80d700ac at udp6_bind+0x13c Sep 16 20:56:00 venom-f5 kernel: #7 0xffffffff80be8b12 at sobind+0x32 Sep 16 20:56:00 venom-f5 kernel: #8 0xffffffff80bf0015 at kern_bindat+0xc5 Sep 16 20:56:00 venom-f5 kernel: #9 0xffffffff80befeab at sys_bind+0x9b Sep 16 20:56:00 venom-f5 kernel: #10 0xffffffff810320d9 at amd64_syscall+0x109 Sep 16 20:56:00 venom-f5 kernel: #11 0xffffffff81008bab at fast_syscall_common+0xf8 Sep 16 20:56:00 venom-f5 kernel: Uptime: 9h25m35s Sep 16 20:56:00 venom-f5 kernel: Dumping 722 out of 16338 MB:..3%..12%..23%..32%..43%..51%..63%..71%..82%..91% Sep 16 20:56:00 venom-f5 kernel: Dump complete Sep 16 20:56:00 venom-f5 kernel: Automatic reboot in 15 seconds - press a key on the console to abort Sep 16 20:56:00 venom-f5 kernel: Rebooting... Sep 16 20:56:00 venom-f5 kernel: cpu_reset: Restarting BSP Sep 16 20:56:00 venom-f5 kernel: cpu_reset_proxy: Stopped CPU 5 Sep 16 20:56:00 venom-f5 kernel: ---<<BOOT>>--- Sep 16 20:56:00 venom-f5 kernel: Copyright (c) 1992-2023 The FreeBSD Project. Sep 16 20:56:00 venom-f5 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Sep 16 20:56:00 venom-f5 kernel: The Regents of the University of California. All rights reserved. Sep 16 20:56:00 venom-f5 kernel: FreeBSD is a registered trademark of The FreeBSD Foundation. Sep 16 20:56:00 venom-f5 kernel: FreeBSD 14.0-BETA2 #1 releng/14.0-n265099-8d60ede293e: Sat Sep 16 11:18:03 CDT 2023 Sep 16 20:56:00 venom-f5 kernel: root@venom-f5:/usr/obj/usr/src/amd64.amd64/sys/VENOM-F5 amd64 Sep 16 20:56:00 venom-f5 kernel: FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a25 9152) Sep 16 20:56:00 venom-f5 kernel: VT(efifb): resolution 800x600 Sep 16 20:56:00 venom-f5 kernel: CPU: 12th Gen Intel(R) Core(TM) i7-12700KF (3609.75-MHz K8-class CPU) Sep 16 20:56:00 venom-f5 kernel: Origin="GenuineIntel" Id=0x90672 Family=0x6 Model=0x97 Stepping=2 Sep 16 20:56:00 venom-f5 kernel: Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX ,FXSR,SSE,SSE2,SS,HTT> Sep 16 20:56:00 venom-f5 kernel: Features2=0xfffab223<SSE3,PCLMULQDQ,VMX,SSSE3,FMA,CX16,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POP CNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> Sep 16 20:56:00 venom-f5 kernel: AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> Sep 16 20:56:00 venom-f5 kernel: AMD Features2=0x121<LAHF,ABM,Prefetch> Sep 16 20:56:00 venom-f5 kernel: Structured Extended Features=0x219c07ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED ,ADX,SMAP,CLFLUSHOPT,CLWB,SHA> Sep 16 20:56:00 venom-f5 kernel: Structured Extended Features2=0x1840073c<UMIP,PKU,OSPKE,WAITPKG,GFNI,VAES,VPCLMULQDQ,RDPID,MOVD IRI,MOVDIR64B> Sep 16 20:56:00 venom-f5 kernel: Structured Extended Features3=0xac004410<FSRM,MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD> Sep 16 20:56:00 venom-f5 kernel: XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> Sep 16 20:56:00 venom-f5 kernel: IA32_ARCH_CAPS=0x6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO> Sep 16 20:56:00 venom-f5 kernel: AMD Extended Feature Extensions ID EBX=0x1009000<IBPB,STIBP,SSBD> Sep 16 20:56:00 venom-f5 kernel: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr Sep 16 20:56:00 venom-f5 kernel: Hypervisor: Origin = "KVMKVMKVM" Sep 16 20:56:00 venom-f5 kernel: real memory = 17179869184 (16384 MB) Sep 16 20:56:00 venom-f5 kernel: avail memory = 16611700736 (15842 MB) Sep 16 20:56:00 venom-f5 kernel: Event timer "LAPIC" quality 600 Sep 16 20:56:00 venom-f5 kernel: ACPI APIC Table: <BOCHS BXPCAPIC> Sep 16 20:56:00 venom-f5 kernel: WARNING: L3 data cache covers more APIC IDs than a package (7 > 3) Sep 16 20:56:00 venom-f5 kernel: FreeBSD/SMP: Multiprocessor System Detected: 6 CPUs Sep 16 20:56:00 venom-f5 kernel: FreeBSD/SMP: 1 package(s) x 6 core(s) Sep 16 20:56:00 venom-f5 kernel: random: registering fast source Intel Secure Key RNG Sep 16 20:56:00 venom-f5 kernel: random: fast provider: "Intel Secure Key RNG" Sep 16 20:56:00 venom-f5 kernel: random: unblocking device. Sep 16 20:56:00 venom-f5 kernel: ioapic0 <Version 1.1> irqs 0-23 Sep 16 20:56:00 venom-f5 kernel: Launching APs: 3 2 4 5 1 Sep 16 20:56:00 venom-f5 kernel: random: entropy device external interface Sep 16 20:56:00 venom-f5 kernel: kbd1 at kbdmux0 Sep 16 20:56:00 venom-f5 kernel: efirtc0: <EFI Realtime Clock> Sep 16 20:56:00 venom-f5 kernel: efirtc0: registered as a time-of-day clock, resolution 1.000000s Sep 16 20:56:00 venom-f5 kernel: kvmclock0: <KVM paravirtual clock> Sep 16 20:56:00 venom-f5 kernel: Timecounter "kvmclock" frequency 1000000000 Hz quality 975 Sep 16 20:56:00 venom-f5 kernel: kvmclock0: registered as a time-of-day clock, resolution 0.000001s Sep 16 20:56:00 venom-f5 kernel: smbios0: <System Management BIOS> at iomem 0xbf91b000-0xbf91b01e Sep 16 20:56:00 venom-f5 kernel: smbios0: Version: 2.8, BCD Revision: 2.8 Sep 16 20:56:00 venom-f5 kernel: aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256> Sep 16 20:56:00 venom-f5 kernel: acpi0: <BOCHS BXPCFACP> Sep 16 20:56:00 venom-f5 kernel: acpi0: Power Button (fixed) Sep 16 20:56:00 venom-f5 kernel: cpu0: <ACPI CPU> on acpi0 Sep 16 20:56:00 venom-f5 kernel: atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0 Sep 16 20:56:00 venom-f5 kernel: atrtc0: registered as a time-of-day clock, resolution 1.000000s Sep 16 20:56:00 venom-f5 kernel: Event timer "RTC" frequency 32768 Hz quality 0 Sep 16 20:56:00 venom-f5 kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Sep 16 20:56:00 venom-f5 kernel: Timecounter "HPET" frequency 100000000 Hz quality 950 Sep 16 20:56:00 venom-f5 kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 Sep 16 20:56:00 venom-f5 kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0 Sep 16 20:56:00 venom-f5 kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 Sep 16 20:56:00 venom-f5 kernel: pci_link4: BIOS IRQ 10 for 0.1.INTA is invalid Sep 16 20:56:00 venom-f5 kernel: pci_link1: BIOS IRQ 10 does not match initial IRQ 11 Sep 16 20:56:00 venom-f5 kernel: pci_link2: BIOS IRQ 11 does not match initial IRQ 10 Sep 16 20:56:00 venom-f5 kernel: pci_link3: BIOS IRQ 11 does not match initial IRQ 10 Sep 16 20:56:00 venom-f5 kernel: pci_link0: BIOS IRQ 10 does not match initial IRQ 11 Sep 16 20:56:00 venom-f5 kernel: pci0: <ACPI PCI bus> on pcib0 Sep 16 20:56:00 venom-f5 kernel: isab0: <PCI-ISA bridge> at device 1.0 on pci0 Sep 16 20:56:00 venom-f5 kernel: isa0: <ISA bus> on isab0 Sep 16 20:56:00 venom-f5 kernel: atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc100-0xc10f at device 1.1 on pci0 Sep 16 20:56:00 venom-f5 kernel: ata0: <ATA channel> at channel 0 on atapci0 Sep 16 20:56:00 venom-f5 kernel: ata1: <ATA channel> at channel 1 on atapci0 Sep 16 20:56:00 venom-f5 kernel: pci0: <bridge> at device 1.3 (no driver attached) Sep 16 20:56:00 venom-f5 kernel: vgapci0: <VGA-compatible display> port 0xc0e0-0xc0ff mem 0xc4000000-0xc7ffffff,0xc0000000-0xc3fff fff,0xc8060000-0xc8061fff irq 10 at device 2.0 on pci0 Sep 16 20:56:00 venom-f5 kernel: vgapci0: Boot video device Sep 16 20:56:00 venom-f5 kernel: em0: <Intel(R) Legacy PRO/1000 MT 82540EM> port 0xc080-0xc0bf mem 0xc8040000-0xc805ffff irq 11 at device 3.0 on pci0 Sep 16 20:56:00 venom-f5 kernel: em0: Using 1024 TX descriptors and 1024 RX descriptors Sep 16 20:56:00 venom-f5 kernel: em0: Ethernet address: 00:a0:98:63:39:f0 Sep 16 20:56:00 venom-f5 kernel: em0: netmap queues/slots: TX 1/1024, RX 1/1024 Sep 16 20:56:00 venom-f5 kernel: xhci0: <NEC uPD720200 USB 3.0 controller> mem 0x800008000-0x80000bfff irq 11 at device 4.0 on pci 0 Sep 16 20:56:00 venom-f5 kernel: xhci0: 32 bytes context size, 32-bit DMA Sep 16 20:56:00 venom-f5 kernel: xhci_interrupt: host controller halted Sep 16 20:56:00 venom-f5 syslogd: last message repeated 51 times Sep 16 20:56:00 venom-f5 kernel: usbus0 on xhci0 Sep 16 20:56:00 venom-f5 kernel: usbus0: 5.0Gbps Super Speed USB v3.0 Sep 16 20:56:00 venom-f5 kernel: ahci0: <Intel ICH9 AHCI SATA controller> port 0xc0c0-0xc0df mem 0xc8063000-0xc8063fff irq 10 at d evice 5.0 on pci0 Sep 16 20:56:00 venom-f5 kernel: ahci0: AHCI v1.00 with 6 1.5Gbps ports, Port Multiplier not supported Sep 16 20:56:00 venom-f5 kernel: ahcich0: <AHCI channel> at channel 0 on ahci0 Sep 16 20:56:00 venom-f5 kernel: ahcich1: <AHCI channel> at channel 1 on ahci0 Sep 16 20:56:00 venom-f5 kernel: ahcich2: <AHCI channel> at channel 2 on ahci0 Sep 16 20:56:00 venom-f5 kernel: ahcich3: <AHCI channel> at channel 3 on ahci0 Sep 16 20:56:00 venom-f5 kernel: ahcich4: <AHCI channel> at channel 4 on ahci0 Sep 16 20:56:00 venom-f5 kernel: ahcich5: <AHCI channel> at channel 5 on ahci0 Sep 16 20:56:00 venom-f5 kernel: virtio_pci0: <VirtIO PCI (legacy) Console adapter> port 0xc040-0xc07f mem 0xc8062000-0xc8062fff,0 x800000000-0x800003fff irq 10 at device 6.0 on pci0 Sep 16 20:56:00 venom-f5 kernel: virtio_pci1: <VirtIO PCI (legacy) Balloon adapter> port 0xc000-0xc03f mem 0x800004000-0x800007fff irq 11 at device 7.0 on pci0 Sep 16 20:56:00 venom-f5 kernel: vtballoon0: <VirtIO Balloon Adapter> on virtio_pci1 Sep 16 20:56:00 venom-f5 kernel: acpi_syscontainer0: <System Container> on acpi0 Sep 16 20:56:00 venom-f5 kernel: acpi_syscontainer1: <System Container> port 0xaf00-0xaf0b on acpi0 Sep 16 20:56:00 venom-f5 kernel: acpi_syscontainer2: <System Container> port 0xafe0-0xafe3 on acpi0 Sep 16 20:56:00 venom-f5 kernel: acpi_syscontainer3: <System Container> port 0xae00-0xae13 on acpi0 Sep 16 20:56:00 venom-f5 kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 Sep 16 20:56:00 venom-f5 kernel: uart0: console (115200,n,8,1) Sep 16 20:56:00 venom-f5 kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 Sep 16 20:56:00 venom-f5 kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0 Sep 16 20:56:00 venom-f5 kernel: kbd0 at atkbd0 Sep 16 20:56:00 venom-f5 kernel: atkbd0: [GIANT-LOCKED] Sep 16 20:56:00 venom-f5 kernel: psm0: <PS/2 Mouse> irq 12 on atkbdc0 Sep 16 20:56:00 venom-f5 kernel: psm0: [GIANT-LOCKED] Sep 16 20:56:00 venom-f5 kernel: WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0. Sep 16 20:56:00 venom-f5 kernel: psm0: model IntelliMouse Explorer, device ID 4 Sep 16 20:56:00 venom-f5 kernel: fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0 Sep 16 20:56:00 venom-f5 kernel: fdc0: does not respond Sep 16 20:56:00 venom-f5 kernel: device_attach: fdc0 attach returned 6 Sep 16 20:56:00 venom-f5 kernel: attimer0: <AT timer> at port 0x40 on isa0 Sep 16 20:56:00 venom-f5 kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 Sep 16 20:56:00 venom-f5 kernel: Event timer "i8254" frequency 1193182 Hz quality 100 Sep 16 20:56:00 venom-f5 kernel: attimer0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14. Sep 16 20:56:00 venom-f5 kernel: fdc0: No FDOUT register! Sep 16 20:56:00 venom-f5 kernel: Timecounters tick every 10.000 msec Sep 16 20:56:00 venom-f5 kernel: ZFS filesystem version: 5 Sep 16 20:56:00 venom-f5 kernel: ZFS storage pool version: features support (5000) Sep 16 20:56:00 venom-f5 kernel: ipfw2 (+ipv6) initialized, divert enabled, nat loadable, default to accept, logging disabled Sep 16 20:56:00 venom-f5 kernel: Trying to mount root from zfs:zroot/ROOT/default []... Sep 16 20:56:00 venom-f5 kernel: ugen0.1: <(0x1033) XHCI root HUB> at usbus0 Sep 16 20:56:00 venom-f5 kernel: uhub0 on usbus0 Sep 16 20:56:00 venom-f5 kernel: uhub0: <(0x1033) XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 Sep 16 20:56:00 venom-f5 kernel: ada0 at ahcich0 bus 0 scbus2 target 0 lun 0 Sep 16 20:56:00 venom-f5 kernel: ada0: <QEMU HARDDISK 2.5+> ATA-7 SATA device Sep 16 20:56:00 venom-f5 kernel: ada0: Serial Number QM00005 Sep 16 20:56:00 venom-f5 kernel: ada0: 150.000MB/s transfers (SATA 1.x, UDMA5, PIO 8192bytes) Sep 16 20:56:00 venom-f5 kernel: ada0: Command Queueing enabled Sep 16 20:56:00 venom-f5 kernel: ada0: 2621440MB (5368709120 512 byte sectors) Sep 16 20:56:00 venom-f5 kernel: uhub0: 8 ports with 8 removable, self powered Sep 16 20:56:00 venom-f5 kernel: ugen0.2: <QEMU QEMU USB Tablet> at usbus0 Sep 16 20:56:00 venom-f5 kernel: Dual Console: Serial Primary, Video Secondary Sep 16 20:56:00 venom-f5 kernel: intsmb0: <Intel PIIX4 SMBUS Interface> irq 9 at device 1.3 on pci0 Sep 16 20:56:00 venom-f5 kernel: intsmb0: Could not allocate I/O space Sep 16 20:56:00 venom-f5 kernel: device_attach: intsmb0 attach returned 6 Sep 16 20:56:00 venom-f5 kernel: vtcon0: <VirtIO Console Adapter> on virtio_pci0 Sep 16 20:56:00 venom-f5 kernel: em0: link state changed to UP
Do you have a backtrace with line number information? This can be obtained from the full dump, in core.*.txt.
(In reply to Mark Johnston from comment #1) I got this error on 14.0-RC4: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 10 fault virtual address = 0x8 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80be4343 stack pointer = 0x28:0xfffffe0080454ae0 frame pointer = 0x28:0xfffffe0080454b10 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi1: netisr 0) rdi: fffff801d713c290 rsi: fffff8018fed5100 rdx: ffffffff82a10090 rcx: 0000000000001000 r8: ffffffff82a10090 r9: fffffe0080454b50 rax: 0000000000000001 rbx: fffff801d713c290 rbp: fffffe0080454b10 r10: 000000000000000a r11: fffffe0080454740 r12: 0000000000000000 r13: 0000000000000000 r14: fffff8018fed5100 r15: 0000000000001000 trap number = 12 panic: page fault cpuid = 0 time = 1699775882 KDB: stack backtrace: #0 0xffffffff80b9002d at kdb_backtrace+0x5d #1 0xffffffff80b43132 at vpanic+0x132 #2 0xffffffff80b42ff3 at panic+0x43 #3 0xffffffff8100c85c at trap_fatal+0x40c #4 0xffffffff8100c8af at trap_pfault+0x4f #5 0xffffffff80fe3828 at calltrap+0x8 #6 0xffffffff80be454b at sbdrop+0x3b #7 0xffffffff80d11c34 at tcp_do_segment+0x2e04 #8 0xffffffff80d0e4e5 at tcp_input_with_port+0xfe5 #9 0xffffffff80d0ee1b at tcp_input+0xb #10 0xffffffff80cfe40d at ip_input+0x23d #11 0xffffffff80c845b8 at swi_net+0x128 #12 0xffffffff80b01497 at ithread_loop+0x257 #13 0xffffffff80afdb0f at fork_exit+0x7f #14 0xffffffff80fe488e at fork_trampoline+0xe Where are the core files?
(In reply to Vedran Miletic from comment #2) Assuming you have coredumps enabled (e.g., "dumpdev" is set in /etc/rc.conf), they should appear in /var/crash after a reboot.
Created attachment 246274 [details] xz compressed core.txt
(In reply to Vedran Miletic from comment #4) Thanks. There is no need to compress core.txt. Are you able to test FreeBSD-CURRENT? It has assertions enabled which may catch the problem more usefully. CC a couple of TCP folks. PR 268699 looks similar.
A full core would certainly be helpful to see the mbufs in play at the time of the panic.
(In reply to Mark Johnston from comment #5) Unfortunately, not really, and this is the first time that it happened so I am not that worried. I'll CC to bug 268699.
It has happened to me 3 times since the original report on various releases. Not all that often but it is a lightly used webserver I am just now installing gdb, should have done that earlier, sorry. I do have vmcore files, but they are too large to attach, if someone wants one and has another way for me to send, let me know. Otherwise, I will attach core.txt on next crash since i should have gdb installed then. Last crash yesterday, this is the version info. I will be upgrading latest RELENG shortly FreeBSD venom-f5 14.0-RELEASE FreeBSD 14.0-RELEASE #2 releng/14.0-n265380-f9716eee8ab: Tue Nov 21 13:04:48 CST 2023 root@venom-f5:/usr/obj/usr/src/amd64.amd64/sys/VENOM-F5 amd64
I ran kdbg on the vmcore file, I got this...I don't know if this helps. root@venom-f5:/var/crash # kgdb /boot/kernel/kernel /var/crash/vmcore.3 GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD] Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd14.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Unread portion of the kernel message buffer: cpuid = 1; apic id = 01 fault virtual address = 0xb8 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80d4e890 stack pointer = 0x28:0xfffffe01017ddc70 frame pointer = 0x28:0xfffffe01017ddcf0 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 676 (isc-net-0005) rdi: ffffffff816c7a30 rsi: ffffffff816c7a30 rdx: 0000000000010200 rcx: 0000000000000000 r8: fffff8001b0c1400 r9: 0000000000000000 rax: fffff8014f794a80 rbx: fffff8010c069540 rbp: fffffe01017ddcf0 r10: 0000000000000000 r11: fffffe0103a59700 r12: fffff80245b80e20 r13: 000000000000c2e5 r14: 000000000000c2e5 r15: fffff8001b0c1400 trap number = 12 panic: page fault cpuid = 1 time = 1702438402 KDB: stack backtrace: #0 0xffffffff80b9302d at kdb_backtrace+0x5d #1 0xffffffff80b46132 at vpanic+0x132 #2 0xffffffff80b45ff3 at panic+0x43 #3 0xffffffff8103285c at trap_fatal+0x40c #4 0xffffffff810328af at trap_pfault+0x4f #5 0xffffffff810098e8 at calltrap+0x8 #6 0xffffffff80d708ac at udp6_bind+0x13c #7 0xffffffff80be9132 at sobind+0x32 #8 0xffffffff80bf05c5 at kern_bindat+0xc5 #9 0xffffffff80bf045b at sys_bind+0x9b #10 0xffffffff81033119 at amd64_syscall+0x109 #11 0xffffffff8100a1fb at fast_syscall_common+0xf8 Uptime: 19d16h43m33s Dumping 1669 out of 16338 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
Adding gleb; the latest backtrace has no hint of TCP in there, but UDP; maybe the mbuf field reuse we've been discussing this week may have something to do with this?
Thanks for adding me, Richard! This unlikely is related to the mbuf field reuse that we discussed. However, it is very likely the problem is in the area I recently worked on. Weldon, I'm interested in the vmcore file + contents of /boot/kernel + contents of /usr/lib/debug/boot/kernel. Do you have an opportunity to put them on web? Please compress them aggressively and you can use my PGP key to encrypt it. Key id is 5D05CC22 and you can obtain or verify it here https://docs.freebsd.org/en/articles/pgpkeys/#_gleb_smirnoff_glebiusfreebsd_org
(In reply to Gleb Smirnoff from comment #11) thanks, email with link to file sent to you.
Getting very similar (with upd6_bind) crashes on multiple machines (vmware guests) every few days after upgrade to recent 14/stable. Kernel was compiled with "nooptions SCTP", otherwise it is generic.
Weldon gave me access to the core. Writing up summary for Mark and other interested parties. The panic happens at in6_pcb.c:257: (t->inp_socket->so_options & SO_REUSEPORT) || The temporary inpcb t has NULL inp_socket. It also has INP_FREED flag set. The inpcb had been found with in6_pcblookup_local() which doesn't do INP_FREED check, neither acquires the inpcb lock. It relies on the hash lock, that we hold. And the freed inpcb has INP_INHASHLIST flag set, which is definitely a problem. These two flags should be a xor. Unless me or Mark quickly find a problem in the code with our eyes, we would need somebody, e.g. Weldon Godfrey or Dmitry Petrov to run a kernel compiled with INVARIANTS option and probably with additional patch that would catch creation of invalid inpcb. Please let me know if you can assist with this.
I see the problem. The inpcb destruction order has a flaw. We first clear inp_socket, then set INP_FREED flag, then call in_pcbremhash(). This isn't compatible with inpcb_lookup_local() which doesn't use inpcb lock. Coming with a patch soon.
Weldon, Dmitry, please test the patch from this review. Let me know if you have any problems with applying it. https://reviews.freebsd.org/D43122
Applied patch. Patch successfully updated intended files. recompiled/installed kernel and rebooted. Looks good so far
Same here - no issues so far with the patch on top of latest 14/stable on one of the machines. It can take awhile for me to be confident it fixes the issue. In my case it was typically crashing after 2-5 days, and I never managed to reproduce it on demand. Note, I do not really use IPv6 on my local network. Could it be that CARP protocol triggers the issue [as it sends some traffic to ff02::12]?
Dmitry, if you see udp6_bind() in the panic trace, then definitely some process requested to bind to a IPv6 address. You can see the process name in the panic banner in the core.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=a13039e2709277b1c3b159e694cc909a5e044151 commit a13039e2709277b1c3b159e694cc909a5e044151 Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2023-12-27 16:34:37 +0000 Commit: Gleb Smirnoff <glebius@FreeBSD.org> CommitDate: 2023-12-27 16:34:37 +0000 inpcb: reoder inpcb destruction First, merge in_pcbdetach() with in_pcbfree(). The comment for in_pcbdetach() was no longer correct. Then, make sure we remove the inpcb from the hash before we commit any destructive actions on it. There are couple functions that rely on the hash lock skipping SMR + inpcb lock to lookup an inpcb. Although there are no known functions that similarly rely on the global inpcb list lock, also do list removal before destructive actions. PR: 273890 Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D43122 sys/netinet/in_pcb.c | 39 +++++++++++++++------------------------ sys/netinet/in_pcb.h | 1 - sys/netinet/raw_ip.c | 1 - sys/netinet/tcp_syncache.c | 2 -- sys/netinet/tcp_usrreq.c | 2 -- sys/netinet/udp_usrreq.c | 1 - sys/netinet6/raw_ip6.c | 1 - sys/netinet6/udp6_usrreq.c | 1 - 8 files changed, 15 insertions(+), 33 deletions(-)
I encountered a crash today. I'm wondering (based on subject line if this is the same issue, or this is another bug?). It's from 14.0-RELEASE-p3 #6: > Backtrace: > Reading symbols from /boot/kernel/kernel... > Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 > 57 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, > (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:57 > #1 doadump (textdump=textdump@entry=1) > at /usr/src/sys/kern/kern_shutdown.c:405 > #2 0xffffffff8078cdec in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:526 > #3 0xffffffff8078d2df in vpanic (fmt=0xffffffff80c16199 "%s", > ap=ap@entry=0xfffffe00b4558ac0) at /usr/src/sys/kern/kern_shutdown.c:970 > #4 0xffffffff8078d133 in panic (fmt=<unavailable>) > at /usr/src/sys/kern/kern_shutdown.c:894 > #5 0xffffffff80bb70dc in trap_fatal (frame=0xfffffe00b4558bb0, eva=184) > at /usr/src/sys/amd64/amd64/trap.c:952 > #6 0xffffffff80bb712f in trap_pfault (frame=0xfffffe00b4558bb0, > usermode=false, signo=<optimized out>, ucode=<optimized out>) > at /usr/src/sys/amd64/amd64/trap.c:760 > #7 <signal handler called> > #8 0xffffffff809a0820 in in6_pcbbind (inp=inp@entry=0xfffff800cf7dde00, > sin6=sin6@entry=0xfffff801d552db60, cred=0xfffff800920c9b00) > at /usr/src/sys/netinet6/in6_pcb.c:257 > #9 0xffffffff809c283c in udp6_bind (so=<optimized out>, > nam=0xfffff801d552db60, td=<optimized out>) > at /usr/src/sys/netinet6/udp6_usrreq.c:1059 > #10 0xffffffff80832992 in sobind (so=0xffffffff80e6c220 <prison0>, > so@entry=0xfffff80138f1ab40, nam=0xffffffff80e6c220 <prison0>, > nam@entry=0xfffff801d552db60, td=0x10200, td@entry=0xfffffe00b42a2900) > at /usr/src/sys/kern/uipc_socket.c:940 > #11 0xffffffff80839e25 in kern_bindat (td=td@entry=0xfffffe00b42a2900, > dirfd=dirfd@entry=-100, fd=<optimized out>, > sa=sa@entry=0xfffff801d552db60) at /usr/src/sys/kern/uipc_syscalls.c:223 > #12 0xffffffff80839cbb in sys_bind (td=0xfffffe00b42a2900, > uap=0xfffffe00b42a2d00) at /usr/src/sys/kern/uipc_syscalls.c:190 > #13 0xffffffff80bb7fec in syscallenter (td=0xfffffe00b42a2900) > at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:159 > #14 amd64_syscall (td=0xfffffe00b42a2900, traced=0) > at /usr/src/sys/amd64/amd64/trap.c:1197 > #15 <signal handler called> > #16 0x0000000827adb5aa in ?? () > Backtrace stopped: Cannot access memory at address 0x82c3d5af8 > (kgdb)
On Tue Jan 2 21:46:25 2024 UTC, takeda@takeda.tk wrote: > I encountered a crash today. I'm wondering (based on subject line if this is the > same issue, or this is another bug?). This looks similar. Please try out the patch. I plan to merge the fix to stable/14 this week. But extra testing of the patch bore I merge is much appreciated.
I had no issues with the original patch (as of 2023-12-19) so far.
Ok so I just rebuilt the kernel and restarted the machine. Please keep in mind that previous kernel was built on Dec 17th and this is first time it crashed. I will update this bug if it crashes again though.
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=2bfe735277b8858dd7ad937e0bf2286bdfb45182 commit 2bfe735277b8858dd7ad937e0bf2286bdfb45182 Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2023-12-27 16:34:37 +0000 Commit: Gleb Smirnoff <glebius@FreeBSD.org> CommitDate: 2024-01-09 00:29:38 +0000 inpcb: reoder inpcb destruction First, merge in_pcbdetach() with in_pcbfree(). The comment for in_pcbdetach() was no longer correct. Then, make sure we remove the inpcb from the hash before we commit any destructive actions on it. There are couple functions that rely on the hash lock skipping SMR + inpcb lock to lookup an inpcb. Although there are no known functions that similarly rely on the global inpcb list lock, also do list removal before destructive actions. PR: 273890 Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D43122 (cherry picked from commit a13039e2709277b1c3b159e694cc909a5e044151) sys/netinet/in_pcb.c | 39 +++++++++++++++------------------------ sys/netinet/in_pcb.h | 1 - sys/netinet/raw_ip.c | 1 - sys/netinet/tcp_syncache.c | 2 -- sys/netinet/tcp_usrreq.c | 2 -- sys/netinet/udp_usrreq.c | 1 - sys/netinet6/raw_ip6.c | 1 - sys/netinet6/udp6_usrreq.c | 1 - 8 files changed, 15 insertions(+), 33 deletions(-)
Merged to stable/14. Will be fixed in FreeBSD 14.1-RELEASE.
*** Bug 276270 has been marked as a duplicate of this bug. ***
Since this has come up several times, let's reopen this until an erratum is released.
A commit in branch releng/14.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=9db5ae3ec45f069f48808a2916ea1c5374e75e6c commit 9db5ae3ec45f069f48808a2916ea1c5374e75e6c Author: Gleb Smirnoff <glebius@FreeBSD.org> AuthorDate: 2023-12-27 16:34:37 +0000 Commit: Gordon Tetlow <gordon@FreeBSD.org> CommitDate: 2024-02-14 05:40:23 +0000 inpcb: reoder inpcb destruction First, merge in_pcbdetach() with in_pcbfree(). The comment for in_pcbdetach() was no longer correct. Then, make sure we remove the inpcb from the hash before we commit any destructive actions on it. There are couple functions that rely on the hash lock skipping SMR + inpcb lock to lookup an inpcb. Although there are no known functions that similarly rely on the global inpcb list lock, also do list removal before destructive actions. PR: 273890 Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D43122 Approved by: so Security: FreeBSD-EN-24:04.ip (cherry picked from commit a13039e2709277b1c3b159e694cc909a5e044151) (cherry picked from commit 2bfe735277b8858dd7ad937e0bf2286bdfb45182) sys/netinet/in_pcb.c | 39 +++++++++++++++------------------------ sys/netinet/in_pcb.h | 1 - sys/netinet/raw_ip.c | 1 - sys/netinet/tcp_syncache.c | 2 -- sys/netinet/tcp_usrreq.c | 2 -- sys/netinet/udp_usrreq.c | 1 - sys/netinet6/raw_ip6.c | 1 - sys/netinet6/udp6_usrreq.c | 1 - 8 files changed, 15 insertions(+), 33 deletions(-)
An EN for 14.0 was released.