Created attachment 236548 [details] core.txt.9 I am running desktop with KDE Plasma on FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64. It was updated few month ago and running without any problem until 4 days ago. Then there was many instant reboots in a couple minutes. I found kernel panic in /var/log/messages. I configured dumpdevice to catch some more information. Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 04 fault virtual address = 0xf3 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8225173e stack pointer = 0x28:0xfffffe00a3cfac30 frame pointer = 0x28:0xfffffe00a3cfac50 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (arc_prune) trap number = 12 panic: page fault cpuid = 2 time = 1663139603 KDB: stack backtrace: #0 0xffffffff80c69465 at kdb_backtrace+0x65 #1 0xffffffff80c1bb1f at vpanic+0x17f #2 0xffffffff80c1b993 at panic+0x43 #3 0xffffffff810afdf5 at trap_fatal+0x385 #4 0xffffffff810afe4f at trap_pfault+0x4f #5 0xffffffff81087528 at calltrap+0x8 #6 0xffffffff821ae7aa at zfs_zinactive+0xca #7 0xffffffff821a8458 at zfs_freebsd_reclaim+0x38 #8 0xffffffff8117e09f at VOP_RECLAIM_APV+0x1f #9 0xffffffff80cf8c72 at vgonel+0x342 #10 0xffffffff80cf4d47 at vnlru_free_impl+0x2f7 #11 0xffffffff80cf4a01 at vnlru_free_vfsops+0x41 #12 0xffffffff821d5c8b at arc_prune_task+0x4b #13 0xffffffff82187a0f at taskq_run+0x1f #14 0xffffffff80c7da41 at taskqueue_run_locked+0x181 #15 0xffffffff80c7ed52 at taskqueue_thread_loop+0xc2 #16 0xffffffff80bd8a5e at fork_exit+0x7e #17 0xffffffff8108859e at fork_trampoline+0xe Uptime: 24m32s Dumping 2455 out of 12189 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80c1b71c in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:487 #3 0xffffffff80c1bb8e in vpanic (fmt=0xffffffff811b4fb9 "%s", ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:920 #4 0xffffffff80c1b993 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:844 #5 0xffffffff810afdf5 in trap_fatal (frame=0xfffffe00a3cfab70, eva=243) at /usr/src/sys/amd64/amd64/trap.c:944 #6 0xffffffff810afe4f in trap_pfault (frame=0xfffffe00a3cfab70, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:763 #7 <signal handler called> #8 sa_handle_destroy (hdl=0x3) at /usr/src/sys/contrib/openzfs/module/zfs/sa.c:1368 #9 0xffffffff821ae7aa in zfs_znode_dmu_fini (zp=0xfffff80166b661d8) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:390 #10 zfs_zinactive (zp=zp@entry=0xfffff80166b661d8) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:1267 #11 0xffffffff821a8458 in zfs_freebsd_reclaim (ap=<optimized out>) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:5188 #12 0xffffffff8117e09f in VOP_RECLAIM_APV ( vop=0xffffffff82465340 <zfs_vnodeops>, a=a@entry=0xfffffe00a3cfacf0) at vnode_if.c:2180 #13 0xffffffff80cf8c72 in VOP_RECLAIM (vp=0xfffff8003ad81b70) at ./vnode_if.h:1087 #14 vgonel (vp=vp@entry=0xfffff8003ad81b70) at /usr/src/sys/kern/vfs_subr.c:4144 #15 0xffffffff80cf4d47 in vtryrecycle (vp=0xfffff8003ad81b70) at /usr/src/sys/kern/vfs_subr.c:1694 #16 vnlru_free_impl (count=5737, count@entry=10000, mnt_op=mnt_op@entry=0xffffffff82465198 <zfs_vfsops>, mvp=mvp@entry=0xfffff800038eb000) at /usr/src/sys/kern/vfs_subr.c:1331 #17 0xffffffff80cf4a01 in vnlru_free_vfsops (count=count@entry=10000, mnt_op=0xffffffff82465198 <zfs_vfsops>, mvp=0xfffff800038eb000) at /usr/src/sys/kern/vfs_subr.c:1355 #18 0xffffffff821d5c8b in arc_prune_task (arg=0x2710) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/arc_os.c:166 #19 0xffffffff82187a0f in taskq_run (arg=0xfffff8014f2cde40, pending=-1546670864) at /usr/src/sys/contrib/openzfs/module/os/freebsd/spl/spl_taskq.c:315 #20 0xffffffff80c7da41 in taskqueue_run_locked ( queue=queue@entry=0xfffff80003d6bc00) at /usr/src/sys/kern/subr_taskqueue.c:477 #21 0xffffffff80c7ed52 in taskqueue_thread_loop (arg=<optimized out>, arg@entry=0xfffff80004bdfa50) at /usr/src/sys/kern/subr_taskqueue.c:794 #22 0xffffffff80bd8a5e in fork_exit ( callout=0xffffffff80c7ec90 <taskqueue_thread_loop>, arg=0xfffff80004bdfa50, frame=0xfffffe00a3cfaf40) at /usr/src/sys/kern/kern_fork.c:1093 #23 <signal handler called> #24 mi_startup () at /usr/src/sys/kern/init_main.c:322 #25 0xffffffff80f79189 in swapper () at /usr/src/sys/vm/vm_swapout.c:755 #26 0xffffffff80385022 in btext () at /usr/src/sys/amd64/amd64/locore.S:80 I have 10 vmcore files saved. Every panic ends with: __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, Some times it was something like reboot loop because KDE Plasma starts application which were running before shutdown / reboot. So it seems it is related to some user's app running inside of KDE. I needed to login as different user to break this panic loop. Computer was runing for many ours under different user account. I guessed it was related to Firefox so I tried to run as my normal user without Firefox last night and it was running for about 5 hours without panic. But this morning I got 2 panics in 1 hour without Firefox running (used Iridium browser instead) I did run memtest86 which passed OK. I did run zpool scrub - no issue found. I can send full vmcore files to someone interested in this panics. core.txt.9 is attached (real username was replaced with "usr1". Any help is really appreciated because this is my main PC for work.
gpart show gpart show -l zpool list -v (In reply to Miroslav Lachman from comment #0) > I did run zpool scrub - no issue found. Still, I would be inclined to take a closer look at hardware. Check connections, et cetera. Nits: 1) the OS is slightly outdated, should be at patch level 2, you might prefer to defer the update until after this bug report is triaged 2) probably in your kld_list, there's fuse; should be fusefs.
@Reporter Could you please include: - /var/run/dmesg.boot output (as an attachment) - zfs list output (as an attachment) - /etc/rc.conf contents (as an attachment)
Created attachment 236599 [details] dmesg.boot
Created attachment 236600 [details] gpart show -l
Created attachment 236601 [details] gpart show
Created attachment 236602 [details] zpool list -v
Created attachment 236603 [details] zfs list
Created attachment 236604 [details] rc.conf rc.conf is a mess. This machine started in the era of PC-BSD (FreeBSD 10) and then was upgraded many times to the current version 13.1 with many automatically added knobs. I will clean it after this issue is fixed.
Created attachment 236605 [details] fstab
Created attachment 236606 [details] loader.conf
Created attachment 236609 [details] Last 10 back traces / core.txt packed in tar.gz The last 10 back traces from core.txt files saved in /var/crash, including 2 from today.
I am sorry for my late reply. The computer is almost unusable under my main user account. Works fine when in single user mode - I suspected some FS problem so I tried to read all files on all 3 disks (3 ZFS pools) by md5sum - it took almost 20 hours but all files was read without kernel panic. I can work in KDE if I use different user account (tester), but kernel panic occurred when I accessed some files of my main user account (usr1) The machine in question is Dell T20 with built-in diagnostic. I booted in to this diagnostic yesterday, it runs about 2 - 3 hours, all test PASSed. I think the problem is not in a hardware. All requested files are attached. tank0 / ada2: SSD Samsung SSD 860 EVO 1TB RVT04B6Q tank2 / ada1: HDD ST2000VX000-1ES164 CV26 tank1 / ada0: HDD ST1000DM003-1CH162 CC47 Swap is encrypted if it matters # swapinfo -h Device Size Used Avail Capacity /dev/gpt/swap0.eli 5.5G 0B 5.5G 0% Let me know if I should provide some additional information.
Additional update - some times it paniced under different user account too and it also paniced in single user mode when I run locate.updatedb but it does not write anything in to /var/crash. Is there a way to dump info in single user too?
(In reply to Miroslav Lachman from comment #13) I have a bt from single user panic: Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 04 fault virtual address = 0xc3 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff821d7dd0 stack pointer = 0x28:0xfffffe00f7640350 frame pointer = 0x28:0xfffffe00f7640410 code segment = base rx0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 74 (find) trap number = 12 panic: page fault cpuid = 2 time = 1663591376 KDB: stack backtrace: #0 0xffffffff80c69465 at kdb_backtrace+0x65 #1 0xffffffff80c1bb1f at vpanic+0x17f #2 0xffffffff80c1b993 at panic+0x43 #3 0xffffffff810afdf5 at trap_fatal+0x385 #4 0xffffffff810afe4f at trap_pfault+0x4f #5 0xffffffff81087528 at calltrap+0x8 #6 0xffffffff8224b7bf at sa_handle_destroy+0x8f #7 0xffffffff821a87aa at zfs_zinactive+0xca #8 0xffffffff821a2458 at zfs_freebsd_reclaim+0x38 #9 0xffffffff8117e09f at VOP_RECLAIM_APV+0x1f #10 0xffffffff80cf8c72 at vgonel+0x342 #11 0xffffffff80cf4d47 at vnlru_free_impl+0x2f7 #12 0xffffffff80cffc38 at vn_alloc_hard+0xc8 #13 0xffffffff80cf5423 at getnewvnode_reserve+0x93 #14 0xffffffff821a78a2 at zfs_zget+0x22 #15 0xffffffff8219284b at zfs_dirent_lookup+0x16b #16 0xffffffff8219291a at zfs_dirlook+0x7a #17 0xffffffff821a4a10 at zfs_lookup+0x3d0 Uptime: 10m33s Dumping 1729 out of 12189 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80c1b71c in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:487 #3 0xffffffff80c1bb8e in vpanic (fmt=0xffffffff811b4fb9 "%s", ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:920 #4 0xffffffff80c1b993 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:844 #5 0xffffffff810afdf5 in trap_fatal (frame=0xfffffe00f7640290, eva=195) at /usr/src/sys/amd64/amd64/trap.c:944 #6 0xffffffff810afe4f in trap_pfault (frame=0xfffffe00f7640290, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:763 #7 <signal handler called> #8 dbuf_evict_user (db=0xfffff800263b4128) at /usr/src/sys/contrib/openzfs/module/zfs/dbuf.c:569 #9 dbuf_rele_and_unlock (db=0xfffff800263b4128, tag=<optimized out>, evicting=0) at /usr/src/sys/contrib/openzfs/module/zfs/dbuf.c:3712 #10 0xffffffff821da0bb in dbuf_rele (db=<optimized out>, tag=<optimized out>, tag@entry=0x0) at /usr/src/sys/contrib/openzfs/module/zfs/dbuf.c:3662 #11 0xffffffff8224b7bf in sa_handle_destroy (hdl=0xfffff8004f980900) at /usr/src/sys/contrib/openzfs/module/zfs/sa.c:1379 #12 0xffffffff821a87aa in zfs_znode_dmu_fini (zp=0xfffff80035b90760) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:390 #13 zfs_zinactive (zp=zp@entry=0xfffff80035b90760) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:1267 #14 0xffffffff821a2458 in zfs_freebsd_reclaim (ap=<optimized out>) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:5188 #15 0xffffffff8117e09f in VOP_RECLAIM_APV ( vop=0xffffffff8245f340 <zfs_vnodeops>, a=a@entry=0xfffffe00f76404e0) at vnode_if.c:2180 #16 0xffffffff80cf8c72 in VOP_RECLAIM (vp=0xfffff80058e705b8) at ./vnode_if.h:1087 #17 vgonel (vp=vp@entry=0xfffff80058e705b8) at /usr/src/sys/kern/vfs_subr.c:4144 #18 0xffffffff80cf4d47 in vtryrecycle (vp=0xfffff80058e705b8) at /usr/src/sys/kern/vfs_subr.c:1694 #19 vnlru_free_impl (count=count@entry=1, mnt_op=mnt_op@entry=0x0, mvp=0xfffff80003be3400) at /usr/src/sys/kern/vfs_subr.c:1331 #20 0xffffffff80cffc38 in vnlru_free_locked (count=1) at /usr/src/sys/kern/vfs_subr.c:1344 #21 vn_alloc_hard (mp=mp@entry=0x0) at /usr/src/sys/kern/vfs_subr.c:1745 #22 0xffffffff80cf5423 in vn_alloc (mp=0x0) at /usr/src/sys/amd64/include/atomic.h:416 #23 getnewvnode_reserve () at /usr/src/sys/kern/vfs_subr.c:1887 #24 0xffffffff821a78a2 in zfs_zget (zfsvfs=zfsvfs@entry=0xfffff8001cdee000, obj_num=257193, zpp=zpp@entry=0xfffffe00f76406f0) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_znode.c:942 #25 0xffffffff8219284b in zfs_dirent_lookup ( dzp=dzp@entry=0xfffff80058c6b938, name=0xfffffe00f7640860 "patch-src_init.c", zpp=zpp@entry=0xfffffe00f7640740, flag=flag@entry=2) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_dir.c:191 #26 0xffffffff8219291a in zfs_dirlook (dzp=dzp@entry=0xfffff80058c6b938, name=0x0, name@entry=0xfffffe00f7640860 "patch-src_init.c", zpp=zpp@entry=0xfffffe00f76407e0) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_dir.c:247 #27 0xffffffff821a4a10 in zfs_lookup (dvp=<optimized out>, nm=nm@entry=0xfffffe00f7640860 "patch-src_init.c", vpp=<optimized out>, cnp=cnp@entry=0xfffffe00f7640c58, nameiop=0, cr=<optimized out>, flags=0, cached=1) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:929 #28 0xffffffff821a011b in zfs_freebsd_lookup (ap=0xfffffe00f7640990, cached=1) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:4593 #29 zfs_freebsd_cachedlookup (ap=0xfffffe00f7640990) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_vnops_os.c:4601 #30 0xffffffff80cdc4ad in VOP_CACHEDLOOKUP (dvp=0xfffff80058dd83d0, vpp=0xfffffe00f7640a10, cnp=0xfffffe00f7640c58) at ./vnode_if.h:99 #31 vfs_cache_lookup (ap=<optimized out>) at /usr/src/sys/kern/vfs_cache.c:3069 #32 0xffffffff80ce0cb0 in VOP_LOOKUP (dvp=dvp@entry=0xfffff80058dd83d0, vpp=0x0, vpp@entry=0xfffffe00f7640a10, cnp=0x0, cnp@entry=0xfffffe00f7640c58) at ./vnode_if.h:65 #33 0xffffffff80ce06e3 in cache_fplookup_noentry ( fpl=fpl@entry=0xfffffe00f7640a88) at /usr/src/sys/kern/vfs_cache.c:4928 #34 0xffffffff80cddf66 in cache_fplookup_next (fpl=0xfffffe00f7640a88) at /usr/src/sys/kern/vfs_cache.c:5284 #35 cache_fplookup_impl (dvp=<optimized out>, fpl=0xfffffe00f7640a88) at /usr/src/sys/kern/vfs_cache.c:5932 #36 cache_fplookup (ndp=ndp@entry=0xfffffe00f7640bd8, status=status@entry=0xfffffe00f7640b84, pwdp=pwdp@entry=0xfffffe00f7640b88) at /usr/src/sys/kern/vfs_cache.c:6104 #37 0xffffffff80ce8cba in namei (ndp=ndp@entry=0xfffffe00f7640bd8) at /usr/src/sys/kern/vfs_lookup.c:570 #38 0xffffffff80d06953 in kern_statat (td=0xfffffe00f75fc560, flag=<optimized out>, fd=-100, path=0xfffffe00f75fc560 "\300`\264\026", pathseg=pathseg@entry=UIO_USERSPACE, sbp=sbp@entry=0xfffffe00f7640d18, hook=0x0) at /usr/src/sys/kern/vfs_syscalls.c:2441 #39 0xffffffff80d0704f in sys_fstatat (td=0x0, uap=0xfffffe00f75fc948) at /usr/src/sys/kern/vfs_syscalls.c:2418 #40 0xffffffff810b06ec in syscallenter (td=0xfffffe00f75fc560) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189 #41 amd64_syscall (td=0xfffffe00f75fc560, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1185 #42 <signal handler called> #43 0x00000008011ad39a in ?? () Backtrace stopped: Cannot access memory at address 0x7fffffffe628 (kgdb)
Is somebody interested in this repeatable panic? If not I will try to wipe it and reinstall from scratch because I need to work on this machine.
I'd be curious to see "zfs get all" from whichever dataset is / right now, and whichever dataset is serving /usr/home right now.
Created attachment 236722 [details] zfs get all /
Created attachment 236723 [details] zfs get all /usr/home
Created attachment 236724 [details] zfs get all /usr/home/usr1
(In reply to Rich Ercolani from comment #16) I think nothing unusual. See attached files.
I can confirm the panic is related to filesystem content. I moved all the data from SSD to HDD by zfs send & zfs receive few days ago and no panic occurred so far.