Bug 252700 - page fault in zfsctl_snapdir_lookup
Summary: page fault in zfsctl_snapdir_lookup
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Many People
Assignee: Robert Wing
URL: https://github.com/freebsd/freebsd-sr...
Keywords:
Depends on:
Blocks:
 
Reported: 2021-01-15 03:12 UTC by Alan Somers
Modified: 2023-12-01 16:55 UTC (History)
11 users (show)

See Also:
asomers: mfc-stable14+
asomers: mfc-stable13+
asomers: mfc-stable12-


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alan Somers freebsd_committer freebsd_triage 2021-01-15 03:12:46 UTC
I hit the following page fault on FreeBSD 13.0-CURRENT at revision d2b3ceddccac60b563f642898e3a314647666a10.

It's reproducible by running the sys/cddl/zfs/tests/snapshot/snapshot_test:snapshot_019_pos test case.

#0  __curthread ()
    at /usr/home/somers/src/freebsd.org/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=textdump@entry=0)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/kern_shutdown.c:399
#2  0xffffffff804b6f4a in db_dump (dummy=<optimized out>, 
    dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>)
    at /usr/home/somers/src/freebsd.org/src/sys/ddb/db_command.c:575
#3  0xffffffff804b6d10 in db_command (last_cmdp=<optimized out>, 
    cmd_table=<optimized out>, dopager=dopager@entry=1)
    at /usr/home/somers/src/freebsd.org/src/sys/ddb/db_command.c:482
#4  0xffffffff804b6a6d in db_command_loop ()
    at /usr/home/somers/src/freebsd.org/src/sys/ddb/db_command.c:535
#5  0xffffffff804b9dd6 in db_trap (type=<optimized out>, code=<optimized out>)
    at /usr/home/somers/src/freebsd.org/src/sys/ddb/db_main.c:270
#6  0xffffffff80c406a4 in kdb_trap (type=type@entry=3, code=code@entry=0, 
    tf=<optimized out>, tf@entry=0xfffffe0084f87f20)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/subr_kdb.c:727
#7  0xffffffff8104719e in trap (frame=0xfffffe0084f87f20)
    at /usr/home/somers/src/freebsd.org/src/sys/amd64/amd64/trap.c:576
#8  <signal handler called>
#9  kdb_enter (why=0xffffffff8122e0ac "panic", msg=<optimized out>)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/subr_kdb.c:506
#10 0xffffffff80bf41a2 in vpanic (fmt=<optimized out>, ap=<optimized out>, 
    ap@entry=0xfffffe0084f88080)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/kern_shutdown.c:907
#11 0xffffffff80bf3f33 in panic (
    fmt=0xffffffff81c94178 <cnputs_mtx> "\375\342\036\201\377\377\377\377")
    at /usr/home/somers/src/freebsd.org/src/sys/kern/kern_shutdown.c:843
#12 0xffffffff810475f7 in trap_fatal (frame=0xfffffe0084f88280, eva=0)
    at /usr/home/somers/src/freebsd.org/src/sys/amd64/amd64/trap.c:915
#13 0xffffffff81047697 in trap_pfault (frame=frame@entry=0xfffffe0084f88280, 
    usermode=false, signo=<optimized out>, signo@entry=0x0, 
    ucode=<optimized out>, ucode@entry=0x0)
    at /usr/home/somers/src/freebsd.org/src/sys/amd64/amd64/trap.c:732
#14 0xffffffff81046c8b in trap (frame=0xfffffe0084f88280)
    at /usr/home/somers/src/freebsd.org/src/sys/amd64/amd64/trap.c:398
#15 <signal handler called>
#16 0x0000000000000000 in ?? ()
#17 0xffffffff81109215 in VOP_CLOSE_APV (
    vop=0xffffffff826f0a90 <zfsctl_ops_snapshot>, 
    a=a@entry=0xfffffe0084f88370) at vnode_if.c:498
#18 0xffffffff80cd9839 in VOP_CLOSE (vp=0xfffff801d8df91e8, fflag=4, 
    cred=0x0, td=0xfffffe0085416e00) at ./vnode_if.h:249
#19 vgonel (vp=vp@entry=0xfffff801d8df91e8)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/vfs_subr.c:3913
#20 0xffffffff80cda05f in vgone (vp=vp@entry=0xfffff801d8df91e8)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/vfs_subr.c:3788
#21 0xffffffff80cc8104 in vfs_hash_insert (vp=0xfffff801d8df91e8, 
    hash=hash@entry=138, flags=flags@entry=2097152, td=<optimized out>, 
    td@entry=0xfffffe0085416e00, vpp=vpp@entry=0xfffffe0084f889f8, 
    fn=0xffffffff82438590 <sfs_compare_ids>, arg=0xfffff80136d08a80)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/vfs_hash.c:175
#22 0xffffffff8243852b in sfs_vnode_insert (vp=0xfffffe0084f88370, 
    flags=2097152, parent_id=<optimized out>, id=<optimized out>, 
    vpp=0xfffffe0084f889f8)
    at /usr/home/somers/src/freebsd.org/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c:152
#23 sfs_vgetx (mp=<optimized out>, flags=62819872, flags@entry=2097152, 
    parent_id=parent_id@entry=2, id=138, tag=<optimized out>, 
    vops=0xffffffff826f0a90 <zfsctl_ops_snapshot>, 
    setup=0xffffffff824397d0 <zfsctl_snapshot_vnode_setup>, 
    arg=0xfffffe0084f885f0, vpp=0xfffffe0084f889f8)
    at /usr/home/somers/src/freebsd.org/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c:200
#24 0xffffffff824390bd in zfsctl_snapdir_lookup (ap=<optimized out>)
    at /usr/home/somers/src/freebsd.org/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c:949
#25 0xffffffff80cca581 in VOP_LOOKUP (dvp=0xfffff8008c771988, 
    vpp=0xfffffe0084f889f8, cnp=0xfffffe0084f88a20) at ./vnode_if.h:69
#26 lookup (ndp=ndp@entry=0xfffffe0084f889a0)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/vfs_lookup.c:1091
#27 0xffffffff80cc9960 in namei (ndp=ndp@entry=0xfffffe0084f889a0)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/vfs_lookup.c:631
#28 0xffffffff80ce5259 in kern_chdir (td=0xfffffe0085416e00, 
    path=0x8008b73e1 <error: Cannot access memory at address 0x8008b73e1>, 
    pathseg=UIO_USERSPACE)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/vfs_syscalls.c:948
#29 0xffffffff81047fbe in syscallenter (td=<optimized out>)
    at /usr/home/somers/src/freebsd.org/src/sys/amd64/amd64/../../kern/subr_syscall.c:189
#30 amd64_syscall (td=0xfffffe0085416e00, traced=0)
    at /usr/home/somers/src/freebsd.org/src/sys/amd64/amd64/trap.c:1156
#31 <signal handler called>
#32 0x000000080058fa0a in ?? ()
Comment 1 Andriy Gapon freebsd_committer freebsd_triage 2021-01-15 07:48:24 UTC
The crash is because sfs does not implement VOP_CLOSE.

#16 0x0000000000000000 in ?? ()
#17 0xffffffff81109215 in VOP_CLOSE_APV (
    vop=0xffffffff826f0a90 <zfsctl_ops_snapshot>, 
    a=a@entry=0xfffffe0084f88370) at vnode_if.c:498
#18 0xffffffff80cd9839 in VOP_CLOSE (vp=0xfffff801d8df91e8, fflag=4, 
    cred=0x0, td=0xfffffe0085416e00) at ./vnode_if.h:249
#19 vgonel (vp=vp@entry=0xfffff801d8df91e8)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/vfs_subr.c:3913
#20 0xffffffff80cda05f in vgone (vp=vp@entry=0xfffff801d8df91e8)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/vfs_subr.c:3788
#21 0xffffffff80cc8104 in vfs_hash_insert (vp=0xfffff801d8df91e8, 
    hash=hash@entry=138, flags=flags@entry=2097152, td=<optimized out>, 
    td@entry=0xfffffe0085416e00, vpp=vpp@entry=0xfffffe0084f889f8, 
    fn=0xffffffff82438590 <sfs_compare_ids>, arg=0xfffff80136d08a80)
    at /usr/home/somers/src/freebsd.org/src/sys/kern/vfs_hash.c:175

It's interesting that vfs_hash_insert wants to vgone that vnode -- so it must doomed already, but apparently vgonel does not see it as doomed?

My instincts suggest that this could be either some very rare / exotic situation or a result of a recent breakage (or, at least, an incompatible change) in VFS.
Comment 2 bugs.freebsd 2022-07-24 17:46:32 UTC
I'm seeing the same issue on FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64

GNU gdb (GDB) 11.2 [GDB v11.2 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd13.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 9; apic id = 09
fault virtual address   = 0x0
fault code              = supervisor read instruction, page not present
instruction pointer     = 0x20:0x0
stack pointer           = 0x28:0xfffffe028bcc8588
frame pointer           = 0x28:0xfffffe028bcc85a0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1426 (rover)
trap number             = 12
panic: page fault
cpuid = 9
time = 1658680663
KDB: stack backtrace:
#0 0xffffffff80c69465 at kdb_backtrace+0x65
#1 0xffffffff80c1bb1f at vpanic+0x17f
#2 0xffffffff80c1b993 at panic+0x43
#3 0xffffffff810afdf5 at trap_fatal+0x385
#4 0xffffffff810afe4f at trap_pfault+0x4f
#5 0xffffffff81087528 at calltrap+0x8
#6 0xffffffff80cf8ab6 at vgonel+0x186
#7 0xffffffff80cf9171 at vgone+0x31
#8 0xffffffff80ce799d at vfs_hash_insert+0x26d
#9 0xffffffff8216b069 at sfs_vgetx+0x149
#10 0xffffffff8216bc54 at zfsctl_snapdir_lookup+0x1e4
#11 0xffffffff80ce9bbc at lookup+0x45c
#12 0xffffffff80ce8de9 at namei+0x259
#13 0xffffffff80d06953 at kern_statat+0xf3
#14 0xffffffff80d0704f at sys_fstatat+0x2f
#15 0xffffffff810b06ec at amd64_syscall+0x10c
#16 0xffffffff81087e3b at fast_syscall_common+0xf8
Uptime: 21h32m34s
Dumping 6645 out of 130655 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55      /usr/src/sys/amd64/include/pcpu_aux.h: No such file or directory.
(kgdb) BT
Undefined command: "BT".  Try "help".
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:399
#2  0xffffffff80c1b71c in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:487
#3  0xffffffff80c1bb8e in vpanic (fmt=0xffffffff811b4fb9 "%s", ap=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:920
#4  0xffffffff80c1b993 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:844
#5  0xffffffff810afdf5 in trap_fatal (frame=0xfffffe028bcc84c0, eva=0) at /usr/src/sys/amd64/amd64/trap.c:944
#6  0xffffffff810afe4f in trap_pfault (frame=0xfffffe028bcc84c0, usermode=false, signo=<optimized out>, ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:763
#7  <signal handler called>
#8  0x0000000000000000 in ?? ()
#9  0xffffffff8117c26c in VOP_CLOSE_APV (vop=0xffffffff82438a70 <zfsctl_ops_snapshot>, a=a@entry=0xfffffe028bcc85b0) at vnode_if.c:498
#10 0xffffffff80cf8ab6 in VOP_CLOSE (vp=0xfffff812e9da97a0, fflag=4, cred=0x0, td=0xfffffe0205592720) at ./vnode_if.h:249
#11 vgonel (vp=vp@entry=0xfffff812e9da97a0) at /usr/src/sys/kern/vfs_subr.c:4088
#12 0xffffffff80cf9171 in vgone (vp=vp@entry=0xfffff812e9da97a0) at /usr/src/sys/kern/vfs_subr.c:3963
#13 0xffffffff80ce799d in vfs_hash_insert (vp=0xfffff812e9da97a0, hash=2907, hash@entry=2345438256, flags=flags@entry=2097152, td=<optimized out>, td@entry=0xfffffe0205592720, vpp=vpp@entry=0xfffffe028bcc8c30, fn=<optimized out>, arg=0xfffff81b0101a000) at /usr/src/sys/kern/vfs_hash.c:181
#14 0xffffffff8216b069 in sfs_vnode_insert (vp=0xfffffe028bcc85b0, flags=2097152, vpp=0xfffffe028bcc8c30, parent_id=<optimized out>, id=<optimized out>) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c:152
#15 sfs_vgetx (mp=0xfffffe028ae5ab00, flags=flags@entry=2097152, parent_id=parent_id@entry=2, id=<optimized out>, tag=<optimized out>, vops=0xffffffff82438a70 <zfsctl_ops_snapshot>, setup=0xffffffff8216c360 <zfsctl_snapshot_vnode_setup>, arg=0xfffffe028bcc8820, vpp=0xfffffe028bcc8c30)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c:200
#16 0xffffffff8216bc54 in zfsctl_snapdir_lookup (ap=<optimized out>) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c:954
#17 0xffffffff80ce9bbc in VOP_LOOKUP (dvp=0xfffff808e5418000, vpp=0xfffffe028bcc8c30, cnp=0xfffffe028bcc8c58) at ./vnode_if.h:65
#18 lookup (ndp=ndp@entry=0xfffffe028bcc8bd8) at /usr/src/sys/kern/vfs_lookup.c:1086
#19 0xffffffff80ce8de9 in namei (ndp=ndp@entry=0xfffffe028bcc8bd8) at /usr/src/sys/kern/vfs_lookup.c:616
#20 0xffffffff80d06953 in kern_statat (td=0xfffffe0205592720, flag=<optimized out>, fd=-100, path=0x0, pathseg=(unknown: 0x5592c30), pathseg@entry=UIO_USERSPACE, sbp=sbp@entry=0xfffffe028bcc8d18, hook=0x0) at /usr/src/sys/kern/vfs_syscalls.c:2441
#21 0xffffffff80d0704f in sys_fstatat (td=0xfffffe028bcc85b0, uap=0xfffffe0205592b08) at /usr/src/sys/kern/vfs_syscalls.c:2418
#22 0xffffffff810b06ec in syscallenter (td=0xfffffe0205592720) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:189
#23 amd64_syscall (td=0xfffffe0205592720, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1185
#24 <signal handler called>
#25 0x000000080134139a in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffde18
Comment 3 Allan Jude freebsd_committer freebsd_triage 2022-09-09 23:31:39 UTC
(In reply to bugs.freebsd from comment #2)

Do you know what you were doing to the .zfs/snasphot directory to cause this crash?

I think I have a fix, but i've not been able to reproduce the error to validate the fix.
Comment 4 Robert Wing freebsd_committer freebsd_triage 2023-01-16 23:08:03 UTC
(In reply to Alan Somers from comment #0)

Hey Alan,

I'm trying to reproduce this, do you recall how you were executing snapshot_019_pos test case with kyua?

I tried the following:

# cd /usr/tests/sys/cddl/zfs/tests/snapshot/snapshot_test
# kyua test
snapshot_test:clone_001_pos  ->  skipped: A minimum of 1 disks is required to run.  You specified 0 disk(s)  [0.264s]
snapshot_test:rollback_001_pos  ->  skipped: A minimum of 1 disks is required to run.  You specified 0 disk(s)  [0.247s]
snapshot_test:rollback_002_pos  ->  skipped: A minimum of 1 disks is required to run.  You specified 0 disk(s)  [0.254s]
snapshot_test:rollback_003_pos  ->  skipped: A minimum of 1 disks is required to run.  You specified 0 disk(s)  [0.266s]
[[ .. snipped .. ]]
....
...
Comment 5 Robert Wing freebsd_committer freebsd_triage 2023-01-16 23:10:13 UTC
or if *someone* has a coredump of this panic...that'd be helpful as well
Comment 6 Alan Somers freebsd_committer freebsd_triage 2023-01-24 00:18:04 UTC
Robert Wing, you need to define some disks to work with.  Assuming vtbd2 is available, and you don't mind its contents being blown away, you should be able to do this:

echo test_suites.FreeBSD.disks = '/dev/vtbd2' >> /etc/kyua.conf
Comment 7 Robert Wing freebsd_committer freebsd_triage 2023-02-17 00:14:00 UTC
https://github.com/openzfs/zfs/pull/14501
Comment 8 Alan Somers freebsd_committer freebsd_triage 2023-02-17 02:05:04 UTC
(In reply to Robert Wing from comment #4)
Firstly, you need to have some disks that you don't mind overwriting.  Then, you need to edit /etc/kyua/kyua.conf and add a line similar to the following:

test_suites.FreeBSD.disks = '/dev/vtbd2 /dev/vtbd3 /dev/vtbd4 /dev/vtbd5'
Comment 9 Robert Wing freebsd_committer freebsd_triage 2023-02-17 02:32:52 UTC
(In reply to Alan Somers from comment #8)

Thanks for the tips on setting that up.

The PR I posted above is the proposed fix.
Comment 11 Andriy Gapon freebsd_committer freebsd_triage 2023-02-22 16:36:26 UTC
(In reply to Robert Wing from comment #10)
Thank you for the fix! But I think that we move bugs to closed/fixed after a fix is actually in FreeBSD.  Even more, when the fix is in all active branches where it is needed / wanted.
Comment 12 Robert Wing freebsd_committer freebsd_triage 2023-02-22 19:01:46 UTC
(In reply to Andriy Gapon from comment #11)

Alright, I'll open this back up until the fix is merged into FreeBSD just in case someone else stumbles across this problem in the meantime.

also - thanks for reviewing this change.
Comment 13 Graham Perrin freebsd_committer freebsd_triage 2023-04-22 18:26:36 UTC
(In reply to Robert Wing from comment #12)

> zfs: merge openzfs/zfs@431083f75 · freebsd/freebsd-src@2a58b31

Includes: 

#14501 FreeBSD: don't verify recycled vnode for zfs control directory
Comment 14 geoffroy desvernay 2023-07-27 11:18:37 UTC
I do experience pseudo random crashes with find(8), may this be related ?

Unread portion of the kernel message buffer:
panic: page fault
cpuid = 4
time = 1690247328
KDB: stack backtrace:
#0 0xffffffff80c53d95 at kdb_backtrace+0x65
#1 0xffffffff80c06711 at vpanic+0x151
#2 0xffffffff80c065b3 at panic+0x43
#3 0xffffffff810b1fa7 at trap_fatal+0x387
#4 0xffffffff810b1fff at trap_pfault+0x4f
#5 0xffffffff81088e48 at calltrap+0x8
#6 0xffffffff80ce4d96 at vgonel+0x186
#7 0xffffffff80ce5451 at vgone+0x31
#8 0xffffffff80cd3df9 at vfs_hash_insert+0x279
#9 0xffffffff82176e59 at sfs_vgetx+0x149
#10 0xffffffff82177a44 at zfsctl_snapdir_lookup+0x1e4
#11 0xffffffff80cd60ac at lookup+0x45c
#12 0xffffffff80cd52cd at namei+0x24d
#13 0xffffffff80cf2c16 at kern_statat+0xf6
#14 0xffffffff80cf331f at sys_fstatat+0x2f
#15 0xffffffff810b289c at amd64_syscall+0x10c
#16 0xffffffff8108975b at fast_syscall_common+0xf8
Uptime: 18h19m29s

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) bt
#0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55
#1  doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:396
#2  0xffffffff80c062da in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:484
#3  0xffffffff80c0677e in vpanic (fmt=<optimized out>, ap=ap@entry=0xfffffe01a84643d0)
    at /usr/src/sys/kern/kern_shutdown.c:923
#4  0xffffffff80c065b3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:847
#5  0xffffffff810b1fa7 in trap_fatal (frame=0xfffffe01a84644c0, eva=0) at /usr/src/sys/amd64/amd64/trap.c:942
#6  0xffffffff810b1fff in trap_pfault (frame=0xfffffe01a84644c0, usermode=false, signo=<optimized out>, 
    ucode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:761
#7  <signal handler called>
#8  0x0000000000000000 in ?? ()
#9  0xffffffff811813fc in VOP_CLOSE_APV (vop=0xffffffff8242aad0 <zfsctl_ops_snapshot>, a=a@entry=0xfffffe01a84645b0)
    at vnode_if.c:498
#10 0xffffffff80ce4d96 in VOP_CLOSE (vp=0xfffff805ce07c3d0, fflag=4, cred=0x0, td=0xfffffe01a3ecaac0) at ./vnode_if.h:249
#11 vgonel (vp=vp@entry=0xfffff805ce07c3d0) at /usr/src/sys/kern/vfs_subr.c:4092
#12 0xffffffff80ce5451 in vgone (vp=vp@entry=0xfffff805ce07c3d0) at /usr/src/sys/kern/vfs_subr.c:3967
#13 0xffffffff80cd3df9 in vfs_hash_insert (vp=0xfffff805ce07c3d0, hash=7373, hash@entry=2823179312, 
    flags=flags@entry=2097152, td=td@entry=0xfffffe01a3ecaac0, vpp=vpp@entry=0xfffffe01a8464c30, fn=<optimized out>, 
    arg=0xfffff808188efd80) at /usr/src/sys/kern/vfs_hash.c:181
#14 0xffffffff82176e59 in sfs_vnode_insert (vp=0xfffffe01a84645b0, flags=2097152, id=<optimized out>, 
    parent_id=<optimized out>, vpp=<optimized out>) at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c:152
#15 sfs_vgetx (mp=0xfffffe01637fab00, flags=flags@entry=2097152, parent_id=parent_id@entry=2, id=<optimized out>, 
    tag=<optimized out>, vops=0xffffffff8242aad0 <zfsctl_ops_snapshot>, 
    setup=0xffffffff82178180 <zfsctl_snapshot_vnode_setup>, arg=0xfffffe01a8464820, vpp=0xfffffe01a8464c30)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c:200
#16 0xffffffff82177a44 in zfsctl_snapdir_lookup (ap=<optimized out>)
    at /usr/src/sys/contrib/openzfs/module/os/freebsd/zfs/zfs_ctldir.c:954
#17 0xffffffff80cd60ac in VOP_LOOKUP (dvp=0xfffff805e17b67a0, vpp=0xfffffe01a8464c30, cnp=0xfffffe01a8464c58)
    at ./vnode_if.h:65
#18 lookup (ndp=ndp@entry=0xfffffe01a8464bd8) at /usr/src/sys/kern/vfs_lookup.c:1086
#19 0xffffffff80cd52cd in namei (ndp=ndp@entry=0xfffffe01a8464bd8) at /usr/src/sys/kern/vfs_lookup.c:616
#20 0xffffffff80cf2c16 in kern_statat (td=0xfffffe01a3ecaac0, flag=<optimized out>, fd=-100, path=0x0, 
    pathseg=(unknown: 0xa3ecafd0), pathseg@entry=UIO_USERSPACE, sbp=0xfffffe01a8465000, sbp@entry=0xfffffe01a8464d18, 
    hook=0x0) at /usr/src/sys/kern/vfs_syscalls.c:2438
#21 0xffffffff80cf331f in sys_fstatat (td=0xfffffe01a84645b0, uap=0xfffffe01a3ecaea8)
    at /usr/src/sys/kern/vfs_syscalls.c:2415
#22 0xffffffff810b289c in syscallenter (td=0xfffffe01a3ecaac0) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:190
#23 amd64_syscall (td=0xfffffe01a3ecaac0, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1183
#24 <signal handler called>
#25 0x0000107a36849cba in ?? ()
Backtrace stopped: Cannot access memory at address 0x107a342835e8
Comment 15 geoffroy desvernay 2023-07-27 11:19:47 UTC
Sorry, this is 13.2-RELEASEp1
Comment 16 geoffroy desvernay 2023-07-27 12:36:41 UTC
is it a good idea to try with this patch ?
https://github.com/openzfs/zfs/commit/28251d81d723292a6813f93495f2c6c132938945
Comment 17 Robert Wing freebsd_committer freebsd_triage 2023-07-28 16:18:31 UTC
(In reply to geoffroy desvernay from comment #16)

Yea, I would try with that patch applied - your panic appears to be identical to the one reported in this bug report.
Comment 18 Christos Chatzaras 2023-08-18 00:06:45 UTC
I hit the same bug in FreeBSD 13.2-RELEASE-p2 today.

Is it possible to have this patch as an ERRATA?

Also this patch (without it servers used for backups have slow disk operations):

https://github.com/openzfs/zfs/pull/14648/commits/5ee2c4b0e372e437cf11cd2ca572ae149c352853
Comment 19 Alan Somers freebsd_committer freebsd_triage 2023-12-01 16:55:39 UTC
Assigning to the committer and closing.  I verified that the fix is in stable/13 and stable/14 .  It's in releng/14.0 too.

As for Christos's request for an EN, I'll leave that up to rew.