I can reliably and instantly trigger this panic with fusefs using a test that mixes mmap writes, fspacectl, and normal reads. panic: Assertion prev_offset < next_offset failed at /usr/home/somers/src/freebsd.org/src/sys/vm/vnode_pager.c:1472 cpuid = 2 time = 1704675343 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe010e0145f0 vpanic() at vpanic+0x131/frame 0xfffffe010e014720 panic() at panic+0x43/frame 0xfffffe010e014780 vnode_pager_generic_putpages() at vnode_pager_generic_putpages+0xb0e/frame 0xfffffe010e014850 vnode_pager_putpages() at vnode_pager_putpages+0x80/frame 0xfffffe010e0148c0 vm_pageout_flush() at vm_pageout_flush+0xf3/frame 0xfffffe010e0149c0 vm_object_page_collect_flush() at vm_object_page_collect_flush+0x2ad/frame 0xfffffe010e014b30 vm_object_page_clean() at vm_object_page_clean+0x195/frame 0xfffffe010e014b90 vinactivef() at vinactivef+0x92/frame 0xfffffe010e014bd0 vput_final() at vput_final+0x2aa/frame 0xfffffe010e014c20 vn_close1() at vn_close1+0x12c/frame 0xfffffe010e014c90 vn_closefile() at vn_closefile+0x3f/frame 0xfffffe010e014d10 _fdrop() at _fdrop+0x1b/frame 0xfffffe010e014d30 closef() at closef+0x1e3/frame 0xfffffe010e014dc0 closefp_impl() at closefp_impl+0x76/frame 0xfffffe010e014e00 amd64_syscall() at amd64_syscall+0x153/frame 0xfffffe010e014f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010e014f30 --- syscall (6, FreeBSD ELF64, close), rip = 0x3c5477232d9a, rsp = 0x3c5470633328, rbp = 0x3c5470633340 --- KDB: enter: panic
Created attachment 247521 [details] Test case demonstrating the bug Steps to reproduce: 1) Apply the attached patch 2) cd tests/sys/fs/fusefs 3) make && sudo make install 4) cd /usr/tests/sys/fs/fusefs 5) kyua debug io:main Or, to run just the relevant test case, replace step 5 above with: 5a) sudo mkdir mountpoint 5b) sudo chmod 1777 mountpoint 5c) sudo touch backing_file control 5d) sudo chown $USER backing_file control 5e) ./io -v mountpoint --gtest_filter=Io/IoFspaceCtl.fspacectl_after_mmap_write/1
There is probably some mess with the v_object' page queue pages pindexes. Could you please, obtain the core dump for the panic, and show locals for the vnode_pager_generic_putpages() frame, to start the investigation?
(In reply to Konstantin Belousov from comment #2) Here's some more info: https://people.freebsd.org/~pho/stress/log/log0510.txt
https://reviews.freebsd.org/D43358
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=bdb46c21a3e68d4395d6e0b6a205187e655532b0 commit bdb46c21a3e68d4395d6e0b6a205187e655532b0 Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2024-01-08 13:21:06 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2024-01-11 09:49:37 +0000 vnode_pager_generic_putpages(): correctly handle clean block at EOF The loop 'skip clean blocks' checking for the clean blocks in the dirty pages might end up setting the in_hole to true when exactly at EOF at the middle of the block, without advancing the prev_offset value. Then the next block is not dirty, and next_offset is clipped back to poffset + maxsize, equal to prev_offset, failing the assertion. Instead of asserting prev_offset < next_offset, we must skip the write. Reported by: asomers PR: 276191 Reviewed by: alc, markj Tested by: asomers Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D43358 sys/vm/vnode_pager.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6b1c534927ad47e9b0dd8be1a39f56c3ea813d44 commit 6b1c534927ad47e9b0dd8be1a39f56c3ea813d44 Author: Alan Somers <asomers@FreeBSD.org> AuthorDate: 2024-01-08 01:03:40 +0000 Commit: Alan Somers <asomers@FreeBSD.org> CommitDate: 2024-01-13 22:47:16 +0000 Add a regression test for PR 276191. The bug isn't fusefs-specific, but this is the easiest way to reproduce it. PR: 276191 MFC after: 1 week MFC with: bdb46c21a3e68d4395d6e0b6a205187e655532b0 Differential Revision: https://reviews.freebsd.org/D43446 Reviewed by: kib tests/sys/fs/fusefs/io.cc | 10 ++++++++++ 1 file changed, 10 insertions(+)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8ecd7bfd6c7cd1b66ef62b8e8fc8ca99c4f26669 commit 8ecd7bfd6c7cd1b66ef62b8e8fc8ca99c4f26669 Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2024-01-08 13:21:06 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2024-01-18 00:51:32 +0000 vnode_pager_generic_putpages(): correctly handle clean block at EOF PR: 276191 (cherry picked from commit bdb46c21a3e68d4395d6e0b6a205187e655532b0) sys/vm/vnode_pager.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=bfa9a16569bab5240a6fdebcd407fd143d0d5ac4 commit bfa9a16569bab5240a6fdebcd407fd143d0d5ac4 Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2024-01-08 13:21:06 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2024-01-18 01:02:52 +0000 vnode_pager_generic_putpages(): correctly handle clean block at EOF PR: 276191 (cherry picked from commit bdb46c21a3e68d4395d6e0b6a205187e655532b0) sys/vm/vnode_pager.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d89317a66ff7874c9c04461739d7546e3042c0ad commit d89317a66ff7874c9c04461739d7546e3042c0ad Author: Alan Somers <asomers@FreeBSD.org> AuthorDate: 2024-01-08 01:03:40 +0000 Commit: Alan Somers <asomers@FreeBSD.org> CommitDate: 2024-01-19 20:27:09 +0000 Add a regression test for PR 276191. The bug isn't fusefs-specific, but this is the easiest way to reproduce it. PR: 276191 MFC with: bdb46c21a3e68d4395d6e0b6a205187e655532b0 Differential Revision: https://reviews.freebsd.org/D43446 Reviewed by: kib (cherry picked from commit 6b1c534927ad47e9b0dd8be1a39f56c3ea813d44) tests/sys/fs/fusefs/io.cc | 10 ++++++++++ 1 file changed, 10 insertions(+)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=62f890cfbc358160ef42f11de78ad2015a6885cb commit 62f890cfbc358160ef42f11de78ad2015a6885cb Author: Alan Somers <asomers@FreeBSD.org> AuthorDate: 2024-01-08 01:03:40 +0000 Commit: Alan Somers <asomers@FreeBSD.org> CommitDate: 2024-01-20 02:01:14 +0000 Add a regression test for PR 276191. The bug isn't fusefs-specific, but this is the easiest way to reproduce it. PR: 276191 Differential Revision: https://reviews.freebsd.org/D43446 Reviewed by: kib (cherry picked from commit 6b1c534927ad47e9b0dd8be1a39f56c3ea813d44) tests/sys/fs/fusefs/io.cc | 10 ++++++++++ 1 file changed, 10 insertions(+)