While poudriere is cleaning up my server crashes with vm_page_dequeue_deferred: page 0xffffa09ec0609d40 has unexpected queue state Stack trace retyped from the console: vm_page_dequeue_deferred vm_page_free_prep vm_page_free_toq vm_object_page_remove bufobj_invalbuf vgonel vrecycle zfs_freebsd_inactive vinactivef vput_final kern_funlinkat I reproduced this and I have two crash dumps but kgdb hits an internal assertion trying to debug them. My guess is this is distantly related to bug 261538 in that the combination of zfs deduplication and cloning creates unusual state in the kernel. Some notes: 1. ARM server (Ampere eMAG) running 13.1-STABLE up through eb2a9b78586. 2. Kernel is built with INVARIANTS (this is a KASSERT in vm_page.c). 3. I have a patch to ZFS to disable another assertion to fix bug 261538. 4. All of poudriere's temporary filesystems have been destroyed by the time of the crash.
I have two kernels, one that crashes and one that does not. The only difference is my attempt to make the message from KASSERT more informative: diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 81b2f518aff..f8c8cc67bd6 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -3713,8 +3713,8 @@ vm_page_dequeue_deferred(vm_page_t m) do { if (old.queue == PQ_NONE) { KASSERT((old.flags & PGA_QUEUE_STATE_MASK) == 0, - ("%s: page %p has unexpected queue state", - __func__, m)); + ("%s: page %p has unexpected queue state (bits = %08x)", + __func__, m, old._bits)); break; } new = old; The kernel with the diff above works. The only difference in the generated code for vm_page_dequeue_deferred is a small difference in register numbering to so old._bits lives in w3 where it will be passed as the fourth argument to panic. All the instructions are the same except for register numbers and panic format string addresses. The number of instructions in the kernel as a whole is the same.
In vm_page_dequeue_deferred, old._bits = 0x00ff0010, i.e. flags=PGA_DEQUEUE, act_count=0, queue=PQ_NONE. The assertion is that queue=PQ_NONE implies certain bits including PGA_DEQUEUE are clear. If the VM system makes two pages backed by different logical blocks of ZFS filesystem(s) that are in turn backed by the same physical block due to deduplication, could this happen? The assertion is in a loop that tries to set PGA_DEQUEUE until vm_page_pqstate_commit_request returns true. If that function returns a spurious false or another CPU is racing to dequeue the same page, the same crash would happen.
(In reply to John F. Carr from comment #0) Sorry that this didn't get looked at. Is it still reproducible? (In reply to John F. Carr from comment #2) Calls to vm_page_dequeue_deferred() for a given page should be serialized by virtue of the fact that this function is only called while freeing a page to the allocator. I'm having a hard time seeing how this panic can happen absent a bug elsewhere (e.g., a double-free of a page). Any code which sets PGA_DEQUEUE either 1) atomically checks that the queue index is not equal to PQ_NONE, or 2) holds a reference to the page which prevents it from being freed. See vm_page_unwire_managed() for an example of the latter. But, we have at least one other report of a similar panic (https://lists.freebsd.org/archives/freebsd-current/2023-August/004475.html), albeit a year later.
The crashes stopped long ago. Recreating the poudriere jails may have eliminated the troublesome state.
> Hardware: arm64 Any > Importance: --- Affects Only Me Should we broaden those two, and raise the version to 15.0-CURRENT? (In reply to Mark Johnston from comment #3) > … at least one other report of a similar panic > (https://lists.freebsd.org/archives/freebsd-current/2023-August/004475.html), … From yesterday's <https://lists.freebsd.org/archives/freebsd-current/2023-September/004663.html> (again, cy@ running poudriere building i386 packages): > panic: vm_page_dequeue_deferred: page 0xfffffe000b7e9748 has unexpected queue state
Affects amd64 machine with poudriere building i386 packages. The panic occurs after poudriere has finished building packages during cleanup while poudriere is unmounting (zfs umount) filesystems. Machine is running 15-CURRENT. It does not affect (yet) a similarly configured machine building amd64 packages using poudriere.