Summary ------- On powerpc64le, 16.0-CURRENT reliably panics in the VM page-queue code (_vm_page_pqstate_commit_dequeue) under heavy parallel build load. The page queue's TAILQ linkage is found corrupted (a node's next->prev no longer points back at the node), which INVARIANTS catches at the TAILQ_REMOVE in the speculative-dequeue path. Panic message ------------- panic: /usr/src/sys/vm/vm_page.c:3716: _vm_page_pqstate_commit_dequeue: Bad link elm 0xc0000079936c7cc0 next->prev != elm Environment ----------- FreeBSD 16.0-CURRENT #1 main-n287158-c9991e01149b: Wed Jul 1 10:39:39 2026 arch: powerpc64le (__FreeBSD_version 1600019) kernel: GENERIC64LE, options INVARIANTS + WITNESS + WITNESS_SKIPSPIN hardware: IBM POWER9, hw.ncpu = 96, 512 GB RAM platform: KVM guest (virtio: vtbd/vtnet) on a Talos II POWER9 host, UFS root Reproduction (on demand, 5/5) ----------------------------- Drive heavy, highly parallel VM page churn. Every occurrence here was during a poudriere build of lang/gcc14 with make parallelism enabled (ALLOW_MAKE_JOBS), which saturates all 96 cores -- 1-minute load average ~98 -- during the gcc bootstrap compile. The box panics partway through that compile every time; it never panics at idle or under light load. Five consecutive attempts each panicked at the same assertion (only the elm pointer differs). Analysis -------- vm_page.c:3716 is the TAILQ_REMOVE(&pq->pq_pl, m, plinks.q) inside _vm_page_pqstate_commit_dequeue(), in the speculative-removal path guarded by "(old->flags & PGA_ENQUEUED) != 0". The comment immediately above it is suggestive: /* * Once the queue index of the page changes there is nothing * synchronizing with further updates to the page's physical * queue state. Therefore we must speculatively remove the page * from the queue now and be prepared to roll back if the queue * state update fails. ... */ The corrupted back-link (next->prev != elm) indicates the pq_pl TAILQ was modified concurrently with this removal. That it reproduces only on powerpc64le under maximal SMP page churn, and not at low concurrency, is consistent with a synchronization/memory-ordering race in the page-queue commit path that the weaker POWER memory model exposes where amd64's TSO does not. This is offered as a lead, not a conclusion -- the exact call path is still needed. Debugging status ---------------- A clean minidump was captured (see below) but is not yet symbolized: devel/gdb 15.1's kgdb loads the kernel and kernel.debug fine but rejects the dump with "Failed to open vmcore: invalid corefile", so no backtrace is available. This looks like a gdb/ppc64le kernel-minidump limitation and may warrant its own PR. Dump header (info): Architecture powerpc64le, Dump Length 1040429056, Compression none, Dump Status good. vmcore.4 (~1 GB) + the matching /boot/kernel/kernel and kernel.debug are preserved and can be provided. The reproducer is fast and deterministic, so additional instrumentation (e.g. QUEUE_MACRO_DEBUG_TRASH, extra asserts, or a targeted patch) can be turned around quickly.
Could you please try to reproduce the crash with this patch? https://reviews.freebsd.org/D58226
That fix seems to work.
(In reply to Piotr Kubaj from comment #2) Thanks, but now I think the patch is wrong (or at least the justification isn't right). Can you please share the backtraces from the crashes?
Please try this patch instead: https://reviews.freebsd.org/D58261
(In reply to Mark Johnston from comment #4) With that patch, I also can't reproduce the issue anymore :)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=d809a10218884162ed47c658233746c53a98b1aa commit d809a10218884162ed47c658233746c53a98b1aa Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2026-07-17 12:57:06 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2026-07-17 13:18:15 +0000 vm_page: Fix dequeue on arches with weak ordering A vm_page's a.queue field records the page queue index for the page queue to which the page belongs. The PGA_ENQUEUED flag indicates whether the page is actually enqueued in that queue's TAILQ. When modifying the a.queue field, you need to hold the page queue lock for the queue corresponding to the old value, unless the old value is PQ_NONE. Suppose a managed page is freed. vm_page_free_prep() calls vm_page_dequeue_deferred(), which checks whether the page belongs to a queue; if so it schedules an asynchronous dequeue operation so that page queue lock acquisitions can be batched if possible. The dequeue operation must be completed before the page's plinks.q fields are reused. So, during page allocation, we call vm_page_dequeue() to finish the dequeue operation. Similarly, since the buddy allocator uses the plinks.q fields for its own internal linkage, vm_freelist_add() calls vm_page_dequeue(). _vm_page_pqstate_commit_dequeue() is the function which actually removes the page from its queue. It sets a.queue = PG_NONE and removes the page from its queue. However, the update to the page's atomic state is relaxed, so on systems with store reordering, it may race with a concurrent enqueue of the page into the buddy queues (probably more likely) or a page queue. Fix this: use a release store to update the page's queue state in _vm_page_pqstate_commit_dequeue(), and make sure that vm_page_dequeue() uses an acquire load when comparing m->a.queue == PQ_NONE. PR: 296767 Reported and tested by: pkubaj Reviewed by: alc, kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D58261 sys/vm/vm_page.c | 26 ++++++++++++++++++++++++-- sys/vm/vm_page.h | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 2 deletions(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3701716fdc45c203951f44d01710da310204b629 commit 3701716fdc45c203951f44d01710da310204b629 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2026-07-17 12:57:06 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2026-07-27 17:34:32 +0000 vm_page: Fix dequeue on arches with weak ordering A vm_page's a.queue field records the page queue index for the page queue to which the page belongs. The PGA_ENQUEUED flag indicates whether the page is actually enqueued in that queue's TAILQ. When modifying the a.queue field, you need to hold the page queue lock for the queue corresponding to the old value, unless the old value is PQ_NONE. Suppose a managed page is freed. vm_page_free_prep() calls vm_page_dequeue_deferred(), which checks whether the page belongs to a queue; if so it schedules an asynchronous dequeue operation so that page queue lock acquisitions can be batched if possible. The dequeue operation must be completed before the page's plinks.q fields are reused. So, during page allocation, we call vm_page_dequeue() to finish the dequeue operation. Similarly, since the buddy allocator uses the plinks.q fields for its own internal linkage, vm_freelist_add() calls vm_page_dequeue(). _vm_page_pqstate_commit_dequeue() is the function which actually removes the page from its queue. It sets a.queue = PG_NONE and removes the page from its queue. However, the update to the page's atomic state is relaxed, so on systems with store reordering, it may race with a concurrent enqueue of the page into the buddy queues (probably more likely) or a page queue. Fix this: use a release store to update the page's queue state in _vm_page_pqstate_commit_dequeue(), and make sure that vm_page_dequeue() uses an acquire load when comparing m->a.queue == PQ_NONE. PR: 296767 Reported and tested by: pkubaj Reviewed by: alc, kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D58261 (cherry picked from commit d809a10218884162ed47c658233746c53a98b1aa) sys/vm/vm_page.c | 26 ++++++++++++++++++++++++-- sys/vm/vm_page.h | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 2 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e782be2e48012a95b8e9bfdc247c2e8207a825e8 commit e782be2e48012a95b8e9bfdc247c2e8207a825e8 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2026-07-17 12:57:06 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2026-07-27 19:35:19 +0000 vm_page: Fix dequeue on arches with weak ordering A vm_page's a.queue field records the page queue index for the page queue to which the page belongs. The PGA_ENQUEUED flag indicates whether the page is actually enqueued in that queue's TAILQ. When modifying the a.queue field, you need to hold the page queue lock for the queue corresponding to the old value, unless the old value is PQ_NONE. Suppose a managed page is freed. vm_page_free_prep() calls vm_page_dequeue_deferred(), which checks whether the page belongs to a queue; if so it schedules an asynchronous dequeue operation so that page queue lock acquisitions can be batched if possible. The dequeue operation must be completed before the page's plinks.q fields are reused. So, during page allocation, we call vm_page_dequeue() to finish the dequeue operation. Similarly, since the buddy allocator uses the plinks.q fields for its own internal linkage, vm_freelist_add() calls vm_page_dequeue(). _vm_page_pqstate_commit_dequeue() is the function which actually removes the page from its queue. It sets a.queue = PG_NONE and removes the page from its queue. However, the update to the page's atomic state is relaxed, so on systems with store reordering, it may race with a concurrent enqueue of the page into the buddy queues (probably more likely) or a page queue. Fix this: use a release store to update the page's queue state in _vm_page_pqstate_commit_dequeue(), and make sure that vm_page_dequeue() uses an acquire load when comparing m->a.queue == PQ_NONE. PR: 296767 Reported and tested by: pkubaj Reviewed by: alc, kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D58261 (cherry picked from commit d809a10218884162ed47c658233746c53a98b1aa) sys/vm/vm_page.c | 26 ++++++++++++++++++++++++-- sys/vm/vm_page.h | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 2 deletions(-)