Bug 294039 - databases/postgresql14-server: jemalloc crashes triggered by autovacuum
Summary: databases/postgresql14-server: jemalloc crashes triggered by autovacuum
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 14.3-RELEASE
Hardware: Any Any
: --- Affects Some People
Assignee: Mark Johnston
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2026-03-25 11:11 UTC by martin
Modified: 2026-04-21 19:59 UTC (History)
10 users (show)

See Also:
bugzilla: maintainer-feedback? (pgsql)
markj: mfc-stable15+
zlei: mfc-stable14+
zlei: mfc-stable13+
markj: needs_errata+


Attachments
proposed patch (1.25 KB, patch)
2026-04-06 04:22 UTC, Mark Johnston
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description martin 2026-03-25 11:11:51 UTC
After updating from FreeBSD 14.3-RELEASE-p7 to 14.3-RELEASE-p8, autovacuum has been causing crashes in the postgres processes about once a week during heavy usage.  The postgresql14-server versions were 14.20_1 and 14.21.

This has happened on two different machines, which run completely different database applications (the only common thing is that they do lots of inserts).  The applications haven't changed and have been running 100% reliably every day since 14.3-RELEASE-p5 in Nov 2025 (and prior to that with FreeBSD 13.x with various versions of postgresql14-server and for years with previous versions).  I've not seen any other problems on these machines.  Machine 1 is physical hardware with root on zfs.  Machine 2 is a VM in the cloud with root on ufs and no zfs.

I've not seen any crashes since setting autovacuum = off at the beginning on March.

It started crashing on machine 1 after these upgrades:
freebsd-update:
        FreeBSD 14.3-RELEASE-p7 to 14.3-RELEASE-p8
pkg upgrade:
	ImageMagick7: 7.1.2.11 -> 7.1.2.11_1 [FreeBSD]
	colord: 1.4.8 -> 1.4.8_1 [FreeBSD]
	emacs: 30.2,3 -> 30.2_1,3 [FreeBSD]
	expat: 2.7.3 -> 2.7.4 [FreeBSD]
	ffmpeg: 8.0.1_3,1 -> 8.0.1_4,1 [FreeBSD]
	ghostscript10: 10.06.0_1 -> 10.06.0_2 [FreeBSD]
	git: 2.52.0 -> 2.53.0 [FreeBSD]
	libjxl: 0.11.1_2 -> 0.11.1_3 [FreeBSD]
	libplacebo: 7.351.0_1 -> 7.351.0_2 [FreeBSD]
	libraw: 0.21.5 -> 0.21.5_1 [FreeBSD]
	lmdb: 0.9.33,1 -> 0.9.35,1 [FreeBSD]
	openjpeg: 2.5.4_1 -> 2.5.4_2 [FreeBSD]
	p11-kit: 0.26.1 -> 0.26.2 [FreeBSD]
	pango: 1.56.4 -> 1.56.4_1 [FreeBSD]
	py311-cryptography: 46.0.3_1,1 -> 46.0.4,1 [FreeBSD]
	py311-packaging: 25.0 -> 26.0 [FreeBSD]
	python311: 3.11.14_1 -> 3.11.14_2 [FreeBSD]
	shaderc: 2025.5_1 -> 2026.1 [FreeBSD]

It started crashing on machine 2 after these upgrades:
freebsd-update:
        FreeBSD 14.3-RELEASE-p7 to 14.3-RELEASE-p8
pkg upgrade:
	pkg: 2.4.2 -> 2.5.1 [FreeBSD]
	apr: 1.7.5.1.6.3_5 -> 1.7.6.1.6.3 [FreeBSD]
	bind-tools: 9.20.17 -> 9.20.18_2 [FreeBSD]
	boost-libs: 1.88.0_2 -> 1.89.0_2 [FreeBSD]
	brotli: 1.1.0,1 -> 1.2.0,1 [FreeBSD]
	ca_root_nss: 3.115_4 -> 3.117_2 [FreeBSD]
	curl: 8.16.0 -> 8.17.0 [FreeBSD]
	gdb: 15.1_3 -> 15.1_4 [FreeBSD]
	gettext-runtime: 0.23.1 -> 0.26 [FreeBSD]
	libedit: 3.1.20250104,1 -> 3.1.20251016_1,1 [FreeBSD]
	libiconv: 1.17_1 -> 1.18_1 [FreeBSD]
	liblz4: 1.10.0,1 -> 1.10.0_2,1 [FreeBSD]
	libnghttp2: 1.67.0 -> 1.68.0 [FreeBSD]
	libsodium: 1.0.19 -> 1.0.21 [FreeBSD]
	libxml2: 2.14.5 -> 2.15.1 [FreeBSD]
	p5-CGI: 4.70 -> 4.71 [FreeBSD]
	p5-HTTP-Message: 7.00 -> 7.01 [FreeBSD]
	p5-URI: 5.32_1 -> 5.34 [FreeBSD]
	pcre2: 10.46 -> 10.47_1 [FreeBSD]
	postgresql14-client: 14.20_1 -> 14.20_2 [FreeBSD]
	postgresql14-server: 14.20 -> 14.20_1 [FreeBSD]
	proftpd: 1.3.8c_1 -> 1.3.8c_2 [FreeBSD]
	python311: 3.11.14 -> 3.11.14_1 [FreeBSD]
	readline: 8.2.13_2 -> 8.3.3 [FreeBSD]
	rsync: 3.4.1_4 -> 3.4.1_5 [FreeBSD]
	source-highlight: 3.1.9_13 -> 3.1.9_14 [FreeBSD]
	zstd: 1.5.7 -> 1.5.7_1 [FreeBSD]

There are three backtraces below.

This is a backtrace in an autovacuum worker process on machine 1 with postgresql14-server-14.20_1:

Program terminated with signal SIGBUS, Bus error.
#0  0x0000000826fbfa93 in extent_try_coalesce_impl (tsdn=tsdn@entry=0x22513e089090, arena=0x22513e200980, 
    r_extent_hooks=0x8215e0578, rtree_ctx=rtree_ctx@entry=0x22513e0890c0, extents=extents@entry=0x22513e2058d8, 
    extent=0x22513e30eb80, coalesced=0x0, growing_retained=<optimized out>, inactive_only=<optimized out>) at jemalloc_extent.c:1620
#1  0x0000000826fbc9e8 in extent_try_coalesce (tsdn=0x22513e089090, arena=0x22513e200980, r_extent_hooks=0x8215e0578, 
    rtree_ctx=0x22513e0890c0, extents=0x22513e2058d8, extent=0x22513e30eb80, coalesced=0x0, growing_retained=<optimized out>)
    at jemalloc_extent.c:1678
#2  extent_record (tsdn=tsdn@entry=0x22513e089090, arena=arena@entry=0x22513e200980, 
    r_extent_hooks=r_extent_hooks@entry=0x8215e0578, extents=extents@entry=0x22513e2058d8, extent=0x22513e30eb80, 
    growing_retained=<optimized out>) at jemalloc_extent.c:1717
#3  0x0000000826fbd855 in extent_grow_retained (tsdn=0x22513e089090, arena=0x22513e200980, r_extent_hooks=0x8215e0578, 
    size=<optimized out>, pad=4096, alignment=<optimized out>, slab=<optimized out>, szind=<optimized out>, zero=<optimized out>, 
    commit=<optimized out>) at jemalloc_extent.c:1383
#4  extent_alloc_retained (tsdn=0x22513e089090, arena=0x22513e200980, r_extent_hooks=0x8215e0578, size=262144, pad=4096, 
    alignment=<optimized out>, slab=<optimized out>, commit=<optimized out>, new_addr=<optimized out>, szind=<optimized out>, 
    zero=<optimized out>) at jemalloc_extent.c:1480
#5  __je_extent_alloc_wrapper (tsdn=0x22513e089090, arena=arena@entry=0x22513e200980, 
    r_extent_hooks=r_extent_hooks@entry=0x8215e0578, new_addr=new_addr@entry=0x0, size=size@entry=262144, pad=pad@entry=4096, 
    alignment=64, slab=<optimized out>, szind=52, zero=0x8215e05cf, commit=0x8215e0577) at jemalloc_extent.c:1539
#6  0x0000000826f9a866 in __je_arena_extent_alloc_large (tsdn=0xc4ceb9fe1a85ec53, tsdn@entry=0x22513e089090, 
    arena=arena@entry=0x22513e200980, usize=262144, alignment=<optimized out>, zero=zero@entry=0x8215e05cf) at jemalloc_arena.c:448
#7  0x0000000826fc2545 in __je_large_palloc (tsdn=0x22513e089090, arena=<optimized out>, usize=<optimized out>, 
    alignment=alignment@entry=64, zero=<optimized out>) at jemalloc_large.c:47
#8  0x0000000826fc23a2 in __je_large_malloc (tsdn=0xc4ceb9fe1a85ec53, arena=0xff51afd7ed558ccd, usize=1937131034, zero=false)
    at jemalloc_large.c:17
#9  0x0000000826f9c405 in __je_arena_malloc_hard (tsdn=0xc4ceb9fe1a85ec53, tsdn@entry=0x22513e089090, arena=0xff51afd7ed558ccd, 
    arena@entry=0x0, size=<optimized out>, ind=2181051958, zero=false) at jemalloc_arena.c:1528
#10 0x0000000826f8f6e5 in arena_malloc (tsdn=0x22513e089090, arena=0x0, ind=2181051958, zero=false, tcache=0x22513e089280, 
    slow_path=false, size=<optimized out>) at /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:176
#11 iallocztm (tsdn=0x22513e089090, ind=2181051958, zero=false, tcache=0x22513e089280, is_internal=false, arena=0x0, 
    slow_path=false, size=<optimized out>) at /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_c.h:53
#12 imalloc_no_sample (tsd=0x22513e089090, usize=262144, ind=2181051958, sopts=<optimized out>, dopts=<optimized out>, 
    size=<optimized out>) at jemalloc_jemalloc.c:1953
#13 imalloc_body (tsd=0x22513e089090, sopts=<optimized out>, dopts=<optimized out>) at jemalloc_jemalloc.c:2153
#14 imalloc (sopts=<optimized out>, dopts=<optimized out>) at jemalloc_jemalloc.c:2262
#15 __je_malloc_default (size=<optimized out>) at jemalloc_jemalloc.c:2293
#16 0x000000000097d1e3 in ?? ()
#17 0x0000000000984d5b in palloc ()
#18 0x000000000094926f in ?? ()
#19 0x0000000000949d3e in RelationCacheInitializePhase3 ()
#20 0x0000000000965ac1 in InitPostgres ()
#21 0x000000000077c20e in ?? ()
#22 0x000000000077bee7 in StartAutoVacWorker ()
#23 0x000000000078bc31 in ?? ()
#24 0x0000000822bc34a0 in handle_signal (actp=actp@entry=0x8215e1190, sig=sig@entry=30, info=info@entry=0x8215e1580, 
    ucp=ucp@entry=0x8215e1210) at /usr/src/lib/libthr/thread/thr_sig.c:299
#25 0x0000000822bc2a6b in thr_sighandler (sig=30, info=0x8215e1580, _ucp=0x8215e1210) at /usr/src/lib/libthr/thread/thr_sig.c:242
#26 <signal handler called>
#27 _select () at _select.S:4
#28 0x0000000822bc5824 in __thr_select (numfds=9, readfds=0x8215e16d0, writefds=0x0, exceptfds=0x0, timeout=0x8215e17d0)
    at /usr/src/lib/libthr/thread/thr_syscalls.c:492
#29 0x000000000078d6bb in ?? ()
#30 0x000000000078af14 in PostmasterMain ()
#31 0x00000000006e64b1 in main ()

Just after this, the application process lost its database connection ("server closed the connection unexpectedly").

This is another backtrace in an autovacuum worker process on machine 1, this time with postgresql14-server-14.21:

Program terminated with signal SIGBUS, Bus error.
#0  0x000000082a8b5c04 in _swrite (fp=fp@entry=0x82a984120, 
    buf=buf@entry=0x6e65206665642070 <error: Cannot access memory at address 0x6e65206665642070>, n=n@entry=134947567)
    at /usr/src/lib/libc/stdio/stdio.c:116
#1  0x000000082a8adbfe in __sflush (fp=fp@entry=0x82a984120) at /usr/src/lib/libc/stdio/fflush.c:125
#2  0x000000082a8b133e in _fwalk (function=0x82a8adba0 <__sflush>) at /usr/src/lib/libc/stdio/fwalk.c:62
#3  0x000000082a8fe47a in exit (status=0) at /usr/src/lib/libc/stdlib/exit.c:94
#4  0x00000000007f454a in proc_exit ()
#5  0x000000000077dd3f in ?? ()
#6  0x000000000077d997 in StartAutoVacWorker ()
#7  0x000000000078d6e1 in ?? ()
#8  0x00000008223f74a0 in handle_signal (actp=actp@entry=0x8212108d0, sig=sig@entry=30, info=info@entry=0x821210cc0, 
    ucp=ucp@entry=0x821210950) at /usr/src/lib/libthr/thread/thr_sig.c:299
#9  0x00000008223f6a6b in thr_sighandler (sig=30, info=0x821210cc0, _ucp=0x821210950) at /usr/src/lib/libthr/thread/thr_sig.c:242
#10 <signal handler called>
#11 _select () at _select.S:4
#12 0x00000008223f9824 in __thr_select (numfds=9, readfds=0x821210e10, writefds=0x0, exceptfds=0x0, timeout=0x821210f10)
    at /usr/src/lib/libthr/thread/thr_syscalls.c:492
#13 0x000000000078f16b in ?? ()
#14 0x000000000078c9c4 in PostmasterMain ()
#15 0x00000000006e7f51 in main ()

As before, just after this, the application process lost its database connection ("server closed the connection unexpectedly").

This is another backtrace, this time in the application's postgres process on machine 2 with postgresql14-server-14.20_1:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000082b9f4a93 in extent_try_coalesce_impl (tsdn=tsdn@entry=0x184fa2a89090, arena=0x184fa2c00980, r_extent_hooks=0x821619bf8, rtree_ctx=rtree_ctx@entry=0x184fa2a890c0, extents=extents@entry=0x184fa2c058d8, extent=0x184fa2d10200, coalesced=0x0, growing_retained=<optimized out>, inactive_only=<optimized out>) at jemalloc_extent.c:1620
#1  0x000000082b9f19e8 in extent_try_coalesce (tsdn=0x184fa2a89090, arena=0x184fa2c00980, r_extent_hooks=0x821619bf8, rtree_ctx=0x184fa2a890c0, extents=0x184fa2c058d8, extent=0x184fa2d10200, coalesced=0x0, growing_retained=<optimized out>) at jemalloc_extent.c:1678
#2  extent_record (tsdn=tsdn@entry=0x184fa2a89090, arena=arena@entry=0x184fa2c00980, r_extent_hooks=r_extent_hooks@entry=0x821619bf8, extents=extents@entry=0x184fa2c058d8, extent=0x184fa2d10200, growing_retained=<optimized out>) at jemalloc_extent.c:1717
#3  0x000000082b9f2855 in extent_grow_retained (tsdn=0x184fa2a89090, arena=0x184fa2c00980, r_extent_hooks=0x821619bf8, size=<optimized out>, pad=0x1000, alignment=<optimized out>, slab=<optimized out>, szind=<optimized out>, zero=<optimized out>, commit=<optimized out>) at jemalloc_extent.c:1383
#4  extent_alloc_retained (tsdn=0x184fa2a89090, arena=0x184fa2c00980, r_extent_hooks=0x821619bf8, size=0x200000, pad=0x1000, alignment=<optimized out>, slab=<optimized out>, commit=<optimized out>, new_addr=<optimized out>, szind=<optimized out>, zero=<optimized out>) at jemalloc_extent.c:1480
#5  __je_extent_alloc_wrapper (tsdn=0x184fa2a89090, arena=arena@entry=0x184fa2c00980, r_extent_hooks=r_extent_hooks@entry=0x821619bf8, new_addr=new_addr@entry=0x0, size=size@entry=0x200000, pad=pad@entry=0x1000, alignment=0x40, slab=<optimized out>, szind=0x40, zero=0x821619c4f, commit=0x821619bf7) at jemalloc_extent.c:1539
#6  0x000000082b9cf866 in __je_arena_extent_alloc_large (tsdn=0xc4ceb9fe1a85ec53, tsdn@entry=0x184fa2a89090, arena=arena@entry=0x184fa2c00980, usize=0x200000, alignment=<optimized out>, zero=zero@entry=0x821619c4f) at jemalloc_arena.c:448
#7  0x000000082b9f7545 in __je_large_palloc (tsdn=0x184fa2a89090, arena=<optimized out>, usize=<optimized out>, alignment=alignment@entry=0x40, zero=<optimized out>) at jemalloc_large.c:47
#8  0x000000082b9f73a2 in __je_large_malloc (tsdn=0xc4ceb9fe1a85ec53, arena=0xff51afd7ed558ccd, usize=0xdd9d807a, zero=0x0) at jemalloc_large.c:17
#9  0x000000082b9d1405 in __je_arena_malloc_hard (tsdn=0xc4ceb9fe1a85ec53, tsdn@entry=0x184fa2a89090, arena=0xff51afd7ed558ccd, arena@entry=0x0, size=<optimized out>, ind=0x1df040b8, zero=0x0) at jemalloc_arena.c:1528
#10 0x000000082b9c46e5 in arena_malloc (tsdn=0x184fa2a89090, arena=0x0, ind=0x1df040b8, zero=0x0, tcache=0x184fa2a89280, slow_path=0x0, size=<optimized out>) at /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:176
#11 iallocztm (tsdn=0x184fa2a89090, ind=0x1df040b8, zero=0x0, tcache=0x184fa2a89280, is_internal=0x0, arena=0x0, slow_path=0x0, size=<optimized out>) at /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_inlines_c.h:53
#12 imalloc_no_sample (tsd=0x184fa2a89090, usize=0x200000, ind=0x1df040b8, sopts=<optimized out>, dopts=<optimized out>, size=<optimized out>) at jemalloc_jemalloc.c:1953
#13 imalloc_body (tsd=0x184fa2a89090, sopts=<optimized out>, dopts=<optimized out>) at jemalloc_jemalloc.c:2153
#14 imalloc (sopts=<optimized out>, dopts=<optimized out>) at jemalloc_jemalloc.c:2262
#15 __je_malloc_default (size=<optimized out>) at jemalloc_jemalloc.c:2293
#16 0x000000000097d1e3 in ?? ()
#17 0x0000000000984e1b in palloc0 ()
#18 0x00000000004ec0f0 in heap_form_minimal_tuple ()
#19 0x000000000098cc3e in ?? ()
#20 0x000000000098fc4a in tuplesort_puttupleslot ()
#21 0x00000000006c2b45 in ?? ()
#22 0x00000000006b851e in ?? ()
#23 0x00000000006b7d10 in ?? ()
#24 0x000000000069216b in standard_ExecutorRun ()
#25 0x000000000081e203 in ?? ()
#26 0x000000000081de25 in PortalRun ()
#27 0x000000000081cfd9 in ?? ()
#28 0x000000000081aa98 in PostgresMain ()
#29 0x000000000078e392 in ?? ()
#30 0x000000000078dbdc in ?? ()
#31 0x000000000078af14 in PostmasterMain ()
#32 0x00000000006e64b1 in main ()
Comment 1 Palle Girgensohn freebsd_committer freebsd_triage 2026-03-26 13:41:59 UTC
Hi!

This does not really sound like an application issue since the update is FreeBSD-14.3p7 -> 14.3p8.  Perhaps this is somehow involved: https://www.freebsd.org/security/advisories/FreeBSD-EN-26:03.vm.asc


BTW, did you really have postgresql-server-14.20 ? From what I can see, that package never existed. 14.20 mistakenly had PORTREVISION=1 from the start. Do you build your own packages?

This seems like an issue with jemalloc. Could libc be compromised?

Try this in rc.conf

postgresql_env="MALLOC_CONF=junk:false"

and see if the crash character changes.

Can you rollback to p7 to verify that it is related specifically to the OS patch?

I'm tagging this as kern, since I suspect this is not primarily a postgresql problem.

Palle
Comment 3 Peter Much 2026-03-30 04:41:56 UTC
It is not related to autovacuum. After I followed recommendation from postgres group and upgraded PG r15 to r16, I did somewhere miss that MIT-krb5 distinguishes FQDN and shortname, and some of my applications did hammer the DB with futile logins. Then I got backtraces straight from gethostbyaddr() into malloc():

https://forums.freebsd.org/threads/trouble-bus-errors-and-segfaults-in-libc-so-from-postgres.101876/post-750679

I cannot perfectly align that security update. It was built and deployed on Jan 31, but the server was rebooted into -p8 on Feb 2 16:05, while the first crash happened on Feb 2 03:52. It is unclear when exactly the binaries on that respective jail have been replaced.
Comment 4 Peter Much 2026-03-30 06:14:53 UTC
(In reply to Peter Much from comment #3)
Please disregard previous second paragraph, I noticed that the mentioned errata concerns the kernel, then more thorough search got me an earlier -p8 boot at Jan 31 14:17:09
Comment 5 Peter Much 2026-03-30 18:03:11 UTC
What I did today: I patched that so-called "errata" OUT of my 14.4 and rebuilt/rebooted the kernel only. 
Then I pushed some data onto the server to fill the memory, as in my impression this might have something to do with swapspace occupancy.
Then I changed my apps back to hammer the DB with unsuccessful logins. A week ago this behaviour brought me a crash about once a day.

We will see. 

Also last night, while still not knowing about this PR here, I walked the way all thru that jemalloc to see where the bogus does actually originate. This may be interesting or it may not, anyway I posted all the data to the forum link given above. For now it seems that my russian anti-imperialist friends have taken over that forum, so we might get this data back, or we might not.

Then I tried to read that so-called "errata", which I do not understand, bur which does very certainly not read like an errata (where a clearly-identified-as-wrong functionality is replaced by a clearly-identified-as-correct functionality), but rather like a programming experiment. :(
Comment 6 Peter Much 2026-04-03 10:27:58 UTC
I have now run the system for three days without further crash, after reverting the FreeBSD-EN-26:03.vm efix.

Then I have re-activated FreeBSD-EN-26:03.vm, and just got another jemalloc crash.

Since it looks to me like there are some un-zeroed pages appearing which probably should be zeroed, I also have added two printf to the fix, in order to get some insight into the difference in behaviour:

--- a/sys/vm/vm_fault.c
+++ b/sys/vm/vm_fault.c
@@ -1207,6 +1207,12 @@ vm_fault_zerofill(struct faultstate *fs)
        /*
         * Zero the page if necessary and mark it valid.
         */
+        if (((fs->m->flags & PG_ZERO) == 0) && !(fs->m_needs_zeroing)) {
+          printf("PMc: KERNEL ALERT page should be zeroed, but will not\n");
+        }
+        if (((fs->m->flags & PG_ZERO) != 0) && (fs->m_needs_zeroing)) {
+          printf("PMc: KERNEL INFO page would not be zeroed, but will\n");
+        }
        if (fs->m_needs_zeroing) {
                pmap_zero_page(fs->m);
        } else {

And the output looks like this:

Apr  3 02:40:34 <kern.crit> edge kernel: [7678] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 02:40:34 <kern.crit> edge kernel: [7678] PMc: KERNEL INFO page would not be zeroed, but will
Apr  3 02:40:34 <kern.crit> edge kernel: [7678] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 03:29:55 <kern.crit> edge kernel: [10639] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 04:49:52 <kern.crit> edge kernel: [15437] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 06:18:01 <kern.crit> edge kernel: [20725] PMc: KERNEL INFO page would not be zeroed, but will
Apr  3 06:58:27 <kern.crit> edge kernel: [23151] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 07:01:09 <kern.crit> edge kernel: [23313] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 07:03:18 <kern.crit> edge kernel: [23442] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 07:27:09 <kern.crit> edge kernel: [24874] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 07:27:09 <kern.crit> edge kernel: [24874] PMc: KERNEL INFO page would not be zeroed, but will
Apr  3 11:09:41 <kern.crit> edge kernel: [38225] PMc: KERNEL INFO page would not be zeroed, but will
Apr  3 11:09:41 <kern.crit> edge kernel: [38225] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 11:09:41 <kern.crit> edge kernel: [38225] PMc: KERNEL INFO page would not be zeroed, but will
Apr  3 11:09:41 <kern.crit> edge kernel: [38225] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 11:09:41 <kern.crit> edge kernel: [38225] PMc: KERNEL INFO page would not be zeroed, but will
Apr  3 11:09:41 <kern.crit> edge kernel: [38225] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  3 11:09:41 <kern.info> edge kernel: [38225] pid 3380 (postgres), jid 5, uid 770: exited on signal 11 (core dumped)


I am now adding the group that approved D53891 to this PR.
Comment 7 Konstantin Belousov freebsd_committer freebsd_triage 2026-04-03 11:14:41 UTC
(In reply to Peter Much from comment #6)
If you are so sure that this change causes problems, you should run the HEAD kernel
with INVARIANTS enabled and sysctl debug.vm_check_pg_zero set to true.  It will
catch the situation where the fresh anonymous page is non-zero.

If the checker detects the problem, it panics and we should get the core dump to
analyze.

The change you blame makes the new mapped page to be zeroed in more cases than
without the change, but the situation where this happen is quite rare.
Comment 8 Peter Much 2026-04-03 12:08:32 UTC
(In reply to Konstantin Belousov from comment #7)

What I am sure of is that my databases suddenly started crashing repeatedly since applying FreeBSD-EN-26:03.vm (and only thanks to Palle Girgenson here I was able to figure that it might be a software issue, because people told me it would be a hardware problem).

Furthermore, this is happening on my infra backend server with hundreds of different applications, jails, tunnels, guests, databases, web applications, backup services etcetc. - and it is practically impossible to migrate that whole thing to run HEAD.
There must be a different approach.
Comment 9 Mark Johnston freebsd_committer freebsd_triage 2026-04-03 14:30:22 UTC
I see a case where we can restart the fault handler without resetting m_needs_zeroing: if vm_fault_object() encounters a busy page in a backing object, it will call vm_fault_busy_sleep(), which resets fault state using a custom routine.  If a page allocation had previously set m_needs_zeroing = false, and the newly allocated page was reclaimed while restarting the fault handler, then we'll allocate a new page using the stale m_needs_zeroing value.

A minimal patch to fix this is:

diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c
index 64580c34c0a0..cf20304706ad 100644
--- a/sys/vm/vm_fault.c
+++ b/sys/vm/vm_fault.c
@@ -1436,6 +1436,7 @@ vm_fault_busy_sleep(struct faultstate *fs)
                VM_OBJECT_UNLOCK(fs->object);
        VM_CNT_INC(v_intrans);
        vm_object_deallocate(fs->first_object);
+       fs->m_needs_zeroing = true;
 }
 
Peter, I wonder if you could test this with the original patch re-applied?
Comment 10 Peter Much 2026-04-04 15:15:46 UTC
(In reply to Mark Johnston from comment #9)
Bad news, Mark. It took almost a day to crash, but it's all the same now:

Apr  4 09:28:22 <kern.crit> edge kernel: [48669] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  4 11:42:28 <kern.crit> edge kernel: [56716] PMc: KERNEL INFO page would not be zeroed, but will
Apr  4 11:48:21 <kern.crit> edge kernel: [57068] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  4 14:28:01 <kern.crit> edge kernel: [66648] PMc: KERNEL ALERT page should be zeroed, but will not
Apr  4 14:28:01 <kern.info> edge kernel: [66648] pid 34089 (postgres), jid 5, uid 770: exited on signal 11 (core dumped)

Sadly, I forgot to put another printf onto Your patch to see if it does anything.

This is an ugly issue and I am quite tired after two months of hunting red herrings.
It might be helpful to know what specifically are the "rare situations" where this eFix should do anything.

Recap: what I found in the postgres coredumps is a memory area used by jemalloc for some tree structure, maybe 4 or 8 MB in total, and it is sparsely populated by pointer structures just as the sourcecode describes them. But within this area are one or two blocks (of exactly 4kB each) which contain some almost random data from an unknown source. And when the pointer is fetched out of this area and then utilized, it results in either SIGSEGV or SIGBUS.
I do not grok how this comes to be, but it is certainly not good. I would have agreed that it might be a rare hardware flaw in the memory controller circuitry, but with the PR reporter here showing the same backtrace, that can be ruled out.
Comment 11 Mark Johnston freebsd_committer freebsd_triage 2026-04-06 04:22:21 UTC
(In reply to Peter Much from comment #10)
Thank you for testing.  I have a follow-up patch which addresses another issue with the original patch.  If you could test that too (reverting my first patch) I would appreciate it.

It's hard to characterize the cases where the original patch or these follow-up ones will have an effect.  It's timing-specific and relates to other factors like memory pressure and swap usage.  I don't have a concise way to explain whether the problems fixed by the original patch are likely to affect a given workload.
Comment 12 Mark Johnston freebsd_committer freebsd_triage 2026-04-06 04:22:55 UTC
Created attachment 269401 [details]
proposed patch
Comment 13 Peter Much 2026-04-07 13:31:51 UTC
(In reply to Mark Johnston from comment #11)

I was asking because I was looking for a better way to reproduce, and also because I am wondering why this seems to appear only here.
For test I am pushing full load on all cores (port building), and then only within a day might get an issue, it burns a lot of energy (and it is noisy).
I can see that this is somehow related to swap; the machine runs under load with 10-20 GB swap occupancy, and I have lowered vfs.zfs.arc.free_target to be slightly above vm.v_free_min so it still holds a substantial amout of ARC.

Concerning Your new patch: now I see nothing at all! There was not a single one of the printf() triggered during 22 hours. Taking the system back to regular operations now.
Comment 14 commit-hook freebsd_committer freebsd_triage 2026-04-08 14:04:52 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=04132e01004316ddd0e0cde6ef15b100b7b1844d

commit 04132e01004316ddd0e0cde6ef15b100b7b1844d
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2026-04-08 04:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-04-08 14:04:29 +0000

    vm_fault: Reset m_needs_zeroing properly

    - When allocating a page, we should only consider the PG_ZERO flag when
      handling the top-level page.
    - Unconditionally reset the flag when restarting the fault handler.
      Previously, vm_fault_busy_sleep() would fail to reset it.

    PR:             294039
    Reviewed by:    kib
    Tested by:      Peter Much <pmc@citylink.dinoex.sub.org>
    MFC after:      3 days
    Fixes:          cff67bc43df1 ("vm_fault: only rely on PG_ZERO when the page was newly allocated")
    Differential Revision:  https://reviews.freebsd.org/D56234

 sys/vm/vm_fault.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
Comment 15 commit-hook freebsd_committer freebsd_triage 2026-04-13 02:57:34 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=9b7c0f4f81f06424899094d4381dede79669b623

commit 9b7c0f4f81f06424899094d4381dede79669b623
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2026-04-08 04:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-04-13 02:53:32 +0000

    vm_fault: Reset m_needs_zeroing properly

    - When allocating a page, we should only consider the PG_ZERO flag when
      handling the top-level page.
    - Unconditionally reset the flag when restarting the fault handler.
      Previously, vm_fault_busy_sleep() would fail to reset it.

    PR:             294039
    Reviewed by:    kib
    Tested by:      Peter Much <pmc@citylink.dinoex.sub.org>
    MFC after:      3 days
    Fixes:          cff67bc43df1 ("vm_fault: only rely on PG_ZERO when the page was newly allocated")
    Differential Revision:  https://reviews.freebsd.org/D56234

    (cherry picked from commit 04132e01004316ddd0e0cde6ef15b100b7b1844d)

 sys/vm/vm_fault.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
Comment 16 commit-hook freebsd_committer freebsd_triage 2026-04-13 02:59:37 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=50f7b62f0862f764215cee98547d5b8c0979ec26

commit 50f7b62f0862f764215cee98547d5b8c0979ec26
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2026-04-08 04:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-04-13 02:58:16 +0000

    vm_fault: Reset m_needs_zeroing properly

    - When allocating a page, we should only consider the PG_ZERO flag when
      handling the top-level page.
    - Unconditionally reset the flag when restarting the fault handler.
      Previously, vm_fault_busy_sleep() would fail to reset it.

    PR:             294039
    Reviewed by:    kib
    Tested by:      Peter Much <pmc@citylink.dinoex.sub.org>
    MFC after:      3 days
    Fixes:          cff67bc43df1 ("vm_fault: only rely on PG_ZERO when the page was newly allocated")
    Differential Revision:  https://reviews.freebsd.org/D56234

    (cherry picked from commit 04132e01004316ddd0e0cde6ef15b100b7b1844d)

 sys/vm/vm_fault.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
Comment 17 commit-hook freebsd_committer freebsd_triage 2026-04-13 10:58:00 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=58718cf365933dc2c75d43f7154371344bf47286

commit 58718cf365933dc2c75d43f7154371344bf47286
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2026-04-08 04:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-04-13 10:57:26 +0000

    vm_fault: Reset m_needs_zeroing properly

    - When allocating a page, we should only consider the PG_ZERO flag when
      handling the top-level page.
    - Unconditionally reset the flag when restarting the fault handler.
      Previously, vm_fault_busy_sleep() would fail to reset it.

    PR:             294039
    Reviewed by:    kib
    Tested by:      Peter Much <pmc@citylink.dinoex.sub.org>
    MFC after:      3 days
    Fixes:          cff67bc43df1 ("vm_fault: only rely on PG_ZERO when the page was newly allocated")
    Differential Revision:  https://reviews.freebsd.org/D56234

    (cherry picked from commit 04132e01004316ddd0e0cde6ef15b100b7b1844d)

 sys/vm/vm_fault.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
Comment 18 Julien Cigar 2026-04-14 11:40:01 UTC
Hello, I was wondering if an EN will be published?
Comment 19 Mark Johnston freebsd_committer freebsd_triage 2026-04-15 13:25:57 UTC
(In reply to Julien Cigar from comment #18)
Yes, we will publish an EN for this.
Comment 20 commit-hook freebsd_committer freebsd_triage 2026-04-21 15:45:14 UTC
A commit in branch releng/15.0 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ffb21713d9fdaef1e416a4443c7bc234cfb5457d

commit ffb21713d9fdaef1e416a4443c7bc234cfb5457d
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2026-04-08 04:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-04-15 19:02:32 +0000

    vm_fault: Reset m_needs_zeroing properly

    - When allocating a page, we should only consider the PG_ZERO flag when
      handling the top-level page.
    - Unconditionally reset the flag when restarting the fault handler.
      Previously, vm_fault_busy_sleep() would fail to reset it.

    Approved by:    so
    Security:       FreeBSD-EN-26:05.vm
    PR:             294039
    Reviewed by:    kib
    Tested by:      Peter Much <pmc@citylink.dinoex.sub.org>
    MFC after:      3 days
    Fixes:          cff67bc43df1 ("vm_fault: only rely on PG_ZERO when the page was newly allocated")
    Differential Revision:  https://reviews.freebsd.org/D56234

    (cherry picked from commit 04132e01004316ddd0e0cde6ef15b100b7b1844d)
    (cherry picked from commit 58718cf365933dc2c75d43f7154371344bf47286)

 sys/vm/vm_fault.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
Comment 21 commit-hook freebsd_committer freebsd_triage 2026-04-21 15:46:17 UTC
A commit in branch releng/14.4 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=1abe7ead45c3236e5325f43535e2513697a5705e

commit 1abe7ead45c3236e5325f43535e2513697a5705e
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2026-04-08 04:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-04-21 15:45:02 +0000

    vm_fault: Reset m_needs_zeroing properly

    - When allocating a page, we should only consider the PG_ZERO flag when
      handling the top-level page.
    - Unconditionally reset the flag when restarting the fault handler.
      Previously, vm_fault_busy_sleep() would fail to reset it.

    Approved by:    so
    Security:       FreeBSD-EN-26:05.vm
    PR:             294039
    Reviewed by:    kib
    Tested by:      Peter Much <pmc@citylink.dinoex.sub.org>
    MFC after:      3 days
    Fixes:          cff67bc43df1 ("vm_fault: only rely on PG_ZERO when the page was newly allocated")
    Differential Revision:  https://reviews.freebsd.org/D56234

    (cherry picked from commit 04132e01004316ddd0e0cde6ef15b100b7b1844d)
    (cherry picked from commit 9b7c0f4f81f06424899094d4381dede79669b623)

 sys/vm/vm_fault.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
Comment 22 commit-hook freebsd_committer freebsd_triage 2026-04-21 15:46:19 UTC
A commit in branch releng/14.3 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=4d22b3925df828c3d37d7b63820283ed97a78035

commit 4d22b3925df828c3d37d7b63820283ed97a78035
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2026-04-08 04:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-04-21 15:45:46 +0000

    vm_fault: Reset m_needs_zeroing properly

    - When allocating a page, we should only consider the PG_ZERO flag when
      handling the top-level page.
    - Unconditionally reset the flag when restarting the fault handler.
      Previously, vm_fault_busy_sleep() would fail to reset it.

    Approved by:    so
    Security:       FreeBSD-EN-26:05.vm
    PR:             294039
    Reviewed by:    kib
    Tested by:      Peter Much <pmc@citylink.dinoex.sub.org>
    MFC after:      3 days
    Fixes:          cff67bc43df1 ("vm_fault: only rely on PG_ZERO when the page was newly allocated")
    Differential Revision:  https://reviews.freebsd.org/D56234

    (cherry picked from commit 04132e01004316ddd0e0cde6ef15b100b7b1844d)
    (cherry picked from commit 9b7c0f4f81f06424899094d4381dede79669b623)

 sys/vm/vm_fault.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
Comment 23 commit-hook freebsd_committer freebsd_triage 2026-04-21 15:47:21 UTC
A commit in branch releng/13.5 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=6c9dd75283503091c202cda1606a539f65bda85f

commit 6c9dd75283503091c202cda1606a539f65bda85f
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2026-04-08 04:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2026-04-21 15:46:57 +0000

    vm_fault: Reset m_needs_zeroing properly

    - When allocating a page, we should only consider the PG_ZERO flag when
      handling the top-level page.
    - Unconditionally reset the flag when restarting the fault handler.
      Previously, vm_fault_busy_sleep() would fail to reset it.

    Approved by:    so
    Security:       FreeBSD-EN-26:05.vm
    PR:             294039
    Reviewed by:    kib
    Tested by:      Peter Much <pmc@citylink.dinoex.sub.org>
    MFC after:      3 days
    Fixes:          cff67bc43df1 ("vm_fault: only rely on PG_ZERO when the page was newly allocated")
    Differential Revision:  https://reviews.freebsd.org/D56234

    (cherry picked from commit 04132e01004316ddd0e0cde6ef15b100b7b1844d)
    (cherry picked from commit 50f7b62f0862f764215cee98547d5b8c0979ec26)

 sys/vm/vm_fault.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
Comment 24 Mark Johnston freebsd_committer freebsd_triage 2026-04-21 19:32:45 UTC
This is addressed by EN-26:05.vm
Comment 25 Robert Clausecker freebsd_committer freebsd_triage 2026-04-21 19:59:50 UTC
Nice fix!  I do wonder if this issue is the cause of the random memory corruption issues some people were observing on ppc64le with ZFS.