## Description On amd64, `do_posix_spawn()` runs the spawn child via `rfork_thread(RFSPAWN, stack + stacksz, _posix_spawn_thr, &psa)` on a small `malloc`'d buffer (`lib/libc/gen/posix_spawn.c`): - `_RFORK_THREAD_STACK_SIZE` = 4096 (`:240`) - for a PATH-searched (relative) command, `stacksz = 4096 + MAX(3, argc+2)*sizeof(char*)`, 16-byte aligned (`:285`, `:295-298`) — e.g. 4128 bytes for `argv = {"true", NULL}` - `stack = malloc(stacksz)` (`:308`); the child is run on it (`:341`) - with `use_env_path` set, the child calls `__libc_execvpe()` (`:264`), i.e. the PATH search `RFSPAWN`/`rfork_thread` gives the child a **shared address space** (vfork semantics) until it execs. In a process that has **many shared objects loaded**, the spawn child's stack usage while running `__libc_execvpe()` exceeds this ~4 KB buffer and **underflows it**, scribbling over whatever heap sits below the buffer in the shared parent address space. The caller then crashes or misbehaves later, far from `posix_spawn`. This is **distinct from CVE-2020-7458 / SA-20:18** (that was a *long $PATH* overflow, fixed in 2020). The problem here is **independent of $PATH length** — it reproduces with a single PATH entry — and is driven by how much stack `__libc_execvpe` uses in a heavily-linked process, against a fixed ~4 KB child stack. ## Real-world impact (PHP) — likely the cause behind PR 277888 PHP's `proc_open()` with an array (argv) command calls `posix_spawnp()` with a relative program name. On FreeBSD with a normal set of PHP extensions loaded: ``` pkg install php84 <plus a typical set of extensions: dom intl pdo pdo_pgsql pgsql session simplexml sodium xml xmlwriter zip imagick memcached redis ...> php -r 'proc_open(["true"], [], $p);' # → Segmentation fault (~100%) ``` The crash itself surfaces much later, at PHP module shutdown (in `_efree`/ `zend_interned_strings_dtor`), because the underflow had overwritten the header of a permanent interned string back during the `proc_open` call. Tellingly, it depends on the spawn path, not on PHP logic: | call | spawn path | crash | |---|---|---| | `proc_open(["true"], …)` (relative) | `posix_spawnp` → `__libc_execvpe` (PATH search) | yes | | `proc_open(["/usr/bin/true"], …)` (absolute) | `posix_spawnp` → `execvPe` direct branch | no | | `proc_open("true", …)` (string) | `posix_spawn` (`/bin/sh -c`) → `_execve` | no | This very likely explains the `proc_open(["date"], $p)` reproducer discussed in **PR 277888** ("lang/php83 and lang/php84: segmentation fault on unloading modules"). That PR was closed FIXED via an `ext/xsl` change, yet its own comments report that the `proc_open(["date"], $p)` repro still segfaults on 14.2-RELEASE-p4 / php 8.4.10 / php 8.3.21 — i.e. that part appears mis-attributed; the cause looks to be here in libc. (Upstream PHP tracking: php/php-src#21995.) ## Evidence that it is the spawn child stack Interpose `rfork_thread()` via `LD_PRELOAD` and give the spawn child a different stack (everything else in libc unchanged; PHP heap layout unchanged because libc still does its own `malloc(stacksz)`): - child stack = **1 MB** → the PHP crash disappears: **0/30** vs **30/30** baseline. - child stack = **~4 KB with a PROT_NONE guard page just below it** → the spawn child dies with **SIGSEGV** (it grows down into the guard page). Because the guard sits immediately below the child stack, this is strong, **location-independent** evidence that the child exhausts its stack downward on this path (it rules out the "maybe it only mattered what was adjacent below the stack" confound). It does not by itself pin down which frame overflows (`execvPe` / `_execve` / run-time-linker work / the `rfork_thread` trampoline). Minimal interposer (essence): ```c pid_t rfork_thread(int flags, void *stack, int (*fn)(void*), void *arg) { long pg = sysconf(_SC_PAGESIZE); size_t sz = GUARDSPAWN_KB*1024; /* 1024 -> fixes it; 4 -> child SIGSEGV */ char *b = mmap(0, pg+sz, PROT_READ|PROT_WRITE, MAP_ANON|MAP_PRIVATE, -1, 0); mprotect(b, pg, PROT_NONE); /* guard page below the child stack */ return real_rfork_thread(flags, b+pg+sz, fn, arg); } ``` Note: a *minimal* standalone program calling `posix_spawnp("true", …)` does **not** underflow the 4 KB stack — the excess usage only appears once enough shared objects are loaded (so it bites real applications like PHP, not toy reproducers). I could not pin down *why*; a plausible (unproven) hypothesis is run-time-linker work on the child's small stack in the shared address space. The recent commit 4daf2d3e7db5 ("posix_spawn: use rfork_thread on all arches", main, 2026-01) does not change `_RFORK_THREAD_STACK_SIZE`; it uses `rfork_thread()` on all arches but the ~4 KB custom child stack remains the x86 path (other arches set `stack = NULL; stacksz = 0`). So current `main` still has this on x86/amd64. ## Suggested fix (maintainers' call) Size the spawn child stack adequately for `__libc_execvpe` under realistic load (the `stacksz` reservation currently only accounts for the ENOEXEC `alloca`), and/or place a guard page below it so an underflow faults deterministically instead of silently corrupting the caller's heap. --- *Disclosure: this analysis was produced with LLM assistance. All stacks, traces, repro counts, source references and the interposer experiment are from real runs on FreeBSD 15.0-RELEASE amd64; the interpretation is the part worth double-checking.*
I can confirm this to be an issue in FreeBSD 15.0-RELEASE-p10 and PHP 8.4.22 (cli) (built: Jun 6, 2026, 19:05:19).
The report is an unreadable AI mess but the point is that there is some random corruption that shows up with php on exit when loaded with many extensions. Raising the stack size in posix_spawn avoids the problem. This shows up easily with www/nextcloud port running `occ status` in a loop. The proc_open repro isn't reliable as it depends on some random mechanism and how many extensions are loaded. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289425 and https://github.com/php/php-src/issues/21995 have more details. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289425#c22 shows some ways it manifests at shutdown. The revert of the xsl shutdown function was a hack that attempted to avoid the root cause but does not. Based on the initial report this AI-generated code avoids the problem by using a 1MB stack size. ``` /* cc -O2 -fPIC -shared -Wall -o /usr/local/libexec/spawnstack.so spawnstack.c */ #include <sys/param.h> #include <sys/mman.h> #include <unistd.h> #include <stdlib.h> #include <dlfcn.h> typedef pid_t (*rfork_thread_t)(int, void *, int (*)(void *), void *); #if __FreeBSD_version >= 1600000 typedef pid_t (*pdrfork_thread_t)(int *, int, int, void *, int (*)(void *), void *); #endif static rfork_thread_t sys_rfork_thread; #if __FreeBSD_version >= 1600000 static pdrfork_thread_t sys_pdrfork_thread; #endif static size_t guard; static size_t depth; __attribute__((constructor)) static void spawnstack_init(void) { const char *s; char *end; long ps; unsigned long kb; sys_rfork_thread = (rfork_thread_t)dlsym(RTLD_NEXT, "rfork_thread"); #if __FreeBSD_version >= 1600000 sys_pdrfork_thread = (pdrfork_thread_t)dlsym(RTLD_NEXT, "pdrfork_thread"); #endif ps = sysconf(_SC_PAGESIZE); guard = ps > 0 ? (size_t)ps : 4096; kb = 1024; s = getenv("LIBC_SPAWN_STACK_KB"); if (s != NULL && *s != '\0') { unsigned long v = strtoul(s, &end, 10); if (*end == '\0' && v >= 64) kb = v; } depth = (kb * 1024 + guard - 1) & ~(guard - 1); } static void * spawnstack_alloc(size_t *len) { char *p; *len = guard + depth; p = mmap(NULL, *len, PROT_READ | PROT_WRITE, MAP_ANON | MAP_PRIVATE, -1, 0); if (p == MAP_FAILED) return (NULL); (void)mprotect(p, guard, PROT_NONE); return (p + *len); } pid_t rfork_thread(int flags, void *stack, int (*func)(void *), void *arg) { void *top; size_t len; pid_t pid; if ((flags & RFSPAWN) == 0 || stack == NULL) return (sys_rfork_thread(flags, stack, func, arg)); top = spawnstack_alloc(&len); if (top == NULL) return (sys_rfork_thread(flags, stack, func, arg)); pid = sys_rfork_thread(flags, top, func, arg); (void)munmap((char *)top - len, len); return (pid); } #if __FreeBSD_version >= 1600000 pid_t pdrfork_thread(int *fdp, int pdflags, int rfflags, void *stack, int (*func)(void *), void *arg) { void *top; size_t len; pid_t pid; if ((rfflags & RFSPAWN) == 0 || stack == NULL) return (sys_pdrfork_thread(fdp, pdflags, rfflags, stack, func, arg)); top = spawnstack_alloc(&len); if (top == NULL) return (sys_pdrfork_thread(fdp, pdflags, rfflags, stack, func, arg)); pid = sys_pdrfork_thread(fdp, pdflags, rfflags, top, func, arg); (void)munmap((char *)top - len, len); return (pid); } #endif ``` /usr/local/bin/occ needs LD_PRELOAD added in. ``` #!/bin/sh args= for arg in "$@" ; do if [ "${arg#* }" != "${arg}" ] ; then args="${args} '${arg}'" else args="${args} ${arg}" fi done ( cd /home/nextcloud/public_html/nextcloud su -m www -c \ "/usr/bin/env LD_PRELOAD=/usr/local/libexec/spawnstack.so /usr/local/bin/php --define apc.enable_cli=1 /home/nextcloud/public_html/nextcloud/occ ${args}" ) ```
I'm tempted to switch the posix_spawn stack size to `DFLSSIZ` or kern.dflssiz, depending on how fancy we want to be: ``` amd64/include/vmparam.h:65:#define DFLSSIZ (8UL*1024*1024) /* initial stack size limit */ arm/include/vmparam.h:54:#define DFLSSIZ (4UL*1024*1024) /* initial stack size limit */ arm64/include/vmparam.h:56:#define DFLSSIZ (128*1024*1024) /* initial stack size limit */ i386/include/vmparam.h:55:#define DFLSSIZ (8UL*1024*1024) /* initial stack size limit */ powerpc/include/vmparam.h:62:#define DFLSSIZ (8*1024*1024) /* default stack size */ riscv/include/vmparam.h:52:#define DFLSSIZ (128*1024*1024) /* initial stack size limit */ ``` 128M on arm64 is maybe a little excessive? Better safe than sorry, though. CC kib@ in case he has a better opinion.
Can somebody give me a stand-alone reproducer, please? Without php and all that stuff.
(In reply to Kyle Evans from comment #3) FWIW, it changes nothing to me. diff --git a/sys/amd64/include/vmparam.h b/sys/amd64/include/vmparam.h index 763b89da9e12..75246b23df09 100644 --- a/sys/amd64/include/vmparam.h +++ b/sys/amd64/include/vmparam.h @@ -62,7 +62,7 @@ #define MAXDSIZ (32768UL*1024*1024) /* max data size */ #endif #ifndef DFLSSIZ -#define DFLSSIZ (8UL*1024*1024) /* initial stack size limit */ +#define DFLSSIZ (16UL*1024*1024) /* initial stack size limit */ #endif #ifndef MAXSSIZ #define MAXSSIZ (512UL*1024*1024) /* max stack size */ (In reply to Bryan Drewery from comment #2) Perhaps we can incorporate this patch into the port ?
(In reply to Konstantin Belousov from comment #4) I'm still trying to pin down why we actually blow out the stack... I don't really see any way that # DSOs can be directly related. We do allocate one MAXPATHLEN(1k) buffer on the stack + a struct stat, but we don't push args or environment onto the stack- that's about the extent of our usage. I'm playing around with claude-code to see if it can reason about what they're talking about and find a way to reproduce it under ASAN or something. Its initial theory is that `strchrnul` is maybe obscure enough that it still needs to be resolved through the PLT, and that `_rtld_bind`'s stack usage scales with # DSOs loaded to an extent that our remaining 3k stack isn't sufficient. When I walk it along a little further, it points out that _rtld_bind -> find_symdef -> symlook_default will invoke donelist_init(): ``` #define donelist_init(dlp) \ ((dlp)->objs = alloca(obj_count * sizeof(dlp)->objs[0]), \ assert((dlp)->objs != NULL), (dlp)->num_alloc = obj_count, \ (dlp)->num_used = 0) ``` I think that might be enough to try and write a test for it, it's just going to be a little sketchy. I guess I could write one DSO and copy it into a bunch of shmfds and fdlopen() that.
If the theory about alloca() is right, then https://reviews.freebsd.org/D57908 should help (but I did not tested). Even if it is not, I believe that it is a useful change.
(In reply to Konstantin Belousov from comment #7) This seems to fix my somewhat amusing reproducer: https://paste.fbsd.dev/t3i5 -- patch is kind of hacky because we don't have an asan COMPILER_FEATURES yet. I don't know that the sketchy .XXX SONAME and runtime replacement is really necessary, but I had added it when I couldn't figure out why my object wasn't being loaded again. In this test case, the failure mode is that the child crashes and atf hangs: ``` root@ifrit:/usr/src/lib/libc/tests/gen # env ASAN_OPTIONS="detect_leaks=0" /usr/tests/lib/libc/gen/posix_spawn_test posix_spawnp_stackoverflow posix_spawn_test: WARNING: Running test cases outside of kyua(1) is unsupported posix_spawn_test: WARNING: No isolation nor timeout control is being applied; you may get unexpected failures; see atf-test-case(4) failed: /usr/src/lib/libc/tests/gen/posix_spawn_test.c:239: WIFEXITED(status) && WEXITSTATUS(status) == 0 not met o load: 0.57 cmd: posix_spawn_test 1912 [uwrlck] 4.05r 0.00u 0.00s 0% 10076k mi_switch+0x172 sleepq_switch+0x109 sleepq_catch_signals+0x276 sleepq_wait_sig+0x9 _sleep+0x29a umtxq_sleep+0x302 do_rw_wrlock+0x3d3 sys__umtx_op+0x7e amd64_syscall+0x168 fast_syscall_common+0xf8 ``` I guess we end up wiping out one of the rtld locks or something.
(In reply to Konstantin Belousov from comment #7) This patch fixes the issue reported in bug 289425 when applied on the recent stable/15. Thank you for solving this. Please plan a MFC to stable/15.
Please hold on. This patch introduces problem in the jail running www/onlyoffice. A few backtraces from this jail are posted below. (lldb) target create "/ezjail/onlyoffice/usr/local/www/onlyoffice/documentserver/server/FileConverter/bin/x2t" --core "x2t.37483.core" bt Core file '/ezjail/onlyoffice/tmp/x2t.37483.core' (x86_64) was loaded. (lldb) bt * thread #1, name = 'V8 DefaultWorke', stop reason = signal SIGSEGV * frame #0: 0x00001c87c6981875 ld-elf.so.1 frame #1: 0x0000000000000408 frame #2: 0x00001c87c69818d2 ld-elf.so.1 frame #3: 0x00001c87c6984299 ld-elf.so.1 frame #4: 0x00001c87c697ae82 ld-elf.so.1 frame #5: 0x00001c87c697fd1b ld-elf.so.1 frame #6: 0x00001c87c697faf3 ld-elf.so.1 frame #7: 0x00001c87c6979146 ld-elf.so.1 frame #8: 0x00001c87c6978d05 ld-elf.so.1 frame #9: 0x00001c87c69789b6 ld-elf.so.1 frame #10: 0x00001c87c6973bad ld-elf.so.1`___lldb_unnamed_symbol_6b80 + 45 frame #11: 0x00000008342f0e99 libthr.so.3`___lldb_unnamed_symbol_10d40 + 345 (lldb) target create "/ezjail/onlyoffice/usr/local/lib/erlang27/erts-15.2.7.9/bin/beam.smp" --core "beam.smp.38994.core" Core file '/ezjail/onlyoffice/tmp/beam.smp.38994.core' (x86_64) was loaded. (lldb) bt * thread #1, name = 'erts_async_1', stop reason = signal SIGSEGV * frame #0: 0x00000b67fd895875 ld-elf.so.1 frame #1: 0x0000000000000078 frame #2: 0x00000b67fd8958d2 ld-elf.so.1 frame #3: 0x00000b67fd898299 ld-elf.so.1 frame #4: 0x00000b67fd88ee82 ld-elf.so.1 frame #5: 0x00000b67fd893d1b ld-elf.so.1 frame #6: 0x00000b67fd893af3 ld-elf.so.1 frame #7: 0x00000b67fd88d146 ld-elf.so.1 frame #8: 0x00000b67fd88cd05 ld-elf.so.1 frame #9: 0x00000b67fd88c9b6 ld-elf.so.1 frame #10: 0x00000b67fd887bad ld-elf.so.1`___lldb_unnamed_symbol_6b80 + 45 frame #11: 0x00000008262c9e99 libthr.so.3`___lldb_unnamed_symbol_10d40 + 345 (lldb) target create "/ezjail/onlyoffice/usr/local/lib/erlang27/erts-15.2.7.9/bin/beam.smp" --core "beam.smp.27430.core" Core file '/ezjail/onlyoffice/tmp/beam.smp.27430.core' (x86_64) was loaded. (lldb) bt * thread #1, name = 'erts_sched_32', stop reason = signal SIGSEGV * frame #0: 0x00000b7d3eab74d4 ld-elf.so.1 frame #1: 0x00000008876f0b80 frame #2: 0x00000b7d3eaab6a6 ld-elf.so.1 frame #3: 0x00000b7d3eaaace5 ld-elf.so.1 frame #4: 0x00000b7d3eaa77d2 ld-elf.so.1 frame #5: 0x000000089deec55b libcrypto.so.35`___lldb_unnamed_symbol_2ec520 + 59 frame #6: 0x000000089deecfa6 libcrypto.so.35`DSO_load + 374 frame #7: 0x000000089deed475 libcrypto.so.35`DSO_dsobyaddr + 133 frame #8: 0x000000089de31639 libcrypto.so.35`___lldb_unnamed_symbol_231600 + 57 frame #9: 0x0000000826c3f2f3 libthr.so.3`pthread_once + 179 frame #10: 0x000000089de24569 libcrypto.so.35`CRYPTO_THREAD_run_once + 9 frame #11: 0x000000089de311a9 libcrypto.so.35`OPENSSL_init_crypto + 377 frame #12: 0x000000089de367a0 libcrypto.so.35`___lldb_unnamed_symbol_2366e0 + 192 frame #13: 0x000000089de31b32 libcrypto.so.35`OSSL_PROVIDER_try_load_ex + 50
There is at least one mt issue with the original patch, I mentioned it in the review. Try the updated diff, it might fix the problem.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1e370f038778e16c4b31f8992dda339d429e5cb8 commit 1e370f038778e16c4b31f8992dda339d429e5cb8 Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2026-06-27 15:11:37 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2026-06-29 20:33:12 +0000 rtld: stop using unbound alloca() For DoneList allocations, its size depends on the number of loaded DSOs. Small images could be served by alloca(), but large donelists need to go into heap. For map_object(), alloca size is the number of segments in the object. In both cases, over-grown situations would cause a stack overflow. PR: 295991 Noted and reviewed by: kevans Tested by: Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D57908 libexec/rtld-elf/map_object.c | 4 ++- libexec/rtld-elf/rtld.c | 82 ++++++++++++++++++++++++++++++------------- libexec/rtld-elf/rtld.h | 2 ++ 3 files changed, 63 insertions(+), 25 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3de9dc5bf4b438e0d98222dc028982380d5a25d2 commit 3de9dc5bf4b438e0d98222dc028982380d5a25d2 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2026-06-30 13:12:51 +0000 Commit: Kyle Evans <kevans@FreeBSD.org> CommitDate: 2026-06-30 13:12:51 +0000 libc: gen: add a test for rtld underflowing our posix_spawn thread This is a distillation of the environment described in the PR, using a dummy shlib and mapping it repeatedly. This takes advantage of the guard page added in 2767a1f3686e5b16 to reliably crash if rtld tries to scale its stack usage excessively with the # DSOs loaded. PR: 295991 Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D57954 lib/libc/tests/gen/Makefile | 2 + lib/libc/tests/gen/libdummy/Makefile (new) | 9 ++++ lib/libc/tests/gen/libdummy/libdummy.c (new) | 14 ++++++ lib/libc/tests/gen/posix_spawn_test.c | 68 ++++++++++++++++++++++++++++ 4 files changed, 93 insertions(+)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=682ebe10032d4e682381eb5f108a8297be7b682c commit 682ebe10032d4e682381eb5f108a8297be7b682c Author: Konstantin Belousov <kib@FreeBSD.org> AuthorDate: 2026-06-27 15:11:37 +0000 Commit: Konstantin Belousov <kib@FreeBSD.org> CommitDate: 2026-07-06 01:05:50 +0000 rtld: stop using unbound alloca() PR: 295991 (cherry picked from commit 1e370f038778e16c4b31f8992dda339d429e5cb8) libexec/rtld-elf/map_object.c | 4 ++- libexec/rtld-elf/rtld.c | 82 ++++++++++++++++++++++++++++++------------- libexec/rtld-elf/rtld.h | 2 ++ 3 files changed, 63 insertions(+), 25 deletions(-)
Any plan for an 15.1-RELEASE ERRATA ? or we have to live with crashing process. It make some php cli crash on 15.1 (but not on 15.0)
(In reply to mickael.maillot from comment #15) In general, this is a regular bugfix, and we do not push each bug fix into releases. The issue should be really exceptional for that. But I would defer it to users and to re@. Anyway, I prefer somebody who is native English speaker to write the user-facing texts, i.e. not me.
(In reply to Konstantin Belousov from comment #16) I'll volunteer to write up the notice; it sounds pretty severe, but given that bdrewery commented due to impact in nextcloud maybe we can get a balanced report on the impact before we solicit secteam. IMO it might be a bit hard to argue for an EN since the patch was not quite trivial and our current release cadence means that it's a non-issue for everyone in ~6-9 months if we get it MFC'd to stable/14 prior to 14.5.