Bug 291720 - 66eb78377bf1 causes environment corruption on some machines
Summary: 66eb78377bf1 causes environment corruption on some machines
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 16.0-CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: Robert Clausecker
URL: https://reviews.freebsd.org/D54303
Keywords: regression
Depends on:
Blocks:
 
Reported: 2025-12-16 13:50 UTC by Cy Schubert
Modified: 2026-01-04 13:29 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Cy Schubert freebsd_committer freebsd_triage 2025-12-16 13:50:09 UTC
66eb78377bf109af1d9e25626bf254b4369436ec corrupts the environment (i.e. getenv environment) on four of my six machines. What the four machines have in common is their 8 GB RAM or less (one, the firewall, has 5 GB). The 32 GB and 96 GB machines have no problem with this.

Except for the bisection (revert locally), this is hardly a deep dive analysis of the problem. I don't know why the smaller (RAM-wise) machines have an issue with this commit.

The error seen is the following from /bin/sh. Multiple of,

sh: environment corrupt; missing value for 107
sh: environment corrupt; missing value for 107

or messages like it with the "107" listed as either nothing or some other garbage. Running installworld again to revert this change causes installworld to fail with the above error plus multiple (100 maybe) of,

cp: =>: No such file or directory

Meaning this revision cannot be reverted without making a temporary change to Makefile.inc1 to revert this commit.
Comment 1 Cy Schubert freebsd_committer freebsd_triage 2025-12-16 13:50:40 UTC
cc the committer responsible for this commit.
Comment 2 Cy Schubert freebsd_committer freebsd_triage 2025-12-16 13:54:23 UTC
One more data point. The affected machines are older AMD CPUs with Phenom or Athlon processors.
Comment 3 Cy Schubert freebsd_committer freebsd_triage 2025-12-16 14:59:47 UTC
(In reply to Cy Schubert from comment #0)

What I mean by making the Makefile.inc1 change is that this commit also breaks installworld. Rolling back this change after reverting it is impossible without a temporary workaround in Makefile.inc1. Once this revision is installed one cannot simply remove it.
Comment 4 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2025-12-16 16:36:39 UTC
A simple workaround is `export ARCHLEVEL=scalar`, as documented in simd(7).  Adding that to the top of /etc/rc.subr should keep your system running until the issue is fixed.  The SIMD code does not use getenv(3) to check ARCHLEVEL so it will still work.
Comment 5 Cy Schubert freebsd_committer freebsd_triage 2025-12-16 16:47:16 UTC
(In reply to Dag-Erling Smørgrav from comment #4)

Could this go into rc.conf instead?

Wouldn't this also need to go into /etc/profile? The problem also occurs when building some ports and causes corruption when using the tab key for command completion in some shells.
Comment 6 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2025-12-16 17:06:52 UTC
(In reply to Cy Schubert from comment #5)
Adding it to /etc/rc.subr ensures that every process in the system that doesn't explicitly clear its environment inherits it.  Adding it to /etc/rc.conf is a weaker form of that which only takes effect after load_rc_conf is called, so you may still have trouble booting.

Adding it to the setenv entry in /etc/login.conf should take care of login sessions, which usually clear their environment, and to anything using su(1).  Adding it to /etc/profile should not be necessary.
Comment 7 Cy Schubert freebsd_committer freebsd_triage 2025-12-16 17:17:04 UTC
(In reply to Dag-Erling Smørgrav from comment #6)

I'll try the weaker form as it,

a) Keeps my local source changes to a minimum,
b) Applies only to the older machines.

I've updated my ansible to push it to login.conf on affected machines.
Comment 8 Robert Clausecker freebsd_committer freebsd_triage 2025-12-16 18:40:39 UTC
The problem is probably that machines without TZCNT resp. BMI1 behave differently on one of the tzcnt instruction.  This was previously fine, but I think the change has triggered the relevant edge case.

As a stop gap solution, could someone please comment out the line

ARCHFUNC(__stpncpy, baseline)

in lib/libc/amd64/string/stpncpy.S?  I don't have the environment to commit right now, removing or commenting out this line effectively disables the affected code.

I will investigate this shortly.
Comment 9 commit-hook freebsd_committer freebsd_triage 2025-12-16 19:16:25 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=ce9557d4ee174267504a63b2c8f6009be27368c9

commit ce9557d4ee174267504a63b2c8f6009be27368c9
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2025-12-16 19:04:51 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2025-12-16 19:14:44 +0000

    libc/amd64: Disable baseline version of stpncpy()

    This implementation appears to be broken on some CPUs.  Disable it
    until the issue can be investigated and fixed.

    PR:             291720
    Fixes:          66eb78377bf1 ("libc/amd64: fix overread conditions in stpncpy()")
    Fixes:          90253d49db09 ("lib/libc/amd64/string: add stpncpy scalar, baseline implementation")

 lib/libc/amd64/string/stpncpy.S | 2 ++
 1 file changed, 2 insertions(+)
Comment 10 Cy Schubert freebsd_committer freebsd_triage 2025-12-16 21:20:59 UTC
As a point of reference for any who follow on, I also had to add the export to /etc/profile because the way my sshd is configured. I don't allow keyboardinteractive here.
Comment 11 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2025-12-16 22:17:25 UTC
(In reply to Cy Schubert from comment #10)
The login.conf entry should have sufficed, but perhaps you forgot to run `cap_mkdb /etc/login.conf` after adding it?
Comment 12 Cy Schubert freebsd_committer freebsd_triage 2025-12-17 00:17:47 UTC
(In reply to Dag-Erling Smørgrav from comment #11)

Just an ansible error.
Comment 13 Mark Linimon freebsd_committer freebsd_triage 2025-12-18 05:16:14 UTC
^Triage: use Keyword instead of deprecated [keyword] convention.
Comment 14 Robert Clausecker freebsd_committer freebsd_triage 2025-12-19 14:55:40 UTC
A fix is proposed in D54303.  Please let me know if it works for you.
Comment 15 commit-hook freebsd_committer freebsd_triage 2026-01-04 13:22:36 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=2f83319214d9adb8ab7a77e35d1014658b3f9cae

commit 2f83319214d9adb8ab7a77e35d1014658b3f9cae
Author:     Robert Clausecker <fuz@FreeBSD.org>
AuthorDate: 2025-12-18 23:37:33 +0000
Commit:     Robert Clausecker <fuz@FreeBSD.org>
CommitDate: 2026-01-04 13:21:41 +0000

    libc/amd64: fix stpncpy.S again

    The previous fix introduced a regression on machines without the BMI1
    instruction set extension.  The TZCNT instruction used in this function
    behaves different on old machines when the source operand is zero, but
    the code was originally designed to never trigger this case.  The bug
    fix caused this case to be possible, leading to a regression on
    sufficiently old hardware.

    Fix the code by messing with things such that the source operand is
    never zero.

    PR:             291720
    Fixes:          66eb78377bf109af1d9e25626bf254b4369436ec
    Tested by:      cy
    Approved by:    markj (mentor)
    Differential Revision:  https://reviews.freebsd.org/D54303

 lib/libc/amd64/string/stpncpy.S | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
Comment 16 commit-hook freebsd_committer freebsd_triage 2026-01-04 13:24:39 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=e626fa022361bca98bba93842afb3d749427b801

commit e626fa022361bca98bba93842afb3d749427b801
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2025-12-16 19:04:51 +0000
Commit:     Robert Clausecker <fuz@FreeBSD.org>
CommitDate: 2026-01-04 13:22:50 +0000

    libc/amd64: Disable baseline version of stpncpy()

    This implementation appears to be broken on some CPUs.  Disable it
    until the issue can be investigated and fixed.

    PR:             291720
    Fixes:          66eb78377bf1 ("libc/amd64: fix overread conditions in stpncpy()")
    Fixes:          90253d49db09 ("lib/libc/amd64/string: add stpncpy scalar, baseline implementation")
    (cherry picked from commit ce9557d4ee174267504a63b2c8f6009be27368c9)

 lib/libc/amd64/string/stpncpy.S | 2 ++
 1 file changed, 2 insertions(+)
Comment 17 commit-hook freebsd_committer freebsd_triage 2026-01-04 13:24:40 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=9c68745a50640ccff37565e3cbfe53dddc18d7d3

commit 9c68745a50640ccff37565e3cbfe53dddc18d7d3
Author:     Robert Clausecker <fuz@FreeBSD.org>
AuthorDate: 2025-12-18 23:37:33 +0000
Commit:     Robert Clausecker <fuz@FreeBSD.org>
CommitDate: 2026-01-04 13:22:51 +0000

    libc/amd64: fix stpncpy.S again

    The previous fix introduced a regression on machines without the BMI1
    instruction set extension.  The TZCNT instruction used in this function
    behaves different on old machines when the source operand is zero, but
    the code was originally designed to never trigger this case.  The bug
    fix caused this case to be possible, leading to a regression on
    sufficiently old hardware.

    Fix the code by messing with things such that the source operand is
    never zero.

    PR:             291720
    Fixes:          66eb78377bf109af1d9e25626bf254b4369436ec
    Tested by:      cy
    Approved by:    markj (mentor)
    Differential Revision:  https://reviews.freebsd.org/D54303

    (cherry picked from commit 2f83319214d9adb8ab7a77e35d1014658b3f9cae)

 lib/libc/amd64/string/stpncpy.S | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
Comment 18 commit-hook freebsd_committer freebsd_triage 2026-01-04 13:27:41 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=a5c6c95631195181bd3707d96ab79184235df6a6

commit a5c6c95631195181bd3707d96ab79184235df6a6
Author:     Robert Clausecker <fuz@FreeBSD.org>
AuthorDate: 2025-12-18 23:37:33 +0000
Commit:     Robert Clausecker <fuz@FreeBSD.org>
CommitDate: 2026-01-04 13:25:33 +0000

    libc/amd64: fix stpncpy.S again

    The previous fix introduced a regression on machines without the BMI1
    instruction set extension.  The TZCNT instruction used in this function
    behaves different on old machines when the source operand is zero, but
    the code was originally designed to never trigger this case.  The bug
    fix caused this case to be possible, leading to a regression on
    sufficiently old hardware.

    Fix the code by messing with things such that the source operand is
    never zero.

    PR:             291720
    Fixes:          66eb78377bf109af1d9e25626bf254b4369436ec
    Tested by:      cy
    Approved by:    markj (mentor)
    Differential Revision:  https://reviews.freebsd.org/D54303

    (cherry picked from commit 2f83319214d9adb8ab7a77e35d1014658b3f9cae)

 lib/libc/amd64/string/stpncpy.S | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)
Comment 19 commit-hook freebsd_committer freebsd_triage 2026-01-04 13:27:43 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=793877e7667ce667cdf3539de68af78fb511c06f

commit 793877e7667ce667cdf3539de68af78fb511c06f
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2025-12-16 19:04:51 +0000
Commit:     Robert Clausecker <fuz@FreeBSD.org>
CommitDate: 2026-01-04 13:25:33 +0000

    libc/amd64: Disable baseline version of stpncpy()

    This implementation appears to be broken on some CPUs.  Disable it
    until the issue can be investigated and fixed.

    PR:             291720
    Fixes:          66eb78377bf1 ("libc/amd64: fix overread conditions in stpncpy()")
    Fixes:          90253d49db09 ("lib/libc/amd64/string: add stpncpy scalar, baseline implementation")
    (cherry picked from commit ce9557d4ee174267504a63b2c8f6009be27368c9)

 lib/libc/amd64/string/stpncpy.S | 2 ++
 1 file changed, 2 insertions(+)
Comment 20 Robert Clausecker freebsd_committer freebsd_triage 2026-01-04 13:29:40 UTC
Thank you for your report.