Bug 233336

Summary: statically linked binaries built WITH_RETPOLINE crash at startup
Product: Base System Reporter: Ed Maste <emaste>
Component: confAssignee: Ed Maste <emaste>
Status: Open ---    
Severity: Affects Some People CC: koobs, ml+freebsd, shawn.webb
Priority: --- Keywords: crash
Version: CURRENT   
Hardware: Any   
OS: Any   
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231325

Description Ed Maste freebsd_committer 2018-11-19 18:38:45 UTC
Reproduction steps:
1. Set WITH_RETPOLINE and WITH_BSD_CRTBEGIN in /etc/src.conf
2. Build and install a statically linked binary (e.g. rescue)
3. Execute it

# rescue/ls
Illegal instruction (core dumped)
Comment 1 Shawn Webb 2018-11-19 18:47:02 UTC
This can happen without BSD_CERTBEGIN as well. Building ports-mgmt/pkg with RETPOLINE enabled causes pkg-static to exhibit the same type of crash.

https://github.com/HardenedBSD/hardenedbsd-ports/commit/48db46348223d77730a95a32597a29e5d23aae25
Comment 2 Ed Maste freebsd_committer 2018-11-19 19:11:12 UTC
Reproducible with a standalone test case with something like
cc -g -static -Wl,-znow,-zretpolineplt hello.c
Comment 3 commit-hook freebsd_committer 2018-11-19 20:49:49 UTC
A commit references this bug:

Author: emaste
Date: Mon Nov 19 20:48:47 UTC 2018
New revision: 340650
URL: https://svnweb.freebsd.org/changeset/base/340650

Log:
  Avoid retpolineplt with static linking

  Statically linked binaries linked with -zretpolineplt crash at startup
  as lld produces a broken PLT.

  PR:		233336
  Sponsored by:	The FreeBSD Foundation

Changes:
  head/share/mk/bsd.prog.mk
Comment 4 Charlie Li 2018-11-19 21:28:10 UTC
This revision fixed my problem of static binaries crashing since base r339898 that I had been discussing with kib@. My test program was make(1), both installed and directly from the objdir. The crashes happened with both base (6.0.1) and lld70 at least.

However, this revision then introduced a different crash of rescue(8) during the beginning of installworld, but that could be a side effect of META_MODE not rebuilding certain files. Will reproduce with actual error message and rebuild from a clean objdir later.
Comment 5 Ed Maste freebsd_committer 2018-11-19 21:33:43 UTC
(In reply to Charlie Li from comment #4)
The rescue build generates a Makefile (from crunchgen) and I suspect this doesn't work with the NO_SHARED check I added in r233336. Perhaps see if adding NO_SHARED=YES to rescue/rescue/Makefile addresses it?
Comment 6 Charlie Li 2018-11-19 22:11:35 UTC
(In reply to Ed Maste from comment #5)
NO_SHARED=YES fixes the crash.
Comment 7 commit-hook freebsd_committer 2018-11-19 22:19:02 UTC
A commit references this bug:

Author: emaste
Date: Mon Nov 19 22:18:18 UTC 2018
New revision: 340652
URL: https://svnweb.freebsd.org/changeset/base/340652

Log:
  rescue: set NO_SHARED in Makefile

  The rescue binary is built statically via the Makefile generated by
  crunchgen, but that does not trigger other shared/static logic in
  bsd.prog.mk - in particular disabling retpolineplt with static linking.

  PR:		233336
  Reported by:	Charlie Li
  Sponsored by:	The FreeBSD Foundation

Changes:
  head/rescue/rescue/Makefile
Comment 8 Ed Maste freebsd_committer 2018-11-19 22:19:59 UTC
(In reply to Charlie Li from comment #6)
Thanks for the followup, change committed in r340652. Please let me know if you observe any other issues. I will update this PR once I have a candidate lld patch for testing.
Comment 9 commit-hook freebsd_committer 2018-11-20 20:17:00 UTC
A commit references this bug:

Author: emaste
Date: Tue Nov 20 20:16:05 UTC 2018
New revision: 340697
URL: https://svnweb.freebsd.org/changeset/base/340697

Log:
  Introduce src.conf knob to build userland with retpoline

  MFC r339511: Introduce src.conf knob to build userland with retpoline

  WITH_RETPOLINE enables -mretpoline vulnerability mitigation in userland
  for CVE-2017-5715.

  MFC r340099: libcompat: disable retpoline when building build tools

  These are built with the host toolchain which may not support retpoline.
  While here, move the MK_ overrides to a separate line and sort them
  alphabetically to support future changes.

  MFC r340650: Avoid retpolineplt with static linking

  Statically linked binaries linked with -zretpolineplt crash at startup
  as lld produces a broken PLT.

  MFC r340652: rescue: set NO_SHARED in Makefile

  The rescue binary is built statically via the Makefile generated by
  crunchgen, but that does not trigger other shared/static logic in
  bsd.prog.mk - in particular

  PR:		233336
  Reported by:	Peter Malcom (r339511), Charlie Li (r340652)
  Approved by:	re (gjb, early MFC)
  Sponsored by:	The FreeBSD Foundation

Changes:
_U  stable/12/
  stable/12/Makefile.inc1
  stable/12/Makefile.libcompat
  stable/12/rescue/rescue/Makefile
  stable/12/share/mk/bsd.lib.mk
  stable/12/share/mk/bsd.opts.mk
  stable/12/share/mk/bsd.prog.mk
  stable/12/tools/build/options/WITH_RETPOLINE
Comment 10 Ed Maste freebsd_committer 2018-11-21 14:24:58 UTC
Upstream fix: https://reviews.llvm.org/D54782

I expect to cherry-pick this into FreeBSD after we commit the clang/lld 7.0 update.
Comment 11 commit-hook freebsd_committer 2018-11-22 19:00:06 UTC
A commit references this bug:

Author: emaste
Date: Thu Nov 22 18:59:06 UTC 2018
New revision: 340773
URL: https://svnweb.freebsd.org/changeset/base/340773

Log:
  MF12 r340697: Introduce src.conf knob to build userland with retpoline

  MFC r339511: Introduce src.conf knob to build userland with retpoline

  WITH_RETPOLINE enables -mretpoline vulnerability mitigation in userland
  for CVE-2017-5715.

  MFC r340099: libcompat: disable retpoline when building build tools

  These are built with the host toolchain which may not support retpoline.
  While here, move the MK_ overrides to a separate line and sort them
  alphabetically to support future changes.

  MFC r340650: Avoid retpolineplt with static linking

  Statically linked binaries linked with -zretpolineplt crash at startup
  as lld produces a broken PLT.

  MFC r340652: rescue: set NO_SHARED in Makefile

  The rescue binary is built statically via the Makefile generated by
  crunchgen, but that does not trigger other shared/static logic in
  bsd.prog.mk - in particular

  PR:		233336
  Reported by:	Peter Malcom (r339511), Charlie Li (r340652)
  Approved by:	re (kib)
  Sponsored by:	The FreeBSD Foundation

Changes:
_U  releng/12.0/
  releng/12.0/Makefile.inc1
  releng/12.0/Makefile.libcompat
  releng/12.0/rescue/rescue/Makefile
  releng/12.0/share/mk/bsd.lib.mk
  releng/12.0/share/mk/bsd.opts.mk
  releng/12.0/share/mk/bsd.prog.mk
  releng/12.0/tools/build/options/WITH_RETPOLINE