Bug 223673 - pkg base (i.e. "make packages") produces nonreproducible output with make -j
Summary: pkg base (i.e. "make packages") produces nonreproducible output with make -j
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: pkgbase
Depends on:
Blocks:
 
Reported: 2017-11-15 01:25 UTC by Ed Maste
Modified: 2017-11-17 23:21 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ed Maste freebsd_committer freebsd_triage 2017-11-15 01:25:19 UTC
The files within the packages are identical but the ordering in the package txz differs when invoked as "make -j... packages"

E.g. from the reproducbile-builds.org Jenkins instance,
https://tests.reproducible-builds.org/freebsd/dbd/repo/FreeBSD:12:amd64/current/FreeBSD-jail-current.txz.html

The first package contained
/​usr/​sbin/​jail
/​usr/​sbin/​jexec
/​usr/​share/​man/​man8/​jexec.​8.​gz
/​usr/​share/​man/​man8/​jail.​8.gz
/​usr/​share/​man/​man5/​jail.​conf.5.gz
​/​usr/​sbin/​jls

while the second package contained
/​usr/​sbin/​jail
/​usr/​share/​man/​man8/​jail.​8.gz
/​usr/​share/​man/​man5/​jail.​conf.5.gz
​/​usr/​sbin/​jexec
/​usr/​share/​man/​man8/​jexec.​8.​gz
/​usr/​sbin/​jls

one possible solution is to sort the files in release/scripts/mtree-to-plist.awk
Comment 1 commit-hook freebsd_committer freebsd_triage 2017-11-15 18:04:16 UTC
A commit references this bug:

Author: emaste
Date: Wed Nov 15 18:03:31 UTC 2017
New revision: 325859
URL: https://svnweb.freebsd.org/changeset/base/325859

Log:
  Sort pkgbase mtree metadata, for reproducible builds

  Packaged base packages are created by running the stageworld and
  stagekernel targets with -DNO_ROOT, and converting the resulting mtree
  file into a set of pkg plists.  If stage* is run with multiple processes
  the order of entries in the mtree file may be nondeterministic, and the
  resulting package tbz also had nondeterministic file ordering.

  The mtree file generated by -DNO_ROOT builds consists of one line per
  file, with the filename starting in the first column, so is easily
  sorted.  There's one exception: the first line of the mtree file is a
  comment, but the # character sorts before the filenames anyhow and needs
  no special treatment.

  PR:		223673
  Reviewed by:	bapt, gjb
  Sponsored by:	The Linux Foundation, Core Infrastructure Initiative
  Sponsored by:	The FreeBSD Foundation
  Differential Revision:	https://reviews.freebsd.org/D13103

Changes:
  head/Makefile.inc1