Bug 232254

Summary: ports-mgmt/pkg: pkg-static unable to sign repos in -current
Product: Ports & Packages Reporter: Sean Bruno <sbruno>
Component: Individual Port(s)Assignee: freebsd-pkg (Nobody) <pkg>
Status: Closed FIXED    
Severity: Affects Many People CC: ddrinnon, emaste, flo, grahamperrin, portmgr, re, rhurlin, royger, sbruno, swills, w.schwarzenfeld
Priority: --- Keywords: regression
Version: LatestFlags: bugzilla: maintainer-feedback? (pkg)
Hardware: amd64   
OS: Any   

Description Sean Bruno freebsd_committer 2018-10-14 18:18:54 UTC
After the base update of openssl, pkg-static and pkg rebuilt against this version of openssl are unable to sign repos with a given key:

access("/root/ssl/pkg.key",R_OK)                 = 0 (0x0)
open("/root/ssl/pkg.key",O_RDONLY,0666)          = 5 (0x5)
close(5)                                         = 0 (0x0)
write(1,"\n",1)                                  = 1 (0x1)
write(2,"pkg-static: ",12)                       = 12 (0xc)
write(2,"can't load key from /root/ssl/pk"...,37) = 37 (0x25)
write(2,"\n",1)                                  = 1 (0x1)
write(4,"\M-}7zXZ\0\0\^D\M-f\M-V\M-4F\^B"...,76) = 76 (0x4c)
close(4)                                         = 0 (0x0)
unlink("/tmp/foo/meta")                          = 0 (0x0)
ioctl(1,TIOCGETA,0x7fffffffd718)                 = 0 (0x0)
write(1,"\rPacking files for repository: "...,35) = 35 (0x23)
ioctl(1,TIOCGETA,0x7fffffffd718)                 = 0 (0x0)
write(1,"\n",1)                                  = 1 (0x1)
close(3)                                         = 0 (0x0)
process exit, rval = 65

root@bob.nyi:/usr/local/poudriere/data/packages/12-amd64-cluster-default # pkg info pkg
Name           : pkg
Version        : 1.10.5_4
Installed on   : Sun Oct 14 17:13:09 2018 UTC
Origin         : ports-mgmt/pkg
Architecture   : FreeBSD:12:amd64
Prefix         : /usr/local
Categories     : ports-mgmt
Licenses       : BSD2CLAUSE
Maintainer     : pkg@FreeBSD.org
WWW            : https://wiki.freebsd.org/pkgng
Comment        : Package manager
Options        :
        DOCS           : on
Shared Libs provided:
Annotations    :
        FreeBSD_version: 1200085
Flat size      : 12.7MiB
Description    :
Package management tool

WWW: https://wiki.freebsd.org/pkgng
root@bob.nyi:/usr/local/poudriere/data/packages/12-amd64-cluster-default # /usr/local/sbin/pkg-static repo /tmp/foo /root/ssl/pkg.key
Creating repository in /tmp/foo: 100%
Packing files for repository:   0%
pkg-static: can't load key from /root/ssl/pkg.key
Packing files for repository: 100%
Comment 1 Glen Barber freebsd_committer 2018-10-15 17:18:04 UTC
Please see the Github freebsd/pkg pull request that resolves this:
Comment 2 Glen Barber freebsd_committer 2018-10-15 19:51:21 UTC
Updated pull request for 1.10.x specifically:
Comment 3 Walter Schwarzenfeld freebsd_triage 2018-10-16 11:00:11 UTC
Should fixed with ports r482214.
Comment 4 Roger Pau Monné freebsd_committer 2018-10-16 15:54:35 UTC
Is there anyway that we could prevent this from happening in the future? Like not updating the front facing package repository if there are critical errors detected in the building phase?
Comment 5 Sean Bruno freebsd_committer 2018-10-16 15:56:46 UTC
(In reply to Roger Pau Monné from comment #4)
This was a side effect of the base openssl upgrade.  I'm unsure how the ports team would have detected this without doing the full upgrade and trying to build the repository.
Comment 6 Roger Pau Monné freebsd_committer 2018-10-16 16:04:16 UTC
(In reply to Sean Bruno from comment #5)
I have to admit I know nothing about the package building infrastructure, but if I understand correctly what happened here is a failure to sign the index in the builders, which I would expect should have caused the update of the front facing repository to fail, leaving it in the state it was previously.
Comment 7 Sean Bruno freebsd_committer 2018-10-16 16:12:53 UTC
(In reply to Roger Pau Monné from comment #6)
This would have happened if the package builders were updated to the openssl update revision, not just the poudriere jails on the package builders AFAIK.  I only ran into this in the freebsd cluster when we attempted to use -current on the host that was building our repositories *and* I updated pkg to the version build in a jail that was at the same revision.
Comment 8 Roger Pau Monné freebsd_committer 2018-10-16 16:31:46 UTC
(In reply to Sean Bruno from comment #7)
As said, I'm afraid I don't really understand how all this infrastructure works, so my reply might be completely wrong.

I would expect the builders to pick the svn updates and build a new set of packages, together with the index and all the needed metadata, and once this is done everything is pushed to the front facing repository for people to consume. In this case there was an error during index generation, which should have halted this process and instead kept the previously working set of packages and metadata in the public repository for clients to consume?
Comment 9 Roger Pau Monné freebsd_committer 2018-10-19 09:34:38 UTC
Could the pkg binary in the mirrors be updated:


This is a build from 11/10 which contains the bug and makes pkg-static completely unusable.
Comment 10 Roger Pau Monné freebsd_committer 2018-10-24 11:34:23 UTC
The long-standing lack of a working pkg-static binary in the package repository has forced Xen to drop the Freebsd tests from the CI:

Comment 11 Glen Barber freebsd_committer 2018-10-24 15:24:54 UTC
It is unclear to me why the timestamp of the latest/Latest/pkg.txz package is seemingly stale.

http://pkg0.nyi.freebsd.org/FreeBSD:12:amd64/latest/Latest/pkg.txz has a timestamp of 2018-Oct-11 01:41.

Can portmgr force a rebuild of this single package to bump it to pkg-1.10.5_5 to get the pkg-static fix?
Comment 12 Antoine Brodin freebsd_committer 2018-10-24 15:27:40 UTC
(In reply to Glen Barber from comment #11)
We can't do it easily this way,  the jail / packages were upgraded to 13.0-CURRENT.
Comment 13 Glen Barber freebsd_committer 2018-10-24 15:46:56 UTC
(In reply to Antoine Brodin from comment #12)
> (In reply to Glen Barber from comment #11)
> We can't do it easily this way,  the jail / packages were upgraded to
> 13.0-CURRENT.

Where does the pkg-static binary in the jail come from?  Is it installed by the broken version on the mirrors?  Or is it baked into the build jails?
Comment 14 Antoine Brodin freebsd_committer 2018-10-24 15:59:55 UTC
(In reply to Glen Barber from comment #13)
The pkg-static binary in the head jails was built on the head jails the last time when pkg version or jail version was bumped.
Comment 15 ddrinnon 2019-01-18 00:48:50 UTC
This bug is also in FreeBSD_12.0-RELEASE