Bug 244549 - ports-mgmt/pkg(-devel): repository FreeBSD contains packages with wrong ABI: FreeBSD:1x.0:amd64
Summary: ports-mgmt/pkg(-devel): repository FreeBSD contains packages with wrong ABI: ...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Package Infrastructure (show other bugs)
Version: Latest
Hardware: Any Any
: Normal Affects Many People
Assignee: Port Management Team
URL:
Keywords: regression
: 244578 (view as bug list)
Depends on:
Blocks:
 
Reported: 2020-03-02 03:42 UTC by Neel Chauhan
Modified: 2020-03-27 17:25 UTC (History)
37 users (show)

See Also:
bugzilla: maintainer-feedback? (portmgr)


Attachments
Build Status of Packages (6.69 KB, text/plain)
2020-03-06 12:19 UTC, mfv
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Neel Chauhan freebsd_committer freebsd_triage 2020-03-02 03:42:36 UTC
I am running FreeBSD 13-CURRENT and I am getting this error on a "pkg update":

pkg: wrong architecture: FreeBSD:13.0:amd64 instead of FreeBSD:13:amd64
pkg: repository FreeBSD contains packages with wrong ABI: FreeBSD:13.0:amd64

I am running FreeBSD 13-CURRENT r358510.
Comment 1 Kyle Evans freebsd_committer freebsd_triage 2020-03-02 03:50:56 UTC
Hi,

pkg-devel's last update included one that started adding the minor version to the ABI string. I think this must have been an accident, as more coordination is needed to pull off a change like that. I have an open Pull Request for this exact issue: https://github.com/freebsd/pkg/pull/1817

We'll need a new pkg-devel release after that and a rebuild of any packages with the wrong ABI, since IIRC that's in each pkg's metadata
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2020-03-02 05:52:08 UTC
^Triage: Issue is with libpkg (in pkg-devel) on the package build cluster.

Upstream now has a PR [1], pending merge.

With confirmation that the change is OK, portmgr may backport the change if a new release will take time.

[1]  https://github.com/freebsd/pkg/pull/1817
Comment 3 Kubilay Kocak freebsd_committer freebsd_triage 2020-03-02 05:53:43 UTC
Issue is in ports-mgmt/-pkg-devel, but actually (also) a package infrastructure issue. Reclassify
Comment 4 Kyle Evans freebsd_committer freebsd_triage 2020-03-02 18:13:14 UTC
antoine@ flagged pkg-devel as FORBIDDEN in ports r527609; the upstream fix to pkg has been committed.

This turns out to not actually be an infrastructure problem- pkg-devel will use itself to create its pkg. We should probably re-evaluate whether we want to keep doing this if WITH_PKG != devel (possible only if we're PACKAGE_BUILDING) to avoid potential issues like this -- but that's all logic contained in pkg-devel/Makefile.
Comment 5 Pietro Cerutti freebsd_committer freebsd_triage 2020-03-03 07:33:51 UTC
Happening to me in a 12.1-RELEASE jail, too:

pkg: wrong architecture: FreeBSD:12.0:amd64 instead of FreeBSD:12:amd64
pkg: repository FreeBSD contains packages with wrong ABI: FreeBSD:12.0:amd64
Comment 6 mfv 2020-03-03 14:17:06 UTC
I encountered the same issue when doing a routine pkg update yesterday (2020-03-02):

> sudo pkg update
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB 322.7kB/s    00:20    
Processing entries:  63%
pkg: wrong architecture: FreeBSD:12.0:amd64 instead of FreeBSD:12:amd64
pkg: repository FreeBSD contains packages with wrong ABI: FreeBSD:12.0:amd64
Processing entries: 100%
Unable to update repository FreeBSD
Error updating repositories!
Exit 70
Comment 7 Simon Ser 2020-03-03 14:19:51 UTC
SourceHut is also hitting this bug on the builds.sr.ht CI system. See e.g. https://builds.sr.ht/~andrewrk/job/162187

This is disturbing CI builds for projects like Mutt, Nim, Zig, Sway.
Comment 8 Ed Maste freebsd_committer freebsd_triage 2020-03-03 14:44:13 UTC
This also took out Cirrus-CI across some number of projects - see e.g. https://github.com/curl/curl/issues/5028 for curl.
Comment 9 Alexander V. Chernikov freebsd_committer freebsd_triage 2020-03-03 19:56:49 UTC
Could you guys consider providing some update on the issue?

https://github.com/freebsd/pkg/pull/1817 has been merged, but it is not exactly clear (at least to me), what are the next steps towards the resolution. Are there any blockers remaining? Any help required?
Comment 10 Renato Botelho freebsd_committer freebsd_triage 2020-03-04 12:17:19 UTC
(In reply to Alexander V. Chernikov from comment #9)

A new release of pkg-devel was done - 1.13.99.3 - https://github.com/freebsd/pkg/commit/93c4c82e29b0022480de930501e8ac62d9527e60

It was committed to ports tree few hours ago https://svnweb.freebsd.org/ports?view=revision&revision=527668

Now I believe we need to wait for a new repository build to get it fixed.
Comment 11 mfv 2020-03-04 14:07:26 UTC
Will this commit also repair the same issue with ports-mgmt/pkg?

With thanks
Comment 12 Kyle Evans freebsd_committer freebsd_triage 2020-03-04 14:09:11 UTC
(In reply to mfv from comment #11)

ports-mgmt/pkg never had the offending commit and will have both it and the fix when it later gets updated. =-) pkg updates are far and few between, though.
Comment 13 Baptiste Daroussin freebsd_committer freebsd_triage 2020-03-04 14:11:25 UTC
So to clarify, pkg-devel was the issue, when it has been marked as FORBIDDEN it was enough and the problem was solved. What is needed is to wait for the next cycle of build to push the new build without pkg-devel. In the mean time I have push a fixed version of pkg-devel, but on the ports side everything was already back to normal.

Build cycles takes time, that is all.

Note this was not affecting at all the quarterly branches.
Comment 14 mfv 2020-03-04 14:20:35 UTC
Hello Baptiste,

Thanks for the bug fix and clarification.  Much appreciated.

Cheers ...
Comment 15 Kyle Evans freebsd_committer freebsd_triage 2020-03-04 19:09:00 UTC
*** Bug 244578 has been marked as a duplicate of this bug. ***
Comment 16 t.wienert 2020-03-04 23:29:27 UTC
(In reply to Baptiste Daroussin from comment #13)

Dear Baptiste,

I am new to FreeBSD so please bear with me. You mentioned that build cycles takes time. How long does such an build cycle usually take?
According to your github commit, the check for 13 current did not pass. This is because the build cycle is just not finished?

Thank you in advance and for your patience.
kind regards
Comment 17 t.wienert 2020-03-05 08:07:58 UTC
it's working now.
Comment 18 bob 2020-03-05 13:35:12 UTC
It worked for me on only one of my 12.1-stable systems and failed on 5 others
that I tried it on.
Comment 19 Daniel Ebdrup Jensen freebsd_committer freebsd_triage 2020-03-05 14:55:21 UTC
For reference, a full build-cycle of the 32000 or so ports, takes about 65-80 hours and they're built roughly every fortnight.
Between each full build, there is a set of intermediate builds which build the ports that have changed since the last completed build (unless a previous build is still running).

These numbers are from the server that builds packages for 12.1 (as of now, 12.0 before), and they naturally greatly depend on the hardware and configuration setup.

This also doesn't take into account the synchronization of the mirrors, which naturally also takes some time since the full set of packages built from ports takes up quite a bit of space.
Comment 20 Theron Tarigo 2020-03-05 19:26:51 UTC
Just to be sure, this really is the bug entry for "pkg: repository FreeBSD contains packages with wrong ABI: FreeBSD:12.0:amd64" (Then it should be renamed to make this clear).  Otherwise, Bug 244578 was marked as duplicate in error.
Comment 21 bob 2020-03-05 20:12:11 UTC
Strange, most of my 12.0-stable systems are updating the pkg repository ok now but the one that I just installed last night from a recent 12.1 snapshot is still failing.
Comment 22 mike 2020-03-05 20:14:24 UTC
(In reply to bob from comment #21)

In /etc/pkg/FreeBSD.conf, are you using "latest" or "quarterly" ?  Quarterly will work, latest should fail still

ie
url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
vs
url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
Comment 23 Kyle Evans freebsd_committer freebsd_triage 2020-03-05 20:15:47 UTC
(In reply to Theron Tarigo from comment #20)

The bug is version independent- you can find it with any of 11.x, 12.x, and 13.x and amd64/i386/other -- I'm not sure there's any way to make it more clear, 95% of the error message is still identical.
Comment 24 Sergey A. Osokin freebsd_committer freebsd_triage 2020-03-05 20:30:27 UTC
(In reply to Kyle Evans from comment #23)

Confirmed, the issue still exists on my 12.1-STABLE:
% sudo pkg update -f
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB 268.9kB/s    00:24    
Processing entries:  63%
pkg: wrong architecture: FreeBSD:12.0:amd64 instead of FreeBSD:12:amd64
pkg: repository FreeBSD contains packages with wrong ABI: FreeBSD:12.0:amd64
Processing entries: 100%
Unable to update repository FreeBSD
Error updating repositories!

Is there any chance to remove the buggy record for pkg-devel?
Comment 25 Alexander Vereeken 2020-03-05 20:32:56 UTC
Yep, i also still get the error with 12.1-Stable r358331
Comment 26 Sergey A. Osokin freebsd_committer freebsd_triage 2020-03-05 20:58:35 UTC
(In reply to Sergey A. Osokin from comment #24)
The packagesite.txz,
http://pkg.freebsd.org/FreeBSD:12:amd64/latest/packagesite.txz
contains the packagesite.yaml file:

% grep "FreeBSD:12.0" packagesite.yaml
{"name":"pkg-devel","origin":"ports-mgmt/pkg-devel","version":"1.13.99.2","comment":"Package manager","maintainer":"pkg@FreeBSD.org","www":"https://wiki.freebsd.org/pkgng","abi":"FreeBSD:12.0:amd64","arch":"freebsd:12.0:x86:64","prefix":"/usr/local","sum":"877847a3a97efe9809f76e59810f541a0e8e2a3ad9a44c5f6d7aec3c19490de4","flatsize":40245881,"path":"All/pkg-devel-1.13.99.2.txz","repopath":"All/pkg-devel-1.13.99.2.txz","licenselogic":"single","licenses":["BSD2CLAUSE"],"pkgsize":9048340,"desc":"Package management tool\n\nWWW: https://wiki.freebsd.org/pkgng","categories":["ports-mgmt"],"shlibs_provided":["libpkg.so.4"],"options":{"DOCS":"on"},"annotations":{"FreeBSD_version":"1200086"},"messages":[{"message":"If you are upgrading from the old package format, first run:\n\n  # pkg2ng","type":"install"}]}

I know it's hacky way, however it's possible to remove that buggy record from the YAML file and replace the archive with the new one and then upload to the pkg site.  Am I missing something?
Comment 27 Sergey A. Osokin freebsd_committer freebsd_triage 2020-03-05 21:21:57 UTC
(In reply to Sergey A. Osokin from comment #26)

Here's the workaround.

1. Setup nginx web server with the following configuration file:

server {
     listen 8080;

     location / {
         location / {
             root   /usr/local/www/nginx;
             autoindex on;
        }
     }
}

2. Create the directory for FreeBSD package site staff:

# mkdir -p /usr/local/www/nginx/FreeBSD:12:amd64/latest

3. Download all files for FreeBSD:12:amd64 architecture from 
http://pkg.freebsd.org/FreeBSD:12:amd64/latest/ to the directory from 2.

4. Unpack packagesite.txz and remove "pkg-devel" record from the packagesite.yaml file.

5. Create an archive with updated packagesite.yaml.

6. Backup /etc/pkg/FreeBSD.conf and create new one:

FreeBSD: {
  url: "pkg+http://localhost:8080/${ABI}/latest",
  mirror_type: "srv",
  enabled: yes
}

7. Run "pkg update -f"

8. Profit!
Comment 28 Sergey A. Osokin freebsd_committer freebsd_triage 2020-03-05 21:40:46 UTC
(In reply to Sergey A. Osokin from comment #27)

Let me correct myself a bit, since we need to continue get packages from the original FreeBSD pkg site the nginx' configuration file should be updated:

server {
     listen 8080;

     location /FreeBSD:12:amd64/latest/Latest {
          proxy_pass http://pkg.freebsd.org;
     }

     location /FreeBSD:12:amd64/latest/All {
          proxy_pass http://pkg.freebsd.org;
     }

     location / {
         location / {
             root   /usr/local/www/nginx;
             autoindex on;
        }
     }
}
Comment 29 bob 2020-03-05 22:34:31 UTC
Do we have any idea/estimate of when to expect this to be resolved? I can wait till it's resolved but I'd like to have an idea of how long I'll have to wait.
Comment 30 Theron Tarigo 2020-03-06 00:03:31 UTC
(In reply to Kyle Evans from comment #23)
Fair enough.

(In reply to Sergey A. Osokin from comment #28)
So all we are waiting for a full build cycle for is to have a single index file have a bad entry disappear?  If it's this easy to fix, whoever has the necessary access should take emergency action and manually upload the fixed file.
Comment 31 Sergey A. Osokin freebsd_committer freebsd_triage 2020-03-06 00:50:39 UTC
(In reply to Theron Tarigo from comment #30)

https://pkg-status.freebsd.org/ shows the build for FreeBSD:12:amd64 is finished ~ 14 hours ago, so my guess we're waiting until all mirrors receive new packages and meta data.  I hope the issue will be resolved soon.
Comment 32 Neel Chauhan freebsd_committer freebsd_triage 2020-03-06 01:19:16 UTC
I don't have an issue anymore, same 13-CURRENT r358510. Thank you!
Comment 33 Sergey A. Osokin freebsd_committer freebsd_triage 2020-03-06 02:50:57 UTC
(In reply to Neel Chauhan from comment #32)

Thanks for the confirmation, Neel.

FreeBSD:13:i386 and FreeBSD:13:amd64 package repositories have been updated recently, so the issue has been resolved for those two.

Let's wait for others.
Comment 34 Chris 2020-03-06 07:04:41 UTC
(In reply to Kyle Evans from comment #23)

Most of the comments here seem to be about FreeBSD 13 and 12.1. I updated an iocage jail from 11.2 to 11.3, and then tried to update the packages on it and that is failing:

root@subsystem_1:/ # pkg update
Updating FreeBSD repository catalogue...
pkg: repository meta has wrong version 2
pkg: Repository FreeBSD load error: meta cannot be loaded No error: 0
[subsystem_1] Fetching meta.txz: 100%    916 B   0.9kB/s    00:01
pkg: repository meta has wrong version 2
repository FreeBSD has no meta file, using default settings
[subsystem_1] Fetching packagesite.txz: 100%    6 MiB   3.3MB/s    00:02
pkg: repository meta has wrong version 2
pkg: Repository FreeBSD load error: meta cannot be loaded No error: 0
Unable to open created repository FreeBSD
Unable to update repository FreeBSD
Error updating repositories!
root@subsystem_1:/ #

Is that the same bug as here, or do I need to look further into my issue elsewhere? I have a couple other jails already on 11.3 and they don't seem to be failing, just this newest one I coincidentally upgraded after this issue started.
Comment 35 Victor Sudakov 2020-03-06 07:26:42 UTC
Sorry for using/reporting 12.0, but:

# freebsd-version
12.0-RELEASE-p3
# pkg update -f
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    916 B   0.9kB/s    00:01    
Fetching packagesite.txz: 100%    6 MiB   6.5MB/s    00:01    
Processing entries:  63%
pkg: wrong architecture: FreeBSD:12.0:amd64 instead of FreeBSD:12:amd64
pkg: repository FreeBSD contains packages with wrong ABI: FreeBSD:12.0:amd64
Processing entries: 100%
Unable to update repository FreeBSD
Error updating repositories!
# 
# pkg -vv | grep -A10 ^Rep
Repositories:
  FreeBSD: { 
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/latest",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
Comment 36 guido 2020-03-06 07:37:31 UTC
(In reply to Sergey A. Osokin from comment #31)
Exactly where do you see this? the only entry referring 121amd64 ended its build on Tue, 03 Mar 2020 01:06:46 GMT

I do not see a newer entry...
Comment 37 per 2020-03-06 09:39:24 UTC
(In reply to guido from comment #36)

As far as I can see, "Tue, 03 Mar 2020 01:06:46 GMT" is the *start* time for the *quarterly* build for FreeBSD 12 - and you have to go to the 2nd page of "Package Builds" to find the *latest* build for 12, which is actually called "default" in the "Ports" column. This build is also finished (as already reported in comment #31) - but the start time is "Mon, 02 Mar 2020 06:55:59 GMT", which would seem to be before any corrective action had been taken...

I.e. it seems to me that the completed build for 12-latest is also broken, and we will have to wait for *another* build for that - am I missing something?

FWIW, latest is still broken for my 12.1-STABLE.
Comment 38 mfv 2020-03-06 12:19:31 UTC
Created attachment 212183 [details]
Build Status of Packages
Comment 39 mfv 2020-03-06 12:26:00 UTC
Hello,

Here are my comments for the Perl code I previosely sent.  This is my fist time sending an attachment and it did not work as expected.  This code is meant to show the status of the package building process, including the completion time which is not readily available from a website.

-----------------

Here is some data that may be used to resolve the timing of the build and
syncing process of the packages for r12.

1. This website shows the build status of r12.0 and r12.1:

    http://beefy6.nyi.freebsd.org/#latest_builds

2. The last two lines of this website shows the date and time of the latest
builds for r12:

   http://pkg.freebsd.org/FreeBSD:12:amd64/latest/ 

As of the time of composing this message the date and time are as follows:
    meta.txz	            916	    2020-Mar-02 06:01
    packagesite.txz	    6453132	    2020-Mar-02 06:01

These two lines show that the latest build is four days old while the first
website shows that the build has finished.  This means that the building
process is complete but that the repositories are still being synced due to the
the large number of packages.

The package building process actually ended on 2020-03-05 at 06:27.  This piece
of data is not immediately available on the websites.  Attached is a crude
piece of Perl code that shows the build status of a particular release,
including the time of completion.

Thus when the second website shows a date of 2020-Mar-05 for meta.txz and
packagesite.txz it means that the syncing of the repositories is complete AND
the updated packages are ready for downloading.

I personally wish there was ONE website that easily shows when the build process
was completed AND also shows when the synching process is finished.  If there
is such a site please let me know.

Cheers ...
Comment 40 Bernhard Froehlich freebsd_committer freebsd_triage 2020-03-06 13:36:14 UTC
(In reply to mfv from comment #39)
Your analysis seems to be correct but the end date seems to be different to what I calculated. Watching the build jobs is slightly simpler that way:

https://pkg-status.freebsd.org/builds?type=package

Filter for the FreeBSD version and platform you are interested in (example 121amd64)

- Started (UTC): Tue, 03 Mar 2020 01:06:46 GMT
- Elapsed: 69:59:48
- Status: stopped:done:

Tue, 03 Mar 2020 01:06:46 GMT + 69:59:48 = Thu, 05 Mar 2020 23:06:34
Comment 41 mfv 2020-03-06 15:49:10 UTC
Hello Bernhard,

Yes, your calculation is simplier.  The difference between your data and mine is due to:

 - Your using data for the quarterly builds (#527662) and using UTC
 - I used data for the default builds (#527609) and used local time

I have since changed the Perl script submitted at Comment 38 to show UTC for default builds.  Here is the output of this script:

----------------------------------------------------
     Build Status as of Fri 2020-03-06 15:21 UTC
----------------------------------------------------
 Status:  stopped:done       Queued:  32,533   100%
 BuildNo: 527609             Built:   31,440    97%
 Started: 2020-03-02 06:55   Failed:      34     0%
 Elapsed: 3d 04h 31m         Skipped:     81     0%
 Pkgs/Hr: 411                Ignored:    978     3%
 Ended:   2020-03-05 11:27   Pending:      0     0%
----------------------------------------------------

Much of this information is available at the website you specified but I find this script useful to run as part of my daily update and maintenance routines.

Cheers ...
Comment 42 Michael Gmelin freebsd_committer freebsd_triage 2020-03-07 13:13:02 UTC
(In reply to Sergey A. Osokin from comment #27)

For my purposes (install and upgrade packages manually):

  ALTABI=FreeBSD:12.0:amd64 pkg update -f

was enough of a workaround.

See

https://lists.freebsd.org/pipermail/freebsd-current/2020-March/075448.html

for details.
Comment 43 mfv 2020-03-07 13:30:23 UTC
Hello,

All is well updating the package repositories for r12.1.  It took nearly 45 hours to sync all the repositories with the updates.

Cheers ...
Comment 44 Michael Gmelin freebsd_committer freebsd_triage 2020-03-07 14:08:24 UTC
(In reply to mfv from comment #43)

I can confirm this here:

  # time pkg update -f
  Updating FreeBSD repository catalogue...
  Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
  Fetching packagesite.txz: 100%    6 MiB 129.0kB/s    00:50    
  Processing entries: 100%
  FreeBSD repository update completed. 31344 packages processed.
  All repositories are up to date.

  real    1m23.683s
  user    0m6.219s
  sys     0m0.349s

Let's hope setting up the new European mirror will be done soon, 83 seconds is just unacceptable.

For comparison, when manually forcing pkg0.nyi.freebsd.org:

  # time pkg update -f
  Updating FreeBSD repository catalogue...
  Fetching meta.conf: 100%    163 B   0.2kB/s    00:01    
  Fetching packagesite.txz: 100%    6 MiB   2.2MB/s    00:03    
  Processing entries: 100%
  FreeBSD repository update completed. 31344 packages processed.
  All repositories are up to date.

  real    0m10.516s
  user    0m6.372s
  sys     0m0.213s

Wouldn't it make sense, to announce such problems (and their solution) in an official project space, e.g., announcement mailing list, freebsd.org webpage, project twitter account...?
Comment 45 Sergey A. Osokin freebsd_committer freebsd_triage 2020-03-07 15:35:48 UTC
(In reply to Michael Gmelin from comment #42)
Cool, thanks for sharing, Michael!

Also, I'd like to confirm the issue has been fixed here for my laptop with FreeBSD 12.1-STABLE.

Thanks everybody!
Comment 46 Thierry Thomas freebsd_committer freebsd_triage 2020-03-07 16:45:55 UTC
On my 12.1-STABLE machine, still get the following error:
pkg upgrade 

...

Proceed with this action? [y/N]: y
[1/85] Fetching xwayland-1.20.7_1,1.txz: 100%  844 KiB 144.1kB/s    00:06
pkg: cached package xwayland-1.20.7_1,1: size mismatch, fetching from remote
[2/85] Fetching xwayland-1.20.7_1,1.txz: 100%  844 KiB  96.0kB/s    00:09
pkg: cached package xwayland-1.20.7_1,1: size mismatch, cannot continue
Consider running 'pkg update -f'

Is it a related error or another one?

Thanks!
Comment 47 Alexander Vereeken 2020-03-07 16:50:40 UTC
Normally it should be not related, you did already pkg update -f?(In reply to Thierry Thomas from comment #46)

Normally it should be not related because i was able to update and upgrade today (EU), you did already pkg update -f?
Comment 48 Thierry Thomas freebsd_committer freebsd_triage 2020-03-07 17:03:33 UTC
(In reply to Alexander Vereeken from comment #47)
Yes, I tried `pkg update -f' and `pkg clean', without success.
Comment 49 Thierry Thomas freebsd_committer freebsd_triage 2020-03-07 18:02:13 UTC
(In reply to Thierry Thomas from comment #48)
I just did it again, and now it's fixed!
Comment 50 Steve Wills freebsd_committer freebsd_triage 2020-03-27 17:25:52 UTC
Issue is fixed, closing.