Bug 255029 - portsnap fetch update corruption
Summary: portsnap fetch update corruption
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-ports-bugs (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2021-04-13 09:13 UTC by jakub_lach
Modified: 2021-04-18 20:35 UTC (History)
8 users (show)

See Also:
koobs: maintainer-feedback? (cperciva)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description jakub_lach 2021-04-13 09:13:35 UTC
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Mar 31 10:44:13 CEST 2021 to Tue Apr 13 09:54:15 CEST 2021.
Fetching 2 metadata patches. done.
Applying metadata patches... done.
Fetching 2 metadata files... gunzip: (stdin): unexpected end of file
metadata is corrupt.

Problem is persistent, though number of metadate changes.
Comment 1 jakub_lach 2021-04-13 10:27:33 UTC
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Mar 31 10:44:13 CEST 2021 to Tue Apr 13 11:31:09 CEST 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... /usr/sbin/portsnap: cannot open 916a87227b20c9ec75255dbc45537bfbd06b693aaf80d5ad34c29a9bf991b1ed.gz: No such file or directory
metadata is corrupt.

FreeBSD 13.0-STABLE #0 stable/13-9abc85d17
Comment 2 Colin Percival freebsd_committer 2021-04-13 13:08:36 UTC
If you delete /var/db/portsnap and recreate it, do you still see this issue?
Comment 3 jakub_lach 2021-04-13 13:34:23 UTC
Currently fetching anything from ipv4.aws.portsnap.freebsd.org is extremely slow

Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching public key from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Tue Apr 13 02:29:08 CEST 2021:
338d0da0ff87d35854564889da42ad9556937e0d3b8683  0% of   59 MB 2846  Bps 04h38m
Comment 4 jakub_lach 2021-04-13 15:16:31 UTC
(In reply to jakub_lach from comment #3)

I think the extremely slow connection with ipv4.aws.portsnap.freebsd.org is the culprit itself, as I had the prevopus errors once again. I'm on fiber link.
Comment 5 Steve Wills freebsd_committer 2021-04-13 16:50:14 UTC
FWIW, I happened to notice t his report also:

https://unix.stackexchange.com/questions/644871/change-ports-mirror-in-freebsd
Comment 6 Ed Maste freebsd_committer 2021-04-13 17:53:22 UTC
(In reply to jakub_lach from comment #4)
I suspect the portsnap server is just overloaded due to the large size of the update and the large number of clients updating after the outage.

I was successfully able to `portsnap fetch` and `portsnap update` on a test machine just now. It did take a while to run.

root@freebsd-s-1vcpu-1gb-tor1-01:~ # portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Tue Apr 13 15:24:58 UTC 2021 to Tue Apr 13 16:29:40 UTC 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... 
done.
Fetching 18 patches. 
(18/18) 100.00%  done.                             
done.
Applying patches... 
done.
Fetching 0 new ports or files... done.
Comment 7 jakub_lach 2021-04-13 19:34:17 UTC
(In reply to Ed Maste from comment #6)

As of now, the portsnap fetch/update/extract while successful, corrupted my ports tree. (portmaster is adamant the /usr/ports does not contain ports tree, and I see makefiles missing)
Comment 8 darkart 2021-04-13 21:35:14 UTC
I have also had successful portsnap fetch/update and fetch/extract runs, however /usr/ports/Mk has only three files:

bsd.database.mk
bsd.gnome.mk
bsd.mate.mk

Just ran "portsnap fetch" and "portsnap update", still only have those three /usr/port/Mk/ files.

And to be clear, building ports does not work.
Comment 9 Colin Percival freebsd_committer 2021-04-13 22:41:28 UTC
Confirmed, some bits from the tree are missing from the portsnap builds.  I'm investigating.
Comment 10 Colin Percival freebsd_committer 2021-04-13 23:05:39 UTC
I think I have a fix.  Currently testing; ETA for working portsnap builds 1-2 hours.
Comment 11 Colin Percival freebsd_committer 2021-04-14 00:30:34 UTC
I believe the portsnap mirrors are now serving up intact ports trees.

Please let me know if you see any more issues after this point.
Comment 12 Ed Maste freebsd_committer 2021-04-14 00:32:47 UTC
Looks good for me on a test machine now

root@freebsd-s-1vcpu-1gb-intel-tor1-01:/usr/ports # date
Wed Apr 14 00:31:59 UTC 2021
root@freebsd-s-1vcpu-1gb-intel-tor1-01:/usr/ports # portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Latest snapshot on server matches what we already have.
No updates needed.
root@freebsd-s-1vcpu-1gb-intel-tor1-01:/usr/ports # ls /usr/ports/Mk
bsd.ccache.mk           bsd.licenses.mk         bsd.port.subdir.mk
bsd.commands.mk         bsd.local.mk            bsd.ruby.mk
bsd.default-versions.mk bsd.ocaml.mk            bsd.sanity.mk
bsd.destdir.mk          bsd.octave.mk           bsd.sites.mk
bsd.gcc.mk              bsd.options.desc.mk     bsd.ssp.mk
bsd.gecko.mk            bsd.options.mk          bsd.tex.mk
bsd.gstreamer.mk        bsd.port.mk             bsd.wx.mk
bsd.java.mk             bsd.port.options.mk     Scripts
bsd.ldap.mk             bsd.port.post.mk        Uses
bsd.licenses.db.mk      bsd.port.pre.mk         Wrappers
Comment 13 darkart 2021-04-14 00:51:53 UTC
I'm getting "metadata is corrupt." failures on "portsnap fetch", maybe the mirror I'm hitting has not yet completed its sync?
I'll try again in another few hours.

Latest error:
----- begin -----
% date ; host ipv4.aws.portsnap.freebsd.org ; sudo portsnap fetch ; date
Wed Apr 14 00:48:12 UTC 2021
ipv4.aws.portsnap.freebsd.org is an alias for a8bfe73a85453059b.awsglobalaccelerator.com.
a8bfe73a85453059b.awsglobalaccelerator.com has address 13.248.205.9
a8bfe73a85453059b.awsglobalaccelerator.com has address 76.223.89.23
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Tue Apr 13 21:16:09 UTC 2021 to Wed Apr 14 00:09:41 UTC 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... gunzip: (stdin): unexpected end of file
metadata is corrupt.
Wed Apr 14 00:50:29 UTC 2021
%
----- end ------
Comment 14 Colin Percival freebsd_committer 2021-04-14 01:01:00 UTC
I think the "gunzip: (stdin): unexpected end of file" error is one we've seen before but I don't recall if we ever managed to track it down.

Can you try

1. Setting a different portsnap working directory?  (You might have some corrupted state which is confusing portsnap.)

2. Running with --debug?
Comment 15 darkart 2021-04-14 01:11:15 UTC
Looks like it has resolved itself, ran "sudo portsnap --debug fetch" and it worked fine.
Tried a couple of other hosts that were having the same error, ran without "--debug" and they were able to complete both "portsnap fetch" and "portsnap update" without problems.  The various files in /usr/ports/Mk appear to be restored, and currently building ports, no problems encountered yet.

Thanks much for the fix!
Comment 16 Tatsuki Makino 2021-04-14 04:36:15 UTC
This is just information.
Now it's fine in my environment.
I did the following.

rm -r -f /var/db/portsnap/*
portsnap fetch
portsnap update

To do that more reliably, we might want to delete /usr/ports as well and run portsnap extract.
Comment 17 jakub_lach 2021-04-14 07:42:44 UTC
As of now, I've removed both /usr/ports and /var/db/portsnap, though unfortunately I cannot fetch intact portsnap snapshot yet.
Comment 18 Colin Percival freebsd_committer 2021-04-14 19:11:04 UTC
I think the remaining issue is related to mirror load.  Jakub, I assume you're in Europe?  I just launched two more EC2 instances in Frankfurt to help with the load in that region.
Comment 19 Tatsuki Makino 2021-04-14 20:19:31 UTC
I had this happen after comment #16.

# portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Apr 14 17:11:13 JST 2021 to Thu Apr 15 00:35:58 JST 2021.
Fetching 5 metadata patches. done.
Applying metadata patches... done.
Fetching 5 metadata files... /usr/sbin/portsnap: cannot open b1b1cb6ba5967e44b1ece8d63f7cb3d01b2c03fcf6691129dcb1cd5b8fe6024f.gz: No such file or directory
metadata is corrupt.

However, repeated fetching revived it.

# portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Apr 14 17:11:13 JST 2021 to Thu Apr 15 00:35:58 JST 2021.
Fetching 5 metadata patches... done.
Applying metadata patches... done.
Fetching 0 metadata files... done.
Fetching 48 patches. 
(48/48) 100.00%  done.                                
done.
Applying patches... 
done.
Fetching 1 new ports or files... /usr/sbin/portsnap: cannot open 9297cc3ab6173c7a7a1afe8103ffc52616f7652a3200e2965c275d2b8c0915fe.gz: No such file or directory
snapshot is corrupt.
# portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Apr 14 17:11:13 JST 2021 to Thu Apr 15 00:35:58 JST 2021.
Fetching 0 metadata patches. done.
Applying metadata patches... done.
Fetching 0 metadata files... done.
Fetching 0 patches. 
 done. 
done.
Applying patches... 
done.
Fetching 1 new ports or files... done.
# portsnap fetch
Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Latest snapshot on server matches what we already have.
No updates needed.

What I think is
/var/db/portsnap directory should not be removed more than once. If you can successfully retrieve the first large snapshot, don't delete it again.
Pause and fetch until you get "No updates needed." by portsnap cron or sleep `jot -r 1 180 360` ; portsnap fetch
Comment 20 jakub_lach 2021-04-14 21:49:50 UTC
(In reply to Colin Percival from comment #18)

Yes, I think so too; since my last message I've got fresh ports tree extracted and have updated some ports already, though latest incremental portsnap update is corrupted once again (due to load I assume).
Comment 21 Kai Gallasch 2021-04-14 23:07:00 UTC
portsnap fetch from a server, located in Germany..

$ host ipv4.aws.portsnap.freebsd.org
ipv4.aws.portsnap.freebsd.org is an alias for a8bfe73a85453059b.awsglobalaccelerator.com.
a8bfe73a85453059b.awsglobalaccelerator.com has address 13.248.205.9
a8bfe73a85453059b.awsglobalaccelerator.com has address 76.223.89.23

Emptying /var/db/portsnap ..

# date
Thu Apr 15 00:54:46 CEST 2021

$ portsnap fetch

Looking up portsnap.FreeBSD.org mirrors... 4 mirrors found.
Fetching public key from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Wed Apr 14 02:09:41 CEST 2021:
0b344cd7869783c8d72504314f54f27b9ac6225ce21fae          88 MB   69 MBps    02s
Extracting snapshot... done.
Verifying snapshot integrity... done.
Fetching snapshot tag from ipv4.aws.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Wed Apr 14 02:09:41 CEST 2021 to Wed Apr 14 23:59:00 CEST 2021.
Fetching 5 metadata patches.. done.
Applying metadata patches... done.
Fetching 3 metadata files... /usr/sbin/portsnap: cannot open 380383766c9c9683aa1eb29a6b058cab62243c00e0d3ddc1d1c666a79bacef92.gz: No such file or directory
metadata is corrupt.
Comment 22 jakub_lach 2021-04-14 23:14:20 UTC
(In reply to Kai Gallasch from comment #21)

Fetching non corrupted update is hit and miss, though still possible (once for ten tries, I would say.)
Comment 23 Colin Percival freebsd_committer 2021-04-14 23:18:31 UTC
Ok, more mirrors coming online shortly...
Comment 24 jakub_lach 2021-04-15 08:52:19 UTC
(In reply to Colin Percival from comment #23)

Currently the mirrors are more or less as responsive as they were normally, maybe fetching an update a little slower. Thanks!
Comment 25 Ed Maste freebsd_committer 2021-04-18 17:10:16 UTC
This can now be closed?
Comment 26 jakub_lach 2021-04-18 17:12:03 UTC
(In reply to Ed Maste from comment #25)
I think so.
Comment 27 Kai Gallasch 2021-04-18 19:25:47 UTC
(In reply to Ed Maste from comment #25)

I have not seen any problems with portsnap during the last days.
Comment 28 Ed Maste freebsd_committer 2021-04-18 20:35:10 UTC
Working now after the git conversion; the load spike due to the large $FreeBSD$-related change is now in the past. Sorry for the outage.