Bug 254484

Summary: Repication of files put into freefall.freebsd.org/public_distfiles takes a very long time
Product: Services Reporter: Yuri Victorovich <yuri>
Component: Core InfrastructureAssignee: Philip Paeps <philip>
Status: Closed Works As Intended    
Severity: Affects Only Me CC: clusteradm, philip
Priority: ---    
Version: unspecified   
Hardware: Any   
OS: Any   

Description Yuri Victorovich freebsd_committer freebsd_triage 2021-03-22 19:22:37 UTC 
In order to have a custom distfile for a port port maintainers put it into public_distfiles on freefall.freebsd.org.

But then it takes up to an hour or longer for this file to be available in http://distcache.FreeBSD.org/local-distfiles/{user}/, http://distcache.us-east.FreeBSD.org/local-distfiles/{user}/, http://distcache.us-west.FreeBSD.org/local-distfiles/{user}/, etc.

Shouldn't they be replicated on-file-change?

Currently this process can affect some users which wait for a hot-fix, and also delays package building of newly changed ports.
Comment 1 Philip Paeps freebsd_committer freebsd_triage 2021-03-23 01:10:40 UTC 
An hour is not a very long time...

I just checked the mechanics: pkg-master fetches local-distfiles from freefall once an hour.  The mirrors sync from pkg-master four times an hour.

I'll change pkg-master to fetch four times an hour too.
Comment 2 Philip Paeps freebsd_committer freebsd_triage 2021-03-24 03:44:39 UTC 
pkg-master now pulls from freefall four times an hour (xx:07, xx:22, xx:37 and xx:52).
Comment 3 Yuri Victorovich freebsd_committer freebsd_triage 2021-03-24 03:56:08 UTC 
(In reply to Philip Paeps from comment #2)

Thank you!
Comment 4 Yuri Victorovich freebsd_committer freebsd_triage 2021-05-02 17:40:54 UTC 
The schedule xx:07, xx:22, xx:37 and xx:52 isn't accurate because when I've put files at 10:11AM PST, and now at 10:40AM PST files are still not replicated.

The problem still exists: 30+ minutes is too long.
Comment 5 Philip Paeps freebsd_committer freebsd_triage 2021-05-07 08:01:53 UTC 
As far as I can tell, replication is taking place on the configured schedule.  Which mirror were you checking?

Note that it is not possible to absolutely guarantee that files will replicate everywhere in under 30 minutes.  Several factors affect replication.
Comment 6 Philip Paeps freebsd_committer freebsd_triage 2021-08-31 06:26:51 UTC 
As far as I can tell, this is working as intended.  I see the syncs happening four times an hour anyway.  As mentioned, there really is no way to make this "instant" without fundamentally changing how pkgsync works.