Bug 226626 - [request] addition of rsync service to pkg.freebsd.org servers
Summary: [request] addition of rsync service to pkg.freebsd.org servers
Status: Open
Alias: None
Product: Services
Classification: Unclassified
Component: Core Infrastructure (show other bugs)
Version: unspecified
Hardware: Any Any
: --- Affects Many People
Assignee: clusteradm
URL:
Keywords: feature
: 227170 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-03-15 10:23 UTC by ev
Modified: 2020-01-12 22:37 UTC (History)
7 users (show)

See Also:


Attachments
nginx-pkg-mirror.conf (1.46 KB, text/plain)
2020-01-12 04:21 UTC, marek
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description ev 2018-03-15 10:23:19 UTC
To create a local mirror I download all packages pkg.freebsd.org via HTTP. But this is not convenient and creates an additional load on the internet channel.

I would like to download packages via rsync. Can consider this possibility?
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2018-03-15 14:42:57 UTC
Reclassify.
Comment 2 ev 2018-11-18 14:27:08 UTC
Are there any news on this request?
Maybe ned help to realize this?
Comment 3 Li-Wen Hsu freebsd_committer 2019-11-13 10:23:10 UTC
Current the pkg build/mirror infrastructure might not have a good way to keep the satellite rsync mirrors updated. We're in the middle of improving these. Please stay tuned.
Comment 4 Li-Wen Hsu freebsd_committer 2019-11-13 10:27:27 UTC
*** Bug 227170 has been marked as a duplicate of this bug. ***
Comment 5 Li-Wen Hsu freebsd_committer 2019-11-13 10:29:03 UTC
bugs 227170 is marked as duplicate of this, but it has one additional suggestion: open HTTP list to let public download. Fold that request to this ticket.
Comment 6 Mateusz Piotrowski freebsd_committer 2019-11-13 14:02:43 UTC
It would be very nice to have it.
Comment 7 ev 2019-11-13 15:31:24 UTC
Updates are currently available on http://update3.freebsd.org
This is an undocumented feature, I hope it will not disappear at present time ;)

I periodically download updates via HTTP (their size is not huge and they do not come out often).
But with each update I have to download it all out again :(
Comment 8 ev 2019-11-25 18:08:41 UTC
May be somebody have any news about this is feature request?
Comment 9 Mateusz Piotrowski freebsd_committer 2019-12-09 10:06:37 UTC
(In reply to Li-Wen Hsu from comment #3)

> Current the pkg build/mirror infrastructure might not have a good way to keep the satellite rsync mirrors updated. We're in the middle of improving these. Please stay tuned.

Could you share a little bit more details? Maybe I could help with it. :)
Comment 10 ev 2020-01-05 21:50:11 UTC
(In reply to Li-Wen Hsu from comment #3)
May be somebody have any news about this is feature request?
Comment 11 Ryan Steinmetz freebsd_committer freebsd_triage 2020-01-11 15:35:23 UTC
This isn't something that we will want to offer as a service as it introduces a ton of risks/issues for both consumers and us on the infrastructure side.
It is far too easy to end up with an inconsistent mirror and the additional IO load that will be placed on us isn't going to be great.

I think a more appropriate solution would be a caching proxy (perhaps using nginx) that people could deploy onsite.
Comment 12 ev 2020-01-11 17:01:08 UTC
> introduces a ton of risks/issues for both consumers and
> us on the infrastructure side.
How can creation of a local mirror affect on infrastructure of FreeBSD? Only in case of change the address of the consumer. But that situation is similar to installation of the proxy by the consumer.

> It is far too easy to end up with an inconsistent mirror
It is too easy to end up with all data removal using ROOT rights, but this does not give a reason to disable ROOT in base release for all consumers :)

> additional IO load that will be placed on us isn't going to be great.
At the moment, downloading the repository takes about a day. At the same time, the internet channel is not fully loaded, which indicates a restriction on the part of freefbs.org servers.

There are places where it’s still bad with the internet and it’s easier for me to send SSD once a quarter, and not forcing them to pay for an expensive satellite channel.

Also see bug #239593 - another case of using a local mirror.

> I think a more appropriate solution would be a caching proxy
> (perhaps using nginx)
So that the proxy can issue a file, first I need to download this file. If I downloaded it, I do not need a proxy - I already posted it to the local repository.

Considering many people, it will soon be easier to create an alternative mirror with rsync support for everyone. But I want it to be on the base site, and not on the alternative.
Comment 13 Ryan Steinmetz freebsd_committer freebsd_triage 2020-01-11 18:37:09 UTC
There's some of this on the hardware/infrastructure side that you'll just need to push the "I believe" button with as I'm not going to get into it.

The main rationale I can think of to have a cache/local copy of stuff is for environments where you have multiple users downloading the same thing over and over.  I'm not sure it makes much sense when you've got a single user.

There's nothing that would stop you from building/maintaining your own internal package repository and distributing/rsyncing/whatever that as you see fit.

It's very common for sites with large installations to either cache content or to build their own as they want the customization.

Still, rsync for the package repo remains something we are not going to provide at this point in time.
Comment 14 ev 2020-01-11 20:35:35 UTC
> just need to push the "I believe" button with as I'm not going to get into it.
Great argument, I will take for use :)

> The main rationale I can think
Reasons:
- fastests creates mirror
- internet fees reduction
- use mirrors in isolated network (from internet)

> There's nothing that would stop you from building/maintaining your own
> internal package repository 
Already wrote - download restriction.
Comment 15 Ryan Steinmetz freebsd_committer freebsd_triage 2020-01-11 20:53:44 UTC
A caching proxy addresses these desires without the issues associated with mirroring the entire package repository as a service.

I think serving packages to sites without internet or limited connectivity is more of an edge case than a typical use case and would need a separate solution.

We'll investigate publishing a suitable nginx configuration to help with this.
Comment 16 ev 2020-01-11 21:09:22 UTC
> A caching proxy addresses these desires without the issues associated with
> mirroring the entire package repository as a service.
A caching proxy not addresses these desires.

> I think serving packages to sites without internet or limited connectivity
> is  more of an edge case than a typical use case
Yes, but does not negate the need for this.

> would need a separate solution.
rsync is great for such a solution.

I can write a python script to create an alternative mirror (from HTTP) and completely configure the server.
Maybe you have the opportunity to allocate for this service VDS/jail with domain name pkg-rsync.freebsd.org (for example) or only public IP-address?
Comment 17 marek 2020-01-12 04:21:14 UTC
Created attachment 210649 [details]
nginx-pkg-mirror.conf
Comment 18 marek 2020-01-12 04:22:00 UTC
(In reply to Ryan Steinmetz from comment #11)

The Nginx cache for FreeBSD packages sounds good for me. This is an interesting idea so I have created the initial configuration for Nginx that caches FreeBSD packages.

Please check the attachment with the Nginx configuration and let me know if anything needs fine-tuning.
Comment 19 Jamie Landeg-Jones 2020-01-12 16:37:36 UTC
(In reply to Ryan Steinmetz from comment #13)

> Still, rsync for the package repo remains something we are not going to provide at this point in time.

Will rsync ever be available for src synchronisation? 'svn co' seems overkill just to sync the local copy of /usr/src, and then hope local changes don't cause inconsistencies!
Comment 20 Ryan Steinmetz freebsd_committer freebsd_triage 2020-01-12 17:00:55 UTC
rsync isn't required to get the source without using svn.  Just download the tarball and extract:

https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/src.txz
Comment 21 Jamie Landeg-Jones 2020-01-12 22:37:30 UTC
I'm talking about tracking current or stable. It would be nice to say "sync this tree with the remote, ignoring any local changes or hacks I've made, or any local additions/deletions, just synchronise it". Doing a rm -r /usr/src to get a fresh checkout, and a tonne of .svn files when there are only a few file changes is not very efficient!

But anyway, sorry, I've hijacked this thread. I'll take it to a mailing list.

cheers!