To create a local mirror I download all packages pkg.freebsd.org via HTTP. But this is not convenient and creates an additional load on the internet channel. I would like to download packages via rsync. Can consider this possibility?
Reclassify.
Are there any news on this request? Maybe ned help to realize this?
Current the pkg build/mirror infrastructure might not have a good way to keep the satellite rsync mirrors updated. We're in the middle of improving these. Please stay tuned.
*** Bug 227170 has been marked as a duplicate of this bug. ***
bugs 227170 is marked as duplicate of this, but it has one additional suggestion: open HTTP list to let public download. Fold that request to this ticket.
It would be very nice to have it.
Updates are currently available on http://update3.freebsd.org This is an undocumented feature, I hope it will not disappear at present time ;) I periodically download updates via HTTP (their size is not huge and they do not come out often). But with each update I have to download it all out again :(
May be somebody have any news about this is feature request?
(In reply to Li-Wen Hsu from comment #3) > Current the pkg build/mirror infrastructure might not have a good way to keep the satellite rsync mirrors updated. We're in the middle of improving these. Please stay tuned. Could you share a little bit more details? Maybe I could help with it. :)
(In reply to Li-Wen Hsu from comment #3) May be somebody have any news about this is feature request?
This isn't something that we will want to offer as a service as it introduces a ton of risks/issues for both consumers and us on the infrastructure side. It is far too easy to end up with an inconsistent mirror and the additional IO load that will be placed on us isn't going to be great. I think a more appropriate solution would be a caching proxy (perhaps using nginx) that people could deploy onsite.
> introduces a ton of risks/issues for both consumers and > us on the infrastructure side. How can creation of a local mirror affect on infrastructure of FreeBSD? Only in case of change the address of the consumer. But that situation is similar to installation of the proxy by the consumer. > It is far too easy to end up with an inconsistent mirror It is too easy to end up with all data removal using ROOT rights, but this does not give a reason to disable ROOT in base release for all consumers :) > additional IO load that will be placed on us isn't going to be great. At the moment, downloading the repository takes about a day. At the same time, the internet channel is not fully loaded, which indicates a restriction on the part of freefbs.org servers. There are places where it’s still bad with the internet and it’s easier for me to send SSD once a quarter, and not forcing them to pay for an expensive satellite channel. Also see bug #239593 - another case of using a local mirror. > I think a more appropriate solution would be a caching proxy > (perhaps using nginx) So that the proxy can issue a file, first I need to download this file. If I downloaded it, I do not need a proxy - I already posted it to the local repository. Considering many people, it will soon be easier to create an alternative mirror with rsync support for everyone. But I want it to be on the base site, and not on the alternative.
There's some of this on the hardware/infrastructure side that you'll just need to push the "I believe" button with as I'm not going to get into it. The main rationale I can think of to have a cache/local copy of stuff is for environments where you have multiple users downloading the same thing over and over. I'm not sure it makes much sense when you've got a single user. There's nothing that would stop you from building/maintaining your own internal package repository and distributing/rsyncing/whatever that as you see fit. It's very common for sites with large installations to either cache content or to build their own as they want the customization. Still, rsync for the package repo remains something we are not going to provide at this point in time.
> just need to push the "I believe" button with as I'm not going to get into it. Great argument, I will take for use :) > The main rationale I can think Reasons: - fastests creates mirror - internet fees reduction - use mirrors in isolated network (from internet) > There's nothing that would stop you from building/maintaining your own > internal package repository Already wrote - download restriction.
A caching proxy addresses these desires without the issues associated with mirroring the entire package repository as a service. I think serving packages to sites without internet or limited connectivity is more of an edge case than a typical use case and would need a separate solution. We'll investigate publishing a suitable nginx configuration to help with this.
> A caching proxy addresses these desires without the issues associated with > mirroring the entire package repository as a service. A caching proxy not addresses these desires. > I think serving packages to sites without internet or limited connectivity > is more of an edge case than a typical use case Yes, but does not negate the need for this. > would need a separate solution. rsync is great for such a solution. I can write a python script to create an alternative mirror (from HTTP) and completely configure the server. Maybe you have the opportunity to allocate for this service VDS/jail with domain name pkg-rsync.freebsd.org (for example) or only public IP-address?
Created attachment 210649 [details] nginx-pkg-mirror.conf
(In reply to Ryan Steinmetz from comment #11) The Nginx cache for FreeBSD packages sounds good for me. This is an interesting idea so I have created the initial configuration for Nginx that caches FreeBSD packages. Please check the attachment with the Nginx configuration and let me know if anything needs fine-tuning.
(In reply to Ryan Steinmetz from comment #13) > Still, rsync for the package repo remains something we are not going to provide at this point in time. Will rsync ever be available for src synchronisation? 'svn co' seems overkill just to sync the local copy of /usr/src, and then hope local changes don't cause inconsistencies!
rsync isn't required to get the source without using svn. Just download the tarball and extract: https://download.freebsd.org/ftp/releases/amd64/12.1-RELEASE/src.txz
I'm talking about tracking current or stable. It would be nice to say "sync this tree with the remote, ignoring any local changes or hacks I've made, or any local additions/deletions, just synchronise it". Doing a rm -r /usr/src to get a fresh checkout, and a tonne of .svn files when there are only a few file changes is not very efficient! But anyway, sorry, I've hijacked this thread. I'll take it to a mailing list. cheers!
(In reply to Li-Wen Hsu from comment #3) In last quarterly status report for 2019 is written "Plan how to add new semi-official pkg mirrors". That means opening rsync service on pkg/update servers?
The best way to tracking -current or -stable branches of src is using svn/git. Rsync is basically doing the similar thing as the SCM tool here, and might be slower if there are only minor changes. Currently the recommended way to setup a local pkg site for speedup is a cache proxy. See the config in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226626#c17 , we are testing and checking the recommended setup and will update the related documents. There are some edge cases need other solutions, but currently just opening the rsync of the pkg directory of a official mirror will not make things better because this will highly possible to end up with an inconsistent local package set due to the internal update/sync between official mirrors. This is related to the ports arch, pkg building, mirror mechanism and many other things and we need to take extra care without breaking any current setup.
> Resolution: --- → Works As Intended This ticket is not a bug, this ticket is request new functional. (In reply to Li-Wen Hsu from comment #23) > Currently the recommended way to setup a local pkg site > for speedup is a cache proxy. Сache proxy won't let create local pkg site, will only create local pkg cache site. To create local pkg site I have to download the entire repository. In this case, cache proxy is no longer relevant. > This is related to the ports arch, pkg building, mirror mechanism > and many other things and we need to take extra care without breaking > any current setup. Similar functionality is planned in the future? Is it possible to take part in the discussion of this issue?
(In reply to ev from comment #24) Please stop reopening this ticket. We have said "no". We are sorry that we cannot service your request at this time. We have no current plans to implement what you are requesting. We are not going to enable RSYNC on our PKG CDN. I do not know what else to tell you to make it clearer.