Bug 244921 - pkg.freebsd.org is slow/unstable from norway
Summary: pkg.freebsd.org is slow/unstable from norway
Status: Closed FIXED
Alias: None
Product: Services
Classification: Unclassified
Component: FTP/WWW Sites & Mirrors (show other bugs)
Version: unspecified
Hardware: Any Any
: --- Affects Some People
Assignee: Cluster Admin
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-19 19:22 UTC by Rasmus Fauske
Modified: 2021-09-10 01:41 UTC (History)
7 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Rasmus Fauske 2020-03-19 19:22:29 UTC
Hi,

In the last months I have noticed that updates using pkg is extremely slow and times out a lot of times.

Is it possible to test the force to use select mirrors so I can see when its better or worse ? 

I notice that now I am directed to the UK
$ traceroute pkg.freebsd.org
traceroute to pkgmir.geo.freebsd.org (213.138.116.73), 64 hops max, 40 byte packets
 1  10.10.1.25 (10.10.1.25)  0.272 ms  0.225 ms  0.339 ms
 2  181.62-50-191.enivest.net (62.50.191.181)  1.083 ms  0.859 ms  0.708 ms
 3  214.82-134-80.bkkb.no (82.134.80.214)  3.793 ms  4.312 ms  4.005 ms
 4  213.82-134-80.bkkb.no (82.134.80.213)  4.172 ms  4.308 ms  4.302 ms
 5  bkk-e-gw1.bkkb.no (193.28.236.254)  8.864 ms  9.096 ms  8.972 ms
 6  ba-e-gw1.bkkb.no (193.28.236.253)  8.873 ms  9.113 ms  8.834 ms
 7  8-2-1.bear1.Oslo2.Level3.net (62.140.27.93)  8.750 ms  8.842 ms *
 8  ae-2-3204.edge4.Stockholm2.Level3.net (4.69.135.162)  15.197 ms  15.185 ms  15.450 ms
 9  Cogent-level3-Stockholm2.Level3.net (4.68.106.130)  15.901 ms  15.893 ms  15.855 ms
10  be3377.ccr22.sto03.atlas.cogentco.com (154.54.36.89)  16.309 ms
    be3376.ccr21.sto03.atlas.cogentco.com (130.117.50.225)  16.462 ms
    be3377.ccr22.sto03.atlas.cogentco.com (154.54.36.89)  16.182 ms
11  be2281.ccr41.ham01.atlas.cogentco.com (154.54.63.1)  27.295 ms
    be2282.ccr42.ham01.atlas.cogentco.com (154.54.72.105)  26.186 ms
    be2281.ccr41.ham01.atlas.cogentco.com (154.54.63.1)  31.365 ms
12  be2816.ccr42.ams03.atlas.cogentco.com (154.54.38.209)  45.529 ms
    be2815.ccr41.ams03.atlas.cogentco.com (154.54.38.205)  41.980 ms
    be2816.ccr42.ams03.atlas.cogentco.com (154.54.38.209)  43.963 ms
13  be12194.ccr41.lon13.atlas.cogentco.com (154.54.56.93)  45.112 ms  45.463 ms
    be12488.ccr42.lon13.atlas.cogentco.com (130.117.51.41)  39.711 ms
14  be2870.ccr22.lon01.atlas.cogentco.com (154.54.58.174)  45.629 ms
    be2869.ccr22.lon01.atlas.cogentco.com (154.54.57.162)  42.401 ms
    be2870.ccr22.lon01.atlas.cogentco.com (154.54.58.174)  42.408 ms
15  be5.asr01.thn.as20860.net (149.6.184.50)  42.082 ms  41.919 ms  42.240 ms
16  bytemark.asr1.thn.as20860.net (130.180.202.57)  47.368 ms  50.752 ms  51.541 ms
17  po1.ar2.dc1.yo26.yrk.bytemark.co.uk (91.223.58.33)  47.448 ms  53.119 ms  64.656 ms
18  213.138.116.46 (213.138.116.46)  49.349 ms  45.244 ms  48.327 ms
19  pkg0.bme.FreeBSD.org (213.138.116.73)  45.330 ms  48.456 ms  48.535 ms

and on the next try to nyi
$ traceroute pkg.freebsd.org
traceroute to pkgmir.geo.freebsd.org (96.47.72.71), 64 hops max, 40 byte packets
 1  10.10.1.25 (10.10.1.25)  0.302 ms  0.281 ms  0.290 ms
 2  181.62-50-191.enivest.net (62.50.191.181)  1.170 ms  0.917 ms  0.829 ms
 3  214.82-134-80.bkkb.no (82.134.80.214)  5.056 ms  3.991 ms  3.930 ms
 4  213.82-134-80.bkkb.no (82.134.80.213)  4.561 ms  4.375 ms  4.169 ms
 5  bkk-e-gw1.bkkb.no (193.28.236.254)  9.006 ms  8.834 ms  9.016 ms
 6  ba-e-gw1.bkkb.no (193.28.236.253)  9.018 ms  9.071 ms  8.755 ms
 7  8-2-6.bear1.Oslo2.Level3.net (62.140.27.25)  8.851 ms
    8-2-4.bear1.Oslo2.Level3.net (62.140.27.125)  8.675 ms
    8-2-7.bear1.Oslo2.Level3.net (62.140.27.29)  8.667 ms
 8  4.69.214.29 (4.69.214.29)  134.296 ms  134.617 ms  133.696 ms
 9  THE-NEW-YOR.ear1.Newark1.Level3.net (4.15.150.218)  170.416 ms  136.048 ms  134.040 ms
10  cs90.cs89new.v.ewr.nyinternet.net (96.47.77.210)  134.309 ms  134.680 ms  134.765 ms
11  96.47.66.42.static.nyinternet.net (96.47.66.42)  133.827 ms  135.480 ms  133.690 ms
12  pkg0.nyi.freebsd.org (96.47.72.71)  134.135 ms  133.987 ms  134.026 ms

Might be that its when I go to the nyi the problem is worst as the next try timed out on one hop
$ traceroute pkg.freebsd.org
traceroute to pkgmir.geo.freebsd.org (96.47.72.71), 64 hops max, 40 byte packets
 1  10.10.1.25 (10.10.1.25)  0.425 ms  0.262 ms  0.263 ms
 2  181.62-50-191.enivest.net (62.50.191.181)  1.001 ms  0.810 ms  0.749 ms
 3  214.82-134-80.bkkb.no (82.134.80.214)  4.031 ms  3.804 ms  4.000 ms
 4  213.82-134-80.bkkb.no (82.134.80.213)  4.351 ms  4.225 ms  4.280 ms
 5  bkk-e-gw1.bkkb.no (193.28.236.254)  9.007 ms  8.929 ms  8.988 ms
 6  ba-e-gw1.bkkb.no (193.28.236.253)  9.029 ms  9.041 ms  8.933 ms
 7  8-2-1.bear1.Oslo2.Level3.net (62.140.27.93)  8.858 ms
    8-2-4.bear1.Oslo2.Level3.net (62.140.27.125)  8.647 ms
    8-2-7.bear1.Oslo2.Level3.net (62.140.27.29)  8.650 ms
 8  * * *
 9  THE-NEW-YOR.ear1.Newark1.Level3.net (4.15.150.218)  134.236 ms  135.004 ms  134.206 ms
10  cs90.cs89new.v.ewr.nyinternet.net (96.47.77.210)  134.030 ms  135.237 ms  135.661 ms
11  96.47.66.42.static.nyinternet.net (96.47.66.42)  134.136 ms  133.978 ms  133.987 ms
12  pkg0.nyi.freebsd.org (96.47.72.71)  134.054 ms  134.437 ms  134.317 ms
Comment 1 Matthias Gamsjager 2020-03-27 21:14:16 UTC
Seems like pkg.bme0 mirror is down/unreachable. Currently I am connected to South Africa.

Any news on the EU mirror of why it is down?
Comment 2 ctipper 2020-05-30 20:31:10 UTC
I am located in UK and I am continually being directed to pkg0.nyi.freebsd.org and pkg0.isc.freebsd.org starting this March. This has resulted in very poor performance running pkg upgrade I should be getting 2Mb/s I seem to only get 100's Kb/s on a regular basis, because these mirrors are too far away.

I am very unfamiliar with DNS but the SRV records I am getting from drill have equal priority and weight for each of the pkg mirrors. So the selected mirror is essentially random?

$ drill _http._tcp.pkg.freebsd.org SRV
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 4426
;; flags: qr rd ra ; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; _http._tcp.pkg.freebsd.org.	IN	SRV

;; ANSWER SECTION:
_http._tcp.pkg.freebsd.org.	248	IN	SRV	10 10 80 pkgmir.geo.freebsd.org.
_http._tcp.pkg.freebsd.org.	248	IN	SRV	50 10 80 pkg0.bme.freebsd.org.
_http._tcp.pkg.freebsd.org.	248	IN	SRV	50 10 80 pkg0.isc.freebsd.org.
_http._tcp.pkg.freebsd.org.	248	IN	SRV	50 10 80 pkg0.nyi.freebsd.org.
_http._tcp.pkg.freebsd.org.	248	IN	SRV	50 10 80 pkg0.pkt.freebsd.org.
_http._tcp.pkg.freebsd.org.	248	IN	SRV	50 10 80 pkg0.tuk.freebsd.org.

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 21 msec
;; SERVER: 127.0.0.1
;; WHEN: Sat May 30 17:43:03 2020
;; MSG SIZE  rcvd: 286

And to confirm:

$drill _http._tcp.pkg.freebsd.org @1.1.1.1 SRV 
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 47647
;; flags: qr rd ra ; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; _http._tcp.pkg.freebsd.org.	IN	SRV

;; ANSWER SECTION:
_http._tcp.pkg.freebsd.org.	278	IN	SRV	10 10 80 pkgmir.geo.freebsd.org.
_http._tcp.pkg.freebsd.org.	278	IN	SRV	50 10 80 pkg0.bme.freebsd.org.
_http._tcp.pkg.freebsd.org.	278	IN	SRV	50 10 80 pkg0.isc.freebsd.org.
_http._tcp.pkg.freebsd.org.	278	IN	SRV	50 10 80 pkg0.nyi.freebsd.org.
_http._tcp.pkg.freebsd.org.	278	IN	SRV	50 10 80 pkg0.pkt.freebsd.org.
_http._tcp.pkg.freebsd.org.	278	IN	SRV	50 10 80 pkg0.tuk.freebsd.org.

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 21 msec
;; SERVER: 1.1.1.1
;; WHEN: Sat May 30 18:23:08 2020
;; MSG SIZE  rcvd: 312

pkgmir.geo.freebsd.org shows this message

> We use MaxMind GeoLite based geo-dns to choose a close mirror for you.

I am afraid I don't believe that any more.
Comment 3 ctipper 2020-06-12 14:36:40 UTC
Log in to router and enable PING on external WAN. 

For some reason speedtest.net had no problem detecting my IP without this enabled so I'm still not happy with the remedy.
Comment 4 Allan Jude freebsd_committer freebsd_triage 2020-06-17 13:53:33 UTC
On April 22nd we deployed a new mirror in Europe (pkg0.pkt.freebsd.org in Amsterdam) and modified the geo-load balancer to send 2/3rds of the traffic from Europe to this mirror, and the remainer to the existing BME mirror (after it was restored following a downtime in March).

This seemed to alleviate the problem, but lately there have been continued reports of slowness. I have temporarily removed the BME mirror from the pkg mirror list for Europe until we have get BME some additional disks to improve its IO performance.
Comment 5 Philip Paeps freebsd_committer freebsd_triage 2020-12-03 08:18:55 UTC
Is this still a problem?

(Going through old mirror-related tickets.)
Comment 6 Matthias Gamsjager 2020-12-03 08:35:51 UTC
I had issues in the past but it seems a lot better now with the extra Amsterdam mirror
Comment 7 Philip Paeps freebsd_committer freebsd_triage 2020-12-18 10:05:44 UTC
Sounds like this was resolved.

Thanks for reporting.  Please let us know if this happens again.
Comment 8 Mateusz Kwiatkowski 2021-02-18 19:27:48 UTC
Hello,

BME mirror is terribly slow again. Checked it from Poland from couple different ISPs. Can't get more than 16kB/s of download. I manually switched to Packet and it's much, much better.
Comment 9 Philip Paeps freebsd_committer freebsd_triage 2021-02-19 02:02:33 UTC
I'll take a look at bme again.
Comment 10 Philip Paeps freebsd_committer freebsd_triage 2021-08-31 05:06:49 UTC
I rebuilt the zpool on peter.bme.freebsd.org as mirrored pairs instead of a raidz in February.  This should have improved performance.  Since then, we've also swapped out some disks.

As far as I can tell, this machine should be performing reasonably well again.

Has anyone noticed it being slow recently?
Comment 11 Matthias Gamsjager 2021-09-09 20:28:27 UTC
Haven't had any issues with pkg for some time now. The Amsterdam mirror addition made quite the difference.
Comment 12 Philip Paeps freebsd_committer freebsd_triage 2021-09-10 01:41:06 UTC
Thanks for confirming!  I'll close this bug now.

Please file a new bug if new issues come up so we can look into them.