Bug 277625

Summary: pkg.freebsd.org via geoDNS should hit FRA from VIE not
Product: Services Reporter: Dave Cottlehuber <dch>
Component: Core InfrastructureAssignee: Cluster Admin <clusteradm>
Status: Closed FIXED    
Severity: Affects Some People CC: freebsdbugs, philip
Priority: ---    
Version: unspecified   
Hardware: Any   
OS: Any   

Description Dave Cottlehuber freebsd_committer freebsd_triage 2024-03-11 10:24:05 UTC
- DNS is 9.9.9.9 which has a VIE POP just 7ms away

- pkg0.fra.freebsd.org is 20ms away
- pkg.freebsd.org heads to the new SJB mirror, 33ms away
 
dch@wintermute ~> drill _http._tcp.pkg.freebsd.org SRV
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 51090
;; flags: qr rd ra ; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; _http._tcp.pkg.freebsd.org.  IN      SRV

;; ANSWER SECTION:
_http._tcp.pkg.freebsd.org.     300     IN      SRV     50 10 80 pkg0.fra.freebsd.org.
_http._tcp.pkg.freebsd.org.     300     IN      SRV     50 10 80 pkg0.nyi.freebsd.org.
_http._tcp.pkg.freebsd.org.     300     IN      SRV     50 10 80 pkg0.sjb.freebsd.org.
_http._tcp.pkg.freebsd.org.     300     IN      SRV     50 10 80 pkg0.tuk.freebsd.org.
_http._tcp.pkg.freebsd.org.     300     IN      SRV     10 10 80 pkgmir.geo.freebsd.org.
_http._tcp.pkg.freebsd.org.     300     IN      SRV     50 10 80 pkg0.fmt.freebsd.org.

Observing pkg update running like a slow dog, it is connecting to SJB not FRA.

PING pkg0.sjb.freebsd.org (85.30.190.140): 56 data bytes
64 bytes from 85.30.190.140: icmp_seq=0 ttl=45 time=34.834 ms

PING pkg0.fra.freebsd.org (147.28.184.43): 56 data bytes
64 bytes from 147.28.184.43: icmp_seq=0 ttl=49 time=19.966 ms

NB my Dutch & German servers go to FRA as expected.
Comment 1 FiLiS 2024-03-21 08:15:09 UTC
Just observed the same thing on a residential Vodafone line in Western Germany. Fortunately, pkg picked up the FRA mirror after the upgrade of pkg itself.
Comment 2 Philip Paeps freebsd_committer freebsd_triage 2024-03-21 12:09:38 UTC
1) I'll redirect AT to fra rather than to the 50/50 europe split.  That should address the immediate problem.

2) However: I would like to know why your connection to SJB is "slow as a dog" despite only being 13ms more distant.

Could you please try to capture a traceroute and/or tcpdump of the slow connection?
Comment 3 Dave Cottlehuber freebsd_committer freebsd_triage 2024-03-27 13:18:50 UTC
I noticed this is still an issue in poudriere builds today, it definitely prefers
the Nordics still, I will hardwire poudriere. From the last hop router 
in my VIE network:

$ traceroute  pkg0.fra.freebsd.org
traceroute to pkg0.fra.freebsd.org (147.28.184.43), 64 hops max, 40 byte packets
 1  ppp06-l0.loopback.service.kapper.net (94.136.0.40)  8.422 ms  7.719 ms  7.767 ms
 2  xe-0-0-1.cr03.vivi1.kapper.net (94.136.0.245)  7.393 ms  7.690 ms  7.993 ms
 3  ten0-0-1.bgp01.vivi1.kapper.net (94.136.1.114)  8.489 ms  7.881 ms  7.472 ms
 4  xe-66-0-0-4010.bbr01.anx03.vie.at.anexia-it.net (144.208.211.109)  9.969 ms  9.670 ms  8.614 ms
 5  ae0-0.bbr02.anx03.vie.at.anexia-it.net (144.208.208.132)  20.256 ms  20.051 ms  20.130 ms
 6  ae1-0.bbr02.anx84.nue.de.anexia-it.net (144.208.208.137)  21.280 ms  21.068 ms  36.298 ms
 7  ae0-0.bbr01.anx84.nue.de.anexia-it.net (144.208.208.139)  21.734 ms  19.813 ms  20.029 ms
 8  ae2-0.bbr02.anx25.fra.de.anexia-it.net (144.208.208.141)  21.291 ms  21.127 ms  20.668 ms
 9  ipv4.de-cix.fra.de.as54825.packet.net (80.81.194.235)  20.348 ms  20.991 ms  21.183 ms
10  * * *
11  * * *
12  * * *
13  * * *
14  pkg0.fra.freebsd.org (147.28.184.43)  21.624 ms  20.876 ms  21.382 ms

-----------

traceroute  pkg0.sjb.freebsd.org
traceroute to pkg0.sjb.freebsd.org (85.30.190.140), 64 hops max, 40 byte packets
 1  ppp06-l0.loopback.service.kapper.net (94.136.0.40)  8.602 ms  8.259 ms  6.608 ms
 2  xe-0-0-1.cr03.vivi1.kapper.net (94.136.0.245)  26.182 ms  8.594 ms  7.948 ms
 3  ten0-0-1.bgp01.vivi1.kapper.net (94.136.1.114)  7.711 ms  8.107 ms  7.989 ms
 4  xe-66-0-0-4010.bbr01.anx03.vie.at.anexia-it.net (144.208.211.109)  7.448 ms  8.171 ms  7.724 ms
 5  ae0-0.bbr02.anx03.vie.at.anexia-it.net (144.208.208.132)  7.679 ms  8.819 ms  7.965 ms
 6  win-b2-link.ip.twelve99.net (62.115.62.152)  8.304 ms  8.852 ms  8.984 ms
 7  * * *
 8  ffm-bb1-link.ip.twelve99.net (62.115.137.202)  21.047 ms  20.212 ms  19.503 ms
 9  * * kbn-bb5-link.ip.twelve99.net (62.115.143.33)  33.821 ms
10  kbn-b4-link.ip.twelve99.net (62.115.134.81)  32.811 ms  33.540 ms
    kbn-b4-link.ip.twelve99.net (62.115.136.231)  33.186 ms
11  * * *
12  * * *
13  85.30.128.83 (85.30.128.83)  35.331 ms  35.008 ms  34.787 ms
14  85.30.128.158 (85.30.128.158)  34.068 ms  34.050 ms  33.290 ms
15  host-85-30-128-160.teleservice.net (85.30.128.160)  36.089 ms  36.095 ms  34.904 ms
16  igw0-ext.sjb.freebsd.org (85.30.190.148)  33.862 ms  34.945 ms  35.563 ms
17  pkg0.sjb.freebsd.org (85.30.190.140)  35.824 ms  35.045 ms  34.514 ms

---------

pcaps from curl of pkg-1.20.9_1.pkg at:

fra takes ~ 3 secs
sjb takes ~ 23.8 secs first run, 8.7 secs 2nd run, 6.4 secs 3rd run, 22 secs 4th run

pcaps: https://skunkwerks.at/~dch/tmp/fra.pcap https://skunkwerks.at/~dch/tmp/sjb.pcap

there's no other notable activity on this net atm.
Comment 4 Philip Paeps freebsd_committer freebsd_triage 2024-03-27 15:25:06 UTC
Index: config
===================================================================
--- config	(revision 14848)
+++ config	(working copy)
@@ -191,6 +191,7 @@
           EU => {
             default => [ europe, nyi-01, fmt-01 ],
             DE => [ fra-01, europe, nyi-01, fmt-01 ],
+            AT => [ fra-01, europe, nyi-01, fmt-01 ],
             DK => [ sjb-01, europe, nyi-01, fmt-01 ],
             FI => [ sjb-01, europe, nyi-01, fmt-01 ],
             NO => [ sjb-01, europe, nyi-01, fmt-01 ],

You should be hitting Frankfurt when automation notices.

Thanks for the pcaps!
Comment 5 Philip Paeps freebsd_committer freebsd_triage 2024-04-04 00:52:37 UTC
This should be fixed I think.  Let me know if other things break. :)