Bug 239114 - Many FreeBSD mirrors are outdated or broken
Summary: Many FreeBSD mirrors are outdated or broken
Status: Open
Alias: None
Product: Services
Classification: Unclassified
Component: FTP/WWW Sites & Mirrors (show other bugs)
Version: unspecified
Hardware: Any Any
: --- Affects Some People
Assignee: Li-Wen Hsu
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-07-10 17:02 UTC by Gautam Mani
Modified: 2019-07-11 21:13 UTC (History)
3 users (show)

See Also:


Attachments
log file (173.85 KB, text/plain)
2019-07-10 17:02 UTC, Gautam Mani
no flags Details
Mirrors to be checked (5.70 KB, text/plain)
2019-07-10 17:03 UTC, Gautam Mani
no flags Details
Program used to check that mirrors are working. (2.46 KB, text/plain)
2019-07-10 17:04 UTC, Gautam Mani
no flags Details
mirrors.csv (6.68 KB, text/plain)
2019-07-10 17:04 UTC, Gautam Mani
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Gautam Mani 2019-07-10 17:02:50 UTC
Created attachment 205664 [details]
log file

I stumbled upon the fact that many of the FreeBSD mirrors are broken or un-available. For some, a host lookup also fails. 

I made a small program to check each of the FTP and HTTP mirrors. I collected the list from the list of mirrors available in the handbook. I did not attempt to download each file - but just checked for existance of one file - in this case, 12.0 dvd1-iso for amd64. rsync has not been checked.

The program and the csv file (list of mirrors) is attached just in case.
The log file has a full trace of each of the errors for the servers.

I thought of writing to the admin emails myself, but given that the number is non-significant - I think it better that this is somehow taken up from the FreeBSD services team. 

Given that this is one of the first things that a new user experiences, it is important that we get these mirrors fixed or else update the documentation accordingly. 

I am hoping that there are no false alarms thrown by the program - I checked quite a few cases myself manually to validate the same and it looks OK except for one character encoding case.

Do let me know how I can help further with this.

Below is the list of mirrors that need to be checked.

ALERT:Mirror successful(ftp://ftp5.FreeBSD.org/pub/FreeBSD/), 550 releases/ISO-IMAGES/12.0/: No such file or directory.: False
ALERT:Mirror successful(ftp://ftp1.am.freebsd.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp.au.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp2.au.FreeBSD.org/pub/FreeBSD/), 550 /pub/FreeBSD/: No such file or directory: False
ALERT:Mirror successful(ftp://ftp2.br.FreeBSD.org/FreeBSD/), 'ascii' codec can't encode characters in position 22-23: ordinal not in range(128): False
ALERT:Mirror successful(ftp://ftp3.br.FreeBSD.org/pub/FreeBSD/), 550 FreeBSD-12.0-RELEASE-amd64-dvd1.iso: No such file or directory: False
ALERT:Mirror successful(ftp://ftp4.br.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp.cn.freebsd.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp2.cz.FreeBSD.org/pub/FreeBSD/), 550 Failed to change directory.: False
ALERT:Mirror successful(ftp://ftp.ee.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp6.fr.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(http://ftp4.de.FreeBSD.org/pub/FreeBSD/), : False
ALERT:Mirror successful(ftp://ftp7.de.FreeBSD.org/pub/FreeBSD/), 550 releases/ISO-IMAGES/12.0/: Datei oder Verzeichnis nicht gefunden: False
ALERT:Mirror successful(http://ftp7.de.FreeBSD.org/pub/FreeBSD/), : False
ALERT:Mirror successful(ftp://ftp8.de.FreeBSD.org/pub/FreeBSD/), 550 Failed to change directory.: False
ALERT:Mirror successful(ftp://ftp2.kr.FreeBSD.org/pub/FreeBSD/), 550 Failed to change directory.: False
ALERT:Mirror successful(ftp://ftp.lt.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(http://ftp.lt.FreeBSD.org/pub/FreeBSD/), HTTPConnectionPool(host='ftp.lt.freebsd.org', port=80): Max retries exceeded with url: /pub/FreeBSD/releases/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-dvd1.iso (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x8022464a8>, 'Connection to ftp.lt.freebsd.org timed out. (connect timeout=15)')): False
ALERT:Mirror successful(ftp://ftp.nz.FreeBSD.org/pub/FreeBSD/), 550 Failed to change directory.: False
ALERT:Mirror successful(http://ftp.nz.FreeBSD.org/pub/FreeBSD/), : False
ALERT:Mirror successful(http://ftp.ru.FreeBSD.org/FreeBSD/), : False
ALERT:Mirror successful(ftp://ftp.isu.net.sa/pub/ftp.freebsd.org/), 550 Failed to change directory.: False
ALERT:Mirror successful(ftp://ftp.za.FreeBSD.org/pub/FreeBSD/), 550 Can't change directory to /pub/FreeBSD/: No such file or directory: False
ALERT:Mirror successful(ftp://ftp.es.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(http://ftp.es.FreeBSD.org/pub/FreeBSD/), ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)): False
ALERT:Mirror successful(ftp://ftp3.es.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp2.se.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp3.se.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp6.se.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(http://ftp2.tw.FreeBSD.org/pub/FreeBSD/), HTTPConnectionPool(host='ftp2.tw.freebsd.org', port=80): Max retries exceeded with url: /pub/FreeBSD/releases/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-dvd1.iso (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x8022645c0>: Failed to establish a new connection: [Errno 61] Connection refused',)): False
ALERT:Mirror successful(ftp://ftp6.tw.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp11.tw.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(http://ftp11.tw.FreeBSD.org/FreeBSD/), HTTPConnectionPool(host='ftp11.tw.freebsd.org', port=80): Max retries exceeded with url: /FreeBSD/releases/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-dvd1.iso (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x802246da0>, 'Connection to ftp11.tw.freebsd.org timed out. (connect timeout=15)')): False
ALERT:Mirror successful(ftp://ftp15.tw.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp.ua.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(http://ftp.ua.FreeBSD.org/pub/FreeBSD/), HTTPConnectionPool(host='ftp.ua.freebsd.org', port=80): Max retries exceeded with url: /pub/FreeBSD/releases/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-dvd1.iso (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x801273240>, 'Connection to ftp.ua.freebsd.org timed out. (connect timeout=15)')): False
ALERT:Mirror successful(ftp://ftp7.ua.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp2.us.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(ftp://ftp5.us.FreeBSD.org/pub/FreeBSD/), 550 releases/ISO-IMAGES/12.0/: No such file or directory.: False
ALERT:Mirror successful(ftp://ftp10.us.FreeBSD.org/pub/FreeBSD/), 550 Failed to change directory.: False
ALERT:Mirror successful(ftp://ftp13.us.FreeBSD.org/pub/FreeBSD/), timed out: False
ALERT:Mirror successful(http://ftp13.us.FreeBSD.org/pub/FreeBSD/), HTTPConnectionPool(host='ftp13.us.freebsd.org', port=80): Max retries exceeded with url: /pub/FreeBSD/releases/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-dvd1.iso (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x8022e6940>, 'Connection to ftp13.us.freebsd.org timed out. (connect timeout=15)')): False
ALERT:Mirror successful(ftp://ftp14.us.FreeBSD.org/pub/FreeBSD/), 550 Failed to change directory.: False
ALERT:Mirror successful(http://ftp14.us.FreeBSD.org/pub/FreeBSD/), : False
ALERT:Mirror successful(ftp://ftp15.us.FreeBSD.org/pub/FreeBSD/), 550 Failed to change directory.: False
Comment 1 Gautam Mani 2019-07-10 17:03:42 UTC
Created attachment 205665 [details]
Mirrors to be checked
Comment 2 Gautam Mani 2019-07-10 17:04:16 UTC
Created attachment 205666 [details]
Program used to check that mirrors are working.
Comment 3 Gautam Mani 2019-07-10 17:04:54 UTC
Created attachment 205667 [details]
mirrors.csv
Comment 4 Philip Paeps freebsd_committer 2019-07-10 17:33:45 UTC
Thank you very much for doing this work!

I think we should take the out of date mirrors out of the DNS and contact their operators.  We should run your script regularly and add mirrors back to the DNS when they're up to date again.

Li-Wen: can you pull the outdated mirrors from the dns?