Bug 259283 - multimedia/libv4l download source from github always fails after 15 min, no alternative site
Summary: multimedia/libv4l download source from github always fails after 15 min, no a...
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-multimedia (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-10-19 18:39 UTC by Bob Frazier
Modified: 2021-10-23 04:03 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (multimedia)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Bob Frazier 2021-10-19 18:39:39 UTC
FreeBSD hack.SFT.local 12.2-STABLE FreeBSD 12.2-STABLE r370671 GENERIC  amd64
(using 2021Q4 ports tree)

while attempting to build multimedia/v4l_compat for x11/mate (v4l_compat uses libv4l downloads) the file at

  https://codeload.github.com/torvalds/linux/tar.gz/v5.13-rc2?dummy=/torvalds-linux-v5.13-rc2_GH0.tar.gz

repeatedly failed to complete the download.  it always stopped at about 100Mb (total file size closer to 180Mb).  There was no alternative for download.

I have a shared host web server with a fat pipeline off site, and it was able to download the entire file directly.  I then downloaded it from the shared host locally (it takes around 30 minutes).  The port built as expected.  Github was apparently cutting the file short after around 15 minutes of downloading, with no ability to pick up where you left off.

Unfortunately, there are no other mirrors available for this file.  I attempted 'make fetch' more than once with similar results.  wget is unable to recover from a partial download as the file size is not known during download.

because download from github is NOT reliable, please at least mirror this on the freebsd.org repo and/or alter the priority of download so that a subsequent fetch from a mirror repo actually works.


this is the typical build error:

===> Fetching all distfiles required by v4l_compat-1.20.0_2 for building
===>  Extracting for v4l_compat-1.20.0_2
=> SHA256 Checksum OK for hselasky-v4l-utils-v4l-utils-1.20.0_GH0.tar.gz.
=> SHA256 Checksum OK for hselasky-webcamd-v5.13.2.2_GH0.tar.gz.
=> SHA256 Checksum mismatch for torvalds-linux-v5.13-rc2_GH0.tar.gz.
===>  Refetch for 1 more times files:  torvalds-linux-v5.13-rc2_GH0.tar.gz 
===>  License GPLv2 LGPL21 accepted by the user
===>   v4l_compat-1.20.0_2 depends on file: /usr/local/sbin/pkg - found
=> torvalds-linux-v5.13-rc2_GH0.tar.gz doesn't seem to exist in /e-drive/distfiles/.
=> Attempting to fetch https://codeload.github.com/torvalds/linux/tar.gz/v5.13-rc2?dummy=/torvalds-linux-v5.13-rc2_GH0.tar.gz
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/torvalds-linux-v5.13-rc2_GH0.tar.gz
=> Couldn't fetch it - please try to retrieve this
=> port manually into /e-drive/distfiles/ and try again.
*** Error code 1
Comment 1 Hans Petter Selasky freebsd_committer 2021-10-19 19:31:14 UTC
The alternative site is the tarballs from kernel.org. They have a CDN for this.

But I thought that using GH would be easier to maintain and update when dealing with this ... Maybe I was wrong.
Comment 2 Tomasz "CeDeROM" CEDRO 2021-10-19 20:00:54 UTC
I have encountered similar problems recently when porting tools for NuttX RTOS development on FreeBSD, specifically genromfs and kconfig-frontends. It turned out that kconfig-frontends maintenance is abandoned, folks used simple copy of the package, but there was no central upstream repository. (what is more there were two known diverged sources on Debian using version 4.11 and Espressif GH using version 4.6). People from GNU / Linux world does not see that as problem.

GitHub is really easy to maintain and process patches as HPS mentioned, also provides reliable packages hosting, but it also may create a diverged fork when not merged back to the upsteam :-)
Comment 3 Bob Frazier 2021-10-20 09:20:51 UTC
(In reply to Hans Petter Selasky from comment #1)

you were not wrong, but unfortunately github has thrown a wrench into the works, so that downloading for >15 minutes appears to cut off the file

if this link were to work, then fetch would grab it from there, however:

http://distcache.freebsd.org/ports-distfiles/torvalds-linux-v5.13-rc2_GH0.tar.gz

I am guessing it just wasn't copied there for some reason.  OK 180Mb may be a reason, but still.

But I like the idea of grabbing it from kernel.org if it can be directly downloaded from there, i.e. no patches etc..
Comment 4 Bob Frazier 2021-10-23 04:03:07 UTC
A similar problem seems to exist with java/openjdk11 attempting to download

 https://codeload.github.com/battleblow/jdk11u/tar.gz/jdk-11.0.12+7-1?dummy=/battleblow-jdk11u-jdk-11.0.12+7-1_GH0.tar.gz

(file size is ~120Mb; the download stopped at ~100Mb)

as before I manually fetched it from my company weh server, then downloaded it from there.  Extract worked (size and checksum ok).

this seems to be an inherent problem with files in codeload.github.com

(please cache on freebsd.org so the alternate site works - thanks)

(build log)
===>   openjdk11-11.0.12+7.1 depends on file: /usr/local/sbin/pkg - found
=> battleblow-jdk11u-jdk-11.0.12+7-1_GH0.tar.gz doesn't seem to exist in /e-drive/distfiles/.
=> Attempting to fetch https://codeload.github.com/battleblow/jdk11u/tar.gz/jdk-11.0.12+7-1?dummy=/battleblow-jdk11u-jdk-11.0.12+7-1_GH0.tar.gz
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/battleblow-jdk11u-jdk-11.0.12+7-1_GH0.tar.gz
=> Couldn't fetch it - please try to retrieve this
=> port manually into /e-drive/distfiles/ and try again.
*** Error code 1