Created attachment 162411 [details] Patch adding GitHub/PyPI site handlers See review URL for full context & change details
Created attachment 162439 [details] Add site handlers v4 Patch v4 includes: * Add authenticated API request support to the GitHub site handler * Document the new options in portscout.conf
v4 also includes: * Take MAINTAINER'ship so we can get more changes back into the port as quickly as possible
Maintainer timeout (15 days) I'll take it.
A commit references this bug: Author: koobs Date: Sun Nov 8 07:37:42 UTC 2015 New revision: 401037 URL: https://svnweb.freebsd.org/changeset/ports/401037 Log: ports-mgmt/portscout: Add GitHub and PyPI site handlers & MORE! Over the past several months portscout.freebsd.org appears to have been more frequently NOT finding updates, particularly for ports that use CHEESESHOP (PyPI) as their MASTER_SITES. Portscout has also never worked for ports using GitHub for distribution files due to the following: a) Portscout, prior to 'guessing', requests a randomly named file from the Site and expects a 4xx (404) in response. If it doesn't receive a 4xx response, it increments a 'lie counter' and does not check the site again in the next run. b) The GitHUB handlers (SUBDIR/MASTER_SITES) in bsd.sites.mk construct a URL that ends in a a dummy query paramater (for the filename), so that fetch saves the correct filename to DISTDIR. This means for any DISTFILE name provided, a 200 OK response is returned These two factors unfortunately leave us in a position where there is no good way to workaround this in the ports framework, including overriding DISTFILES, DISTNAME, FETCH_ARGS, or the SUBDIR URL itself for various reasons (not matching distinfo, file conflicts in DISTDIR, etc) Fortunately, the portroach project (OpenBSD's fork of portscout) contains a site handler for GitHub and PyPI (among others) already [1]. These site handlers use API endpoints at GitHub and PyPI that respond JSON respectively, providing a faster and more accurate way to determine the latest version of a package, without having to go through the 'guessing' process. This commit: - Adds GitHub and PyPI site handlers, and modifies or extends them to accept/match our MASTER_SITES URL's. - Adds authenticated API request support and two settings for the GitHub site handler - Add p5-JSON to RUN_DEPENDS (needed by new site handlers) - Add HTTPS option for supporting https:// MASTER_SITES. Currently portscout does not check (fails) https:// MASTER_SITES [2] - Take MAINTAINER'ship - Adds badly needed logging/debugging messages to key parts of the process retaining the conditional logic that ties the verbosity to "quiet" or "debug" portscout.conf settings. - Renables the SQLITE3 option (previously commented out) and renames it to SQLITE (the standard, as per bsd.options.desk.mk) - Creates a DATABASE option group allowing either/or SQLITE or POSTGRESQL to be selected - Switches option conditionals where possible to options helpers - Backport a fix for maintainer matching/mapping [3] - Adds LICENSE (BSD2CLAUSE) - Updates and sorts pkg-plist This change was tested again ports maintained by me, and resulted in 'new versions' being found and reported for 42 of my ports (of 123). [1] https://github.com/jasperla/portroach/tree/master/Portroach/SiteHandler [2] Reported by: truckman [3] https://github.com/jasperla/portroach/commit/2f6ee134ddc178f74688b37b986b66ecb0481782 PR: 203996 Approved by: maintainer (timeout, 15 days) Differential Revision: https://reviews.freebsd.org/D3996 Changes: head/ports-mgmt/portscout/Makefile head/ports-mgmt/portscout/files/ head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-GitHub.pm head/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm head/ports-mgmt/portscout/files/patch-Portscout_DataSrc_Ports.pm head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler.pm head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_GitHub.pm head/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm head/ports-mgmt/portscout/files/patch-portscout.conf head/ports-mgmt/portscout/files/patch-portscout.pl head/ports-mgmt/portscout/pkg-plist