Bug 205762 - CHEESESHOP bsd.sites.mk substitution breaks databases/pyspatialite
Summary: CHEESESHOP bsd.sites.mk substitution breaks databases/pyspatialite
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Ports Framework (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Antoine Brodin
URL:
Keywords: needs-qa, patch
Depends on:
Blocks:
 
Reported: 2015-12-31 18:29 UTC by Raphael Kubo da Costa
Modified: 2016-09-04 10:24 UTC (History)
4 users (show)

See Also:
antoine: merge-quarterly-


Attachments
patch for bsd.sites.mk (611 bytes, patch)
2016-09-04 08:59 UTC, Antoine Brodin
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Raphael Kubo da Costa freebsd_committer 2015-12-31 18:29:23 UTC
MASTER_SITE_SUBDIRS in bsd.sites.mk has the following substitution for CHEESESHOP:

  CHEESESHOP:source/${DISTNAME:C/(.).*/\1/}/${DISTNAME:C/(.*)-[0-9].*/\1/}

This breaks databases/pyspatialite, which sets DISTNAME to "${PORTNAME}-${PORTVERSION}-alpha-0". Since the regular expressions are greedy, the last CHEESESHOP substitution matches "pyspatialite-3.0.1-alpha" instead of "pyspatialite".

Perhaps PORTNAME should be used instead of DISTNAME?
Comment 1 Antoine Brodin freebsd_committer 2015-12-31 19:57:14 UTC
pyspatialite is an exception, and using PORTNAME instead of DISTNAME breaks more ports that it fixes

You can use this workaround:
MASTER_SITES=  CHEESESHOP/source/p/${PORTNAME}
Comment 2 commit-hook freebsd_committer 2016-01-03 17:51:39 UTC
A commit references this bug:

Author: sunpoet
Date: Sun Jan  3 17:50:44 UTC 2016
New revision: 405177
URL: https://svnweb.freebsd.org/changeset/ports/405177

Log:
  - Update MASTER_SITES and unbreak this port [1]
  - While I'm here, use DISTVERSIONSUFFIX instead of DISTNAME

  PR:		205762
  Submitted by:	rakuco
  Obtained from:	antoine [1]
  Approved by:	portmgr (blanket)
  With hat:	python

Changes:
  head/databases/pyspatialite/Makefile
Comment 3 Sunpoet Po-Chuan Hsieh freebsd_committer 2016-01-03 17:58:53 UTC
I think this should be fixed ASAP.

But we also need to improve CHEESESHOP substitution. It does not recognize DISTVERSIONPREFIX and DISTVERSIONSUFFIX which are usually non-digit (if set). It might work if we replace [0-9].* with DISTVERSIONFULL. At least it works for this port and x11-fonts/py-bdflib.
Comment 4 Antoine Brodin freebsd_committer 2016-01-03 18:32:51 UTC
Something like this?

-                       CHEESESHOP:source/${DISTNAME:C/(.).*/\1/}/${DISTNAME:C/(.*)-[0-9].*/\1/} \
+                       CHEESESHOP:source/${DISTNAME:C/(.).*/\1/}/${DISTNAME:S/-${DISTVERSIONFULL}$//} \

This should be tested with empty DISTDIR / disabled MASTER_SITE_BACKUP
Comment 6 Raphael Kubo da Costa freebsd_committer 2016-01-04 11:53:58 UTC
It looks like all errors except for textproc/py-diff-match-patch are unrelated to the patch and the ports have been broken for a while.

It's worth considering the problems in devel/py-cycler and www/py-blogofile though: basically the tarballs and the directories differ in casing, which can also cause problems. Not sure if the only solution to those is to set MASTER_SITE_SUBDIR manually.

(In reply to Antoine Brodin from comment #5)
> I have the following errors with the patch applied:
> 
> fetch: https://pypi.python.org/packages/source/a/anyconfig/anyconfig-0.0.5.tar.gz: Not Found

Fixed in ports r405193.

> fetch: https://pypi.python.org/packages/source/c/cycler/cycler-0.9.0.tar.gz: Not Found

This was broken in ports r404408, I've replied to the commit email.

> fetch: https://pypi.python.org/packages/source/m/mongokit/mongokit-0.8.3.tar.gz: size mismatch: expected 68024, actual 68015

Fixed in ports r405195.

> fetch: https://pypi.python.org/packages/source/S/SimPy/SimPy-2.3.1.tar.gz: Not Found

Fixed in ports r405196 and ports r405227.

> fetch: https://pypi.python.org/packages/source/u/unittestplus/unittestplus-1.1.1.zip: Not Found

The project has vanished from PyPI and I couldn't find an official website anymore. I'm tempted to deprecate it and mark the port for removal.

> fetch: https://pypi.python.org/packages/source/P/PyX/PyX-0.12.1.tar.gz: Not Found

This was fixed a few weeks ago in ports r403796.

> fetch: https://pypi.python.org/packages/source/B/BitVector/BitVector-3.1.tar.gz: Not Found

Fixed in ports r405198.

> fetch: https://pypi.python.org/packages/source/p/python-novaclient/python-novaclient-2.6.8.tar.gz: Not Found

I couldn't find this release/tag in PyPI, GitHub or OpenStack's git repositories.

> fetch: https://pypi.python.org/packages/source/d/diff-match-patch-20121119/diff-match-patch-20121119.tar.gz: Not Found

This one looks like the only legitimate error: the tarballs' parent director should be diff-match-patch, not diff-match-patch-20121119.

> fetch: https://pypi.python.org/packages/source/b/blogofile/blogofile-0.7.1.tar.gz: Not Found

This has the same problem as devel/py-cycler: the tarball is called "blogofile-0.7.1.tar.gz" (lowercase 'b'), whereas the PyPI project is called "Blogofile" (uppercase 'B'), which is also the tarball's parent directory name.

> fetch: https://pypi.python.org/packages/source/d/django-cms/django-cms-2.4.1.tar.gz: size mismatch: expected 4436793, actual 4650951

The new tarball has quite a few differences, so I left it broken.

> fetch: https://pypi.python.org/packages/source/T/TraitsBackendWX/TraitsBackendWX-3.2.0.tar.gz: Not Found

I couldn't find the tarball anywhere.

> fetch: https://pypi.python.org/packages/source/T/TraitsGUI/TraitsGUI-3.1.0.tar.gz: Not Found

Fixed in ports r405199.
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2016-01-04 12:46:08 UTC
Assign to committer that's resolving/committing
Comment 8 Raphael Kubo da Costa freebsd_committer 2016-01-05 18:10:39 UTC
The ports I mentioned in comment #6 have all either been fixed or marked BROKEN so they shouldn't interfere with your tests.
Comment 9 Kubilay Kocak freebsd_committer freebsd_triage 2016-01-19 07:01:48 UTC
Re-open, merge-quarterly flag is pending.
Comment 10 Antoine Brodin freebsd_committer 2016-02-06 11:47:41 UTC
We don't want this in quarterly as this requires lots of MFH.
But I don't think it's fixed in head?
Comment 11 Raphael Kubo da Costa freebsd_committer 2016-02-07 14:08:40 UTC
databases/pyspatialite itself was fixed with the commit in comment #2, it's not clear if there's anything we want to change in the CHEESESHOP substitution though. Was the change proposed in comment #4 tested after comment #8?
Comment 12 Antoine Brodin freebsd_committer 2016-09-04 08:59:19 UTC
Created attachment 174364 [details]
patch for bsd.sites.mk
Comment 13 Antoine Brodin freebsd_committer 2016-09-04 09:52:12 UTC
Ports failing to fetch with the patch applied:



=> Attempting to fetch https://pypi.python.org/packages/source/P/PyX/PyX-0.12.1.tar.gz
fetch: https://pypi.python.org/packages/source/P/PyX/PyX-0.12.1.tar.gz: Not Found
=> Attempting to fetch https://files.pythonhosted.org/packages/source/P/PyX/PyX-0.12.1.tar.gz
fetch: https://files.pythonhosted.org/packages/source/P/PyX/PyX-0.12.1.tar.gz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/graphics/py-PyX12

-> it fails to fetch without the patch applied too




=> Attempting to fetch https://pypi.python.org/packages/source/P/PyPrind-2.9.8/PyPrind-2.9.8.tar.gz
fetch: https://pypi.python.org/packages/source/P/PyPrind-2.9.8/PyPrind-2.9.8.tar.gz: Not Found
=> Attempting to fetch https://files.pythonhosted.org/packages/source/P/PyPrind-2.9.8/PyPrind-2.9.8.tar.gz
fetch: https://files.pythonhosted.org/packages/source/P/PyPrind-2.9.8/PyPrind-2.9.8.tar.gz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/misc/py-pyprind

-> this port has a bogus DISTVERSIONPREFIX





=> Attempting to fetch https://pypi.python.org/packages/source/h/haproxy_log_analysis-2.0b0/haproxy_log_analysis-2.0b0.tar.gz
fetch: https://pypi.python.org/packages/source/h/haproxy_log_analysis-2.0b0/haproxy_log_analysis-2.0b0.tar.gz: Not Found
=> Attempting to fetch https://files.pythonhosted.org/packages/source/h/haproxy_log_analysis-2.0b0/haproxy_log_analysis-2.0b0.tar.gz
fetch: https://files.pythonhosted.org/packages/source/h/haproxy_log_analysis-2.0b0/haproxy_log_analysis-2.0b0.tar.gz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/net/py-haproxy-log-analysis

-> this port should use DISTVERSION





=> Attempting to fetch http://files.turpial.org.ve/sources/stable/libturpial-1.7.0.tar.gz
fetch: http://files.turpial.org.ve/sources/stable/libturpial-1.7.0.tar.gz: Bad Gateway
=> Attempting to fetch https://pypi.python.org/packages/source/l/libturpial/libturpial-1.7.0.tar.gz
fetch: https://pypi.python.org/packages/source/l/libturpial/libturpial-1.7.0.tar.gz: Not Found
=> Attempting to fetch https://files.pythonhosted.org/packages/source/l/libturpial/libturpial-1.7.0.tar.gz
fetch: https://files.pythonhosted.org/packages/source/l/libturpial/libturpial-1.7.0.tar.gz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/net-im/py-libturpial

-> it fails to fetch without the patch applied too





=> Attempting to fetch http://www.c-s.li/ports/stem-1.4.1.tar.bz2
fetch: http://www.c-s.li/ports/stem-1.4.1.tar.bz2: size unknown
fetch: http://www.c-s.li/ports/stem-1.4.1.tar.bz2: size of remote file is not known
stem-1.4.1.tar.bz2                                       0  B    0  Bpsstem-1.4.1.tar.bz2                                      13 kB  196 MBps 00m00s
=> Fetched file size mismatch (expected 1570151, actual 14020)
=> Trying next site
=> Attempting to fetch https://pypi.python.org/packages/source/s/stem/stem-1.4.1.tar.bz2
fetch: https://pypi.python.org/packages/source/s/stem/stem-1.4.1.tar.bz2: Not Found
=> Attempting to fetch https://files.pythonhosted.org/packages/source/s/stem/stem-1.4.1.tar.bz2
fetch: https://files.pythonhosted.org/packages/source/s/stem/stem-1.4.1.tar.bz2: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/security/py-stem

-> it fails to fetch without the patch applied too



=> Attempting to fetch https://pypi.python.org/packages/source/p/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214.tar.gz
fetch: https://pypi.python.org/packages/source/p/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214.tar.gz: Not Found
=> Attempting to fetch https://files.pythonhosted.org/packages/source/p/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214.tar.gz
fetch: https://files.pythonhosted.org/packages/source/p/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214/pycryptopp-0.7.1.869544967005693312591928092448767568728501330214.tar.gz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/security/py-pycryptopp

-> abuses PORTVERSION/DISTNAME, should be fixed




=> Attempting to fetch https://pypi.python.org/packages/source/d/diff-match-patch-20121119/diff-match-patch-20121119.tar.gz
fetch: https://pypi.python.org/packages/source/d/diff-match-patch-20121119/diff-match-patch-20121119.tar.gz: Not Found
=> Attempting to fetch https://files.pythonhosted.org/packages/source/d/diff-match-patch-20121119/diff-match-patch-20121119.tar.gz
fetch: https://files.pythonhosted.org/packages/source/d/diff-match-patch-20121119/diff-match-patch-20121119.tar.gz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/textproc/py-diff-match-patch

-> abuses PORTVERSION, should be fixed




=> Attempting to fetch https://pypi.python.org/packages/source/d/django-cms/django-cms-2.4.1.tar.gz
fetch: https://pypi.python.org/packages/source/d/django-cms/django-cms-2.4.1.tar.gz: size mismatch: expected 4436793, actual 4650951
=> Attempting to fetch https://files.pythonhosted.org/packages/source/d/django-cms/django-cms-2.4.1.tar.gz
fetch: https://files.pythonhosted.org/packages/source/d/django-cms/django-cms-2.4.1.tar.gz: size mismatch: expected 4436793, actual 4650951
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/www/py-django-cms

-> it fails to fetch without the patch applied too, rerolled?




=> Attempting to fetch https://pypi.python.org/packages/source/T/TraitsBackendWX/TraitsBackendWX-3.2.0.tar.gz
fetch: https://pypi.python.org/packages/source/T/TraitsBackendWX/TraitsBackendWX-3.2.0.tar.gz: Not Found
=> Attempting to fetch https://files.pythonhosted.org/packages/source/T/TraitsBackendWX/TraitsBackendWX-3.2.0.tar.gz
fetch: https://files.pythonhosted.org/packages/source/T/TraitsBackendWX/TraitsBackendWX-3.2.0.tar.gz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /tmp/distfiles/ and try again.
*** Error code 1
Stop.
make: stopped in /usr/ports/x11-toolkits/py-traitsbackendwx

-> it fails to fetch without the patch applied too, already marked BROKEN
Comment 14 commit-hook freebsd_committer 2016-09-04 09:55:47 UTC
A commit references this bug:

Author: antoine
Date: Sun Sep  4 09:54:56 UTC 2016
New revision: 421318
URL: https://svnweb.freebsd.org/changeset/ports/421318

Log:
  Remove bogus DISTVERSIONPREFIX

  PR:		205762

Changes:
  head/misc/py-pyprind/Makefile
Comment 15 commit-hook freebsd_committer 2016-09-04 09:58:49 UTC
A commit references this bug:

Author: antoine
Date: Sun Sep  4 09:58:40 UTC 2016
New revision: 421319
URL: https://svnweb.freebsd.org/changeset/ports/421319

Log:
  Use DISTVERSION

  PR:		205762

Changes:
  head/net/py-haproxy-log-analysis/Makefile
Comment 16 commit-hook freebsd_committer 2016-09-04 10:03:51 UTC
A commit references this bug:

Author: antoine
Date: Sun Sep  4 10:03:20 UTC 2016
New revision: 421320
URL: https://svnweb.freebsd.org/changeset/ports/421320

Log:
  Use the same PORTVERSION as the one on Pypi

  PR:		205762

Changes:
  head/security/py-pycryptopp/Makefile
Comment 17 commit-hook freebsd_committer 2016-09-04 10:05:53 UTC
A commit references this bug:

Author: antoine
Date: Sun Sep  4 10:05:46 UTC 2016
New revision: 421321
URL: https://svnweb.freebsd.org/changeset/ports/421321

Log:
  Use the same PORTVERSION as the one on Pypi

  PR:		205762

Changes:
  head/textproc/py-diff-match-patch/Makefile
  head/textproc/py-diff-match-patch/distinfo
Comment 18 commit-hook freebsd_committer 2016-09-04 10:23:56 UTC
A commit references this bug:

Author: antoine
Date: Sun Sep  4 10:23:14 UTC 2016
New revision: 421322
URL: https://svnweb.freebsd.org/changeset/ports/421322

Log:
  Improve CHEESESHOP subdir substitution, it didn't work well with
  DISTVERSION containing non digits.

  PR:		205762
  With hat:	portmgr

Changes:
  head/Mk/bsd.sites.mk