Bug 233718

Summary: math/py-matplotlib: Flavor @py36 does not install because of wrong PYTHON*_SITELIBDIR
Product: Ports & Packages Reporter: Rainer Hurling <rhurlin>
Component: Individual Port(s)Assignee: freebsd-ports-bugs (Nobody) <ports-bugs>
Status: Closed Overcome By Events    
Severity: Affects Only Me CC: koobs, lbartoletti, mainland, rhurlin, se, swills
Priority: --- Keywords: needs-qa
Version: LatestFlags: mainland: maintainer-feedback+
koobs: merge-quarterly?
Hardware: Any   
OS: Any   
Attachments:
Description Flags
patch SITELIBDIR for @py36
none
install py-matplotlib@py36 after building dependencies
none
install py-matplotlib@py36 with existing dependencies none

Description Rainer Hurling freebsd_committer freebsd_triage 2018-12-02 11:42:12 UTC
Created attachment 199746 [details]
patch SITELIBDIR for @py36

In the process of preparing my new port graphics/qgis with QGIS 3, I found math/py-matplotlib as a showstopper, because it is not able to install for flavor @py36.

I think, this is because math/py-matplotlib@py36 tries to rename (and install) the Python files wrongly in PYTHON_SITELIBDIR instead of PYTHONPREFIX_SITELIBDIR.

The attached patch seems to solve the problem, now both @py27 and @py36 are installable at the same time.


Tested with Poudriere (F11.2i/a, F12.0-RC1i/a, and HEADi/a).

'portlint -AC' seems fine.
Comment 1 Geoffrey Mainland 2018-12-02 15:06:08 UTC
Nice catch, thanks!
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2018-12-03 01:58:23 UTC
Thank you for the report Rainer. 

Could you attach a full build/error log of the failure and success cases please, ideally with the post-install: @$FIND command un-quietened (remove the prefixed @)
Comment 3 Rainer Hurling freebsd_committer freebsd_triage 2018-12-03 07:03:49 UTC
Created attachment 199778 [details]
install py-matplotlib@py36 after building dependencies
Comment 4 Rainer Hurling freebsd_committer freebsd_triage 2018-12-03 07:04:36 UTC
Created attachment 199779 [details]
install py-matplotlib@py36 with existing dependencies
Comment 5 Rainer Hurling freebsd_committer freebsd_triage 2018-12-03 07:05:31 UTC
(In reply to Kubilay Kocak from comment #2)

Hi Kubilay,

Thanks for taking this up.

I think your suspicion about my PR is right. Now that I try to reproduce the reported error, it is not as it seems before. Instead, it turns out, that it is not an installation problem, but most likely a problem with portmaster and its bookkeeping.

Portmaster does not create the usual directory entry 'py36-matplotlib-2.1.2_4' under /var/db/pkg, when 'portmaster math/py-matplotlib@py36' also has to build needed dependencies, like e.g. py36-cython-0.29, py36-numpy-1.15.1,1, py36-cycler-0.10.0, py36-dateutil-2.7.3, py36-pyparsing-2.3.0, and py36-tornado-5.1.1.

If the dependencies already exist, portmaster creates an entry on /var/db/pkg after successful installation of py36-matplotlib.


As requested, I created log files for the two cases, with the following command lines:

#env DISABLE_MAKE_JOBS=yes portmaster math/py-matplotlib@py36 > & build-error_deps-also-to-build.log
#env DISABLE_MAKE_JOBS=yes portmaster math/py-matplotlib@py36 > & build-error_deps-not-to-build.log


Obviously, there is no problem with math/py-matplotlib@py36 itself, so this PR should be closed. Sorry for the noise.
Comment 6 Kubilay Kocak freebsd_committer freebsd_triage 2018-12-03 07:48:46 UTC
(In reply to Rainer Hurling from comment #5)

If there's a bug somewhere, whether in py-matplotlib or elsewhere (portmaster), we want to isolate and identify the root cause(s).

The only odd thing I can observe in the attached logs is the last line of the last block:

===>>> The following actions were performed:
	Installation of lang/cython@py36 (py36-cython-0.29)
	Installation of math/py-numpy@py36 (py36-numpy-1.15.1,1)
	Installation of devel/py-cycler@py36 (py36-cycler-0.10.0)
	Installation of devel/py-dateutil@py36 (py36-dateutil-2.7.3)
	Installation of devel/py-pyparsing@py36 (py36-pyparsing-2.3.0)
	Installation of www/py-tornado@py36 (py36-tornado-5.1.1)
	Installation of math/py-matplotlib@py36 (py27-matplotlib-2.1.2_4) <---

Every other reference to port origins and package names is foo@py36 and py36-foo respectively, with the only reference to py27 being the (resolved) package name in the above line, which definitely appears incorrect.

However, the logs don't seem to include the actual full logs of the py-matplotlib(@py36) process so its difficult to say at what point that issue first presents itself.

On the PYTHONPREFIX_SITELIBDIR change proposal, both the PREFIX and non-PREFIX versions of that variable include the correct Python version specific path, use of PREFIX only makes the port PREFIX-overriden safe. I think this means that its unlikely to be the cause, nor the solution.

What we need from here is:

- Full log of py-matplotlib failing. Please un-quieten the @FIND command in the port before doing this, so we can confirm the full path/filenames that get operated on.
- Test: Does the current port (without patch) fail under poudriere?
Comment 7 Rainer Hurling freebsd_committer freebsd_triage 2018-12-03 14:33:21 UTC
(In reply to Kubilay Kocak from comment #6)

>If there's a bug somewhere, whether in py-matplotlib or elsewhere (portmaster), 
>we want to isolate and identify the root cause(s).
I agree.

>The only odd thing I can observe in the attached logs is the last line of the last block:
>===>>> The following actions were performed:
>	Installation of lang/cython@py36 (py36-cython-0.29)
>	Installation of math/py-numpy@py36 (py36-numpy-1.15.1,1)
>	Installation of devel/py-cycler@py36 (py36-cycler-0.10.0)
>	Installation of devel/py-dateutil@py36 (py36-dateutil-2.7.3)
>	Installation of devel/py-pyparsing@py36 (py36-pyparsing-2.3.0)
>	Installation of www/py-tornado@py36 (py36-tornado-5.1.1)
>	Installation of math/py-matplotlib@py36 (py27-matplotlib-2.1.2_4) <---

>Every other reference to port origins and package names is foo@py36 and py36-foo
>respectively, with the only reference to py27 being the (resolved) package name 
>in the above line, which definitely appears incorrect.
Yup, as I tried to describe in comment #5, the math/py-matplotlib with flavor @py36, in contrary to my initial report, builds and installs fine in both cases.

'pkg info -o math/py-matplotlib' proofs this:
py27-matplotlib-2.1.2_4        math/py-matplotlib
py36-matplotlib-2.1.2_4        math/py-matplotlib

The only thing is, that there is no entry under /var/db/pkg in the first case (py-matplotlib@py36 needs to install some deps first). That's why I think, that portmaster could be the culprit.

>However, the logs don't seem to include the actual full logs of the 
>py-matplotlib(@py36) process so its difficult to say at what point that 
>issue first presents itself.
Hmm, I thought I already attached full logs and described, how they were created. If these are not full logs, which way do you create them?

>On the PYTHONPREFIX_SITELIBDIR change proposal, both the PREFIX and non-PREFIX 
>versions of that variable include the correct Python version specific path, 
>use of PREFIX only makes the port PREFIX-overriden safe. I think this means 
>that its unlikely to be the cause, nor the solution.
If I am right with the above, there is no need to change from PYTHON_SITELIBDIR to PYTHONPREFIX_SITELIBDIR. Even with PYTHONPREFIX_SITELIBDIR, the behaviour is the same.

>What we need from here is:
>- Full log of py-matplotlib failing. Please un-quieten the @FIND command 
>  in the port before doing this, so we can confirm the full path/filenames 
>  that get operated on.
>- Test: Does the current port (without patch) fail under poudriere?
In the logs, I used FIND instead of @FIND already. The relevant line should be:
/usr/bin/find /usr/ports/math/py-matplotlib/work-py36/stage/usr/local/lib/python3.6/site-packages/matplotlib -name '*.so'  | /usr/bin/xargs /usr/bin/strip

On Poudriere, math/py-matplotlib@py36 builds fine (without patch).

Do you think, the missing entry for "py36-matplotlib-2.1.2_4" under /var/db/pkg is something, the port itself (and not portmaster) is responsible for?
Comment 8 Steve Wills freebsd_committer freebsd_triage 2019-05-03 12:15:37 UTC
I'm not able to reproduce an issue here.
Comment 9 Rainer Hurling freebsd_committer freebsd_triage 2020-05-20 09:35:12 UTC
I think this PR has been overtaken by events, the problem disappears.

As far as I can say, concurrent installations with different flavors of math/py-matplotlib are able now. If I am right, the PR should be closed.