Bug 219641 - devel/py-qt5-core: Fails to package/install, incorrect pk-plist entries
Summary: devel/py-qt5-core: Fails to package/install, incorrect pk-plist entries
Status: Closed Overcome By Events
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-kde (group)
URL:
Keywords: easy, patch-ready
: 220109 223764 233311 233312 233313 (view as bug list)
Depends on: pyqt-concurrent
Blocks:
  Show dependency treegraph
 
Reported: 2017-05-29 17:27 UTC by gdinolt
Modified: 2018-11-24 00:16 UTC (History)
15 users (show)

See Also:
tcberner: maintainer-feedback+
koobs: merge-quarterly?
koobs: exp-run?


Attachments
py35-qt-core install failure example (6.79 KB, text/x-log)
2017-05-30 04:08 UTC, gdinolt
no flags Details
Patch of /usr/ports/Mk/Uses/python.mk which seems to fix the problem (981 bytes, patch)
2017-05-30 04:22 UTC, gdinolt
no flags Details | Diff
Explicitly set the sip-executable location (387 bytes, patch)
2017-06-05 22:48 UTC, Mikhail T.
no flags Details | Diff
Patch py-qt5-core to include QtCore.pyi (462 bytes, patch)
2017-06-05 22:56 UTC, Mikhail T.
no flags Details | Diff
Use PYTHON_VER instead PYTHON_VERSION. Duh... (1.33 KB, patch)
2017-06-05 23:01 UTC, Mikhail T.
no flags Details | Diff
Fix py-qt5-* with python3 (1.41 KB, patch)
2017-10-31 16:06 UTC, Antoine Brodin
no flags Details | Diff
Fix py-qt5-* and py-qt4-demo with python3 (1.72 KB, patch)
2017-10-31 20:36 UTC, Antoine Brodin
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description gdinolt 2017-05-29 17:27:52 UTC
devel/py-qt5-core builds as expected with python3.5 but does not install because the work/.PLIST.mktmp file refers to files that don't exist in the stage directory. 

By default, when python3.5 compiles, for example, file Qt5.py for installation, it generates 

lib/python3.5/site-packages/PyQt5/__pycache__/Qt5.cpython-35.opt-1.pyc       
lib/python3.5/site-packages/PyQt5/__pycache__/Qt5.cpython-35.pyc 

in the stage directory.

When the 'add-plist-python' target from /usr/ports/Mk/Uses/python.mk is called to modify pkg-plist it generates the wrong output:

lib/python3.5/site-packages/PyQt5/__pycache__/Qt.cpython-35.pyc
lib/python3.5/site-packages/PyQt5/__pycache__/Qt.opt-1.cpython-35.pyc

Note that '.opt-1.' appears before cpython-35.pyc rather than as shown above.

A fix is to modify the 'add-plist-python' target in python.mk.

I updated the target to:

add-plist-python:
	@${AWK} '\
		/\.py[co]$$/ && !($$0 ~ "/" pc "/") {id = match($$0, /\/[^\/]+\.py[co]$$/); if (id != 0) {d = substr($$0, 1, RSTART - 1); dirs[d] = 1}; od = match($$0, /opt-1/);\
		if ( od != 0 ) sub(/\.opt-1\./, "." mt "&"); else \
		sub(/\.pyc$$/,  "." mt "&");  sub(/[^\/]+\.py[co]$$/, pc "/&"); print; next} \
		/^@dirrm / {d = substr($$0, 8); if (d in dirs) {print $$0 "/" pc}; print $$0; next} \
		/^@dirrmtry / {d = substr($$0, 11); if (d in dirs) {print $$0 "/" pc}; print $$0; next} \
		{print} \
		' \
		pc="__pycache__" mt="$$(${PYMAGICTAG})" pyo="${PYTHON_PYOEXTENSION}" \
		${TMPPLIST} > ${TMPPLIST}.pyc_tmp
	@${MV} ${TMPPLIST}.pyc_tmp ${TMPPLIST}

I have not tested with other versions of python.
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2017-05-30 03:00:18 UTC
Thank you for your report.

If you could, please:

1) Test with other Python 3.x versions to determine whether the issue is reproducible in those versions.

2) Include a full build log of failures as an attachment

3) Include the proposed change (to python.mk) as an attachment rather than in-comment, in unified diff format.

If indeed a python.mk bug (and modification) is implicated, an exp-run is warranted.

CC'ing Antoine (antoine@) as a significant author of the section in question
Comment 2 gdinolt 2017-05-30 04:08:23 UTC
Created attachment 183051 [details]
py35-qt-core install failure example

This file shows the install failure messages, note that this illustrates that the .PLIST.mktmp and the stage directory do not agree on the file names. 

The actual filenames in the stage directory are '*.cpython-35.opt-1.pyc
Comment 3 gdinolt 2017-05-30 04:22:05 UTC
Created attachment 183052 [details]
Patch of /usr/ports/Mk/Uses/python.mk which seems to fix the problem

Without this patch, py-qt5-core does not install for python versions 3.5 and 3.6 but does work for versions less than 3.5

This file is a patch for python.mk that appears to solve the problem.  I have tested it by building and installing py-qt-core for python versions 2.7, 3.4, 3.5 and 3.6.
Comment 4 Kubilay Kocak freebsd_committer freebsd_triage 2017-05-30 04:45:18 UTC
Thank you for the additional information.

To clarify, when you say:

"does not install for python versions 3.5 and 3.6 but does work for versions less than 3.5"

does this mean:

"without this patch, 3.5 and 3.6 do not work (fail to package, as reported), but 3.4 does work (is unaffected)" ?
Comment 5 Kubilay Kocak freebsd_committer freebsd_triage 2017-05-30 04:46:13 UTC
Note this issue is/appears limited in scope to those using USE_PYTHON=py3kplist
Comment 6 gdinolt 2017-05-30 13:53:07 UTC
(In reply to Kubilay Kocak from comment #4)

Just to be sure:

Without the patch to python.mk, py-gtk5-core:

    builds and installs as expected using python3.4

    fails to install using python3.5 and python3.6

With the patch using python3.4, python3.5 and python3.6 all build and install.

Note that python3.4 uses *.cpython-34.pyo for its "optimized" compiled file while python3.5 and python3.6 use *.cpython-3[56].opt-1.pyc
Comment 7 Dan MacDonald 2017-05-31 10:41:33 UTC
Hi

I'm trying to install pyqt5 built against python3 under TrueOS (FreeBSD 12) but I have so far had no luck.

From reading this thread I get the impression that without using the attached patch I should be able to get pyqt5 to build against python3.4 like so:

cd /usr/ports/devel/py-qt5
make PYTHON_VERSION=python3.4 install

That only gets so far for me before I get this error:

===>  Staging for py34-qt5-5.7.1
===>   py34-qt5-5.7.1 depends on file: /usr/local/bin/python3.4 - found
===>   py34-qt5-5.7.1 depends on package: py34-qt5-core>=5.7.1 - not found
===>   py34-qt5-core-5.7.1 depends on file: /usr/local/bin/python3.4 - found
===>   py34-qt5-core-5.7.1 depends on package: py34-sip>=4.19.2 - found
===>   py34-qt5-core-5.7.1 depends on file: /usr/local/lib/qt5/bin/qmake - found
===>   py34-qt5-core-5.7.1 depends on shared library: libqscintilla2-qt5.so - found (/usr/local/lib/qt5/libqscintilla2-qt5.so)
===>   py34-qt5-core-5.7.1 depends on shared library: libQt5Core.so - found (/usr/local/lib/qt5/libQt5Core.so)
===>  Configuring for py34-qt5-core-5.7.1
cd /usr/ports/devel/py-qt5-core/work/PyQt5_gpl-5.7.1 && /usr/bin/env QT_SELECT=qt5 PYTHON="/usr/local/bin/python3.4" XDG_DATA_HOME=/usr/ports/devel/py-qt5-core/work  XDG_CONFIG_HOME=/usr/ports/devel/py-qt5-core/work  HOME=/usr/ports/devel/py-qt5-core/work SHELL=/bin/sh CONFIG_SHELL=/bin/sh  /usr/local/bin/python3.4 configure.py --enable QtCore --qsci-api --qsci-api-destdir=/usr/local/share/qt5/qsci -b /usr/local/bin  -d /usr/local/lib/python3.4/site-packages  -q /usr/local/lib/qt5/bin/qmake  --confirm-license  --sipdir /usr/local/share/py-sip/PyQt5
Querying qmake about your Qt installation...
Determining the details of your Qt installation...
This is the GPL version of PyQt 5.7.1 (licensed under the GNU General Public
License) for Python 3.4.6 on freebsd12.
Found the license file pyqt-gpl.sip.
Error: Make sure you have a working sip on your PATH or use the --sip argument
to explicitly specify a working sip.
*** Error code 1

Stop.
make[3]: stopped in /usr/ports/devel/py-qt5-core
*** Error code 1

Stop.
make[2]: stopped in /usr/ports/devel/py-qt5-core
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/devel/py-qt5
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/py-qt5

devel/py-sip seems to be installed already so I'm not sure why its not finding it. I am able to build pyqt5 against python27 (the default) without problem, but I have had zero success with py3.

Thanks for your help!
Comment 8 Dan MacDonald 2017-05-31 14:00:55 UTC
Through having a search, it seems FreeBSD has packages for py34-sip, py36-sip etc whilst TrueOS only has py27-sip in its repos. You'll have to forgive me as I'm new to FreeBSD and even newer to using ports but does this matter ie do I need to request they add py34-sip or py36-sip to the TrueOS repo? 

I would've thought that building stuff from ports creates packages that are identical to what you get via pkg, only you have to compile them yourself which gives you more scope for tweaking build options etc. Is this not the case? Are ports builds treated differently by the OS compared to packages installed via pkg?
Comment 9 Dan MacDonald 2017-06-01 09:44:59 UTC
I'm happy to create a new bug report for my problem if its unrelated to this one but I too was getting the misnamed pyc files when I tried building pyqt5 against python 3.6.1, but then I saw gdinolt has apparently got pyqt5 built against python 3.4 without any extra patches so I wanted to try that first, hence my previous replies.

My ports knowledge is sadly lacking right now but I'm sure I'll be much more useful round here after I've found my feet.
Comment 10 Dan MacDonald 2017-06-01 15:33:36 UTC
I've got building pyqt5 to progress further by linking /usr/local/bin/sip-3.4 to /usr/local/bin/sip but I expect that is a bug in that either the sip port should create such a link or the pyqt5 port should know to look for sip-x.x.

Now building pyqt5 has stopped at py34-qt5-dbussupport which apparently needs at least python 3.6. 

When configuring pyqt5, I stuck with all the default options for all of its (sub)ports except for setting the python version to 3.4 so I presume gdinolt only built a subset of the pyqt5 packages?
Comment 11 Dan MacDonald 2017-06-01 16:29:30 UTC
Yay!

After a much longer than expected battle, I've finally got pyqt5 built against python 3.4. To achieve this:

cd /usr/ports/devel/py-qt5
make PYTHON_VERSION=python3.4 config
# Disable both dbus sub-ports and the pyqt5 demo port in the ncurses menu
make PYTHON_VERSION=python3.4 install

If you get the error:

Error: Make sure you have a working sip on your PATH or use the --sip argument

Then run:

ln -s /usr/local/bin/sip-3.4 /usr/local/bin/sip

Before re-running:

make PYTHON_VERSION=python3.4 install
Comment 12 Mikhail T. 2017-06-05 22:48:51 UTC
Created attachment 183244 [details]
Explicitly set the sip-executable location

I can't even get py-qt5-core to BUILD with a non-default Python, because py36-sip installs only bin/sip-3.6 -- there is no bin/sip.

This additional patch is necessary in my case and useful in all cases, I think...
Comment 13 Mikhail T. 2017-06-05 22:56:43 UTC
Created attachment 183245 [details]
Patch py-qt5-core to include QtCore.pyi

And this little patch was necessary for me to pass the check-plist in devel/py-qt5-core, when using python-3.6 -- not sure, how it would handle 3.4 or 3.5...
Comment 14 Mikhail T. 2017-06-05 23:01:43 UTC
Created attachment 183246 [details]
Use PYTHON_VER instead PYTHON_VERSION. Duh...
Comment 15 Tobias C. Berner freebsd_committer freebsd_triage 2017-09-25 13:07:55 UTC
*** Bug 220109 has been marked as a duplicate of this bug. ***
Comment 16 Loïc Bartoletti freebsd_committer freebsd_triage 2017-10-30 22:01:46 UTC
I've tried this patch and works fine except for py-qt5-demo.
Comment 17 Mikhail T. 2017-10-30 22:05:26 UTC
(In reply to lbartoletti from comment #16)

Could you, please, provide more details? Did the patch introduce _new_ breakage to  py-qt5-demo, or was it simply insufficient to make that one port work?

In the former case, could you provide the log showing the new error? Thank you!
Comment 18 Antoine Brodin freebsd_committer freebsd_triage 2017-10-30 22:48:53 UTC
Sorry but this is not a Uses/python.mk bug,  this is a bug in the ports...

You can't mix %%PYTHON_PYOEXTENSION%% and py3kplist together.
Comment 19 Mikhail T. 2017-10-30 22:54:14 UTC
(In reply to Antoine Brodin from comment #18)
> Sorry but this is not a Uses/python.mk bug, this is a bug in the ports...
Do you mean, the individual py-ports, or the ports infrastructure?

> You can't mix %%PYTHON_PYOEXTENSION%% and py3kplist together.
Could you elaborate? None of the proposed patches mention py3kplist...
Comment 20 Antoine Brodin freebsd_committer freebsd_triage 2017-10-30 22:54:24 UTC
Most problems should be fixed with https://svnweb.freebsd.org/changeset/ports/453212
Comment 21 Antoine Brodin freebsd_committer freebsd_triage 2017-10-30 22:55:52 UTC
(In reply to Mikhail T. from comment #19)
The proposed Uses/python.mk patches were touching the py3kplist part.
Comment 22 Mikhail T. 2017-10-30 22:58:39 UTC
(In reply to Antoine Brodin from comment #21)
For better or worse, this PR is now about more than one port... For example, my most recent patch adds explicit setting of sip to pyqt.mk -- which is crucial for people, who have more than on Python installed...
Comment 23 Rainer Hurling freebsd_committer freebsd_triage 2017-10-31 11:52:44 UTC
(In reply to Antoine Brodin from comment #20)
Hi Antoine,
Thanks for https://svnweb.freebsd.org/changeset/ports/453212.

One big showstopper was not solved, btw. In bug #211414, comment #8, I described a conflict with pyuic5:

Installing py35-qt5-core-5.6...
pkg-static: py35-qt5-core-5.6 conflicts with py27-qt5-core-5.6 (installs files into the same place).  Problematic file: /usr/local/bin/pyuic5


Is there any chance to get bin/pyuic5 also moved into some 'Python version dependend' directory?
Comment 24 Antoine Brodin freebsd_committer freebsd_triage 2017-10-31 14:55:22 UTC
Hi kde@,  would you like an exp-run with:

+		--sip ${LOCALBASE}/bin/sip-${PYTHON_VER} \

added to Mk/Uses/pyqt.mk ?
Comment 25 Antoine Brodin freebsd_committer freebsd_triage 2017-10-31 16:06:29 UTC
Created attachment 187619 [details]
Fix py-qt5-* with python3
Comment 26 Mikhail T. 2017-10-31 17:31:29 UTC
Comment on attachment 187619 [details]
Fix py-qt5-* with python3

> +USE_PYTHON=	py3kplist

Why is `py3kplist' necessary, if `autoplist' was not necessary for the regular case of Python-2.7?

The comment inside python.mk confuses me:

#       py3kplist       - Automatically generates Python 3.x compatible
#                         __pycache__ entries from a Python 2.x packaging list
#                         when defined. Use this for ports that do *not* use
#                         standard Python packaging mechanisms such as
#                         distutils, and support *both* Python 2.x and 3.x.
#                         Not needed, if USE_PYTHON=autoplist is set.
Comment 27 Antoine Brodin freebsd_committer freebsd_triage 2017-10-31 20:36:24 UTC
Created attachment 187626 [details]
Fix py-qt5-* and py-qt4-demo with python3
Comment 28 commit-hook freebsd_committer freebsd_triage 2017-11-01 07:33:01 UTC
A commit references this bug:

Author: antoine
Date: Wed Nov  1 07:32:34 UTC 2017
New revision: 453263
URL: https://svnweb.freebsd.org/changeset/ports/453263

Log:
  - py-qt5-*: Fix build when using non default version of python, in this case
    sip is installed as sip-${PYTHON_VER}

  - py-qt*-demo: Fix packaging with python3, those ports have a python2 pkg-plist
    so USE_PYTHON=py3kplist must be used to convert it

  PR:		219641

Changes:
  head/Mk/Uses/pyqt.mk
  head/devel/py-qt5-qscintilla2/Makefile
  head/misc/py-qt4-demo/Makefile
  head/misc/py-qt5-demo/Makefile
Comment 29 Tobias C. Berner freebsd_committer freebsd_triage 2017-11-01 07:41:40 UTC
Thanks.
Comment 30 Antoine Brodin freebsd_committer freebsd_triage 2017-11-01 08:33:46 UTC
Comment on attachment 187626 [details]
Fix py-qt5-* and py-qt4-demo with python3

this patch was committed
Comment 31 John Hein 2017-11-13 01:10:24 UTC
(In reply to Mikhail T. from comment #13)
Without this patch (attachment 183245 [details]), 'make stage check-orphans' for devel/py-qt5-core gives this error when building for python36 (e.g., DEFAULT_VERSIONS+=python=3.6 in make.conf):

===> Checking for items in STAGEDIR missing from pkg-plist
Error: Orphaned: %%PYTHON_SITELIBDIR%%/PyQt5/QtCore.pyi


This is different than the originally reported problem, so perhaps this merits a different bug report.
Comment 32 Loïc Bartoletti freebsd_committer freebsd_triage 2017-12-21 11:17:52 UTC
Hi,

I haven't tested what I'm proposing, but this conflict of pyuic5 is embarrassing for Rainer and myself for the next version of QGis 3.

Since py-qt5-core has flavours, is it possible to rename the file pyuic5 according to its flavor? This should avoid installation conflicts.
I think we need also to add a pkg-message, and aliases like py3uic5 for py36ui5 and py2uic5 for py27uic5.

my 2c.

Regards
Comment 33 Rainer Hurling freebsd_committer freebsd_triage 2018-01-15 11:35:30 UTC
(In reply to Antoine Brodin from comment #24)

I investigated a bit deeper and it seems, that there are several QT5 Python ports, which generate conflicts for py3x, if py27 is installed already:


pkg-static: py36-qt5-core-5.9.2 conflicts with py27-qt5-core-5.9.2 (installs files into the same place).
  Problematic file: /usr/local/bin/pyuic5

pkg-static: py36-qt5-gui-5.9.2 conflicts with py27-qt5-gui-5.9.2 (installs files into the same place).
  Problematic file: /usr/local/share/py-sip/PyQt5/QtGui/QtGuimod.sip

pkg-static: py36-qt5-widgets-5.9.2 conflicts with py27-qt5-widgets-5.9.2 (installs files into the same place).
  Problematic file: /usr/local/share/py-sip/PyQt5/QtWidgets/QtWidgetsmod.sip

pkg-static: py36-qt5-sql-5.9.2 conflicts with py27-qt5-sql-5.9.2 (installs files into the same place).
  Problematic file: /usr/local/share/py-sip/PyQt5/QtSql/QtSqlmod.sip

pkg-static: py36-qt5-printsupport-5.9.2 conflicts with py27-qt5-printsupport-5.9.2 (installs files into the same place).
  Problematic file: /usr/local/share/py-sip/PyQt5/QtPrintSupport/QtPrintSupportmod.sip

pkg-static: py36-qt5-qscintilla2-2.9.1 conflicts with py27-qt5-qscintilla2-2.9.1 (installs files into the same place).
  Problematic file: /usr/local/share/py-sip/PyQt5/Qsci/qsciabstractapis.sip

pkg-static: py36-qt5-svg-5.9.2 conflicts with py27-qt5-svg-5.9.2 (installs files into the same place).
  Problematic file: /usr/local/share/py-sip/PyQt5/QtSvg/QtSvgmod.sip

And so on ...


Obviously it is not enough to solve the problem only for the devel/py-qt5-core port. The patch suggested by Antoine Brodin in comment 24 would give us a working py-qt5-core port, for the others we probably need a change in the path of ../py-sip/..

Ting-Wei Lan submitted a patch in bug #224818, which looks interesting for these py-qt5 problems, also. He patched py-${PORTNAME} into ${PKGNAMEPREFIX}${PORTNAME}, which could be used here to change the path 

    py-sip into py[27|3x]-sip 
or 
    PyQt[45] into Py[27|3x]Qt[45] .

This is not tested, especially not for any regressions. I was not able to prepare a working patch here, sorry.
Comment 34 Loïc Bartoletti freebsd_committer freebsd_triage 2018-01-16 06:17:21 UTC
Hi,

I think Rainer is true. On OpenBSD, it seems that files are in py-sip/PyQt5-[2|3]

see
share/sip/PyQt5${MODPY_BIN_SUFFIX}

Link: http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/x11/py-qt5/pkg/PLIST?rev=1.1&content-type=text/x-cvsweb-markup
Comment 35 John W. O'Brien 2018-02-08 21:18:10 UTC
See also bug 225773, which includes a patch fixing the PYI file problem on devel/py-qt5-core, x11-toolkits/py-qt5-widgets, and x11-toolkits/py-qt5-gui. That patch allows me to fully package and install those three ports under all four python versions in the tree.
Comment 36 Walter Schwarzenfeld 2018-03-07 06:13:03 UTC
See comment35
and
https://svnweb.freebsd.org/ports?view=revision&revision=461841

This is fixed.
Comment 37 Rainer Hurling freebsd_committer freebsd_triage 2018-10-27 09:26:34 UTC
(In reply to w.schwarzenfeld from comment #36)

Hi Walter,

Sorry to answer here again. But this is not solved for me. If devel/py-qt5-core@py27 is installed already and I try to install devel/py-qt5-core@py36 additionally, it conflicts:

===>   Registering installation for py36-qt5-core-5.9.2_6
Installing py36-qt5-core-5.9.2_6...
pkg-static: py36-qt5-core-5.9.2_6 conflicts with py27-qt5-core-5.9.2_6 (installs files into the same place).  Problematic file: /usr/local/bin/pyuic5
*** Error code 70


In the bottom of comment #33 I tried to show, what works for me. Any chance to think again about the problem?
Comment 38 Walter Schwarzenfeld 2018-10-27 11:39:15 UTC
Ok, reopen
Comment 39 Tobias C. Berner freebsd_committer freebsd_triage 2018-10-27 12:19:23 UTC
*** Bug 223764 has been marked as a duplicate of this bug. ***
Comment 40 Tobias C. Berner freebsd_committer freebsd_triage 2018-10-27 12:20:18 UTC
See: 
https://reviews.freebsd.org/D8714
Comment 41 Tobias C. Berner freebsd_committer freebsd_triage 2018-10-27 18:28:20 UTC
*** Bug 223409 has been marked as a duplicate of this bug. ***
Comment 42 Tobias C. Berner freebsd_committer freebsd_triage 2018-11-19 06:16:16 UTC
*** Bug 233312 has been marked as a duplicate of this bug. ***
Comment 43 Tobias C. Berner freebsd_committer freebsd_triage 2018-11-19 06:16:45 UTC
*** Bug 233311 has been marked as a duplicate of this bug. ***
Comment 44 Tobias C. Berner freebsd_committer freebsd_triage 2018-11-19 06:17:21 UTC
*** Bug 233313 has been marked as a duplicate of this bug. ***