Bug 265022 - x11/xcb-proto: Fails to package: python3.7/site-packages/xcbgen/__pycache__/*.pyc: No such file or directory
Summary: x11/xcb-proto: Fails to package: python3.7/site-packages/xcbgen/__pycache__/*...
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-x11 (Nobody)
URL:
Keywords: needs-patch, needs-qa
Depends on:
Blocks:
 
Reported: 2022-07-04 01:29 UTC by Tom
Modified: 2022-07-27 13:05 UTC (History)
7 users (show)

See Also:
bugzilla: maintainer-feedback? (x11)
vincent.jancso: maintainer-feedback? (bapt)
koobs: maintainer-feedback? (python)
koobs: merge-quarterly?


Attachments
pkg-plist fix for Python 3.7 (1.40 KB, patch)
2022-07-27 12:50 UTC, Miroslav Lachman
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tom 2022-07-04 01:29:37 UTC
Hi,

When attempting to install xcb-proto using portmaster I get the following error:

===>  Installing for xcb-proto-1.15.2
===>  Checking if xcb-proto is already installed
===>   Registering installation for xcb-proto-1.15.2 as automatic
pkg-static: Unable to access file /usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/__init__.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/align.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/error.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/expr.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/matcher.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/state.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/xtypes.opt-2.pyc:No such file or directory
*** Error code 1


Cheers.
Comment 1 vincent.jancso 2022-07-04 06:55:16 UTC
Same problem here when building:

===>  Building package for xcb-proto-1.15.2
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/__init__.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/align.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/error.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/expr.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/matcher.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/state.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/xtypes.opt-2.pyc:No such file or directory
*** Error code 1
Comment 2 Baptiste Daroussin freebsd_committer freebsd_triage 2022-07-04 07:58:54 UTC
this is due to python 3.7 please update to the current default which is python 3.9
Comment 3 Tom 2022-07-04 09:50:09 UTC
That's quite strict. 3.7 has the same support tier as 3.9 AFAIK, with 3.10 being the current Python3. Is 3.9 the only version that will work or would 3.10 be ok too?
Comment 4 Michael Osipov 2022-07-04 11:17:36 UTC
(In reply to Tom from comment #3)
Just bumped into this as well. Don't believe that only 3.9+ is supported...
Comment 5 Baptiste Daroussin freebsd_committer freebsd_triage 2022-07-04 17:33:29 UTC
I am not saying we are limited to 3.9+ I am saying that _pycache_ thing is a pythonery I don't know and I don't know yet how this is supposed to be fixed, in the meantime 3.9+ works so I am stating it ;) does not mean it should not be fixed
Comment 6 Michael Osipov 2022-07-04 17:35:10 UTC
(In reply to Baptiste Daroussin from comment #5)

OK, I see. For the time being I have patched the Makefile of this port for python:3.9+ and it continued to work, but this is not for the faint of heart.
Comment 7 Vladimir Druzenko freebsd_committer freebsd_triage 2022-07-05 12:53:16 UTC
(In reply to Tom from comment #3)
Work for me with 3.10.
Comment 8 Michael Osipov 2022-07-05 18:28:18 UTC
This is how it looks in a poudriere jail:
root@123-release-amd64-default-ldadw-base:/wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__ # ll | grep opt-2
-rw-r--r--  1 root  wheel    144 Jul  5 18:23 __init__.cpython-37.opt-2.pyc
-rw-r--r--  1 root  wheel   4631 Jul  5 18:23 align.cpython-37.opt-2.pyc
-rw-r--r--  1 root  wheel    308 Jul  5 18:23 error.cpython-37.opt-2.pyc
-rw-r--r--  1 root  wheel   3448 Jul  5 18:23 expr.cpython-37.opt-2.pyc
-rw-r--r--  1 root  wheel   3801 Jul  5 18:23 matcher.cpython-37.opt-2.pyc
-rw-r--r--  1 root  wheel   5916 Jul  5 18:23 state.cpython-37.opt-2.pyc
-rw-r--r--  1 root  wheel  31137 Jul  5 18:23 xtypes.cpython-37.opt-2.pyc

This is what the pkg-plist wants:
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/__init__.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/align.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/error.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/expr.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/matcher.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/state.opt-2.pyc:No such file or directory
pkg-static: Unable to access file /wrkdirs/usr/ports/x11/xcb-proto/work/stage/usr/local/lib/python3.7/site-packages/xcbgen/__pycache__/xtypes.opt-2.pyc:No such file or directory
Comment 9 Michael Osipov 2022-07-05 19:00:41 UTC
Some more investigation: The used PYTHON_EXT_SUFFIX is empty before Python 3.8.7. Other ports use .cpython-%%PYTHON_SUFFIX%% for this reason. I have tried in the pkg-plist: %%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/__init__.cpython-%%PYTHON_SUFFIX%%.opt-2.pyc

Haven't tested with 3.8+, but logically this should work. 

Now, either this port is broken or these lines are:
https://github.com/freebsd/freebsd-ports/blob/dee560336f0d0bc6e38bdecbe92edf370f4e2905/Mk/Uses/python.mk#L505-L509

Here it is: https://github.com/freebsd/freebsd-ports/commit/fd8c799f257f75622e7f1a9f74b4e1130002ef74

I have the feeling that it has been backported to 3.7.x at some point in time.

root@123-release-amd64-default-ldadw-base:/usr/ports/x11/xcb-proto # grep -r -l PYTHON_SUFFIX /usr/ports | cut -f 4-5 -d / | sort -u
accessibility/accerciser
accessibility/py-speech-dispatcher
accessibility/redshift
archivers/py-lz4
archivers/py-rjsmin
astro/py-jplephem
astro/py-sgp4
astro/py-skyfield
astro/py-skyfield-data
astro/py-spktype01
astro/py-spktype21
astro/weather
biology/hmmer
biology/infernal
biology/py-PySCeS
cad/astk-serveur
cad/freecad
cad/libredwg
chinese/ibus-cangjie
chinese/pycangjie
databases/arrow
databases/grass7
databases/ldb15
databases/ldb20
databases/ldb21
databases/ldb22
databases/py-apache-arrow
databases/py-duckdb
databases/py-gdbm
databases/py-sqlite3
databases/py-xapian
databases/redisdesktopmanager
databases/tdb
deskutils/py-autokey
deskutils/py-term-background
deskutils/recoll
deskutils/treeline
devel/boost-python-libs
devel/cmake
devel/distcc
devel/gitg
devel/gobject-introspection
devel/icestorm
devel/ikos
devel/libsearpc
devel/mercurial
devel/py-cysignals
devel/py-gobject3
devel/py-guppy3
devel/py-jaraco
devel/py-odfpy
devel/py-pysimdjson
devel/pyside2
devel/pyside2-tools
devel/shiboken2
devel/talloc
devel/tevent
devel/tortoisehg
devel/youcompleteme
emulators/virtualbox-ose
emulators/virtualbox-ose-legacy
finance/gnucash
french/aster
french/homard
games/anki
games/monsterz
games/openage
games/whichwayisup
graphics/eog-plugins
graphics/lensfun
graphics/py-cairocffi
graphics/py-mcomix
graphics/py-python-poppler-qt5
japanese/ibus-anthy
lang/python-tools
lang/python310
lang/python311
lang/python37
lang/python38
lang/python39
mail/mailutils
math/cadabra2
math/convertall
math/e-antic
math/mathicsscript
math/py-brial
math/py-cypari2
math/py-fpylll
math/py-mathics
math/py-mathics-scanner
math/py-matplotlib
math/py-pandas
math/py-piranha
math/py-pplpy
math/py-rpy2
math/py-spot
math/rpcalc
math/sage
misc/hashdb
misc/veles
multimedia/gpodder
multimedia/gstreamer1-validate
multimedia/mythtv
multimedia/photofilmstrip
multimedia/pitivi
multimedia/py-mlt6
net-mgmt/net-snmp
net-mgmt/seafile-client
net-mgmt/seafile-server
net-p2p/py-libtorrent-rasterbar
net/ceph14
net/glusterfs
net/pacemaker1
net/samba412
net/samba413
print/lilypond-devel
print/lyx
print/system-config-printer
science/cantera
science/code_saturne
science/elmerfem
science/openbabel
science/py-asap3
science/py-dipy
science/py-dlib
science/py-jupyter_jsmol
science/py-pymol
science/py-pyprecice
science/tfel-edf
science/v_sim
security/apkid
security/py-pass-audit
security/sssd
security/zeronet
sysutils/calamares
sysutils/catfish
sysutils/datadog-integrations
sysutils/py-salt
sysutils/scanmem
sysutils/syslog-ng
sysutils/xen-tools
textproc/gtk-doc
textproc/ibus-typing-booster
textproc/meld
textproc/py-enchant
textproc/py-langid
textproc/py-mathics-pygments
textproc/py-wordcloud
www/npm
www/py-seafdav
www/py-seafobj
www/unit-python
x11-toolkits/py-tkinter
x11-wm/chamfer
x11/guake
x11/mate-dock-applet
x11/menulibre
x11/py-xcffib
x11/xcb-proto
x11/xpra

vs 

root@123-release-amd64-default-ldadw-base:/usr/ports/x11/xcb-proto # grep -r -l PYTHON_EXT_SUFFIX /usr/ports | cut -f 4-5 -d / | sort -u
accessibility/orca
archivers/py-lz4
astro/astrometry
astro/kosmorro
astro/py-ephem
astro/py-kosmorrolib
astro/py-sgp4
audio/picard-plugins
biology/py-python-libsbml
cad/klayout
cad/netgen
comms/hamlib
converters/py-bsdconv
databases/grass8
databases/ldb15
databases/ldb20
databases/ldb21
databases/ldb22
databases/py-apache-arrow
databases/py-xapian
databases/pymongo
databases/rrdtool
databases/tdb
deskutils/gnome-tweaks
deskutils/mozo
deskutils/py-paperless
deskutils/recoll
devel/cbang
devel/cpplint
devel/git-cola
devel/gobject-introspection
devel/kf5-kapidox
devel/llvm-devel
devel/llvm14
devel/omniORB
devel/py-cld
devel/py-cysignals
devel/py-gobject3
devel/py-grpcio
devel/py-guppy3
devel/py-ice37
devel/py-jep
devel/py-jsonnet
devel/py-memory-allocator
devel/py-omniorb
devel/py-pysimdjson
devel/py-qt5-pyqt
devel/talloc
devel/tevent
devel/youcompleteme
editors/pluma
editors/pluma-plugins
finance/odoo
games/openage
graphics/mypaint
graphics/opencv
graphics/openimageio
graphics/py-cairo
graphics/py-python-poppler-qt5
math/cadabra2
math/openturns
math/py-cypari2
math/py-dionysus
math/py-fpylll
math/py-heyoka
math/py-kahip
math/py-mathics
math/py-matplotlib
math/py-nlopt
math/py-pandas
math/py-pplpy
math/py-primecountpy
math/py-primme
math/py-spot
math/py-topologic
math/sage
misc/adios2
misc/py-ezc3d
misc/py-xgboost
multimedia/onevpl
multimedia/openshot
multimedia/py-gstreamer1
multimedia/py-opentimelineio
net-mgmt/crmsh
net-mgmt/net-snmp
net/ceph14
net/kea
net/ns3
net/pacemaker2
net/py-libdnet
net/samba412
net/samba413
print/lilypond
print/pdfarranger
science/agrum
science/cantera
science/code_saturne
science/py-PyNE
science/py-avogadrolibs
science/py-fresnel
science/py-hoomd-blue
science/py-libgetar
science/py-pygmo2
science/py-pymol
science/tfel
science/yoda
security/libpreludedb
security/libpwquality
security/py-gpgme
security/py-ospd-openvas
security/py-yara-python-dex
security/silktools
security/suricata
sysutils/android-file-transfer
sysutils/bareos-server
sysutils/bareos20-server
sysutils/openzfs
sysutils/webmin
sysutils/xen-tools
textproc/meld
textproc/py-hunspell
textproc/py-wordcloud
textproc/redland-bindings
x11-toolkits/nanogui
x11-wm/qtile
x11/xpra

This now needs someone who knows Python internals better than me.
Comment 10 Michael Osipov 2022-07-05 19:15:25 UTC
Good example working on 3.7: devel/py-cysignals
I will retest this port with 3.8 and 3.9+ tomorrow. As of now I am convinced that the pkg-plist is just incomplete.
Comment 11 Michael Osipov 2022-07-05 21:30:04 UTC
Mystery solved, 3.7 through 3.10 tried in clean poudriere jails: pkg-plist is broken. Here is a patch:
root@deblndw011x:/var/poudriere/ports/default/x11/xcb-proto (2022Q3 *=)
# git diff
diff --git a/x11/xcb-proto/pkg-plist b/x11/xcb-proto/pkg-plist
index 723b65d4fb13..0f79c71d935d 100644
--- a/x11/xcb-proto/pkg-plist
+++ b/x11/xcb-proto/pkg-plist
@@ -53,10 +53,10 @@ libdata/pkgconfig/xcb-proto.pc
 %%PYTHON_SITELIBDIR%%/xcbgen/xtypes.py
 %%PYTHON_SITELIBDIR%%/xcbgen/xtypes.pyc
 %%PYTHON_SITELIBDIR%%/xcbgen/xtypes.pyo
-%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/__init__%%PYTHON_EXT_SUFFIX%%.opt-2.pyc
-%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/align%%PYTHON_EXT_SUFFIX%%.opt-2.pyc
-%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/error%%PYTHON_EXT_SUFFIX%%.opt-2.pyc
-%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/expr%%PYTHON_EXT_SUFFIX%%.opt-2.pyc
-%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/matcher%%PYTHON_EXT_SUFFIX%%.opt-2.pyc
-%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/state%%PYTHON_EXT_SUFFIX%%.opt-2.pyc
-%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/xtypes%%PYTHON_EXT_SUFFIX%%.opt-2.pyc
+%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/__init__.cpython-%%PYTHON_SUFFIX%%.opt-2.pyc
+%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/align.cpython-%%PYTHON_SUFFIX%%.opt-2.pyc
+%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/error.cpython-%%PYTHON_SUFFIX%%.opt-2.pyc
+%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/expr.cpython-%%PYTHON_SUFFIX%%.opt-2.pyc
+%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/matcher.cpython-%%PYTHON_SUFFIX%%.opt-2.pyc
+%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/state.cpython-%%PYTHON_SUFFIX%%.opt-2.pyc
+%%PYTHON_SITELIBDIR%%/xcbgen/__pycache__/xtypes.cpython-%%PYTHON_SUFFIX%%.opt-2.pyc

Please verify, apply to main and 2022Q3 and let's move on.

As far as I understand %%PYTHON_EXT_SUFFIX%% is only meant to be used for shared objects.
Comment 12 Michael Osipov 2022-07-19 18:40:49 UTC
Can this be applied and we move on?
Comment 13 Nathan Robertson 2022-07-27 01:32:25 UTC
(In reply to Michael Osipov from comment #11)

Michael's suggestion / patch is exactly the way this same was resolved for the math/py-pandas port.

Bugzilla link for the same bug against that port: https://bugs.freebsd.org/bugzilla/show_bug.cgi?format=multiple&id=263218

And the patch that was merged as part of that bug that resolved the issue: https://cgit.freebsd.org/ports/commit/?id=7ad549afde193b795d59d6148462c970e0b28667

I can't upgrade to Python 3.9 (we have library dependencies for our software that breaks on Python 3.8), and our packages are build with Poudriere, which is a pain to manually patch. Could you please apply the suggested fix (as was done for py-pandas)? It seems a simple fix to maintain backward compatibility.
Comment 14 Miroslav Lachman 2022-07-27 12:50:18 UTC
Created attachment 235505 [details]
pkg-plist fix for Python 3.7

Patch proposed by Michael Osipov fixes the problem for our build with Poudriere and Python 3.7. Please commit it to 2022Q3.
Comment 15 Michael Osipov 2022-07-27 12:51:55 UTC
(In reply to Miroslav Lachman from comment #14)

main as well!
Comment 16 commit-hook freebsd_committer freebsd_triage 2022-07-27 13:03:10 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=a85a4e867a3bc662c20c8d4f61461db94005d8a3

commit a85a4e867a3bc662c20c8d4f61461db94005d8a3
Author:     Baptiste Daroussin <bapt@FreeBSD.org>
AuthorDate: 2022-07-27 13:00:58 +0000
Commit:     Baptiste Daroussin <bapt@FreeBSD.org>
CommitDate: 2022-07-27 13:02:47 +0000

    x11/xcb-proto: fix build with any version of python

    PR:             265022
    Submitted by:   michael.osipov@siemens.com
    Reported by:    many
    Pointyhat to:   bapt

 x11/xcb-proto/pkg-plist | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
Comment 17 commit-hook freebsd_committer freebsd_triage 2022-07-27 13:05:11 UTC
A commit in branch 2022Q3 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=04f717e8125f4d5570d2d3a72c9a8512c026b996

commit 04f717e8125f4d5570d2d3a72c9a8512c026b996
Author:     Baptiste Daroussin <bapt@FreeBSD.org>
AuthorDate: 2022-07-27 13:00:58 +0000
Commit:     Baptiste Daroussin <bapt@FreeBSD.org>
CommitDate: 2022-07-27 13:04:52 +0000

    x11/xcb-proto: fix build with any version of python

    PR:             265022
    Submitted by:   michael.osipov@siemens.com
    Reported by:    many
    Pointyhat to:   bapt

    (cherry picked from commit a85a4e867a3bc662c20c8d4f61461db94005d8a3)

 x11/xcb-proto/pkg-plist | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
Comment 18 Baptiste Daroussin freebsd_committer freebsd_triage 2022-07-27 13:05:42 UTC
fixed, thank you and sorry about it