Bug 258377 - lang/python3?: Disable detection of multiarch for Clang 13
Summary: lang/python3?: Disable detection of multiarch for Clang 13
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Dimitry Andric
URL:
Keywords:
Depends on:
Blocks: 258209
  Show dependency treegraph
 
Reported: 2021-09-08 22:14 UTC by Dimitry Andric
Modified: 2022-01-05 22:12 UTC (History)
2 users (show)

See Also:
koobs: merge-quarterly?


Attachments
Stub out multiarch detection in lang/python3X ports for clang 13 (4.26 KB, patch)
2021-09-08 22:14 UTC, Dimitry Andric
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Dimitry Andric freebsd_committer freebsd_triage 2021-09-08 22:14:46 UTC
Created attachment 227773 [details]
Stub out multiarch detection in lang/python3X ports for clang 13

During an exp-run for llvm 13 (see bug 258209), it turned out that lang/python3([6-9]|10) do not build with clang 13.0.0 [1][2][3][4][5]:

LD_LIBRARY_PATH=/wrkdirs/usr/ports/lang/python36/work/Python-3.6.14 CC='cc' LDSHARED='cc -shared  -lpthread -L/usr/local/lib  -fstack-protector-strong   ' OPT='-DNDEBUG'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''  ./python -E ./setup.py -q build
Failed to import the site module
Traceback (most recent call last):
  File "/wrkdirs/usr/ports/lang/python36/work/Python-3.6.14/Lib/site.py", line 553, in <module>
    main()
  File "/wrkdirs/usr/ports/lang/python36/work/Python-3.6.14/Lib/site.py", line 539, in main
    known_paths = addusersitepackages(known_paths)
  File "/wrkdirs/usr/ports/lang/python36/work/Python-3.6.14/Lib/site.py", line 282, in addusersitepackages
    user_site = getusersitepackages()
  File "/wrkdirs/usr/ports/lang/python36/work/Python-3.6.14/Lib/site.py", line 258, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/wrkdirs/usr/ports/lang/python36/work/Python-3.6.14/Lib/site.py", line 248, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/wrkdirs/usr/ports/lang/python36/work/Python-3.6.14/Lib/sysconfig.py", line 601, in get_config_var
    return get_config_vars().get(name)
  File "/wrkdirs/usr/ports/lang/python36/work/Python-3.6.14/Lib/sysconfig.py", line 550, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/wrkdirs/usr/ports/lang/python36/work/Python-3.6.14/Lib/sysconfig.py", line 421, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata_m_freebsd14_x86_64-unknown-freebsd14'
*** Error code 1

The reason this breaks with clang >= 13, is that it now started supporting the -print-multiarch option, but in its output it adds a major.minor version number, for example x86_64-unknown-freebsd14.0. The dot confuses Python, as it appends the multiarch name to the sysconfig module name, and then can't load the resulting _sysconfigdata__freebsd14_x86_64-unknown-freebsd14.0.py file:

    ModuleNotFoundError: No module named '_sysconfigdata__freebsd14_x86_64-unknown-freebsd14'

Since we do not support multiarch, and the configure script has no way to disable the multiarch check, I would like to propose to just stub it out during post-patch, as per the attached diff.

[1] http://gohan04.nyi.freebsd.org/data/mainamd64PR258209-default/2021-09-05_20h27m09s/logs/errors/python36-3.6.14.log
[2] http://gohan04.nyi.freebsd.org/data/mainamd64PR258209-default/2021-09-05_20h27m09s/logs/errors/python37-3.7.11.log
[3] http://gohan04.nyi.freebsd.org/data/mainamd64PR258209-default/2021-09-05_20h27m09s/logs/errors/python38-3.8.11.log
[4] http://gohan04.nyi.freebsd.org/data/mainamd64PR258209-default/2021-09-05_20h27m09s/logs/errors/python39-3.9.7.log
[5] http://gohan04.nyi.freebsd.org/data/mainamd64PR258209-default/2021-09-05_20h27m09s/logs/errors/python310-3.10.0.r1.log
Comment 1 Dimitry Andric freebsd_committer freebsd_triage 2021-09-08 22:16:04 UTC
Comment on attachment 227773 [details]
Stub out multiarch detection in lang/python3X ports for clang 13

>commit f7c6323844a874743c0d8a0400a41fbc7af197db
>Author: Dimitry Andric <dim@FreeBSD.org>
>Date:   Wed Sep 8 23:59:36 2021 +0200
>
>    lang/python3([6-9]|10): disable detection of multiarch
>    
>    It breaks with clang >= 13, which adds a major.minor version number in
>    -print-multiarch output, and the dot confuses Python:
>    
>    ModuleNotFoundError: No module named '_sysconfigdata__freebsd14_x86_64-unknown-freebsd14'
>    
>    Since we do not support multiarch, and the configure script has no way
>    to disable the multiarch check, stub it out during post-patch.
>
>diff --git a/lang/python310/Makefile b/lang/python310/Makefile
>index 3dbf8d5d004e..64aba656d340 100644
>--- a/lang/python310/Makefile
>+++ b/lang/python310/Makefile
>@@ -124,6 +124,10 @@ post-patch:
> # which introduces hidden dependency and breaks build
> 	@${REINPLACE_CMD} -e 's|uuid/uuid.h|ignore_&|' ${WRKSRC}/configure
> 	@${REINPLACE_CMD} -e '/find_library_file/ s|uuid|ignore_&|' ${WRKSRC}/setup.py
>+# disable detection of multiarch as it breaks with clang >= 13, which adds a
>+# major.minor version number in -print-multiarch output, confusing Python
>+	@${REINPLACE_CMD} -e 's|^MULTIARCH=.*|MULTIARCH=|' ${WRKSRC}/configure \
>+		${WRKSRC}/configure.ac
> # Apply DISABLED_EXTENSIONS
> 	@${ECHO_CMD} '*disabled*' > ${WRKSRC}/Modules/Setup.local
> .  for _module in ${DISABLED_EXTENSIONS}
>diff --git a/lang/python36/Makefile b/lang/python36/Makefile
>index 636f8d198350..b52d72bc3e06 100644
>--- a/lang/python36/Makefile
>+++ b/lang/python36/Makefile
>@@ -121,6 +121,12 @@ DISABLED_EXTENSIONS+=	nis
> PLIST_SUB+=	NO_NIS=""
> .endif
> 
>+post-patch:
>+# disable detection of multiarch as it breaks with clang >= 13, which adds a
>+# major.minor version number in -print-multiarch output, confusing Python
>+	@${REINPLACE_CMD} -e 's|^MULTIARCH=.*|MULTIARCH=|' ${WRKSRC}/configure \
>+		${WRKSRC}/configure.ac
>+
> post-install:
> .if ! ${PORT_OPTIONS:MDEBUG}
> 	${RM} ${STAGEDIR}${PREFIX}/lib/libpython3.so						# Upstream Issue: https://bugs.python.org/issue17975
>diff --git a/lang/python37/Makefile b/lang/python37/Makefile
>index 8909af0dd7ab..4db00e66e9d2 100644
>--- a/lang/python37/Makefile
>+++ b/lang/python37/Makefile
>@@ -111,6 +111,10 @@ post-patch:
> # which introduces hidden dependency and breaks build
> 	@${REINPLACE_CMD} -e 's|uuid/uuid.h|ignore_&|' ${WRKSRC}/configure
> 	@${REINPLACE_CMD} -e '/find_library_file/ s|uuid|ignore_&|' ${WRKSRC}/setup.py
>+# disable detection of multiarch as it breaks with clang >= 13, which adds a
>+# major.minor version number in -print-multiarch output, confusing Python
>+	@${REINPLACE_CMD} -e 's|^MULTIARCH=.*|MULTIARCH=|' ${WRKSRC}/configure \
>+		${WRKSRC}/configure.ac
> # Apply DISABLED_EXTENSIONS
> 	@${ECHO_CMD} '*disabled*' > ${WRKSRC}/Modules/Setup.local
> .  for _module in ${DISABLED_EXTENSIONS}
>diff --git a/lang/python38/Makefile b/lang/python38/Makefile
>index 68359a8579a8..f3dcec0757fa 100644
>--- a/lang/python38/Makefile
>+++ b/lang/python38/Makefile
>@@ -115,6 +115,10 @@ post-patch:
> # which introduces hidden dependency and breaks build
> 	@${REINPLACE_CMD} -e 's|uuid/uuid.h|ignore_&|' ${WRKSRC}/configure
> 	@${REINPLACE_CMD} -e '/find_library_file/ s|uuid|ignore_&|' ${WRKSRC}/setup.py
>+# disable detection of multiarch as it breaks with clang >= 13, which adds a
>+# major.minor version number in -print-multiarch output, confusing Python
>+	@${REINPLACE_CMD} -e 's|^MULTIARCH=.*|MULTIARCH=|' ${WRKSRC}/configure \
>+		${WRKSRC}/configure.ac
> # Apply DISABLED_EXTENSIONS
> 	@${ECHO_CMD} '*disabled*' > ${WRKSRC}/Modules/Setup.local
> .  for _module in ${DISABLED_EXTENSIONS}
>diff --git a/lang/python39/Makefile b/lang/python39/Makefile
>index 4a0d31ba17f2..2d052cf3741d 100644
>--- a/lang/python39/Makefile
>+++ b/lang/python39/Makefile
>@@ -116,6 +116,10 @@ post-patch:
> # which introduces hidden dependency and breaks build
> 	@${REINPLACE_CMD} -e 's|uuid/uuid.h|ignore_&|' ${WRKSRC}/configure
> 	@${REINPLACE_CMD} -e '/find_library_file/ s|uuid|ignore_&|' ${WRKSRC}/setup.py
>+# disable detection of multiarch as it breaks with clang >= 13, which adds a
>+# major.minor version number in -print-multiarch output, confusing Python
>+	@${REINPLACE_CMD} -e 's|^MULTIARCH=.*|MULTIARCH=|' ${WRKSRC}/configure \
>+		${WRKSRC}/configure.ac
> # Apply DISABLED_EXTENSIONS
> 	@${ECHO_CMD} '*disabled*' > ${WRKSRC}/Modules/Setup.local
> .  for _module in ${DISABLED_EXTENSIONS}
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2021-09-09 01:22:28 UTC
Thank you for the report and patch Dim. Would an upstream --disable-multiarch or similar configure option or anything else be useful?

Also, can you confirm whether python27 is affected as well?
Comment 3 Dimitry Andric freebsd_committer freebsd_triage 2021-09-09 07:34:05 UTC
(In reply to Kubilay Kocak from comment #2)
> Would an upstream --disable-multiarch or similar configure option or anything else be useful?

Probably; as jrtc27@ remarked yesterday already:

<@jrtc27> and surprised there's no configure option for it... having it always inferred seems like a bad idea
<@jrtc27> would expect a normal --with-multiarch=(auto|triple) and --without-multiarch set of options


> Also, can you confirm whether python27 is affected as well?

It has exactly the same configure.ac/configure parts, but doesn't actually seem to *do* anything with the detected MULTIARCH variable, so it works out of the box with clang 13. But maybe we can also patch it similarly, for the sake of consistency. (And not bother submitting it upstream, as that no longer exists ;)
Comment 4 commit-hook freebsd_committer freebsd_triage 2021-09-27 17:57:29 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=6bef09666460fb9f9052550854fa69f5d5e4a0e8

commit 6bef09666460fb9f9052550854fa69f5d5e4a0e8
Author:     Dimitry Andric <dim@FreeBSD.org>
AuthorDate: 2021-09-27 17:39:05 +0000
Commit:     Dimitry Andric <dim@FreeBSD.org>
CommitDate: 2021-09-27 17:56:59 +0000

    lang/python3([6-9]|10): disable detection of multiarch

    It breaks with clang >= 13, which adds a major.minor version number in
    -print-multiarch output, and the dot confuses Python:

    ModuleNotFoundError: No module named '_sysconfigdata__freebsd14_x86_64-unknown-freebsd14'

    Since we do not support multiarch, and the configure script has no way
    to disable the multiarch check, stub it out during post-patch.

    PR:             258377
    Approved by:    maintainer timeout (2 weeks)
    MFH:            2021Q3

 lang/python310/Makefile | 4 ++++
 lang/python36/Makefile  | 6 ++++++
 lang/python37/Makefile  | 4 ++++
 lang/python38/Makefile  | 4 ++++
 lang/python39/Makefile  | 4 ++++
 5 files changed, 22 insertions(+)
Comment 5 commit-hook freebsd_committer freebsd_triage 2021-11-26 18:49:43 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=4af982e6bea019a33ae85c6c92a3042fadccb739

commit 4af982e6bea019a33ae85c6c92a3042fadccb739
Author:     Dimitry Andric <dim@FreeBSD.org>
AuthorDate: 2021-11-26 16:02:16 +0000
Commit:     Dimitry Andric <dim@FreeBSD.org>
CommitDate: 2021-11-26 18:48:57 +0000

    lang/python3(6|7|8|9|10|11): work around unwanted multiarch detection again

    In ports 6bef09666460 we patched out python's erroneous multiarch
    detection using sed, but upstream changed the surrounding parts in the
    configure script to defeat the regex we were using. Update the regex to
    cope with the change.

    PR:             259896, 258377
    MFH:            2021Q4
    Approved by:    maintainer timeout (9 days)

 lang/python310/Makefile | 3 +--
 lang/python311/Makefile | 3 +--
 lang/python36/Makefile  | 3 +--
 lang/python37/Makefile  | 3 +--
 lang/python38/Makefile  | 3 +--
 lang/python39/Makefile  | 3 +--
 6 files changed, 6 insertions(+), 12 deletions(-)
Comment 6 gpshead 2022-01-05 21:53:50 UTC
Please file bugs upstream against CPython any time you find yourself patching something to support your platform. Every patch is technical debt.

https://bugs.python.org/issue46263 didn't need to happen so late.
Comment 7 Kubilay Kocak freebsd_committer freebsd_triage 2022-01-05 22:12:22 UTC
(In reply to gpshead from comment #6)

We are usually very good with upstreaming. Thank you for the heads up