Bug 232510 - lang/python37: -L argument order causes build failure in presence of (installed, olrder) python37
Summary: lang/python37: -L argument order causes build failure in presence of (install...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: Normal Affects Many People
Assignee: Wen Heping
URL:
Keywords: needs-qa, regression
: 232795 234409 234467 (view as bug list)
Depends on:
Blocks:
 
Reported: 2018-10-21 19:07 UTC by Simeon Simeonov
Modified: 2019-01-08 07:21 UTC (History)
11 users (show)

See Also:
bugzilla: maintainer-feedback? (python)


Attachments
Build error log on 11.2-RELEASE amd64 host (98.26 KB, text/plain)
2018-10-22 05:09 UTC, Yasuhiro KIMURA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Simeon Simeonov 2018-10-21 19:07:53 UTC
The new port for Python 3.7.1 fails with:

--- libpython3.so ---
cc -pthread -shared  -L/usr/local/lib -Wl,-rpath,/usr/local/lib -fstack-protector  -Wl,--no-as-needed -o libpython3.so -Wl,-hlibpython3.so 
--- python ---
--- Programs/_testembed ---
--- python ---
cc -pthread  -L/usr/local/lib -Wl,-rpath,/usr/local/lib -fstack-protector  -Wl,--export-dynamic -o python Programs/python.o -L. -lpython3.7m -lintl -ldl -L/usr/local/lib -lintl -lutil   -lm  
--- Programs/_testembed ---
cc -pthread  -L/usr/local/lib -Wl,-rpath,/usr/local/lib -fstack-protector  -Wl,--export-dynamic -o Programs/_testembed Programs/_testembed.o -L. -lpython3.7m -lintl -ldl -L/usr/local/lib -lintl -lutil   -lm  
Programs/_testembed.o: In function `test_init_from_config':
_testembed.c:(.text+0xd28): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_dev_mode':
_testembed.c:(.text+0xe0f): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_isolated':
_testembed.c:(.text+0xeb3): undefined reference to `_Py_InitializeFromConfig'
--- pybuilddir.txt ---
--- Programs/_testembed ---
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** [Programs/_testembed] Error code 1

make[2]: stopped in /usr/ports/lang/python37/work/Python-3.7.1
--- pybuilddir.txt ---
LD_LIBRARY_PATH=/usr/ports/lang/python37/work/Python-3.7.1 ./python -E -S -m sysconfig --generate-posix-vars ; if test $? -ne 0 ; then  echo "generate-posix-vars failed" ;  rm -f ./pybuilddir.txt ;  exit 1 ;  fi


[root@SagaBOX /usr/ports/lang/python37]# make showconfig
===> The following configuration options are available for python37-3.7.1:
     DEBUG=off: Build with debugging support
     IPV6=on: IPv6 protocol support
     LIBFFI=on: Use libffi from ports instead of bundled version
     NLS=on: Enable gettext support for the locale module
     PYMALLOC=on: Enable specialized mallocs
====> Hash Algorithm (PEP-456): you can only select none or one of them
     FNV=off: Modified Fowler-Noll-Vo Algorithm
     SIPHASH=off: SipHash24 Algorithm
===> Use 'make config' to modify these settings
Comment 1 w.schwarzenfeld freebsd_triage 2018-10-21 19:33:56 UTC
see also https://lists.freebsd.org/pipermail/freebsd-ports/2018-October/114515.html

(I have no problem with it on 11.2-RELEASE-amd64).
Comment 2 Simeon Simeonov 2018-10-21 20:05:16 UTC
I am running:
FreeBSD 11.2-STABLE #0 r339176 (amd64)

with /etc/make.conf

CFLAGS=-O2 -pipe -fno-strict-aliasing

## kernel ##
KERNCONF=SagaBOX
COPTFLAGS=-O2 -pipe -funroll-loops -ffast-math -fno-strict-aliasing

## world ##
MALLOC_PRODUCTION=yes
SSL_OP_NO_SSLv2=yes
SSL_OP_NO_SSLv3=yes
security_openssl_UNSET=SSL2 SSL3

## ports ##
OPTIONS_UNSET=DEBUG DOCS X11
OPTIONS_SET=OPTIMIZED_CFLAGS
DEFAULT_VERSIONS=python=3.7 python3=3.7 php=7.2 ssl=libressl
DEFAULT_VERSIONS+=perl5=5.26


I managed to build Python 3.7.0 yesterday on this very same system...
Comment 3 w.schwarzenfeld freebsd_triage 2018-10-21 20:19:11 UTC
Maybe something with the path to the compiler
if do
make CC=clang60 CXX=clang++60 CPP=clang-cpp60 (could be any other compiler)

I got:
--- python ---
./libpython3.7m.so: undefined reference to `pthread_getcpuclockid'
./libpython3.7m.so: undefined reference to `pthread_create'
--- Programs/_testembed ---
./libpython3.7m.so: undefined reference to `pthread_getcpuclockid'
--- python ---
clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
--- Programs/_testembed ---
./libpython3.7m.so: undefined reference to `pthread_create'
clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
--- python ---

if i do
make CC=/usr/local/bin/clang60 CXX=/usr/local/bin/clang++60 CPP=/usr/local/bin/clang-cpp60

it compiles fine.
Comment 4 rkoberman 2018-10-21 20:33:31 UTC
No problem building the port on my 11.2-STABLE r338990 amd64 system.

All default configure options unless defaults have changed since the port was first installed.

==> The following configuration options are available for python37-3.7.0_3:
     DEBUG=off: Build with debugging support
     IPV6=on: IPv6 protocol support
     LIBFFI=on: Use libffi from ports instead of bundled version
     NLS=on: Enable gettext support for the locale module
     PYMALLOC=on: Enable specialized mallocs
====> Hash Algorithm (PEP-456): you can only select none or one of them
     FNV=off: Modified Fowler-Noll-Vo Algorithm
     SIPHASH=off: SipHash24 Algorithm
Comment 5 Simeon Simeonov 2018-10-21 20:42:05 UTC
"python37-3.7.0_3"...?
Are you building the latest port?
Comment 6 w.schwarzenfeld freebsd_triage 2018-10-21 20:49:48 UTC
The  latest is 3.7.1.
Comment 7 Simeon Simeonov 2018-10-21 21:10:34 UTC
Hmmm... The port builds fine on a jail running the exactly same system (hosted by the "broken" system) with the same /etc/make.conf...
So yes... everything points to some conflict and / or some legacy garbage...
# cd /usr/src && make check-old && make check-old-libs
doesn't give me anything... and neither does:
# mergemaster -F
Comment 8 Simeon Simeonov 2018-10-21 21:20:26 UTC
After creating a package (pkg create) from the successful build in the jail, I installed the package (pkg upgrade -U python37-3.7.1.txz) on the "broken" host-system... and now the port is building fine on the host-system as well.
This is incredibly strange.
Comment 9 hakan 2018-10-21 22:19:04 UTC
Using portmaster it failed for me similarly:

--- Programs/_testembed ---
Programs/_testembed.o: In function `test_init_from_config':
_testembed.c:(.text+0xd28): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_dev_mode':
_testembed.c:(.text+0xe0f): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_isolated':
_testembed.c:(.text+0xeb3): undefined reference to `_Py_InitializeFromConfig'
--- pybuilddir.txt ---
LD_LIBRARY_PATH=/usr/ports/lang/python37/work/Python-3.7.1 ./python -E -S -m sysconfig --generate-posix-vars ; if test $? -ne 0 ; then  echo "generate-posix-vars failed" ;  rm -f ./pybuilddir.txt ;  exit 1 ;  fi
--- Programs/_testembed ---
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** [Programs/_testembed] Error code 1

make[2]: stopped in /usr/ports/lang/python37/work/Python-3.7.1
1 error

make[2]: stopped in /usr/ports/lang/python37/work/Python-3.7.1
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/lang/python37
*** Error code 1

Stop.
make: stopped in /usr/ports/lang/python37

===>>> make build failed for lang/python37
===>>> Aborting update

===>>> Update for python37-3.7.0_3 failed
===>>> Aborting update

===>>> Update for security/py-openssl@py37 failed
===>>> Aborting update


===>>> You can restart from the point of failure with this command line:
       portmaster <flags> security/py-openssl@py37 lang/python37

This command has been saved to /tmp/portmasterfail.txt

I am on 11.2-RELEASE-p4 with this /etc/make.conf

cat /etc/make.conf
WITH_CCACHE_BUILD=yes

DEFAULT_VERSIONS+= perl5=5.26 php=7.2 pqsql=10 python=3.7 python3=3.7

And here is the port's config:

make showconfig
===> The following configuration options are available for python37-3.7.1:
     DEBUG=off: Build with debugging support
     IPV6=on: IPv6 protocol support
     LIBFFI=on: Use libffi from ports instead of bundled version
     NLS=on: Enable gettext support for the locale module
     PYMALLOC=on: Enable specialized mallocs
====> Hash Algorithm (PEP-456): you can only select none or one of them
     FNV=off: Modified Fowler-Noll-Vo Algorithm
     SIPHASH=off: SipHash24 Algorithm
===> Use 'make config' to modify these settings
Comment 10 rkoberman 2018-10-21 22:58:56 UTC
One other point of reference that would help explain some of this. My installation of python37 was a new install. I was not upgrading a previously installed version. There were no sharables or header files from n older version present.

Looks like the build will select an existing file as opposed to one in the work directory. This is not a new issue.
Comment 11 Kubilay Kocak freebsd_committer freebsd_triage 2018-10-22 02:40:20 UTC
Wen committed the update in ports r482664
Comment 12 Kubilay Kocak freebsd_committer freebsd_triage 2018-10-22 03:22:53 UTC
@Simeon Can you include (as an attachment) a full build log, and clarify from which exact portversion this is an upgrade from
Comment 13 Simeon Simeonov 2018-10-22 03:46:09 UTC
I am no longer able to reproduce the problem unfortunately.
The portversion I was upgrading from was: python37-3.7.0_3
Comment 14 Yasuhiro KIMURA 2018-10-22 05:09:04 UTC
Created attachment 198457 [details]
Build error log on 11.2-RELEASE amd64 host

(In reply to Kubilay Kocak from comment #12)

Attached file is build error log on my 11.2-RELEASE amd64 host. I can provide more information if necessary.
Comment 15 Kubilay Kocak freebsd_committer freebsd_triage 2018-10-22 08:34:34 UTC
For reference:

_Py_InitializeFromConfig() was added in 0c90d6f75931 [1] by vstinner (CC'd) via PR 8659 as part of upstream issue 34247 [2]

[1] https://github.com/python/cpython/commit/0c90d6f75931da4fec84d06c2efe9dd94bb96b77#diff-559231a69a7d7f06ab48269b3bfa1f45

[2] https://bugs.python.org/issue34247

@Yasuhiro (and others), in case it's relevant, can you please try to build the port with MAKE_JOBS_UNSAFE=yes as it was removed in ports r482401 and let us know if the issue remains reproducible or not.
Comment 16 Yasuhiro KIMURA 2018-10-22 10:02:36 UTC
(In reply to Kubilay Kocak from comment #15)

I tried 'make MAKE_JOBS_UNSAFE=yes' under same conditions as comment #14 and got same error.
Comment 17 hakan 2018-10-22 18:14:49 UTC
I tried `portmaster -m MAKE_JOBS_UNSAFE=yes lang/python37` and it resulted in a different error:

In file included from /usr/local/include/readline/readline.h:36:
/usr/local/include/readline/rltypedefs.h:35:22: warning: this function declaration is not a prototype [-Wstrict-prototypes]
typedef int Function () __attribute__ ((deprecated));
                     ^
                      void
/usr/local/include/readline/rltypedefs.h:36:24: warning: this function declaration is not a prototype [-Wstrict-prototypes]
typedef void VFunction () __attribute__ ((deprecated));
                       ^
                        void
/usr/local/include/readline/rltypedefs.h:37:26: warning: this function declaration is not a prototype [-Wstrict-prototypes]
typedef char *CPFunction () __attribute__ ((deprecated));
                         ^
                          void
/usr/local/include/readline/rltypedefs.h:38:28: warning: this function declaration is not a prototype [-Wstrict-prototypes]
typedef char **CPPFunction () __attribute__ ((deprecated));
                           ^
                            void
In file included from /usr/ports/lang/python37/work/Python-3.7.1/Modules/readline.c:31:
/usr/local/include/readline/readline.h:390:23: warning: this function declaration is not a prototype [-Wstrict-prototypes]
extern int rl_message ();
                      ^
                       void
5 warnings generated.
In file included from /usr/ports/lang/python37/work/Python-3.7.1/Modules/socketmodule.c:283:
In file included from /usr/ports/lang/python37/work/Python-3.7.1/Modules/socketmodule.h:66:
In file included from /usr/include/bluetooth.h:51:
/usr/include/netgraph/bluetooth/include/ng_btsocket.h:246:2: warning: "Make sure new member of socket address initialized" [-W#warnings]
#warning "Make sure new member of socket address initialized"
 ^
1 warning generated.
In file included from /usr/ports/lang/python37/work/Python-3.7.1/Modules/_ssl.c:38:
In file included from /usr/ports/lang/python37/work/Python-3.7.1/Modules/socketmodule.h:66:
In file included from /usr/include/bluetooth.h:51:
/usr/include/netgraph/bluetooth/include/ng_btsocket.h:246:2: warning: "Make sure new member of socket address initialized" [-W#warnings]
#warning "Make sure new member of socket address initialized"
 ^
1 warning generated.

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_tkinter              spwd
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time

install  -m 0644 ./Tools/gdb/libpython.py python-gdb.py
cc -pthread -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe  -fstack-protector -fno-strict-aliasing   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration   -I. -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Programs/_testembed.o ./Programs/_testembed.c
cc -pthread  -L/usr/local/lib  -fstack-protector  -Wl,--export-dynamic -o Programs/_testembed Programs/_testembed.o -L. -lpython3.7m -lintl -ldl -L/usr/local/lib -lintl -lutil   -lm
Programs/_testembed.o: In function `test_init_from_config':
_testembed.c:(.text+0xd28): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_dev_mode':
_testembed.c:(.text+0xe0f): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_isolated':
_testembed.c:(.text+0xeb3): undefined reference to `_Py_InitializeFromConfig'
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Stop.
make[2]: stopped in /usr/ports/lang/python37/work/Python-3.7.1
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/lang/python37
*** Error code 1

Stop.
make: stopped in /usr/ports/lang/python37

===>>> make build failed for lang/python37
===>>> Aborting update


===>>> You can restart from the point of failure with this command line:
       portmaster <flags> lang/python37

This command has been saved to /tmp/portmasterfail.txt
Comment 18 Kubilay Kocak freebsd_committer freebsd_triage 2018-10-23 02:32:49 UTC
(In reply to hakan from comment #17)

Thank you Hakan. It's the same error (see bottom of log), the earlier messages are just warnings.
Comment 19 Serge 2018-10-23 18:27:41 UTC
I confirm the same issue when trying to build Python 3.7.1 with 3.7.0_3 installed.

My configuration is a jail under 11.2-RELEASE-p4 with ccache installed. My make.conf is as follows:

WITH_CCACHE_BUILD=yes		

DEFAULT_VERSIONS+=ssl=openssl111
DEFAULT_VERSIONS+= php=7.2
DEFAULT_VERSIONS+= pgsql=10
DEFAULT_VERSIONS+= samba=4.8
DEFAULT_VERSIONS+= perl5=5.28
DEFAULT_VERSIONS+= python=3.7
DEFAULT_VERSIONS+= python3=3.7

OPTIONS_UNSET+=X11
OPTIONS_UNSET+= MYSQL
OPTIONS_SET+= PGSQL
Comment 20 Kubilay Kocak freebsd_committer freebsd_triage 2018-10-24 03:10:26 UTC
(In reply to Serge from comment #19)

That's two reports (comment 9, comment 19) with ccache enabled (not implying ccache is necessarily a root cause).

Does the issue remain reproducible with ccache disabled?
Comment 21 Serge 2018-10-24 03:23:51 UTC
Hi there,
I've just tried building with ccache disabled. At first sight, the problem remains the same. Here are the final lines of the error:

cc -pthread  -L/usr/local/lib -Wl,-rpath,/usr/local/lib -fstack-protector  -Wl,--export-dynamic -o Programs/_testembed Programs/_testembed.o -L. -lpython3.7m -lintl -ldl -L/usr/local/lib -lintl -lutil   -lm
Programs/_testembed.o: In function `test_init_from_config':
_testembed.c:(.text+0xd28): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_dev_mode':
_testembed.c:(.text+0xe0f): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_isolated':
_testembed.c:(.text+0xeb3): undefined reference to `_Py_InitializeFromConfig'
--- pybuilddir.txt ---
--- Programs/_testembed ---
cc: error: linker command failed with exit code 1 (use -v to see invocation)
--- pybuilddir.txt ---
LD_LIBRARY_PATH=/skeleton/usr/ports/lang/python37/work/Python-3.7.1 ./python -E -S -m sysconfig --generate-posix-vars ; if test $? -ne 0 ; then  echo "generate-posix-vars failed" ;  rm -f ./pybuilddir.txt ;  exit 1 ;  fi
--- Programs/_testembed ---
*** [Programs/_testembed] Error code 1
Comment 22 chris.torek 2018-10-25 05:27:18 UTC
Note: running `make deinstall` and then `make` succeeds.  It seems as though the build refers to some existing installed file by mistake.
Comment 23 Serge 2018-10-25 20:15:43 UTC
(In reply to chris.torek from comment #22)

Yes, confirmed in my case as well. After 'make deinstall' I can build python37 normally. Thank you for sharing this work-around.
Comment 24 jSML4ThWwBID69YC 2018-10-29 13:45:09 UTC
I have a similar issue upgrading from 3.7 to 3.7.1. 

5 warnings generated.
In file included from /usr/ports/lang/python37/work/Python-3.7.1/Modules/socketmodule.c:283:
In file included from /usr/ports/lang/python37/work/Python-3.7.1/Modules/socketmodule.h:66:
In file included from /usr/include/bluetooth.h:51:
/usr/include/netgraph/bluetooth/include/ng_btsocket.h:246:2: warning: "Make sure new member of socket address initialized" [-W#warnings]
#warning "Make sure new member of socket address initialized"
 ^
1 warning generated.
In file included from /usr/ports/lang/python37/work/Python-3.7.1/Modules/_ssl.c:38:
In file included from /usr/ports/lang/python37/work/Python-3.7.1/Modules/socketmodule.h:66:
In file included from /usr/include/bluetooth.h:51:
/usr/include/netgraph/bluetooth/include/ng_btsocket.h:246:2: warning: "Make sure new member of socket address initialized" [-W#warnings]
#warning "Make sure new member of socket address initialized"
 ^
1 warning generated.

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm                 _tkinter              spwd               
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd                
time                                                           

install  -m 0644 ./Tools/gdb/libpython.py python-gdb.py
cc -pthread -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe  -fstack-protector -fno-strict-aliasing   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration   -I. -I./Include -I/usr/local/include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Programs/_testembed.o ./Programs/_testembed.c
cc -pthread  -L/usr/local/lib -Wl,-rpath,/usr/local/lib -fstack-protector  -Wl,--export-dynamic -o Programs/_testembed Programs/_testembed.o -L. -lpython3.7 -lintl -ldl -L/usr/local/lib -lintl -lutil   -lm  
Programs/_testembed.o: In function `test_init_from_config':
_testembed.c:(.text+0xd28): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_dev_mode':
_testembed.c:(.text+0xe0f): undefined reference to `_Py_InitializeFromConfig'
Programs/_testembed.o: In function `test_init_isolated':
_testembed.c:(.text+0xeb3): undefined reference to `_Py_InitializeFromConfig'
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

A 'make deinstall', 'make install' in /usr/ports/lang/python37 resolved it.
Comment 25 Kubilay Kocak freebsd_committer freebsd_triage 2018-10-30 02:25:48 UTC
*** Bug 232795 has been marked as a duplicate of this bug. ***
Comment 26 Kubilay Kocak freebsd_committer freebsd_triage 2018-12-29 02:10:32 UTC
*** Bug 234467 has been marked as a duplicate of this bug. ***
Comment 27 Kubilay Kocak freebsd_committer freebsd_triage 2018-12-29 02:11:29 UTC
*** Bug 234409 has been marked as a duplicate of this bug. ***
Comment 28 wen 2019-01-08 06:16:04 UTC
Hi, all:

  I updated a patch to resolv it 2 weeks ago in :
   https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234409

  Would you have a test ?

Thanks !
wen
Comment 29 rkoberman 2019-01-08 07:21:28 UTC
I can confirm that my issue is resolved. Thanks!