Created attachment 145517 [details] fix lang/python27 build with libressl
Root problem: As soon as libressl is installed via ports, lang/pythonXX and other ports will try to pick it up, most likely due to the include and linker flags order. This is problematic in two regards: * lang/python27 does not have an explicit BUILD/RUN_DEPENDS on libressl, since libressl is not officially supported by upstream (yet) * libressl is API and ABI incompatible with OpenSSL, so depending on the system status (libressl installed or not), python will take the one or other. I do not like the patch, since it would disable the function for both, OpenSSL and libressl and thus effectively change the behaviour of lang/pythonXX. We need to get two things right: * handle OpenSSL support correctly and explicitly use OpenSSL, so that an installed SSL replacement does not cause a wrong and unintended behaviour * offer users to link against libressl explicitly (via an OPTION), which can cause different behaviour.
(In reply to Marcus von Appen from comment #1) > Root problem: > As soon as libressl is installed via ports, lang/pythonXX and other ports > will try to pick it up, most likely due to the include and linker flags > order. This is problematic in two regards: > > * lang/python27 does not have an explicit BUILD/RUN_DEPENDS on libressl, > since libressl is not officially supported by upstream (yet) > * libressl is API and ABI incompatible with OpenSSL, so depending on the > system status (libressl installed or not), python will take the one or other. > > I do not like the patch, since it would disable the function for both, > OpenSSL and libressl and thus effectively change the behaviour of > lang/pythonXX. Does RAND_egd even work on FreeBSD? I also tried to preserve the python ssl api by having PySSL_RAND_egd just return an error. So I don't think it's behaviour has changed. Might have missed something, tho. > We need to get two things right: > > * handle OpenSSL support correctly and explicitly use OpenSSL, so that an > installed SSL replacement does not cause a wrong and unintended behaviour > * offer users to link against libressl explicitly (via an OPTION), which can > cause different behaviour. How about this: * Add smt. like WITH_OPENSSL_LIBRE and some logic to bsd.openssl.mk * Have the libressl port fail to build without that flag set. * Add a static library or c file with dummy functions(RAND_egd, ...) to libressl.
(In reply to meyer.arne83 from comment #2) > (In reply to Marcus von Appen from comment #1) > > Root problem: > > As soon as libressl is installed via ports, lang/pythonXX and other ports > > will try to pick it up, most likely due to the include and linker flags > > order. This is problematic in two regards: > > > > * lang/python27 does not have an explicit BUILD/RUN_DEPENDS on libressl, > > since libressl is not officially supported by upstream (yet) > > * libressl is API and ABI incompatible with OpenSSL, so depending on the > > system status (libressl installed or not), python will take the one or other. > > > > I do not like the patch, since it would disable the function for both, > > OpenSSL and libressl and thus effectively change the behaviour of > > lang/pythonXX. > Does RAND_egd even work on FreeBSD? > I also tried to preserve the python ssl api by having PySSL_RAND_egd just > return an error. So I don't think it's behaviour has changed. Might have > missed something, tho. This is just the minor part of the issue. The bigger problem is the library being pulled in. > > We need to get two things right: > > > > * handle OpenSSL support correctly and explicitly use OpenSSL, so that an > > installed SSL replacement does not cause a wrong and unintended behaviour > > * offer users to link against libressl explicitly (via an OPTION), which can > > cause different behaviour. > How about this: > * Add smt. like WITH_OPENSSL_LIBRE and some logic to bsd.openssl.mk > * Have the libressl port fail to build without that flag set. That sounds reasonable. I added the libressl maintainer and portmgr@ to this issue to get their feedback on it. > * Add a static library or c file with dummy functions(RAND_egd, ...) to > libressl. That might contradict the purpose of libressl and adds a can of worms (local patches), which should be avoided. Best'd be to ask the upstream developers of libressl about such things, I think.
(In reply to Marcus von Appen from comment #3) > (In reply to meyer.arne83 from comment #2) > > (In reply to Marcus von Appen from comment #1) [...] > > > We need to get two things right: > > > > > > * handle OpenSSL support correctly and explicitly use OpenSSL, so that an > > > installed SSL replacement does not cause a wrong and unintended behaviour > > > * offer users to link against libressl explicitly (via an OPTION), which can > > > cause different behaviour. > > How about this: > > * Add smt. like WITH_OPENSSL_LIBRE and some logic to bsd.openssl.mk > > * Have the libressl port fail to build without that flag set. > > That sounds reasonable. I added the libressl maintainer and portmgr@ to this > issue to get their feedback on it. > I implemented the WITH_OPENSSL_LIBRE logic on my laptop. Tested it. Looks good so far. The only issue is: Are there ports who have a hard dependency on openssl/ports? > > * Add a static library or c file with dummy functions(RAND_egd, ...) to > > libressl. > > That might contradict the purpose of libressl and adds a can of worms (local > patches), which should be avoided. Best'd be to ask the upstream developers > of libressl about such things, I think. Yeah. You're right about that. I have a patch for libressl that adds back the RAND_egd api. But the functions just return -1 (means no connection to egd). As RAND_egd is the main reason for breakage, and basically useless, that seems like a tolerable approach to me. That way, we don't have to patch every user of that api. With that patch and the change to bsd.openssl.mk I can compile python27 without my first patch. I can open up a new bug with my patches, if anyone is interested...
Canonicalise python@ assignment.
Annoyingly, everything that needs py27 as a dependence (like py-fail2ban) stops installation with "signal 11" if LibreSSL is installed. The main problem is that "touch .../_ssl.so" is required for python27 before "make install" works as it should.
Created a patch that makes the build figure out RAND_egd support. This has been submitted upstream in http://bugs.python.org/issue21356, a Python committer has indicated they willing to include this in their code. "We're still willing to fix this if someone tells us how to test for LibreSSL in C code."
Created attachment 149952 [details] Patch the port to detect RAND_egd and disable it conditionally
Created attachment 149974 [details] Single patch for this PR for the source tree This requires the Makefile patch as well!
Created attachment 149975 [details] Patch for Makefile Primarily passes LDFLAGS and CPPFLAGS to configure otherwise it will not pick up OpenSSL/LibreSSL from ports
Created attachment 149976 [details] Output of the test_ssl unit test There are expected failures/errors in this test, SSLv23 and SSLv3 are not supported by LibreSSL
Created attachment 149998 [details] Single patch for this PR for the source tree Lib/ssl.py was missing Tested again, see build.log
Created attachment 149999 [details] Output of the test_ssl unit test
Created attachment 150000 [details] Build log
Created attachment 150909 [details] Patch for Makefile Adapted for python27-2.7.9
Created attachment 150910 [details] Single patch for this PR for the source tree
^ this fails with Python 2.7.9 yet: ===> Registering installation for python27-2.7.9 pkg-static: Unable to access file /usr/ports/lang/python27/work/stage/usr/local/bin/2to3-2.7: No such file or directory pkg-static: Unable to access file /usr/ports/lang/python27/work/stage/usr/local/bin/idle2.7: No such file or directory pkg-static: Unable to access file /usr/ports/lang/python27/work/stage/usr/local/bin/pydoc2.7: No such file or directory pkg-static: Unable to access file /usr/ports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/ossaudiodev.so: No such file or directory *** Error code 74
Created attachment 150944 [details] Poudriere build log Can't reproduce the pkg-static errors, see poudriere build log. Jens: Did you use the latest patch or the original one? After install of the package I have lrwxr-xr-x 1 root wheel 6 Oct 3 01:49 /usr/local/bin/2to3 -> 2to3-2 lrwxr-xr-x 1 root wheel 8 Oct 3 01:46 /usr/local/bin/2to3-2 -> 2to3-2.7 -rwxr-xr-x 1 root wheel 101 Dec 24 19:25 /usr/local/bin/2to3-2.7
I used the latest one.
Using the version in the ports tree, I'm getting this error: *** WARNING: renaming "_ssl" since importing it failed: build/lib.freebsd-10.1-RELEASE-p2-amd64-2.7/_ssl.so: Undefined symbol "SSLv2_method" This probably happened because LibreSSL ripped out SSLv2 support, which causes the ssl module to not be built, leading to a package failure later on: Failed to build these modules: _ssl pkg-static: Unable to access file /wrkdirs/usr/ports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/_ssl.so: No such file or directory *** Error code 1 I've reported this problem here because all of the above patches seem to deal only with the RAND_egd() issue.
Created attachment 151207 [details] Patch for Makefile Revised patch, previous patch would fail when WITH_OPENSSL_PORT was NOT set. This construct works whether WITH_OPENSSL_PORT is defined or not.
Created attachment 151208 [details] Build log with LibreSSL installed This is just the make output with LibreSSL 2.1.2 installed and in make.conf WITH_OPENSSL_PORT= yes OPENSSL_SHLIBVER= 30 OPENSSL_PORT= security/libressl The Poudriere build log is built without LibreSSL as a regression check for the port.
Jens, Kevin, How are you building? You will need to have LibreSSL (2.1.2 at the moment) installed and the following in your /etc/make.conf WITH_OPENSSL_PORT= yes OPENSSL_SHLIBVER= 30 OPENSSL_PORT= security/libressl Python has been modified to cope with SSLv2 not being available some time ago.
Bernard, Thanks for the quick response. Indeed, I didn't set OPENSSL_SHLIBVER, is this documented somewhere ('bsd.openssl.mk' has default set to 8)? After setting OPENSSL_SHLIBVER and applying both patches Python built correctly. Thanks for investigating this issue and working out a patch!
Latest Python 2.7.9 has the _ssl.so error even with the make.conf entries again.
What are we missing to get python 2.7.9 and libressl running, then?
See https://bugs.freebsd.org/bugzilla/attachment.cgi?id=153769 for a successful Poudriere build log of Python 2.7 with LibreSSL 2.1.4
Bernard will be: - Obsoleting all current attachments - Attaching a new combined diff, AND - Attaching new QA logs
Created attachment 153781 [details] svn diff for lang/python27 Beware that this also renames a number of patches as make makepatch generates them.
Created attachment 153782 [details] Poudriere build log python27 linking to libcrypto.so.32.0.0
Created attachment 153783 [details] svn diff for lang/python27 Forgot to svn rm the old patches... Fixed!
Created attachment 153820 [details] svn diff for lang/python27 Updated patches as per koobs@' request
Created attachment 153828 [details] svn diff for lang/python27 Now with the new patch 1. Comments from code to the patch header 2. Patches generated by make makepatch and info from original patches merged
Created attachment 154266 [details] svn diff for lang/python27
I'm out of my wits now... This whole patch thing is just becoming way to convoluted. Of the existing 10 (TEN) patches currently in files only 3 (THREE) are according to the naming standard of make makepatch, one is not named after the file it modifies but after a PR number so an svn diff is effectively deleting most of the patches and then adding them again. I've re-rolled these patches a large number of times now and although I want to adhere to all of the policies I feel this port is beyond repair. First bring this port in line with all of the policies and _then_ I will provide an svn diff. Until then, use the tarball or the non-svn diff or the shar. Every make makepatch trashes the comments preceding the actual patch and require manual re-adding... ... ... The CPPFLAGS and LDFLAGS additions in the Makefile are required to fix RAND_egd detection in configure, without it checks libcrypto / RAND_egd from base.
Created attachment 154280 [details] Poudriere build log python27 linking to libcrypto.so.32.0.0
Created attachment 154281 [details] non-svn diff for lang/python27 from /usr/ports
Created attachment 154282 [details] shar of lang/python27
Created attachment 154283 [details] Tarball of the re-rolled port
Created attachment 154342 [details] svn diff for lang/python27 Now a proper svn diff, tested and used to generate regression-test and poudriere logs
Created attachment 154344 [details] Poudriere build log python27 linking to libcrypto.so.32.0.0
Created attachment 154345 [details] regression-test output for lang/python27
I just installed these ports on a CURRENT system: mercurial-3.3.2 python27-2.7.9 I then typed: hg clone https://bitbucket.org/mcdeck/cunit-to-junit/ and got this: abort: /usr/local/lib/python2.7/lib-dynload/_ssl.so: Undefined symbol "SSLv2_method"! Is there a fix for this available?
Hi Rodrigo, SSLv2 was completely removed from LibreSSL, so any fixing will need to be done in the mercurial code.
Has the diff been tested against LibreSSL 2.1.6? Just tried to build python27-2.7.9_1 from ports against libressl 2.1.6 and it failed during pkg installation with error: ===> Registering installation for python27-2.7.9_1 as automatic pkg-static: Unable to access file /usr/pkg/usr/ports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/_ssl.so: No such file or directory *** [fake-pkg] Error code 74 Stop in /usr/ports/lang/python27.
(In reply to Seth from comment #45) Hi Seth, Yes this has been tested against 2.1.6 with Python-2.7.9_1 will add an updated poudriere build log.
Created attachment 155432 [details] non-svn diff for lang/python27 2.7.9_1 and LibreSSL 2.1.6 _1 was only a version bump for the -HEAD tracking people
Thanks, I'll give it a try. Is there a rough time frame for committing the patch to the ports tree?
Any reason this has not been applied yet ? This, or the upstream version, https://hg.python.org/cpython/rev/eddcb6671a48 that should be about the same.
(In reply to Mathieu Arnold from comment #49) I've been using this patch locally for several months; it works fine! Considering that the same patch has been upstreamed, someone on python@ should really go ahead and commit this.
+1 for merging this :)
*** Bug 199826 has been marked as a duplicate of this bug. ***
By now we're close to 2.7.10 which is planned for release this month 2.7.10rc1 2015-05-09 2.7.10 2015-05-23 Can someone start the port for 2.7.10rc1?
Comment on attachment 154342 [details] svn diff for lang/python27 Superseeded by 2.7.10
Created attachment 157097 [details] Make regression-test output Built with base openssl When built with libressl regression-test fails on a version-test
See https://reviews.freebsd.org/D2637 or https://github.com/Sp1l/ports/blob/master/patches/patch-lang_python27-2.7.10 CONFIGURE_ENV still requires modification with the new 2.7.10 version Patches renamed as per makepatch standard
(In reply to Bernard Spil from comment #56) There is a simple path error in this patch. https://github.com/Sp1l/ports/blob/master/patches/patch-lang_python27-2.7.10 Attached is the fixed patch. After the applied patch poudriere built python with out issues.
Created attachment 157316 [details] Updated patch for python27-2.7.10 (In reply to Bernard Spil from comment #56) There is a simple path error in this patch. https://github.com/Sp1l/ports/blob/master/patches/patch-lang_python27-2.7.10 Attached is the fixed patch. After the applied patch poudriere built python with out issues.
I ran into this today, when trying to build ports with libressl. Any update?
Better check https://reviews.freebsd.org/D2637 it contains the latest patches.
A commit references this bug: Author: brnrd Date: Fri Jul 3 13:27:44 UTC 2015 New revision: 391238 URL: https://svnweb.freebsd.org/changeset/ports/391238 Log: lang/python: Update to 2.7.10 The FreeBSD Python team welcomes Python 2.7.10 to the Ports tree! Announcement: https://www.python.org/downloads/release/python-2710/ Changelog: https://hg.python.org/cpython/raw-file/v2.7.10/Misc/NEWS Important changes for this release: - Python 2.7.10 is a bug fix release of the Python 2.7.x series. - Fixes issues with LibreSSL 2.2+ PR: 192511, 201088 Differentialr Revision: D2637 Reviewed by: koobs (mentor), vsevolod (mentor), antoine Approved by: koobs (mentor) Changes: head/lang/python27/Makefile head/lang/python27/distinfo head/lang/python27/files/patch-Doc__library__fcntl.rst head/lang/python27/files/patch-Lib__distutils__unixccompiler.py head/lang/python27/files/patch-Lib_distutils_command_build__scripts.py head/lang/python27/files/patch-Lib_distutils_command_build_scripts.py head/lang/python27/files/patch-Makefile.pre.in head/lang/python27/files/patch-Modules-_ctypes-libffi-configure head/lang/python27/files/patch-Modules___ctypes__libffi__src__arm__ffi.c head/lang/python27/files/patch-Modules___ssl.c head/lang/python27/files/patch-Modules__fcntlmodule.c head/lang/python27/files/patch-setup.py head/lang/python27/pkg-plist
A commit references this bug: Author: brnrd Date: Fri Jul 3 13:34:42 UTC 2015 New revision: 391239 URL: https://svnweb.freebsd.org/changeset/ports/391239 Log: lang/python-(tools,doc-html): Update to 2.7.10 The FreeBSD Python team welcomes Python 2.7.10 to the Ports tree! Announcement: https://www.python.org/downloads/release/python-2710/ Changelog: https://hg.python.org/cpython/raw-file/v2.7.10/Misc/NEWS Important changes for this release: - Python 2.7.10 is a bug fix release of the Python 2.7.x series. - Fixes issues with LibreSSL 2.2+ PR: 192511, 201088 Differentialr Revision: D2637 Reviewed by: koobs (mentor), vsevolod (mentor), antoine Approved by: koobs (mentor) Changes: head/lang/python-tools/Makefile
A commit references this bug: Author: brnrd Date: Fri Jul 3 13:37:43 UTC 2015 New revision: 391240 URL: https://svnweb.freebsd.org/changeset/ports/391240 Log: lang/python-doc-html: Update to 2.7.10 The FreeBSD Python team welcomes Python 2.7.10 to the Ports tree! Announcement: https://www.python.org/downloads/release/python-2710/ Changelog: https://hg.python.org/cpython/raw-file/v2.7.10/Misc/NEWS Important changes for this release: - Python 2.7.10 is a bug fix release of the Python 2.7.x series. - Fixes issues with LibreSSL 2.2+ PR: 192511, 201088 Differentialr Revision: D2637 Reviewed by: koobs (mentor), vsevolod (mentor), antoine Approved by: koobs (mentor) Changes: head/lang/python-doc-html/distinfo
Over to Bernard to close if complete.