For ports security/py-yubikey-manager from `make install`, or from `pkg install py39-yubikey-manager` on FreeBSD stable/14 (currently Alpha 4), this port will successfully install, but will error out immediately upon initial execution of `ykman` with the error message: ImportError: /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "ERR_GET_FUNC" I'm assuming that python-cryptography doesn't like a missing export due to some OpenSSL change in stable/14. OpenSSL 3.0.10 1 Aug 2023 (Library: OpenSSL 3.0.10 1 Aug 2023) FreeBSD freebsd14 14.0-ALPHA4 FreeBSD 14.0-ALPHA4 amd64 1400097 #0 stable/14-n265026-4c3f144478d4: Fri Sep 1 05:47:56 UTC 2023 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 Full traceback: Traceback (most recent call last): File "/usr/local/bin/ykman", line 33, in <module> sys.exit(load_entry_point('yubikey-manager==4.0.9', 'console_scripts', 'ykman')()) File "/usr/local/bin/ykman", line 25, in importlib_load_entry_point return next(matches).load() File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load module = import_module(match.group('module')) File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/usr/local/lib/python3.9/site-packages/ykman/cli/__main__.py", line 47, in <module> from ..diagnostics import get_diagnostics File "/usr/local/lib/python3.9/site-packages/ykman/diagnostics.py", line 17, in <module> from fido2.ctap2 import Ctap2, ClientPin File "/usr/local/lib/python3.9/site-packages/fido2/ctap2/__init__.py", line 38, in <module> from .blob import LargeBlobs # noqa File "/usr/local/lib/python3.9/site-packages/fido2/ctap2/blob.py", line 35, in <module> from cryptography.hazmat.primitives.ciphers.aead import AESGCM File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/primitives/ciphers/aead.py", line 10, in <module> from cryptography.hazmat.backends.openssl import aead File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 6, in <module> from cryptography.hazmat.backends.openssl.backend import backend File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 113, in <module> from cryptography.hazmat.bindings.openssl import binding File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module> from cryptography.hazmat.bindings._openssl import ffi, lib ImportError: /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "ERR_GET_FUNC"
https://www.openssl.org/news/cl30.txt * The ERR_GET_FUNC() function was removed. With the loss of meaningful function codes, this function can only cause problems for calling applications.
what's the recommendation for how to replace it? and, have you reported this issue upstream, too?
Looks like this issue is related to one or more of {py-cryptography, py-fido2, py-openssl, py-yubikey-manager} being built without CRYPTOGRAPHY_OPENSSL_NO_LEGACY set in the environment. I hacked together a build, and made things work for now. I will see if I can determine an appropriate minimal patch set, then someone can take a look and see if this is the *right* solution, as I'm sure there are multiple build cases that I am not currently considering (e.g.: with and without OpenSSL legacy support). I also linked a related python OpenSSL bug in the [see also] section since it helped me diagnose.
olá! thanks for letting us know about that; once I got notified here jumped right away to test on a 'poudriere-git-3.3.99.20220831', but... it all just worked fine for me :\ do you mind sharing a little bit more info about your setup? or just update your current tree and give it another shot? -- once the bug #254853 was tagged as closed, afaik. here's the ports tree commit I was when started the fresh build on a clean jail with 14.0-ALPHA4: * bbde282ffa [origin/main] x11-fm/thunar: Update to 4.18.7 this is the console output from poudriere: * [05:01:03] [01] [00:00:06] Finished security/py-yubikey-manager@py39 | py39-yubikey-manager-4.0.9_4: Success my host machine's uname can be identified by: * FreeBSD 15.0-CURRENT amd64 1500000 #8 main-n265115-edd28b857e2d: Fri Sep 1 10:28:44 UTC 2023 GENERIC-NODEBUG 1500000 1500000 ae5fe105e23ea20887610f866b81f6600a8dfc63 buildlog landing in a few on a different post ...
Created attachment 244609 [details] 273505___security_py-yubikey-manager___14amd64_poudriere.log # poudriere jail -c -v 14.0-ALPHA4 -j 14amd64 ... [ we all know the output here, I assume ] # poudriere jail -l JAILNAME VERSION ARCH METHOD TIMESTAMP 14amd64 14.0-ALPHA4 amd64 http 2023-09-02 15:44:15 # poudriere bulk -j 14amd64 -p git -T -t -C security/py-yubikey-manager ... [ log/output attached ]
`make` was clean (no errors), Both `make test` and actually attempting to run `ykman` was the failure point. Vanilla fresh install 14-ALPHA4 from ISO Install ykman (ports or pkg) Failure due to "Undefined symbol "ERR_GET_FUNC"" I'm going to do a fresh install in a VM again today and see if it is reproducible. (VMWare)
100% reproducible from clean install of 14-ALPHA4 amd64 ISO. `pkg config abi` results in: FreeBSD:14:amd64 Failcase #1 Tested with `pkg install py39-yubikey-manager`, then `ykman` results in : ImportError: /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "ERR_GET_FUNC" Failcase #2 Testing with latest ports git repo (run after Failcase #1) a) git clone https://git.FreeBSD.org/ports.git main as of commit 3a6b4db0a193826f2cb8e9e35acc53dd4c9941d2 b) cd /usr/ports/security/py-yubikey-manager c) echo "BATCH=YES" >> /etc/make.conf && make && make test (test results in same failure message, log available upon request) d) make deinstall && make reinstall && ykman (results in same failure message) Let me know if you need logs from any of these steps. I have a snapshot I can revert easily and re-run test cases from both clean and failed states.
Looks like this is not specific to security/py-yubikey-manager, but directly related to build environment for at least security/py-cryptography (possibly others), and related errors can be seen by running `make test` for py-cryptography alone (in addition to `make test` with py-yubikey-manager). I am attaching a failure `make test` log for py-cryptography, and a success `make test` log for py-cryptography when I patch my Makefile to include CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1 as follows: diff --git a/security/py-cryptography/Makefile b/security/py-cryptography/Makefile index 92db171964c8..37d7ffbd133b 100644 --- a/security/py-cryptography/Makefile +++ b/security/py-cryptography/Makefile @@ -42,7 +42,9 @@ CARGO_TEST= no CFLAGS+= -I${OPENSSLINC} LDFLAGS+= -L${OPENSSLLIB} -TEST_ENV= PYTHONPATH=${STAGEDIR}${PYTHON_SITELIBDIR} +MAKE_ENV+= CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1 +TEST_ENV= PYTHONPATH=${STAGEDIR}${PYTHON_SITELIBDIR} \ + ${MAKE_ENV} CPE_VENDOR= cryptography_project Just a datapoint, not a proper fix.
Created attachment 244615 [details] py-cryptography base `make-test` failure
Created attachment 244616 [details] py-cryptography modified Makefile `make test` results
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=16446be04be60fc4831368bf7dc9b2199aa1d6ac commit 16446be04be60fc4831368bf7dc9b2199aa1d6ac Author: Vinícius Zavam <egypcio@FreeBSD.org> AuthorDate: 2023-09-04 08:22:42 +0000 Commit: Vinícius Zavam <egypcio@FreeBSD.org> CommitDate: 2023-09-04 08:22:42 +0000 security/py-yubikey-manager: update 4.0.9 to 5.2.0 While here, * Add OTP HID support for FreeBSD (merged in upstream) [0]; * Move into pep517 (https://wiki.freebsd.org/Python/PEP-517); * Start using USES=pycryptography as introduced by 7bb64b89d0e5ec8 Note that, * should one does not with to set PYCRYPTOGRAHY_DEFAULT=legacy, `ykman` (and other packages/ports depending on that) would require CRYPTOGRAPHY_OPENSSL_NO_LEGACY= to be true. [0] https://github.com/Yubico/yubikey-manager/commit/ecd7897b3f020542f70581f77f47ba57c739b334 PR: 273505 Reported by: David Horn <dhorn2000 % gmail.com> security/py-yubikey-manager/Makefile | 12 +++++++----- security/py-yubikey-manager/distinfo | 6 +++--- .../files/patch-ykman_hid_____init____.py (gone) | 12 ------------ 3 files changed, 10 insertions(+), 20 deletions(-)
I was riding in autopilot here; apologize for that! meanwhile, I got time to upgrade our port and sync it to a more recent code from upstream :) the distfile on PYPI changed and we lost notifications from portscout -- that's why we kept serving only 4.x instead of newer 5.x version for this port. that said: we now applied the most recent changes added to the ports framework to handle python ports using pep517 and pycryptography.
(In reply to David Horn from comment #8) that you could actually try suggesting to the maintainers of 'security/py-cryptography' I mean, to set that on MAKE_ENV for the non-legacy version might even support giving us a global solution --or I'm just being silly and too optimistic :3
(In reply to David Horn from comment #10) I just mentioned that on #bsdports on EFnet and hope that if this patch lands in there you can get the credits for it David. ty again for reporting this one -- I am closing the issue but will keep an eye out for future troubles :) have a great one
Thanks for the update to the latest upstream, works much better now. Latest commit at least now gives a useful error message with regards to the environment variable needed to be successful. I will continue to pull the thread on the root issue to see if we can make this "just work" without setting environment variables at make or runtime. I'm currently experimenting with /etc/ssl/openssl.cnf and security/py-cryptography knobs to see if there is an easy fix. Hitting up folks on #freebsd-python IRC as well. All else fails, I will put in another bug report for security/py-cryptography. Thanks again! -_Dave
The term "legacy" is too overloaded in this context as well. CRYPTOGRAPHY_OPENSSL_NO_LEGACY implies that deprecated algorithms/functions will not be available from the linked openssl implementation, and security/py-cryptography-legacy implies that the older build system (non-rust) will be used (nothing to do with openssl deprecated algorithms/function). In /etc/ssl/openssl.cnf, legacy openssl 3 providers `openssl list -providers` are disabled in 14-Alpha4 in base by default, but enabling them does not seem to satisfy py-cryptography's need for legacy provider support. <sigh>Investigation (and hopefully understanding) continues ;)
Created attachment 244660 [details] 14-Alpha4 Patch to Base OpenSSL for legacy provider Looks like the patch notes from https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254853#c98 by Mark Millard solves the base issue with the OpenSSL 3 legacy provider in stable/14 base for my use case (py-cryptography/py-fido2/py-yubikey-manager. No more needing to set the environment variable or modify `/etc/ssl/openssl.cnf` to execute python. This applies cleanly to stable/14 for me. I uploaded an attachment for future reference.
Actual fix for openssl legacy provider has been committed to -CURRENT here: https://cgit.freebsd.org/src/commit/?id=1a18383a52bc373e316d224cef1298debf6f7e25 Waiting for MFC to stable/14, then will re-test again.