Created attachment 223896 [details] patch for 3.4.7 py-cryptography is now using a bit of rust in an extension; I've filed a bug for a new port for setuptools-rust, #254851 We've made this to work by USES=cargo, but then making sure that setuptools-rust does the compilation, with the CARGO_BUILD CARGO_INSTALL CARGO_TEST =no
Thanks Jeroen, does this depend on bug 254851? If so please add it to the Depends On field, thanks!
I have it from cryptography upstream that the 3.4 branch is safe to use without rust (and doesnt require it), as long as we set the environment variable to not use it. Looks like the patch attached here is the reverse of what was intended? @Jeroen Are you able to do a plain 3.4.x update here (adding the right env var not to use rust, see setup.py), and then create a separate issue for the future 35.0 branch coming soon that includes the changes you included here? (using rust).
Created attachment 225623 [details] Don't build rust modules (3.4.7 specific) You mean setting `CRYPTOGRAPHY_DONT_BUILD_RUST=`. The setup.py still imports setuptools_rust. Under the assumption that it is completely safe to also remove the setuptools_rust import, I've patched setup.py, and not set the environment variable. I don't see anything else using CRYPTOGRAPHY_DONT_BUILD_RUST. The environment variable is documented in https://cryptography.io/en/3.4.7/faq.html. It took me a while to understand that that instruction is gone with the master version of docs.
(In reply to Jeroen Pulles from comment #3) Requesting clarity from upstream on the nature of the import error and the extent to which its related to, or should be conditional on the env var not being present
Upstream advises that: - patching out the error and passing the env var, if the tests pass, is OK - that they'll accept an upstream for conditionalizing the import/error block on the non presence of the env var
Minor changes to commit: - Use specific *_DEPENDS lines (with <version-specs>) over shared macros - Add comment to patch "setuptools_rust try/except should be conditional on environment variable"
Looking forward for this
Maintainer reset.
The python.mk do-install step should register all files with the `--record` argument. On my installs this fails to register the cryptography/hazmat/bindings/_rust.abi3.so file that is in the build and stage directories. The _rust.abi3.so file is not in the .PLIST.pymodtmp file, and thus isn't installed. If I set `PYDISTUTILS_SETUP=${PYSETUP}` in the py-cryptography makefile, I get the same result: no _rust.abi3.so file. However, if I manually run `python3.9 setup.py install --record my-file.log` in the work directory, then _rust.abi3.so is found in "my-file.log". I don't understand where this goes wrong. How can I get this file included in the (auto-generated) plist?
Hopefully this port can be updated soon, as py-openssl has been updated and now requires py-cryptography >= 35.0
(Never mind that, py-openssl has been downgraded so things are no longer broken, see bug #262750)
From security/py-openssl: > # We need to keep old py-cryptography and py-penssl for 11.x release > # due to outdated OpenSSL version in base No longer seems relevant after 2021-09-31 EoL of 11.x What does it take to land this in ports?
Created attachment 233937 [details] git diff for security/py-cryptography ``` security/py-cryptography: Update to 37.0.2 * Remove Rust dependency * Supports LibreSSL 3.2 - 3.5 PR: 254853 Submitted by: Jeroen Pulles ```
Any movement on this?
Created attachment 236907 [details] update to 38.0.1 with rust build works for me
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=266670 required to build with fresh libressl.
Latest 38.0.1 from ports does not work for me: ImportError: cannot import name 'ObjectIdentifier' from 'cryptography.hazmat.bindings._rust' (unknown location) Traceback (most recent call last): File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.9/site-packages/homeassistant/__main__.py", line 203, in <module> sys.exit(main()) File "/usr/local/lib/python3.9/site-packages/homeassistant/__main__.py", line 159, in main args = get_arguments() File "/usr/local/lib/python3.9/site-packages/homeassistant/__main__.py", line 69, in get_arguments from . import config as config_util File "/usr/local/lib/python3.9/site-packages/homeassistant/config.py", line 19, in <module> from . import auth File "/usr/local/lib/python3.9/site-packages/homeassistant/auth/__init__.py", line 10, in <module> import jwt File "/usr/local/lib/python3.9/site-packages/jwt/__init__.py", line 1, in <module> from .api_jwk import PyJWK, PyJWKSet File "/usr/local/lib/python3.9/site-packages/jwt/api_jwk.py", line 6, in <module> from .algorithms import get_default_algorithms File "/usr/local/lib/python3.9/site-packages/jwt/algorithms.py", line 6, in <module> from .utils import ( File "/usr/local/lib/python3.9/site-packages/jwt/utils.py", line 7, in <module> from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in <module> from cryptography.hazmat._oid import ObjectIdentifier File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/_oid.py", line 7, in <module> from cryptography.hazmat.bindings._rust import ( ImportError: cannot import name 'ObjectIdentifier' from 'cryptography.hazmat.bindings._rust' (unknown location) ============================================================== with mine patch that include rust it work.
Created attachment 237172 [details] update to 38.0.1 with rust build Rebased patch includes rust code build.
I've successfully built the 38.0.1+rust patch. This however also requires py-opensll to be upgraded to 22.0 0. With that combination py-certbot works fine again.
Hi, I cannot succeed in patching port from attachmnent 237172 It's a long time since I did any programming, and I do not know what should be in the patch file I etracted the "raw unified" view, but when using patch it asks for the name of the file to pach. If someone can help...
Try: patch -p1 --directory /usr/ports/ --input RAW_UNIFIED.patch
(In reply to Ivan Rozhuk from comment #21) Thanks, now duplicity works again, and I have backups
Created attachment 237314 [details] update to 38.0.1 with rust build
Created attachment 239411 [details] update to 38.0.1 with rust build
I see diffs are added here to build 38.0.1. Is there anything that blocks this from being rolled out? A lot of broken dependencies because the obsolete version of this package, py39-matrix-synapse being one of them.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269705
Looking at the project's setup.py file, py-setuptools-rust is a hard requirement.
Proposed patch posted to Differential Review: https://reviews.freebsd.org/D38939 .
(In reply to Enji Cooper from comment #28) The diff from the review builds a package that fails when running certbot. The diff 'update to 38.01 with rust build' updated to 39.0.2 works as expected. I think it has to do with where the tarball if fetched from, since changing to PYPI from GH creates a non-working package.
(In reply to Ralf van der Enden from comment #29) Hmm... I see. I'll see if I can replicate what you're describing.
Created attachment 240799 [details] update to 38.0.1 with rust build
(In reply to Enji Cooper from comment #30) I incorporated the missing post-install logic and applied the other logic to the diff. Please re-review.
Until Rust builds under QEMU_EMULATING, this really can't be updated to something that hard requires Rust. You're talking obliteration of a large swath of the Python ecosystem for those who can't build on native hardware. The cryptography team only migrated the non-SSL/TLS stuff, like X.509 support, to Rust; the rest is still a wrapper around OpenSSL/LibreSSL. I backported much of what was needed to make cryptography 3.4 work with current OpenSSL/LibreSSL, though not necessarily the API from later cryptography: https://github.com/vishwin/py-cryptography/tree/3.4.x It is better to provide a separate security/py-cryptography-rust (or similarly named) in the same vein as graphics/librsvg2{,-rust} with an appropriate selector.
(In reply to Charlie Li from comment #33) On the other hand, not updating this port puts us in a bind with other projects, which now require newer versions of py-cryptography than the one we're shipping. py-openssl, for example, requires cryptography>=38.0.0 and py-twisted in turn had to be hackfixed/patched to even work with our version of py-openssl anymore. Maintaining an ever growing set of patches doesn't seem like that great of an idea either. Regardless of whether keep the old version around and place the new one in security/py-cryptography-rust or update this port, I think we should decide on one of these solutions rather soon so that other ports can move ahead.
I tend to agree with Sascha: as of right now all projects reliant on modern copies of py-openssl, py-pycryptography, etc, are stuck. This is blocking modern python support (newer versions of py-pycryptography are guaranteed to be better supported with 3.11+) and this is blocking us from switching over to openssl3. I am game for conditionalizing support -- but it shouldn't block the upgrades. I'll start a conversation upstream with the cryptography project so setuptools-rust dependent pieces can (maybe) be split off into their own packages/wheels.
(In reply to Enji Cooper from comment #35) https://github.com/pyca/cryptography/issues/8513
Looking into this, the whole splitting of rust vs non-rust building is an all or nothing change that will affect multiple ports that require x509 support. I don't think it's worth implementing If folks are interested in conditionalizing the rust build for their specific build use cases, I could see adding that support, but introducing additional complexity in the port doesn't make a ton of sense since it would require monkey patching the port a lot to make it conditionally build and install hazmat/fernet modules which rely on rust bindings, e.g., x509, etc.
(In reply to Enji Cooper from comment #37) Incomplete thought... "I don't think it's worth implementing in FreeBSD ports, since it would introduce a larger longterm maintenance burden."
Considering that I did all that backporting so 3.4 could remain operable, with LibreSSL to boot, the maintenance burden is well understood and at least I have been willing to keep it up. Upstream does a good job with their CI, since they test consumers in addition to their own stuff, so keeping that synced makes things easier. I actually just renewed a Let's Encrypt certificate with the latest certbot using this on Python 3.10, so not everything is broken or has a bleak outlook. Splitting the rust bits off isn't a good idea, agreed, since SSL/TLS libraries always had the functionality that cryptography wrapped/bound, until they decided that C was not good enough for anything but the actual SSL/TLS bits. But I also agree that we do need to make an updated cryptography available, hence suggesting (and has been on my list for time) the graphics/librsvg2{,-rust} approach.
Created attachment 240959 [details] update to 39.0.2 with rust build
Created attachment 241170 [details] update to 39.0.2 with rust build rebased
*** Bug 266989 has been marked as a duplicate of this bug. ***
I'm working on the 40.0.1 update [1]. Please test it. Thanks. [1] https://people.freebsd.org/~sunpoet/patch/security-py-cryptography.txt
This should still be a separate port with a DEFAULT_VERSIONS knob. QEMU_EMULATING users are still affected.
(In reply to Charlie Li from comment #44) Upstream has made a convincing argument against _not_ doing that. It sounds like the longterm goal is to expand rust use, not reduce it, in the port. I think FreeBSD just needs to bite the bullet and do something to either help by throwing resources against our private builders, or assist upstream with producing regular prebuilt binaries (which should greatly help everyone the community at large).
(In reply to Enji Cooper from comment #45) To clarify, when I said "upstream" in this comment, I meant rust, not py-cryptography. If rust doesn't have to be built from scratch each time, it will definitely reduce the cost on the qemu builders.
(In reply to Enji Cooper from comment #46) If it's any help, the creator of poudriere is working on an update where some packages can be pre-added to an enviroment (such as rust) so they don't build anew. I had this problem with clamd and a jail I built from the start. Here's the github issue about it: https://github.com/freebsd/poudriere/issues/822
(In reply to László Károlyi from comment #47) Thank you for the link László! I'll reach out to bdrewery about that effort -- it would definitely address the need for this port dealing with rust.
That poudriere issue is not relevant here, not in the slightest, because it does not address how to avoid building or executing rustc in QEMU_EMULATING, which alone prevents building any port with a Rust component in that environment. You would need something akin to the static ccache option *inserted* into a builder jail on creation to avoid QEMU_EMULATING since packages are built for specific architectures.
It certainly solves the building part, but I didn't know it has problems with executing under QEMU. That's very bad.
Because I have a story, I hope to upgrade the security/py-cryptography too. I had upgraded security/openssl to 3.1. and I running certification file update script @ security/py-certbot and got error as follows... Renewing Let's Encrypt certificates: Traceback (most recent call last): File "/usr/local/bin/certbot-3.9", line 33, in <module> sys.exit(load_entry_point('certbot==2.4.0', 'console_scripts', 'certbot')()) File "/usr/local/bin/certbot-3.9", line 25, in importlib_load_entry_point return next(matches).load() File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load module = import_module(match.group('module')) File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/usr/local/lib/python3.9/site-packages/certbot/main.py", line 6, in <module> from certbot._internal import main as internal_main File "/usr/local/lib/python3.9/site-packages/certbot/_internal/main.py", line 21, in <module> import josepy as jose File "/usr/local/lib/python3.9/site-packages/josepy/__init__.py", line 40, in <module> from josepy.json_util import ( File "/usr/local/lib/python3.9/site-packages/josepy/json_util.py", line 14, in <module> from OpenSSL import crypto File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/usr/local/lib/python3.9/site-packages/OpenSSL/crypto.py", line 17, in <module> from OpenSSL._util import ( File "/usr/local/lib/python3.9/site-packages/OpenSSL/_util.py", line 6, in <module> from cryptography.hazmat.bindings.openssl.binding import Binding File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module> from cryptography.hazmat.bindings._openssl import ffi, lib ImportError: /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "ERR_GET_FUNC" Errors were reported when renewing Let's Encrypt certificate(s). After googled, answer is current version of security/py-cryptography too old (3.4.8) Thanks a lot.
Bug 254853 looks like the issue where rust fails to build with QEMU_EMULATING.
Hi, I'm adding this information here, instead of filing a new bug report. security/py-cryptography fails at runtime oh head with OpenSSL 3, most probably version 3.4.8 present in the ports tree does not properly support the new OpenSSL. The error is: ImportError: /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: Undefined symbol "ERR_GET_FUNC" I guess getting this library updated is now even more urgent.
(In reply to Guido Falsi from comment #53) Opps, this information was already present, but I missed it for some reason, sorry for the noise.
Created attachment 243353 [details] v0 Update to 41.0.2 It fixes ansible for me
(In reply to Mikael Urankar from comment #55) this is based on Po-Chuan Hsieh's patch
Patches which upgrade the port have been available for several months now. With OpenSSL 3 in base for the past few weeks, I don’t understand why continuing to build broken ports is acceptable. What is blocking the update from being done?
I think it is time to go ahead with this and deal with Tier-2 archs later.
Moin moin I'll take this with my portmgr hat. mfg Tobias
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=b28a739884e37c25bb57ea5f157252adb63065d3 commit b28a739884e37c25bb57ea5f157252adb63065d3 Author: Tobias C. Berner <tcberner@FreeBSD.org> AuthorDate: 2023-07-22 07:38:27 +0000 Commit: Tobias C. Berner <tcberner@FreeBSD.org> CommitDate: 2023-07-22 07:46:38 +0000 framework: convert tree to use USES=pycryptography A future commit will update to security/py-cryptography will introduce a rust dependency. PR: 254853 cad/uranium/Makefile | 3 +-- comms/py-esptool/Makefile | 3 +-- databases/py-mycli/Makefile | 3 +-- databases/py-ormar/Makefile | 3 +-- databases/py-sqlalchemy-utils/Makefile | 2 +- devel/oci-cli/Makefile | 3 +-- devel/osc/Makefile | 5 ++--- devel/py-adb/Makefile | 5 ++--- devel/py-aiortc/Makefile | 3 +-- devel/py-azure-identity/Makefile | 3 +-- devel/py-azure-keyvault/Makefile | 3 +-- devel/py-azure-multiapi-storage/Makefile | 3 +-- devel/py-azure-storage-common/Makefile | 3 +-- devel/py-castellan/Makefile | 5 ++--- devel/py-cursive/Makefile | 3 +-- devel/py-moto/Makefile | 3 +-- devel/py-msal/Makefile | 5 ++--- devel/py-oci/Makefile | 3 +-- devel/py-openstacksdk/Makefile | 3 +-- devel/py-pip/Makefile | 5 ++--- devel/py-twisted/Makefile | 2 +- dns/letsdns/Makefile | 5 ++--- dns/py-dns-crawler/Makefile | 3 +-- dns/py-dns-lexicon/Makefile | 3 +-- dns/py-dnspython/Makefile | 2 +- finance/electrum/Makefile | 3 +-- finance/py-ccxt/Makefile | 5 ++--- mail/py-flanker/Makefile | 3 +-- misc/py-apache-beam/Makefile | 5 ++--- misc/py-cinder/Makefile | 3 +-- net-im/py-matrix-synapse/Makefile | 3 +-- net-mgmt/py-adal/Makefile | 5 ++--- net-p2p/reticulum/Makefile | 5 ++--- net/onionprobe/Makefile | 5 ++--- net/py-cepa/Makefile | 5 ++--- net/py-ripe.atlas.sagan/Makefile | 5 ++--- net/py-smbprotocol/Makefile | 5 ++--- net/py-transip/Makefile | 5 ++--- net/py-urllib3/Makefile | 6 +++--- net/scapy/Makefile | 5 ++--- news/sabnzbd/Makefile | 2 +- security/caldera/Makefile | 3 +-- security/cowrie/Makefile | 3 +-- security/py-SecretStorage/Makefile | 5 ++--- security/py-acme/Makefile | 5 ++--- security/py-asyncssh/Makefile | 5 ++--- security/py-authlib/Makefile | 4 +--- security/py-azure-keyvault-keys/Makefile | 3 +-- security/py-badkeys/Makefile | 5 ++--- security/py-cert-human/Makefile | 5 ++--- security/py-certbot/Makefile | 3 +-- security/py-dfvfs/Makefile | 3 +-- security/py-fido2/Makefile | 1 + security/py-josepy/Makefile | 5 ++--- security/py-msoffcrypto-tool/Makefile | 5 ++--- security/py-noiseprotocol/Makefile | 4 +--- security/py-oauthlib/Makefile | 6 +++--- security/py-openssl/Makefile | 5 ++--- security/py-paramiko/Makefile | 3 +-- security/py-pgpy/Makefile | 5 ++--- security/py-plaso/Makefile | 3 +-- security/py-pyhanko-certvalidator/Makefile | 3 +-- security/py-pyhanko/Makefile | 3 +-- security/py-pysaml2/Makefile | 5 ++--- security/py-pysaml24/Makefile | 5 ++--- security/py-pysaml26/Makefile | 5 ++--- security/py-pyspnego/Makefile | 3 +-- security/py-python-axolotl/Makefile | 5 ++--- security/py-requests-credssp/Makefile | 5 ++--- security/py-securesystemslib/Makefile | 5 ++--- security/py-service_identity/Makefile | 3 +-- security/py-social-auth-core/Makefile | 5 ++--- security/py-sshpubkeys/Makefile | 5 ++--- security/py-sslyze/Makefile | 5 ++--- security/py-stem/Makefile | 3 +-- security/py-trustme/Makefile | 5 ++--- security/py-txtorcon/Makefile | 3 +-- security/py-webauthn/Makefile | 3 +-- security/py-yubikey-manager/Makefile | 3 +-- sysutils/datadog-integrations/Makefile | 9 +++------ sysutils/py-ansible-core/Makefile | 5 ++--- sysutils/py-azure-cli-core/Makefile | 3 +-- sysutils/py-azure-cli/Makefile | 3 +-- textproc/py-pdfminer.six/Makefile | 5 ++--- www/buku/Makefile | 3 +-- www/mitmproxy/Makefile | 3 +-- www/py-autobahn/Makefile | 5 ++--- www/py-azure-storage/Makefile | 3 +-- www/py-flask-jwt-extended/Makefile | 3 +-- www/py-pyjwt/Makefile | 2 +- www/py-pyjwt1/Makefile | 4 ++-- www/py-requests_ntlm/Makefile | 5 ++--- www/py-scrapy/Makefile | 5 ++--- x11/xpra/Makefile | 3 +-- 94 files changed, 141 insertions(+), 229 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=173ac9651cc9ef0ab952b29e4cffa56d28e8770b commit 173ac9651cc9ef0ab952b29e4cffa56d28e8770b Author: Tobias C. Berner <tcberner@FreeBSD.org> AuthorDate: 2023-07-22 06:00:10 +0000 Commit: Tobias C. Berner <tcberner@FreeBSD.org> CommitDate: 2023-07-22 07:46:37 +0000 framework: add PYCRYPTOGRAPHY default version A future commit will update to security/py-cryptography will introduce a rust dependency. PR: 254853 Mk/bsd.default-versions.mk | 8 +++++++- UPDATING | 12 ++++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=7bb64b89d0e5ec8d77b28f8341269ffbad279ebf commit 7bb64b89d0e5ec8d77b28f8341269ffbad279ebf Author: Tobias C. Berner <tcberner@FreeBSD.org> AuthorDate: 2023-07-22 06:16:46 +0000 Commit: Tobias C. Berner <tcberner@FreeBSD.org> CommitDate: 2023-07-22 07:46:38 +0000 framework: add Mk/Uses/pycryptography With the added new DEFAULT_VERSION "PYCRYPTOGRAHY_DEFAULT=rust|legacy" it is important to transparently depend on the right one. This is the job of this new uses. So instead of manually adding dependency lines like RUN_DEPENDS=${PYTHON_PKGNAMEPREFIX}cryptography>0:security/py-cryptography@${PY_FLAVOR} this should now be USES=pycryptography:run Supported arguments: <none>,build,run,test If no argument is given, it defaults to 'build,run'. A future commit will update to security/py-cryptography will introduce a rust dependency. PR: 254853 CHANGES | 20 ++++++++++++++++++ Mk/Uses/pycryptography.mk (new) | 46 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=c380909c8060259c0eb8aa067de08534677924b4 commit c380909c8060259c0eb8aa067de08534677924b4 Author: Tobias C. Berner <tcberner@FreeBSD.org> AuthorDate: 2023-07-22 05:51:54 +0000 Commit: Tobias C. Berner <tcberner@FreeBSD.org> CommitDate: 2023-07-22 07:46:37 +0000 security/py-cryptography: copy port to -legacy variant prior to update A future commit will update to security/py-cryptography will introduce a rust dependency. PR: 254853 security/Makefile | 1 + security/py-cryptography-legacy/Makefile (new) | 51 ++++ security/py-cryptography-legacy/distinfo (new) | 3 + .../files/patch-libressl (new) | 316 +++++++++++++++++++++ .../files/patch-setup.py (new) | 55 ++++ security/py-cryptography-legacy/pkg-descr (new) | 7 + 6 files changed, 433 insertions(+)
Moin moin I added a new pycryptography uses and a default version PYCRYPTOGRAPHY_DEFAULT=rust|legacy. @mikael, please update the security/py-cryptography port to the one using rust. Please let me know if I broke something :) mfg Tobias
(In reply to Charlie Li from comment #33) I think it would be advisable to switch the legacy-ports upstream to your fork, given the looming incompatibility with base OpenSSL. Could you do that part, please? mfg Tobias
Created attachment 243623 [details] v1
(In reply to Mikael Urankar from comment #66) Great work with your patches! Hope they get committed to the tree ASAP. I am using locally your v0 patch at present. It works but there is an issue with it, you correctly replaced `.include <bsd.port.post.mk>` with `.include <bsd.port.mk>` but you left `.include <bsd.port.pre.mk>` which is wrong and causes the build to fail in poudriere due to failed sanity checks. Apart from this v0 patch looks fine to me. I'm in the process of applying this new v1 patch.
(In reply to Mikael Urankar from comment #66) Hi, I made the following PR earlier: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272885 Your patch applied cleanly with git apply and py-cryptography built and installed fine. This PR was later pointed out to me by markml because I was getting the error "Undefined symbol "ERR_GET_FUNC"" when running certbot. After reinstalling py-cryptography with your patch, I recompiled certbot (just from the ports tree, this is on recent -current with a ports tree as of a few hrs ago). certbot builds and installs fine but when I try to run it, it bails with this error at the end: File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 43, in _legacy_provider_error raise RuntimeError( RuntimeError: OpenSSL 3.0's legacy provider failed to load. This is a fatal error by default, but cryptography supports running without legacy algorithms by setting the environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY. If you did not expect this error, you have likely made a mistake with your OpenSSL configuration.
(In reply to void from comment #68) setenv CRYPTOGRAPHY_OPENSSL_NO_LEGACY 1
(In reply to Mikael Urankar from comment #69) Hi, I tried the following in /usr/ports/security/py-certbot # setenv CRYPTOGRAPHY_OPENSSL_NO_LEGACY 1 # make distclean clean config deinstall reinstall It builds and installs fine. Then ran certbot certonly and the error message is now different: File "/usr/local/lib/python3.9/site-packages/josepy/json_util.py", line 14, in <module> from OpenSSL import crypto File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/usr/local/lib/python3.9/site-packages/OpenSSL/crypto.py", line 3279, in <module> _lib.OpenSSL_add_all_algorithms() AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'
(In reply to void from comment #70) pyOpenSSL probably needs to be upgraded: https://reviews.freebsd.org/D39037 . I really wish I had a ports commit bit sometimes (*sigh*)…
(In reply to void from comment #68) (In reply to Mikael Urankar from comment #69) FreeBSD should default to disabling the legacy crypto providers and should have a build flag to flip the default around. The legacy crypto providers are considered insecure by upstream (and many in the security community) today; it’s best to default a secure crypto stance (and add something to RELNOTES for folks upgrading to 14.0-RELEASE) instead of doing the opposite. That way folks who are getting a fresh install are defaulting to more secure cryptography.
(In reply to Enji Cooper from comment #71) # patch -p0 < py-openssl.patch Hmm... Looks like a unified diff to me... The text leading up to this was: -------------------------- |Index: security/py-openssl/Makefile |=================================================================== |--- security/py-openssl/Makefile |+++ security/py-openssl/Makefile -------------------------- File to patch: Makefile Patching file Makefile using Plan A... Hunk #1 failed at 1. Hunk #2 failed at 19. Hunk #3 failed at 27. Hunk #4 failed at 43. 4 out of 4 hunks failed--saving rejects to Makefile.rej Hmm... The next patch looks like a unified diff to me... The text leading up to this was: -------------------------- |Index: security/py-openssl/distinfo |=================================================================== |--- security/py-openssl/distinfo |+++ security/py-openssl/distinfo -------------------------- File to patch: distinfo Patching file distinfo using Plan A... Hunk #1 failed at 1. 1 out of 1 hunks failed--saving rejects to distinfo.rej done
(In reply to Enji Cooper from comment #72) Just a thought - do you think later python versions might be a workaround? Or maybe libressl built as default ssl before then building all other ssl-related packages like certbot?
(In reply to void from comment #73) Yeah... I need to update the patch (I forgot that conflicts were introduced after I opened the differential revision).
(In reply to void from comment #74) 1. re: later versions of cpython: I don't think so. 2. re: using libressl: yes, that's probably a viable workaround, but given libressl and openssl's shared history (and exposed symbols), my gut says mix-and-matching the libraries might not work as desired.
(In reply to Enji Cooper from comment #75) There is newer py-openssl in bug 266680.
(In reply to Charlie Li from comment #49) QUOTE That poudriere issue is not relevant here, not in the slightest, because it does not address how to avoid building or executing rustc in QEMU_EMULATING END QUOTE That is an armv6 issue but armv7 has been built via the ampere[123] build servers that support aarch32/armv7 code exectution for about a year as of now (2023-August). qemu is not used for building armv7 ports into packages any more for official builds. https://pkg-status.freebsd.org/?all=1&type=package reports the first ampere armv7 build as: default default 130releng-armv7 a51002eeb3d0 32299 29273 294 1791 941 0 stopped:done: Fri, 22 Jul 2022 07:39:45 GMT 66:37:20 ampere3 The first build of armv7 main: default default main-armv7 p83aeeda2ebb7_s30253da1a 32288 29099 (+27041) 274 (+147) 1948 (+26) 967 (+247) 0 stopped:done: Fri, 29 Jul 2022 21:46:08 GMT 88:40:40 ampere2 The first build of armv7 quarterly: default quarterly 130releng-armv7 8662e656d870 32236 29152 (+27142) 238 (+140) 1871 (+9) 975 (+237) 0 stopped:done: Fri, 05 Aug 2022 13:04:55 GMT 70:00:06 ampere1 (I've been building armv7 ports that way for longer, but on different hardware.) From Warner's list activity, it looks to me like not supporting armv6 may be a viable alternative. You might want to ask him about such. From what I see looking at the build history, less than half of the ports manage to build for armv6 via qemu currently.
(In reply to Brad Davis from comment #77) bingo! result is a working certbot :D TYVM :D
(In reply to Mark Millard from comment #78) Good for you, not everyone else has a suitable real armv7 machine to perform builds, especially when not everyone can use prefetch at all. I personally haven't built any armv6 for years, or even ever. The better way in the long term is to make qemu-user expendable in our cross tooling, particularly when executing compilers or interpreters.
(In reply to Charlie Li from comment #80) May be I need to provide more context for my note. Looks like I took to much context from how I got here. John Baldwin had written on freebsd-arch: QUOTE rust isn't supported on armv7 on FreeBSD (and someone would need to do the work to make that happen) END QUOTE and later: QUOTE Please see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254853 as an example. According to that bug at least rustc is not doable under qemu-user which is how armv7 packages are built. END QUOTE I was really just making notes that would avoid the errors of interpretation that occurred: A) "rust isn't supported on armv7 on FreeBSD": it is B) "rustc is not doable under qemu-user which is how armv7 packages are built": qemu is not how (official) armv7 packages are built I was not trying to get into the viability of not having even, say, an RPi4B to build armv7 packages with via poudriere-devel and an armv7 poudriere jail. (Not what I normally use these days but I've done so and am still set up to do so. I've used a Windows Dev Kit 2023 and a HoneyComb as well.)
To be a bit more precise: armv7 can compile rust by itself (with a few patches), including some selected applications. I do this in an infinite loop (as part of my platform stability test), along with a full kde5 build (and a few other ports). Of course, poudriere is not the right tool for 32-bit platforms, so all these ports are built using direct make.
Err, "(with a few fixes)" should be attached to kde5, rust itself doesn't need any out of tree fixes.
I updated the differential revision with the latest version of py-cryptography, which required updating all of the cargo crate pins: https://reviews.freebsd.org/D38939 .
I forgot to provide my patch for both ports. See https://people.freebsd.org/~sunpoet/patch/security-py-cryptography.txt and https://people.freebsd.org/~sunpoet/patch/security-py-openssl.txt
Created attachment 244442 [details] Update py-cryptography{,-vectors} to 41.0.3
(In reply to Li-Wen Hsu from comment #86) I'm doing a test build for all the ports have USES=pycryptography with https://bugs.freebsd.org/bugzilla/attachment.cgi?id=244442 . Will commit it with tcberner's approval in comment #64 (I've checked with him.)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=82c675ccaa87127ca291de1d224aa18cb782149f commit 82c675ccaa87127ca291de1d224aa18cb782149f Author: Li-Wen Hsu <lwhsu@FreeBSD.org> AuthorDate: 2023-08-29 17:28:54 +0000 Commit: Li-Wen Hsu <lwhsu@FreeBSD.org> CommitDate: 2023-08-29 17:28:54 +0000 security/py-cryptography*: Update to 41.0.3 This patch combines the efforts from the people invloved in the PR. I just do the integrating and testing. PR: 254853 Approved by: tcberner (portmgr) Sponsored by: The FreeBSD Foundation security/py-cryptography-vectors/Makefile | 7 +- security/py-cryptography-vectors/distinfo | 6 +- security/py-cryptography/Makefile | 30 +- security/py-cryptography/Makefile.crates (new) | 53 ++++ security/py-cryptography/distinfo | 112 +++++++- .../py-cryptography/files/patch-libressl (gone) | 316 --------------------- .../py-cryptography/files/patch-setup.py (gone) | 55 ---- 7 files changed, 186 insertions(+), 393 deletions(-)
The change from 3.4.8 to 41.0.3 seems to have broken py-dnspython; switching to py-cryptography-legacy (IOW regressing to 3.4.8) seems to have fixed it for now but probably isn't ideal. Traceback (most recent call last): File "/usr/local/bin/spf.py", line 155, in <module> import dns.resolver # http://www.dnspython.org File "/usr/local/lib/python3.9/site-packages/dns/resolver.py", line 30, in <module> import dns._ddr File "/usr/local/lib/python3.9/site-packages/dns/_ddr.py", line 12, in <module> import dns.nameserver File "/usr/local/lib/python3.9/site-packages/dns/nameserver.py", line 5, in <module> import dns.asyncquery File "/usr/local/lib/python3.9/site-packages/dns/asyncquery.py", line 32, in <module> import dns.quic File "/usr/local/lib/python3.9/site-packages/dns/quic/__init__.py", line 4, in <module> import aioquic.quic.configuration # type: ignore File "/usr/local/lib/python3.9/site-packages/aioquic/quic/configuration.py", line 6, in <module> from ..tls import ( File "/usr/local/lib/python3.9/site-packages/aioquic/tls.py", line 40, in <module> from OpenSSL import crypto File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/usr/local/lib/python3.9/site-packages/OpenSSL/crypto.py", line 3279, in <module> _lib.OpenSSL_add_all_algorithms() AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'
(In reply to vometia from comment #89) The problem is that py-openssl needs to be updated as well. py-cryptography 41.0.3 has a minimum version requirement for py-openssl (version 22.0, but possible higher).
(In reply to Ralf van der Enden from comment #90) There is a PR to update it, someone just needs to push it thru https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=266680
Move two depending rust tickets to see also, so we can close this one and the ones depend on this. We can open or create another ticket for the follow-ups but let's clear the thoughts for now.
I'll have to pursue the details later, but on aarch64 it looks like kyua tests involving python are broken after updating to the recent changes: # /usr/bin/kyua test -k /usr/tests/Kyuafile examples/test_examples.py examples/test_examples.py:__test_cases_list__ -> broken: Test program did not exit cleanly [0.002s] By contrast, on amd64 it still runs: # /usr/bin/kyua test -k /usr/tests/Kyuafile examples/test_examples.py you have mail examples/test_examples.py:TestExampleSimple::test_get_properties -> skipped: comment me to run the test [0.274s] examples/test_examples.py:TestExampleSimple::test_one -> skipped: comment me to run the test [0.276s] examples/test_examples.py:TestExampleSimple::test_parametrize[AF_INET6] -> skipped: comment me to run the test [0.275s] examples/test_examples.py:TestExampleSimple::test_parametrize[AF_INET] -> skipped: comment me to run the test [0.274s] examples/test_examples.py:TestExampleSimple::test_parametrize[FAMILY_39] -> skipped: comment me to run the test [0.273s] examples/test_examples.py:TestExampleSimple::test_syscall_failure -> skipped: comment me to run the test [0.276s] examples/test_examples.py:TestExampleSimple::test_two -> skipped: Required file '/path/file1' not found [0.001s] examples/test_examples.py:TestExampleSimple::test_with_cleanup -> passed [0.272s] examples/test_examples.py:TestExampleSimplest::test_one -> skipped: comment me to run the test [0.270s] examples/test_examples.py:TestVnetDual1::test_ifstat -> skipped: comment me to run the test [0.271s] examples/test_examples.py:TestVnetSimple::test_ping -> skipped: comment me to run the test [0.270s] examples/test_examples.py:TestVnetSimple::test_topology -> skipped: comment me to run the test [0.271s] Results file id is usr_tests.20230901-160656-320496 Results saved to /usr/home/root/.kyua/store/results.usr_tests.20230901-160656-320496.db 12/12 passed (0 failed) [an armv7 chroot (on aarch64 system) gets the same kind of result as aarch64. But armv7 had other issues before the change as well.] For reference: # uname -apKU FreeBSD CA78C-WDK23-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT 2023 root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000 # ~/fbsd-based-on-what-commit.sh -C /usr/ports/ 8f7f59bbc250 (HEAD -> main, freebsd/main, freebsd/HEAD) sysutils/bmd-plugin-hookcmd: add new ports. Author: Vanilla I. Shu <vanilla@FreeBSD.org> Commit: Vanilla I. Shu <vanilla@FreeBSD.org> CommitDate: 2023-09-01 02:54:14 +0000 branch: main merge-base: 8f7f59bbc2504672483c7840638c7fc171e1b9f5 merge-base: CommitDate: 2023-09-01 02:54:14 +0000 n632086 (--first-parent --count for merge-base)
(In reply to Mark Millard from comment #93) I have submitted the aarch64 kyua phython use issue in: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273506 I've initially guessed that that Base System changes would be involved in a fix but that may be wrong (only a ports issue then). So my classification could be wrong. The submittal has a Python backtrace that at its end indicates: QUOTE RuntimeError: OpenSSL 3.0's legacy provider failed to load. END QUOTE
(In reply to Mark Millard from comment #94) I tracked the aarch64 example failure reprting down to dlerror message text via use of gdb: /usr/lib/ossl-modules/legacy.so: Undefined symbol \"MD4_Update\"", '\000' <repeats 449 times> See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273506
(In reply to Mark Millard from comment #95) amd64 did not have: devel/py-pytest-twisted devel/py-twisted net/py-dpkt security/nist-kat security/openvpn installed and so was not doing the same sequence of dlopen calls. After adding those, the amd64 run fails like aarch64: It then starts to try to use .../cryptography/hazmat/bindings/_rust.abi3.so leads to the problem. [Detaching after fork from child process 98729] Breakpoint 1.1, dlopen (name=name@entry=0x803239710 "libc.so.7", mode=2) at /usr/main-src/libexec/rtld-elf/rtld.c:3662 3662 return (rtld_dlopen(name, -1, mode)); (gdb) c Continuing. Breakpoint 1.1, dlopen (name=name@entry=0x8032c80d0 "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so", mode=2) at /usr/main-src/libexec/rtld-elf/rtld.c:3662 3662 return (rtld_dlopen(name, -1, mode)); (gdb) c Continuing. Breakpoint 1.1, dlopen (name=name@entry=0x8032c7440 "/usr/local/lib/python3.9/site-packages/_cffi_backend.cpython-39.so", mode=2) at /usr/main-src/libexec/rtld-elf/rtld.c:3662 3662 return (rtld_dlopen(name, -1, mode)); (gdb) c Continuing. Breakpoint 1.1, dlopen (name=name@entry=0x80219d080 "/usr/lib/ossl-modules/legacy.so", mode=2) at /usr/main-src/libexec/rtld-elf/rtld.c:3662 3662 return (rtld_dlopen(name, -1, mode)); (gdb) c Continuing. INTERNALERROR> Traceback (most recent call last): . . . This appears to be tied to devel/py-twisted and/or devel/py-pytest-twisted .
(In reply to Mark Millard from comment #96) Looking at the dlopen of /lib/libcrypto.so.30 shows mode=2, i.e.: mode=(RTLD_NOW|RTLD_LOCAL) The RTLD_LOCAL means that MD4_Update and the like are not available for the later dlopen of /usr/lib/ossl-modules/legacy.so to bind to. . . . The mode value comes from dlfcn_load in: /usr/main-src/crypto/openssl/crypto/dso/dso_dlfcn.c static int dlfcn_load(DSO *dso) { void *ptr = NULL; /* See applicable comments in dso_dl.c */ char *filename = DSO_convert_filename(dso, NULL); int flags = DLOPEN_FLAG; int saveerrno = get_last_sys_error(); if (filename == NULL) { ERR_raise(ERR_LIB_DSO, DSO_R_NO_FILENAME); goto err; } # ifdef RTLD_GLOBAL if (dso->flags & DSO_FLAG_GLOBAL_SYMBOLS) flags |= RTLD_GLOBAL; # endif # ifdef _AIX if (filename[strlen(filename) - 1] == ')') flags |= RTLD_MEMBER; # endif ptr = dlopen(filename, flags); . . . So far I've not found anything that would lead to dso->flags & DSO_FLAG_GLOBAL_SYMBOLS being non-zero. Thus the failures would result: /usr/lib/ossl-modules/legacy.so use is designed for RTLD_GLOBAL use for /lib/libcrypto.so.30 .
The following crude patch leads to kyua's python use working for examples/test_examples.py with devel/py-pytest-twisted and devel/py-twisted and the like installed: # git -C /usr/main-src/ diff crypto/openssl/crypto/init.c diff --git a/crypto/openssl/crypto/init.c b/crypto/openssl/crypto/init.c index cacf637c89f8..77f693fcfa05 100644 --- a/crypto/openssl/crypto/init.c +++ b/crypto/openssl/crypto/init.c @@ -159,7 +159,7 @@ DEFINE_RUN_ONCE_STATIC(ossl_init_load_crypto_nodelete) if (!err_shelve_state(&err)) return 0; - dso = DSO_dsobyaddr(&base_inited, DSO_FLAG_NO_UNLOAD_ON_FREE); + dso = DSO_dsobyaddr(&base_inited, DSO_FLAG_NO_UNLOAD_ON_FREE|DSO_FLAG_GLOBAL_SYMBOLS); /* * In case of No!, it is uncertain our exit()-handlers can still be * called. After dlclose() the whole library might have been unloaded (The above note is also in my separate bugzilla.) For reference: # /usr/bin/kyua test -k /usr/tests/Kyuafile examples/test_examples.py examples/test_examples.py:TestExampleSimple::test_get_properties -> skipped: comment me to run the test [0.582s] examples/test_examples.py:TestExampleSimple::test_one -> skipped: comment me to run the test [0.589s] examples/test_examples.py:TestExampleSimple::test_parametrize[AF_INET6] -> skipped: comment me to run the test [0.582s] examples/test_examples.py:TestExampleSimple::test_parametrize[AF_INET] -> skipped: comment me to run the test [0.595s] examples/test_examples.py:TestExampleSimple::test_parametrize[FAMILY_39] -> skipped: comment me to run the test [0.589s] examples/test_examples.py:TestExampleSimple::test_syscall_failure -> skipped: comment me to run the test [0.590s] examples/test_examples.py:TestExampleSimple::test_two -> skipped: Required file '/path/file1' not found [0.001s] examples/test_examples.py:TestExampleSimple::test_with_cleanup -> passed [0.583s] examples/test_examples.py:TestExampleSimplest::test_one -> skipped: comment me to run the test [0.583s] examples/test_examples.py:TestVnetDual1::test_ifstat -> skipped: comment me to run the test [0.593s] examples/test_examples.py:TestVnetSimple::test_ping -> skipped: comment me to run the test [0.589s] examples/test_examples.py:TestVnetSimple::test_topology -> skipped: comment me to run the test [0.588s] Results file id is usr_tests.20230903-014520-255324 Results saved to /usr/home/root/.kyua/store/results.usr_tests.20230903-014520-255324.db 12/12 passed (0 failed)
(In reply to vometia from comment #89) It's py39-openssl that's broken (see your error message), and basically everything that requires it. That includes dnspython, certbot, synapse (matrix server), and much more. Until that gets fixed, everything remains broken.
(In reply to László Károlyi from comment #99) After my system openssl patch in comment #98 it works just like before the recent changes. /usr/src/crypto/openssl/crypto/init.c is preventing /lib/libcrypto.so.30 from publishing its definitions so that /usr/lib/ossl-modules/legacy.so can put them to use. /lib/libcrypto.so.30 needs to be dlopen'd via: mode=(RTLD_NOW|RTLD_GLOBAL) instead of: mode=(RTLD_NOW|RTLD_LOCAL) In crypto/openssl/crypto/init.c terms this means that "|DSO_FLAG_GLOBAL_SYMBOLS" needs to be used --but it is no currently: dso = DSO_dsobyaddr(&base_inited, DSO_FLAG_NO_UNLOAD_ON_FREE|DSO_FLAG_GLOBAL_SYMBOLS); For use of the system libraries, it is a system bug.
@Mark Millard: Looks like your patch to base openssl 3 legacy provider also fixes the root cause of my py-cryptography legacy provider issue. @All: Hopefully this patch is under review, and can be committed to stable/14 (and CURRENT) soon. My py-cryptography stable/14 issue and tests with patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273505#c17 I will be glad to re-test all of this once the patch lands. --Thanks!
(In reply to Mark Millard from comment #100) On Linux (as per Debian 12) legacy.so is linked to libcrypto.so. In FreeBSD I chose not to, since this situation seems to be supported by OpenSSL 3, and legacy.so from the security/openssl30 port did not get linked to libcrypto.so. It was also easier to build legacy.so for FreeBSD this way, when dealing with the build order: libcrypto.so would need to be built before legacy.so.
Pierre: do we know why Linux does that and FreeBSD doesn’t?
Reopen for legacy.so loading issue.
(In reply to Enji Cooper from comment #103) Revisiting this issue, it could be that I was wrong. (Or the port was updated since I last compared?) Regardless I have updated the patch in bug #273506 as a new candidate for a fix. (Also as review D41879)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1a18383a52bc373e316d224cef1298debf6f7e25 commit 1a18383a52bc373e316d224cef1298debf6f7e25 Author: Pierre Pronchery <pierre@freebsdfoundation.org> AuthorDate: 2023-09-15 15:14:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-16 15:55:12 +0000 libcrypto: link engines and the legacy provider to libcrypto OpenSSL's legacy provider module and engines need to link to libcrypto.so, as it provides some of the actual implementations of legacy routines. This is a little tricky due to build order issues. Introduce a small hack (LIBCRYPTO_WITHOUT_SUBDIRS) that builds libcrypto.so in its usual early phase without any OpenSSL provider modules or engines. This is intended to restore the test suite; a future change should remove the hack and replace it with a better approach. PR: 254853, 273528 Discussed with: Folks at EuroBSDCon in Coimbra Sponsored by: The FreeBSD Foundation Makefile.inc1 | 2 +- secure/lib/libcrypto/Makefile | 2 ++ secure/lib/libcrypto/engines/Makefile.inc | 2 ++ secure/lib/libcrypto/modules/legacy/Makefile | 1 + 4 files changed, 6 insertions(+), 1 deletion(-)
I tested the commit from Comment 106 on -CURRENT/15, and it resolves the OpenSSL 3 legacy loading issue for me. FreeBSD freebsd14 15.0-CURRENT FreeBSD 15.0-CURRENT #1 main-n265364-9080190b970c: Sat Sep 16 13:42:11 EDT 2023 root@freebsd14:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 `make test` on py-cryptography, py-fido2, and py-yubikey-manager all succeed without errors (And as desired, this removes the warnings about setting CRYPTOGRAPHY_OPENSSL_NO_LEGACY since the legacy provider from OpenSSL 3 now works as desired) I will watch for the MFC to stable/14 and re-test all of my fail cases again at that time.
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=fd4b13acd90bdc107a072660d91baa163474ebef commit fd4b13acd90bdc107a072660d91baa163474ebef Author: Pierre Pronchery <pierre@freebsdfoundation.org> AuthorDate: 2023-09-15 15:14:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-19 15:01:55 +0000 libcrypto: link engines and the legacy provider to libcrypto OpenSSL's legacy provider module and engines need to link to libcrypto.so, as it provides some of the actual implementations of legacy routines. This is a little tricky due to build order issues. Introduce a small hack (LIBCRYPTO_WITHOUT_SUBDIRS) that builds libcrypto.so in its usual early phase without any OpenSSL provider modules or engines. This is intended to restore the test suite; a future change should remove the hack and replace it with a better approach. PR: 254853, 273528 Discussed with: Folks at EuroBSDCon in Coimbra Sponsored by: The FreeBSD Foundation (cherry picked from commit 1a18383a52bc373e316d224cef1298debf6f7e25) Makefile.inc1 | 2 +- secure/lib/libcrypto/Makefile | 2 ++ secure/lib/libcrypto/engines/Makefile.inc | 2 ++ secure/lib/libcrypto/modules/legacy/Makefile | 1 + 4 files changed, 6 insertions(+), 1 deletion(-)
A commit in branch releng/14.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b7312b90289e7244c124da9845c0a3011ba8d936 commit b7312b90289e7244c124da9845c0a3011ba8d936 Author: Pierre Pronchery <pierre@freebsdfoundation.org> AuthorDate: 2023-09-15 15:14:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-19 17:19:17 +0000 libcrypto: link engines and the legacy provider to libcrypto OpenSSL's legacy provider module and engines need to link to libcrypto.so, as it provides some of the actual implementations of legacy routines. This is a little tricky due to build order issues. Introduce a small hack (LIBCRYPTO_WITHOUT_SUBDIRS) that builds libcrypto.so in its usual early phase without any OpenSSL provider modules or engines. This is intended to restore the test suite; a future change should remove the hack and replace it with a better approach. PR: 254853, 273528 Discussed with: Folks at EuroBSDCon in Coimbra Sponsored by: The FreeBSD Foundation (cherry picked from commit 1a18383a52bc373e316d224cef1298debf6f7e25) (cherry picked from commit fd4b13acd90bdc107a072660d91baa163474ebef) Approved by: re (gjb) Makefile.inc1 | 2 +- secure/lib/libcrypto/Makefile | 2 ++ secure/lib/libcrypto/engines/Makefile.inc | 2 ++ secure/lib/libcrypto/modules/legacy/Makefile | 1 + 4 files changed, 6 insertions(+), 1 deletion(-)
I retested on stable/14 with the py-cryptography, py-fido2, py-yubikey-manager. With latest commit openssl and py-cryptography looks good from my standpoint. Ports work, `make test` succeeds for all of the cases where it previously complained about openssl legacy provider support missing. commit fd4b13acd90bdc107a072660d91baa163474ebef FreeBSD freebsd14 14.0-STABLE FreeBSD 14.0-STABLE #2 stable/14-n265155-fd4b13acd90b: Tue Sep 19 18:53:18 EDT 2023 root@freebsd14:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 I will also test aarch64 later.
Make file missed: MAKE_ENV= OPENSSL_DIR="${OPENSSLBASE}" without this line it links with OpenSSL from base instead of libressl. # ldd /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: libssl.so.111 => /usr/lib/libssl.so.111 (0x80147a000) libcrypto.so.111 => /lib/libcrypto.so.111 (0x8020b3000) libthr.so.3 => /lib/libthr.so.3 (0x801514000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x801542000) libc.so.7 => /lib/libc.so.7 (0x801082000)
(In reply to Ivan Rozhuk from comment #111) Could you please open up a new issue? It would be nice to close this issue and fix followup issues instead of keeping this open indefinitely.
(In reply to David Horn from comment #110) arm64/aarch64 looks good now as well on 14-BETA3 (has legacy OpenSSL 3 provider fix in base) after latest package builders completed for aarch64. All of my failcases have been resolved. Thanks!
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=5202fbf4e8e4dd1793c24780d2c17ee5e0610a05 commit 5202fbf4e8e4dd1793c24780d2c17ee5e0610a05 Author: Ben Shertenlieb <b.shertenlieb@yahoo.com> AuthorDate: 2023-11-03 07:24:55 +0000 Commit: Li-Wen Hsu <lwhsu@FreeBSD.org> CommitDate: 2023-11-03 07:24:55 +0000 multimedia/plexmediaserver-plexpass: Update to 1.32.8.7639 New release from Plex.tv Changelog: https://forums.plex.tv/t/plex-media-server/30447/607 NEW: (Web) Updated to 4.116.1 FIXES: (Library) “Episode sorting” and “Album sorting” preferences had meaningless “Library default” global library setting. (Library) The default season sort order in TV libraries may not work as intended. (LiveTV) Transcoded video would fail to play in some circumstances. (Metadata) Items could lose metadata if a refresh request failed due to a network failure. (Player) Controlling a player didn’t work in some cases. (Sharing) Sharing a single episode was not playable by sharee. (Transcoder) HW transcoding failed on GeminiLake CPUs when running on linux. (Transcoder) Videos HW encoded on the shield would display a black screen on some devices. (Updater) Windows Updater would hang when run with PMSService. PR: 254853 multimedia/plexmediaserver-plexpass/Makefile | 4 ++-- multimedia/plexmediaserver-plexpass/distinfo | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
ImportError: /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: Undefined symbol "EVP_default_properties_is_fips_enabled" All ports have been rebuilt - everything is updated - the latest FreeBSD 13.2 # certbot Traceback (most recent call last): File "/usr/local/bin/certbot", line 33, in <module> sys.exit(load_entry_point('certbot==2.7.4', 'console_scripts', 'certbot')()) File "/usr/local/bin/certbot", line 25, in importlib_load_entry_point return next(matches).load() File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load module = import_module(match.group('module')) File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/usr/local/lib/python3.9/site-packages/certbot/main.py", line 6, in <module> from certbot._internal import main as internal_main File "/usr/local/lib/python3.9/site-packages/certbot/_internal/main.py", line 21, in <module> import josepy as jose File "/usr/local/lib/python3.9/site-packages/josepy/__init__.py", line 40, in <module> from josepy.json_util import ( File "/usr/local/lib/python3.9/site-packages/josepy/json_util.py", line 14, in <module> from OpenSSL import crypto File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import SSL, crypto File "/usr/local/lib/python3.9/site-packages/OpenSSL/SSL.py", line 9, in <module> from OpenSSL._util import ( File "/usr/local/lib/python3.9/site-packages/OpenSSL/_util.py", line 6, in <module> from cryptography.hazmat.bindings.openssl.binding import Binding File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module> from cryptography.exceptions import InternalError File "/usr/local/lib/python3.9/site-packages/cryptography/exceptions.py", line 9, in <module> from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions ImportError: /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: Undefined symbol "EVP_default_properties_is_fips_enabled"
(In reply to Dmitry from comment #115) security/py-cryptography already has the following update committed on 2023-Sep-22: security/py-cryptography: Update to 43.0.4 The update to 43.0.1 was back 2023-Aug-29 so it has been in place for a while. I'm unsure if a new bugzilla is needed at this point instead of continuing to grow this one for issues that might require additional changes. It may be that this one should be closed now. It is also not clear to me if you built your own ports, used a package from 2023Q4 (quarterly), or used a package from main (latest). Wherever the submittal goes, you probably need to also indicate such context related to the certbot command that you used that ended up reporting EVP_default_properties_is_fips_enabled as missing.
(In reply to Mark Millard from comment #116) https://pypi.org/project/cryptography/ https://github.com/pyca/cryptography/tags 41.0.5 I see no 43.0+ in wild, where source for it?
(In reply to Mark Millard from comment #116) py39-cryptography-41.0.5,1 this is the only one available in the ports - all ports have been updated to the latest ones - FreeBSD version 13.2 where is 43.0.4 available?
(In reply to Dmitry from comment #118) Dumb digit position swapping (spanning several characters) on my part from what I can tell. So copy/paste from the update after the 41.0.3 commit instead: security/py-cryptography: Update to 41.0.5 - Update WWW From: https://cgit.freebsd.org/ports/commit/security/py-cryptography?id=469b9cacad27 Sorry for the misleading text.
(In reply to Mark Millard from comment #119) Actually, I previously copied/pasted the bad reference that has a "3 instead of 1" typo: https://cgit.freebsd.org/ports/commit/security/py-cryptography?id=5935d168accf reports itself as: security/py-cryptography: Update to 43.0.4 instead of as an update to 41.0.4 . Still, sorry for the misleading text.
(In reply to Mark Millard from comment #120) on the contrary - if you downgrade the version, it works https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273770#c8
This PR is still being shown as a blocker to other PRs. It should be closed for not being a reason for procrastination.
update to 42.0.7 is already in the ports tree.