Doesn't work on head since OpenSSL 3: ~$ aws Traceback (most recent call last): File "/usr/local/bin/aws", line 19, in <module> import awscli.clidriver File "/usr/local/lib/python3.9/site-packages/awscli/clidriver.py", line 17, in <module> import botocore.session File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 26, in <module> import botocore.client File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 15, in <module> from botocore import waiter, xform_name File "/usr/local/lib/python3.9/site-packages/botocore/waiter.py", line 18, in <module> from botocore.docs.docstring import WaiterDocstring File "/usr/local/lib/python3.9/site-packages/botocore/docs/__init__.py", line 15, in <module> from botocore.docs.service import ServiceDocumenter File "/usr/local/lib/python3.9/site-packages/botocore/docs/service.py", line 14, in <module> from botocore.docs.client import ClientDocumenter, ClientExceptionsDocumenter File "/usr/local/lib/python3.9/site-packages/botocore/docs/client.py", line 17, in <module> from botocore.docs.example import ResponseExampleDocumenter File "/usr/local/lib/python3.9/site-packages/botocore/docs/example.py", line 13, in <module> from botocore.docs.shape import ShapeDocumenter File "/usr/local/lib/python3.9/site-packages/botocore/docs/shape.py", line 19, in <module> from botocore.utils import is_json_value_header File "/usr/local/lib/python3.9/site-packages/botocore/utils.py", line 37, in <module> import botocore.httpsession File "/usr/local/lib/python3.9/site-packages/botocore/httpsession.py", line 45, in <module> from urllib3.contrib.pyopenssl import ( File "/usr/local/lib/python3.9/site-packages/urllib3/contrib/pyopenssl.py", line 50, in <module> import OpenSSL.crypto File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import SSL, crypto File "/usr/local/lib/python3.9/site-packages/OpenSSL/SSL.py", line 9, in <module> from OpenSSL._util import ( File "/usr/local/lib/python3.9/site-packages/OpenSSL/_util.py", line 6, in <module> from cryptography.hazmat.bindings.openssl.binding import Binding File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 167, in <module> Binding.init_static_locks() File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 134, in init_static_locks cls._ensure_ffi_initialized() File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 123, in _ensure_ffi_initialized _legacy_provider_error(cls._legacy_provider_loaded) File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 43, in _legacy_provider_error raise RuntimeError( RuntimeError: OpenSSL 3.0's legacy provider failed to load. This is a fatal error by default, but cryptography supports running without legacy algorithms by setting the environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY. If you did not expect this error, you have likely made a mistake with your OpenSSL configuration.
Please, your "uname -srmK" and "openssl version" Because: # uname -srmK FreeBSD 15.0-CURRENT amd64 1500000 # openssl version OpenSSL 3.0.10 1 Aug 2023 (Library: OpenSSL 3.0.10 1 Aug 2023) # aws Note: AWS CLI version 2, the latest major version of the AWS CLI, is now stable and recommended for general use. For more information, see the AWS CLI version 2 installation instructions at: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters] To see help text, you can run: aws help aws <command> help aws <command> <subcommand> help aws: error: the following arguments are required: command Did you update py-openssl and py-cryptography to the latest versions?
OK, this is not an awscli problem, it's a problem of interaction between py-cryptography and OpenSSL 3.0 configured specifically in FreeBSD, where there are legacy providers missing. Enji, you seem to be handling OpenSSL 3.0 import, is it possible to also include: - RC2 - RC4 - IDEA - SEED etc, etc, in order to make things work again? The other solution would be to call the programs using py-cryptography with environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY set. This is not a problem of only py-awscli, but with all programs using py-cryptography.
Created attachment 244635 [details] simple patch (improved version should check if base uses OpenSSL 3)
it looks good to me as a workaround, thanks. As a permanent solution, py-cryptography should not rely on the legacy providers, but this is beyond the scope of this PR.
(In reply to Juraj Lutter from comment #2) I've been handling most of the import of OpenSSL 3.0 into the base system (sponsored by the FreeBSD Foundation) so feel free to CC me for everything related. And sorry if I missed anything!
Can you run this command on your system, and let me know if you see anything failing instead? # echo test | openssl md4 -provider legacy MD4(stdin)= 36d729ab4ff7260da6fb010ef5747bb3
On stable/14: otis@cake:~ % echo test | openssl md4 -provider legacy MD4(stdin)= 36d729ab4ff7260da6fb010ef5747bb3 on main: % echo test | openssl md4 -provider legacy MD4(stdin)= 36d729ab4ff7260da6fb010ef5747bb3
See also bug 273506. There is something particular in the way Python loads these .so's, as the following occurs: ... >>> import OpenSSL DBG: calling OSSL_PROVIDER_load(NULL, default) DBG: provider=<cdata 'OSSL_PROVIDER *' 0x800fb41c0> DBG: calling OSSL_PROVIDER_load(NULL, legacy) Breakpoint 1, rtld_dlopen (name=name@entry=0x801734720 "/usr/lib/ossl-modules/legacy.so", fd=fd@entry=-1, mode=2) at /usr/src/libexec/rtld-elf/rtld.c:3678 3678 LD_UTRACE(UTRACE_DLOPEN_START, NULL, NULL, 0, mode, name); (gdb) n 3679 ld_tracing = (mode & RTLD_TRACE) == 0 ? NULL : "1"; (gdb) 3680 if (ld_tracing != NULL) { (gdb) 3688 if (mode & RTLD_NODELETE) (gdb) 3690 if (mode & RTLD_NOLOAD) (gdb) 3692 if (mode & RTLD_DEEPBIND) (gdb) 3690 if (mode & RTLD_NOLOAD) (gdb) 3688 if (mode & RTLD_NODELETE) (gdb) 3694 if (ld_tracing != NULL) (gdb) 3697 return (dlopen_object(name, fd, obj_main, lo_flags, (gdb) 3698 mode & (RTLD_MODEMASK | RTLD_GLOBAL), NULL)); (gdb) 3697 return (dlopen_object(name, fd, obj_main, lo_flags, (gdb) dlfcn_load (dso=0x800e51e10) at /usr/src/crypto/openssl/crypto/dso/dso_dlfcn.c:117 117 if (ptr == NULL) { (gdb) p ptr $1 = (void *) 0x0 So for some reason, python3 gets NULL when calling OSSL_PROVIDER_load(NULL, "legacy"), although OSSL_PROVIDER_load(NULL, "default") works just fine. If you do exactly the same with a small test program like: #include <stdio.h> #include <openssl/provider.h> int main(void) { OSSL_PROVIDER *default_provider = OSSL_PROVIDER_load(NULL, "default"); printf("default_provider=%p\n", default_provider); OSSL_PROVIDER *legacy_provider = OSSL_PROVIDER_load(NULL, "legacy"); printf("legacy_provider=%p\n", legacy_provider); return 0; } You will see that it works just fine and returns non-NULL pointers for both the default and legacy providers.
I think the problem (with both the port and the version in base) is that the config doesn’t enable the legacy provider by default: you need to programmatically enable it in order to access the legacy (deemed unsafe) cryptographic algorithms. The same goes with the FIPS provider unless one programmatically loads it.
(In reply to Enji Cooper from comment #9) As comment #8, it seems python-specific. Even you enabled the legacy provider in /etc/ssl/openssl.cnf , python still cannot load the legacy provider.
(In reply to Li-Wen Hsu from comment #10) I don't understand openssl well enough to say if this is sane or not, but Mark Millard's suggestion in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254853#c100 seems to work for me: diff --git a/crypto/openssl/crypto/init.c b/crypto/openssl/crypto/init.c index cacf637c89f8..77f693fcfa05 100644 --- a/crypto/openssl/crypto/init.c +++ b/crypto/openssl/crypto/init.c @@ -159,7 +159,7 @@ DEFINE_RUN_ONCE_STATIC(ossl_init_load_crypto_nodelete) if (!err_shelve_state(&err)) return 0; - dso = DSO_dsobyaddr(&base_inited, DSO_FLAG_NO_UNLOAD_ON_FREE); + dso = DSO_dsobyaddr(&base_inited, DSO_FLAG_NO_UNLOAD_ON_FREE|DSO_FLAG_GLOBAL_SYMBOLS); /* * In case of No!, it is uncertain our exit()-handlers can still be * called. After dlclose() the whole library might have been unloaded
(In reply to Kristof Provost from comment #11) Indeed, just being curious why the current code works on Linux and needed in FreeBSD. If this can be answered, I think it's fine to apply this.
(In reply to Li-Wen Hsu from comment #12) I replied in bug #254853, comment #102 but meant to do it here: On Linux (as per Debian 12) legacy.so is linked to libcrypto.so. In FreeBSD I chose not to, since this situation seems to be supported by OpenSSL 3, and legacy.so from the security/openssl30 port did not get linked to libcrypto.so. It was also easier to build legacy.so for FreeBSD this way, when dealing with the build order: libcrypto.so would need to be built before legacy.so.
(In reply to Pierre Pronchery from comment #13) I've thrown that patch up in https://reviews.freebsd.org/D41874, because I'd really like to land a fix for this. We have a large number of failing tests as a result of this issue, and we really shouldn't release until this is fixed either.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=1a18383a52bc373e316d224cef1298debf6f7e25 commit 1a18383a52bc373e316d224cef1298debf6f7e25 Author: Pierre Pronchery <pierre@freebsdfoundation.org> AuthorDate: 2023-09-15 15:14:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-16 15:55:12 +0000 libcrypto: link engines and the legacy provider to libcrypto OpenSSL's legacy provider module and engines need to link to libcrypto.so, as it provides some of the actual implementations of legacy routines. This is a little tricky due to build order issues. Introduce a small hack (LIBCRYPTO_WITHOUT_SUBDIRS) that builds libcrypto.so in its usual early phase without any OpenSSL provider modules or engines. This is intended to restore the test suite; a future change should remove the hack and replace it with a better approach. PR: 254853, 273528 Discussed with: Folks at EuroBSDCon in Coimbra Sponsored by: The FreeBSD Foundation Makefile.inc1 | 2 +- secure/lib/libcrypto/Makefile | 2 ++ secure/lib/libcrypto/engines/Makefile.inc | 2 ++ secure/lib/libcrypto/modules/legacy/Makefile | 1 + 4 files changed, 6 insertions(+), 1 deletion(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=fd4b13acd90bdc107a072660d91baa163474ebef commit fd4b13acd90bdc107a072660d91baa163474ebef Author: Pierre Pronchery <pierre@freebsdfoundation.org> AuthorDate: 2023-09-15 15:14:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-19 15:01:55 +0000 libcrypto: link engines and the legacy provider to libcrypto OpenSSL's legacy provider module and engines need to link to libcrypto.so, as it provides some of the actual implementations of legacy routines. This is a little tricky due to build order issues. Introduce a small hack (LIBCRYPTO_WITHOUT_SUBDIRS) that builds libcrypto.so in its usual early phase without any OpenSSL provider modules or engines. This is intended to restore the test suite; a future change should remove the hack and replace it with a better approach. PR: 254853, 273528 Discussed with: Folks at EuroBSDCon in Coimbra Sponsored by: The FreeBSD Foundation (cherry picked from commit 1a18383a52bc373e316d224cef1298debf6f7e25) Makefile.inc1 | 2 +- secure/lib/libcrypto/Makefile | 2 ++ secure/lib/libcrypto/engines/Makefile.inc | 2 ++ secure/lib/libcrypto/modules/legacy/Makefile | 1 + 4 files changed, 6 insertions(+), 1 deletion(-)
A commit in branch releng/14.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b7312b90289e7244c124da9845c0a3011ba8d936 commit b7312b90289e7244c124da9845c0a3011ba8d936 Author: Pierre Pronchery <pierre@freebsdfoundation.org> AuthorDate: 2023-09-15 15:14:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-19 17:19:17 +0000 libcrypto: link engines and the legacy provider to libcrypto OpenSSL's legacy provider module and engines need to link to libcrypto.so, as it provides some of the actual implementations of legacy routines. This is a little tricky due to build order issues. Introduce a small hack (LIBCRYPTO_WITHOUT_SUBDIRS) that builds libcrypto.so in its usual early phase without any OpenSSL provider modules or engines. This is intended to restore the test suite; a future change should remove the hack and replace it with a better approach. PR: 254853, 273528 Discussed with: Folks at EuroBSDCon in Coimbra Sponsored by: The FreeBSD Foundation (cherry picked from commit 1a18383a52bc373e316d224cef1298debf6f7e25) (cherry picked from commit fd4b13acd90bdc107a072660d91baa163474ebef) Approved by: re (gjb) Makefile.inc1 | 2 +- secure/lib/libcrypto/Makefile | 2 ++ secure/lib/libcrypto/engines/Makefile.inc | 2 ++ secure/lib/libcrypto/modules/legacy/Makefile | 1 + 4 files changed, 6 insertions(+), 1 deletion(-)
I believe this is now fixed, can you confirm? (There are a couple of other OpenSSL 3 related issues in progress, but I believe devel/py-awscli should work.)
Works fine with 2ccfa855b2fc.