Bug 254643

Summary: security/openssl 1.1.1k and SSL webservers do not work when cryptodev.ko is loaded
Product: Ports & Packages Reporter: cryx-freebsd
Component: Individual Port(s)Assignee: Bernard Spil <brnrd>
Status: Closed FIXED    
Severity: Affects Some People CC: bjk, chris, emaste, franco, jhb, lev, wollman
Priority: --- Flags: bugzilla: maintainer-feedback? (brnrd)
Version: Latest   
Hardware: amd64   
OS: Any   
Attachments:
Description Flags
Test case
none
svn diff for security/openssl
brnrd: maintainer-approval+
svn diff for security/openssl brnrd: maintainer-approval+

Description cryx-freebsd 2021-03-29 19:48:41 UTC
I've seen this happening on FreeBSD 11.4 and 12.2 that when cryptodev.ko module is loaded and openssl with version 1.1.1k from ports is installed, both nginx and apache24 are unable to establish an SSL session.

nginx error message is like this:

[crit] 5569#100511: *27523 SSL_do_handshake() failed (SSL: error:0201502D:system library:ioctl:Operation not supported error:1427D044:SSL routines:construct_stateless_ticket:internal error error:0201502D:system library:ioctl:Operation not supported) (45: Operation not supported) while SSL handshaking

When unloading the cryptodev.ko and restarting the webserver, everything works okay, reloading cryptodev.ko again and restarting the webserver brings back the issue.

This did not happen with previously installed 1.1.1j version.
Comment 1 Lev A. Serebryakov freebsd_committer freebsd_triage 2021-04-01 10:07:52 UTC
I have same problem. Commenting out this code helps:

#ifdef CRIOGET
    if (ioctl(fd, CRIOGET, &cfd) < 0) {
        fprintf(stderr, "Could not create crypto fd: %s\n", strerror(errno));
        close(fd);
        cfd = -1;
        return;
    }
    close(fd);
#else
    cfd = fd;
#endif

in crypto/engine/eng_devcrypto.c

workaround this problem
Comment 2 Bernard Spil freebsd_committer freebsd_triage 2021-04-01 18:57:47 UTC
(In reply to Lev A. Serebryakov from comment #1)

Hi Lev,

This is related to these commits?

https://github.com/openssl/openssl/commit/a12c6442f24a32867c971b6feb5db61d01b02c1f
https://github.com/openssl/openssl/commit/b6de54b2c1062f15819174784d9bd53c85c432d3

commenting out seems ill advised, you would no longer open a handle for cfd.

Can you try the same thing reversing these 2 commits?

Can't reproduce this at the moment, running 13.0-RC4 with
> device cryptodev # /dev/crypto for access to Hardware
in kernel config
Comment 3 Bernard Spil freebsd_committer freebsd_triage 2021-04-01 18:59:19 UTC
Adding jhb as author of the commits that seem to cause this issue
Comment 4 John Baldwin freebsd_committer freebsd_triage 2021-04-01 20:49:49 UTC
Adding Garrett and Benjamin who I was talking with on a thread about this in regards to breakage Garrett sees with nrpe3.
Comment 5 Garrett Wollman freebsd_committer freebsd_triage 2021-04-01 21:03:51 UTC
(In reply to John Baldwin from comment #4)
Pretty sure this is the same problem and the same fix.

Short-term workaround: add CONFIGURE_ARGS+=no-devcryptoeng to security/openssl/Makefile.
Comment 6 Garrett Wollman freebsd_committer freebsd_triage 2021-04-01 21:06:06 UTC
An easy run-time workaround is `devfs rule apply path crypto hide` and restart the daemons.
Comment 7 Garrett Wollman freebsd_committer freebsd_triage 2021-04-01 21:07:34 UTC
Created attachment 223755 [details]
Test case
Comment 8 John Baldwin freebsd_committer freebsd_triage 2021-04-01 21:18:49 UTC
So some more details from the thread with Garrett.  The crypto fd obtained via the CRIOGET ioctl does not survive fork()'s (it also can't be passed across UNIX domain sockets and the same property which is used to kill kqueue() fd's across fork has the same effect here), so the fd ends up being either invalid or reused for some other purpose in the child process.

The older 1.0.2 engine avoided this by not trying to reuse a global cfd in this manner, but instead it only kept the original /dev/crypto around in a global and called CRIOGET for each new session.

FreeBSD 13.0 has changed /dev/crypto so that CRIOGET is no longer needed.  Instead, operations are now done directly against /dev/crypto, and /dev/crypto fd's are normal character device fd's that work across fork.

In practice, the /dev/crypto engine is not very useful even in 13.0.  The only time it can possibly offload something is for a self-contained AES-CBC operation that is using a coprocessor.  In particular, using AESNI instructions does _not_ need /dev/crypto, and you are much better off executing those instructions in userland directly than using /dev/crypto which just adds extra copies and the overhead of kernel <-> user transitions.  Note that TLS, for example, never uses bare AES-CBC, but uses it conjunction with an HMAC, so even if you managed to offload the AES-CBC pass via the engine, the local CPU would still have to touch the data in userland to compute the HMAC.

For now I think the most expedient thing is to disable the crypto engine on 12.x and older in the port.  I will probably just revert the CRIOGET changes from OpenSSL upstream so that it only uses the engine on 13.0 and later.
Comment 9 Garrett Wollman freebsd_committer freebsd_triage 2021-04-01 21:22:58 UTC
(In reply to John Baldwin from comment #8)
Might be worthwhile to have an OPTION in the port, defaulting to off, even for 13.x, since given the limitations that you state it's not going to be helpful for most users.
Comment 10 John Baldwin freebsd_committer freebsd_triage 2021-04-01 21:27:15 UTC
I worry that an option will just confuse users a bit more.  What we do right now for the base openssl is we only enable /dev/crypto in 13.0 and later and don't hook it uptimes to the build at all in 12.x.  Having the port disable the engine on < 13.0 would match the behavior in /usr/src.  I wouldn't be opposed to an option so long as the option wasn't even available on < 13.
Comment 11 Garrett Wollman freebsd_committer freebsd_triage 2021-04-01 21:32:05 UTC
(In reply to John Baldwin from comment #10)
Sounds reasonable. I just don't want people enabling it unless they actually know they want it. (I'll be removing cryptodev from my kernels but new packages are much easier to push than a new kernel.)
Comment 12 Lev A. Serebryakov freebsd_committer freebsd_triage 2021-04-02 08:52:45 UTC
(In reply to Bernard Spil from comment #2)
Is it actual, or problem is understood?
Comment 13 Bernard Spil freebsd_committer freebsd_triage 2021-04-02 18:58:10 UTC
Created attachment 223773 [details]
svn diff for security/openssl

`poudriere testport` was successful
`perl configdata.pm --dump` showed `devcryptoeng` in disabled on 12.2
Comment 14 Bernard Spil freebsd_committer freebsd_triage 2021-04-02 19:32:56 UTC
Created attachment 223774 [details]
svn diff for security/openssl

Bump portrevision
Comment 15 Garrett Wollman freebsd_committer freebsd_triage 2021-04-04 03:20:40 UTC
(In reply to Bernard Spil from comment #14)
LGTM.
Comment 16 commit-hook freebsd_committer freebsd_triage 2021-04-06 10:47:32 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=64d94165652f40041bf71bdf7a775f867e7b80a9

commit 64d94165652f40041bf71bdf7a775f867e7b80a9
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-04-06 10:39:33 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-04-06 10:45:45 +0000

    security/openssl: Fix /dev/crypto issue with 1.1.1k

    PR:             254643
    Reported by:    <cryx-freebsd h3q com>
    Reviewed by:    wollman

 security/openssl/Makefile | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
Comment 17 commit-hook freebsd_committer freebsd_triage 2021-04-06 10:57:35 UTC
A commit in branch 2021Q1 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=48f05c509b86f1ce4a7c9be31ed8adc891870734

commit 48f05c509b86f1ce4a7c9be31ed8adc891870734
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-04-06 10:39:33 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-04-06 10:54:17 +0000

    security/openssl: Fix /dev/crypto issue with 1.1.1k

    PR:             254643
    Reported by:    <cryx-freebsd h3q com>
    Reviewed by:    wollman

 security/openssl/Makefile | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
Comment 18 Bernard Spil freebsd_committer freebsd_triage 2021-04-06 11:11:43 UTC
Thanks for your help in this!
Comment 19 commit-hook freebsd_committer freebsd_triage 2021-04-09 11:43:48 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=5dd3123de243e32c77eb792f46c9fce41f00c11c

commit 5dd3123de243e32c77eb792f46c9fce41f00c11c
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-04-09 11:40:52 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-04-09 11:42:19 +0000

    security/openssl-devel: Update to 3.0.0-alpha14

     * and fix cryptodev on < 13

    PR:             254643
    Reported by:    <cryx-freebsd h3q com>
    Reviewed by:    wollman

 security/openssl-devel/Makefile  | 12 +++++++++---
 security/openssl-devel/distinfo  |  6 +++---
 security/openssl-devel/pkg-plist |  5 +++++
 3 files changed, 17 insertions(+), 6 deletions(-)