Bug 253135

Summary: OpenSSL KTLS causes regression in svn/svnlite
Product: Base System Reporter: Guido Falsi <madpilot>
Component: binAssignee: John Baldwin <jhb>
Status: Closed FIXED    
Severity: Affects Some People CC: emaste, hyogeollee, jbeich, jhb, junchoon, lwhsu, ohartmann
Priority: --- Keywords: regression
Version: CURRENTFlags: jhb: mfc-stable13+
Hardware: amd64   
OS: Any   
URL: https://reviews.freebsd.org/D28472
See Also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253214
https://issues.apache.org/jira/browse/SERF-198
Attachments:
Description Flags
Fix not to returns 1 for KLTS commands. none

Description Guido Falsi freebsd_committer freebsd_triage 2021-01-31 18:39:17 UTC
After updating head I noticed a regression in subversion client. both with svn and svnlite it hangs when trying to connect using https:// scheme. I observe the client using high CPU and opening and closing connection after connection (observed using netstat). Triggering this is as easy as:

svn co https://svn.freebsd.org/ports/head ports


I did a full bisect and tracked it down to KTLS, head commit aa906e2a4957

Could not observe anomalous behavior in any other case, only subversion seems affected.

I further confirmed this by compiling using WITHOUT_OPENSSL_KTLS and after this subversion client behaves normally.
Comment 1 O. Hartmann 2021-02-03 12:58:44 UTC
With 14-CURRENT and KTLS enabled (default) AND with all ports recompiled (after change from 13-CURRENT to 14-CURRENT), any Apache 2.4 service via TLS/SSL is broken, Firefox of clients report about a broken handshake or handshake not as expected or similar.

saltstack seems also affected, but I haven"t had the chance to check that in detail.

apach24 is definitely affected as SSL/TLS service, when host is 14-CURRENT with KTLS enabled.
Comment 2 Hyogeol Lee 2021-02-03 15:33:02 UTC
Created attachment 222113 [details]
Fix not to returns 1 for KLTS commands.

I found setsocketopt did not used in svnlite.

And bio_bucket_ctrl in ssl_buckets.c always returns 1 by default so initial record bypassed.

I attached fix for svnlite.
Comment 3 Guido Falsi freebsd_committer freebsd_triage 2021-02-03 15:55:24 UTC
jhb also provided a similar patch for base:

https://lists.freebsd.org/pipermail/dev-commits-src-all/2021-February/002195.html

I guess this also needs to be put in ports serf.
Comment 4 Guido Falsi freebsd_committer freebsd_triage 2021-02-03 16:05:41 UTC
(In reply to Guido Falsi from comment #3)

Filed bug #253214 to track patching the ports provided serf library.
Comment 5 John Baldwin freebsd_committer freebsd_triage 2021-02-03 19:24:22 UTC
I've filed an upstream bug report here: https://issues.apache.org/jira/browse/SERF-198

I have a review for the base system here: https://reviews.freebsd.org/D28472
Comment 6 John Baldwin freebsd_committer freebsd_triage 2021-02-03 19:26:48 UTC
Hyogeol, thanks for the patch, but I think it is better to fix the default return value.  There are other additional control operations in newer versions of OpenSSL (e.g. 3.0 adds an EOF query), so fixing the default return value is more future proof I believe.
Comment 7 commit-hook freebsd_committer freebsd_triage 2021-02-03 23:01:18 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=cb7cc72c546e0f87598961c3860e17391f42866c

commit cb7cc72c546e0f87598961c3860e17391f42866c
Author:     John Baldwin <jhb@FreeBSD.org>
AuthorDate: 2021-02-03 22:59:32 +0000
Commit:     John Baldwin <jhb@FreeBSD.org>
CommitDate: 2021-02-03 22:59:32 +0000

    serf: Fix the default return value of the BIO control method.

    OpenSSL BIO classes provide an abstraction for dealing with I/O.
    OpenSSL provides BIO classes for commonly used I/O primitives backed
    by file descriptors, sockets, etc. as well as permitting consumers
    of OpenSSL to define custom BIO classes.

    One of the methods BIO classes implement is a control method invoked
    by BIO_ctrl() for various ancilliary tasks somewhat analgous to
    fcntl() and ioctl() on file descriptors.  According to the BIO_ctrl(3)
    manual page, control methods should return 0 for unknown control
    requests.

    KTLS support in OpenSSL adds new control requests.  Two of those new
    requests are queries to determine if KTLS is enabled for either
    reading or writing.  These control reuquest return 1 if KTLS is
    enabled and 0 if it is not.

    serf includes two custom BIO classes for wrapping I/O requests from
    files and from a buffer in memory.  These BIO classes both use a
    custom control method.  However, this custom control method was
    returning 1 for unknown or unsupported control requests instead of 0.
    As a result, OpenSSL with KTLS believed that these BIOs were using
    KTLS and were thus adding headers and doing encryption/decryption in
    the BIO.  Correcting the return value removes this confusion.

    PR:             253135
    Reported by:    Guido Falsi <mad@madpilot.net>
    Reviewed by:    emaste
    MFC after:      3 days
    Sponsored by:   Netflix
    Differential Revision:  https://reviews.freebsd.org/D28472

 contrib/serf/buckets/ssl_buckets.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Comment 8 commit-hook freebsd_committer freebsd_triage 2021-02-08 22:39:33 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=b122886de25a51e3587d94ab2988a6ccb1e6a4e7

commit b122886de25a51e3587d94ab2988a6ccb1e6a4e7
Author:     John Baldwin <jhb@FreeBSD.org>
AuthorDate: 2021-02-03 22:59:32 +0000
Commit:     John Baldwin <jhb@FreeBSD.org>
CommitDate: 2021-02-08 22:38:59 +0000

    serf: Fix the default return value of the BIO control method.

    OpenSSL BIO classes provide an abstraction for dealing with I/O.
    OpenSSL provides BIO classes for commonly used I/O primitives backed
    by file descriptors, sockets, etc. as well as permitting consumers
    of OpenSSL to define custom BIO classes.

    One of the methods BIO classes implement is a control method invoked
    by BIO_ctrl() for various ancilliary tasks somewhat analgous to
    fcntl() and ioctl() on file descriptors.  According to the BIO_ctrl(3)
    manual page, control methods should return 0 for unknown control
    requests.

    KTLS support in OpenSSL adds new control requests.  Two of those new
    requests are queries to determine if KTLS is enabled for either
    reading or writing.  These control reuquest return 1 if KTLS is
    enabled and 0 if it is not.

    serf includes two custom BIO classes for wrapping I/O requests from
    files and from a buffer in memory.  These BIO classes both use a
    custom control method.  However, this custom control method was
    returning 1 for unknown or unsupported control requests instead of 0.
    As a result, OpenSSL with KTLS believed that these BIOs were using
    KTLS and were thus adding headers and doing encryption/decryption in
    the BIO.  Correcting the return value removes this confusion.

    PR:             253135
    Reported by:    Guido Falsi <mad@madpilot.net>
    Sponsored by:   Netflix

    (cherry picked from commit cb7cc72c546e0f87598961c3860e17391f42866c)

 contrib/serf/buckets/ssl_buckets.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Comment 9 commit-hook freebsd_committer freebsd_triage 2021-02-09 17:40:33 UTC
A commit in branch releng/13.0 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=f69cd089ccf4bf6afbb61fe84e1df2af2b037ec4

commit f69cd089ccf4bf6afbb61fe84e1df2af2b037ec4
Author:     John Baldwin <jhb@FreeBSD.org>
AuthorDate: 2021-02-03 22:59:32 +0000
Commit:     John Baldwin <jhb@FreeBSD.org>
CommitDate: 2021-02-09 17:35:21 +0000

    serf: Fix the default return value of the BIO control method.

    OpenSSL BIO classes provide an abstraction for dealing with I/O.
    OpenSSL provides BIO classes for commonly used I/O primitives backed
    by file descriptors, sockets, etc. as well as permitting consumers
    of OpenSSL to define custom BIO classes.

    One of the methods BIO classes implement is a control method invoked
    by BIO_ctrl() for various ancilliary tasks somewhat analgous to
    fcntl() and ioctl() on file descriptors.  According to the BIO_ctrl(3)
    manual page, control methods should return 0 for unknown control
    requests.

    KTLS support in OpenSSL adds new control requests.  Two of those new
    requests are queries to determine if KTLS is enabled for either
    reading or writing.  These control reuquest return 1 if KTLS is
    enabled and 0 if it is not.

    serf includes two custom BIO classes for wrapping I/O requests from
    files and from a buffer in memory.  These BIO classes both use a
    custom control method.  However, this custom control method was
    returning 1 for unknown or unsupported control requests instead of 0.
    As a result, OpenSSL with KTLS believed that these BIOs were using
    KTLS and were thus adding headers and doing encryption/decryption in
    the BIO.  Correcting the return value removes this confusion.

    PR:             253135
    Reported by:    Guido Falsi <mad@madpilot.net>
    Approved by:    re (gjb)
    Sponsored by:   Netflix

    (cherry picked from commit cb7cc72c546e0f87598961c3860e17391f42866c)
    (cherry picked from commit b122886de25a51e3587d94ab2988a6ccb1e6a4e7)

 contrib/serf/buckets/ssl_buckets.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)