Comment 1 Dirk Meyer 2018-09-27 10:54:47 UTC

Sorry, I can't reproduce your problem.

FreeBSD 11.2 amd64.
sendmail+tls+sasl2-8.15.2_11
tested with openssl-1.0.2p_1,1 (ports)
and with base:
OpenSSL 1.0.2o-freebsd  27 Mar 2018

1)
Please rebuild the port:
cd /usr/ports/mail/sendmail && make clean
cd /usr/ports/mail/sendmail && make install

Still no problem with the packages from:
"pkg+http://pkg.FreeBSD.org/${ABI}/latest"

$ ldd /usr/libexec/sendmail/sendmail
/usr/libexec/sendmail/sendmail:
        libutil.so.9 => /lib/libutil.so.9 (0x8008d1000)
        libssl.so.8 => /usr/lib/libssl.so.8 (0x800ae5000)
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x800e00000)
        libwrap.so.6 => /usr/lib/libwrap.so.6 (0x80126f000)
        libc.so.7 => /lib/libc.so.7 (0x801478000)

2)
Please check your configuration, permissions and the logfiles.
did you rebuild your cf files?

I did the following:
- cd /usr/ports/mail/sendmail; mv work work.notls; make

my compiled sendmail binary uses additional libraries compared to your
version (I listed the used options originally):

# ldd work/stage/usr/local/sbin/sendmail 
work/stage/usr/local/sbin/sendmail:
        libwrap.so.6 => /usr/lib/libwrap.so.6 (0x8008db000)
        libsasl2.so.3 => /usr/local/lib/libsasl2.so.3 (0x800ae4000)
        libldap-2.4.so.2 => /usr/local/lib/libldap-2.4.so.2 (0x800d00000)
        liblber-2.4.so.2 => /usr/local/lib/liblber-2.4.so.2 (0x800f47000)
        libblacklist.so.0 => /usr/lib/libblacklist.so.0 (0x801156000)
        libssl.so.8 => /usr/lib/libssl.so.8 (0x801359000)
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
        libutil.so.9 => /lib/libutil.so.9 (0x801a6f000)
        libc.so.7 => /lib/libc.so.7 (0x801c83000)
        libthr.so.3 => /lib/libthr.so.3 (0x80203f000)

> Please check your configuration, permissions and the logfiles.
> did you rebuild your cf files?

I did rebuild cf (we're using a patched version of cyrusv2). When I start
the re-compiled version like this:
root@zrhcz-mx1:/usr/ports/mail/sendmail # cat /var/run/sendmail.pid 
28458
/usr/sbin/sendmail -L sm-mta -bd -q30m
root@zrhcz-mx1:/usr/ports/mail/sendmail # chmod 2555 work/stage/usr/local/sbin/sendmail 
root@zrhcz-mx1:/usr/ports/mail/sendmail # chgrp smmsp work/stage/usr/local/sbin/sendmail 
root@zrhcz-mx1:/usr/ports/mail/sendmail # ls -l work/stage/usr/local/sbin/sendmail 
-r-xr-sr-x  1 root  smmsp  776720 Sep 27 12:58 work/stage/usr/local/sbin/sendmail
root@zrhcz-mx1:/usr/ports/mail/sendmail # kill 28458; work/stage/usr/local/sbin/sendmail -L sm-mta -bd -q30m

I'm getting the error back:
Sep 27 13:07:45 zrhcz-mx1 sm-mta[64758]: STARTTLS=server, error: RSA_generate_key failed

if I on the other hand remove files/patch-tls.c, remove the work directory and rebuild, the resulting sendmail
starts up fine. These are production mail servers, I can't perform extended tests with the non-functioning version
I'm afraid.

Comment 3 Dirk Meyer 2018-10-01 09:08:41 UTC

Still unable to reproduce your problem
neither in plain host or jails,
neither with ipv4 or ipv6.

Working in production with official, wildcard and let's encrpyt.

Example:
$ egrep '(CA|Server)Cert|Client|DH' /etc/mail/sendmail.cf
O ClientSSLOptions=+SSL_OP_NO_SSLv2 +SSL_OP_NO_SSLv3
#O ClientPortOptions=Family=inet, Address=0.0.0.0
#O ClientSSLOptions
O CACertPath=/etc/mail/certs/verify
O CACertFile=/usr/local/etc/letsencrypt/live/do.main/chain.pem
O ServerCertFile=/usr/local/etc/letsencrypt/live/do.main/cert.pem
# Client Cert
O ClientCertFile=/usr/local/etc/letsencrypt/live/do.main/cert.pem
# Client private key
O ClientKeyFile=/usr/local/etc/letsencrypt/live/do.main/privkey.pem
# DHParameters (only required if DSA/DH is used)
O DHParameters=/etc/mail/certs/dhparam

Minimal example:
O CACertPath=/etc/mail/certs/empty
O CACertFile=/etc/mail/certs/cacert.pem
O ServerCertFile=/etc/mail/certs/host.cert
O ServerKeyFile=/etc/mail/certs/host.key
O ClientCertFile=/etc/mail/certs/host.cert
O ClientKeyFile=/etc/mail/certs/host.key

The default package has:

$ ldd /usr/local/sbin/sendmail 
/usr/local/sbin/sendmail:
        libwrap.so.6 => /usr/lib/libwrap.so.6 (0x8008d3000)
        libsasl2.so.3 => /usr/local/lib/libsasl2.so.3 (0x800adc000)
        libblacklist.so.0 => /usr/lib/libblacklist.so.0 (0x800cf8000)
        libssl.so.8 => /usr/lib/libssl.so.8 (0x800efb000)
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x801200000)
        libutil.so.9 => /lib/libutil.so.9 (0x80166f000)
        libc.so.7 => /lib/libc.so.7 (0x801883000)
        libthr.so.3 => /lib/libthr.so.3 (0x801c3f000)

Make sure you don't have different versions of openssl/libressl installed,
e.g. inherited by libldap.

$ ldd  /usr/local/lib/libldap-2.4.so.2.10.9 
/usr/local/lib/libldap-2.4.so.2.10.9:
        liblber-2.4.so.2 => /usr/local/lib/liblber-2.4.so.2 (0x801247000)
        libssl.so.8 => /usr/lib/libssl.so.8 (0x801456000)
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x801800000)
        libc.so.7 => /lib/libc.so.7 (0x800823000)

I don't seen an obvious local installation issue here. We use packages where possible, just do ports compilation as in this case where we need non-standard port options. Regarding your examples:

$ egrep '(CA|Server)Cert|Client|DH' /etc/mail/sendmail.cf
#O ClientPortOptions=Family=inet, Address=0.0.0.0
O ClientSSLOptions=+SSL_OP_NO_SSLv2 
O CACertPath=/etc/mail/certs
O CACertFile=/etc/mail/certs/mail-certs.crt
O ServerCertFile=/etc/mail/certs/_.virtualtec.ch.crt
# Client Cert
O ClientCertFile=/etc/mail/certs/_.virtualtec.ch.crt
# Client private key
O ClientKeyFile=/etc/mail/certs/_.virtualtec.ch.key
# DHParameters (only required if DSA/DH is used)
O DHParameters=/etc/mail/certs/dhparam

this is using a wildcard cert.

To rule out conflicting libraries:
root@zrhcz-mx1:/usr/ports/mail/sendmail # ldd -a work/stage/usr/local/sbin/sendmail | egrep 'ssl|crypto'
        libssl.so.8 => /usr/lib/libssl.so.8 (0x801359000)
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
        libssl.so.8 => /usr/lib/libssl.so.8 (0x801359000)
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
/usr/lib/libssl.so.8:
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
/lib/libcrypto.so.8:


root@zrhcz-mx1:/usr/ports/mail/sendmail # ldd -a work.notls//stage/usr/local/sbin/sendmail | egrep 'ssl|crypto'
        libssl.so.8 => /usr/lib/libssl.so.8 (0x801359000)
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
        libssl.so.8 => /usr/lib/libssl.so.8 (0x801359000)
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
/usr/lib/libssl.so.8:
        libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
/lib/libcrypto.so.8:


You can mark the problem as "affects only me", and I know how to fix it (by removing the patch-tls.c file). Perhaps
if someone else has the same issue, they'll now at least find this bug report with a possible workaround. Thanks for
looking into it! If it were easier to debug sendmail with gdb, I would have had a go at it, but it turns out that's not 
so trivial...

I think this could be reproduced.
This error is raised only if Sendmail is configured with shared memory, e.g. with
something like
define(`confSHARED_MEMORY_KEY',`2012')dnl
in .mc. Otherwise this message won't appear. This seems the default case.
The shared memory configuration in rare use.

The fix is to correct the error code handling (in the shared memory branch):

--- sendmail-8.15.2/sendmail/tls.c.orig.2       2019-12-18 00:30:12.000000000 +0100
+++ sendmail-8.15.2/sendmail/tls.c      2021-12-28 04:34:05.314847659 +0100
@@ -752,7 +752,7 @@
        RSA *rsa = NULL;

        bne = BN_new();
-       if (bne && BN_set_word(bne, e) != 1)
+       if (bne && BN_set_word(bne, e) == 1)
                rsa = RSA_new();
        if (rsa && RSA_generate_key_ex(rsa, bits, bne, NULL) != 1)
        {

BN_set_word() returns 1 if it succeeds and in this case a new RSA key should be created. Otherwise rsa remains NULL which leads to the mentioned error message, even BN_set_word() signals no error.