Bug 263287 - mail/sendmail: 8.17.1_2 -> 8.17.1_3 upgrade breaks SMTP AUTH
Summary: mail/sendmail: 8.17.1_2 -> 8.17.1_3 upgrade breaks SMTP AUTH
Status: Closed FIXED
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Only Me
Assignee: Dirk Meyer
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-04-14 23:30 UTC by Greg Rivers
Modified: 2022-07-15 14:43 UTC (History)
2 users (show)

See Also:
dinoex: maintainer-feedback+
dinoex: merge-quarterly+


Attachments
Working sendmail binary from October 30 2021 8.17.1_2 package (778.01 KB, application/x-executable)
2022-04-17 21:51 UTC, Greg Rivers
no flags Details
Suggested patch (1.25 KB, patch)
2022-04-19 21:13 UTC, Dirk Meyer
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Greg Rivers 2022-04-14 23:30:00 UTC
Subject upgrade via the "latest" pkg repo breaks SMTP AUTH for me on two different systems. Testing indicates that sendmail no longer offers AUTH after the upgrade:

Before upgrade:
$ openssl s_client -brief -connect example.org:465
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:ECDSA+SHA1:RSA+SHA224:RSA+SHA1
Peer certificate: CN = example.org
Hash used: SHA384
Signature type: ECDSA
Verification: OK
Server Temp Key: ECDH, P-256, 256 bits
220 example.org ESMTP Sendmail 8.17.1/8.17.1; Thu, 14 Apr 2022 17:00:48 -0500 (CDT)
ehlo localhost
250-example.org Hello [IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-AUTH LOGIN PLAIN
250-DELIVERBY
250 HELP
quit
221 2.0.0 example.org closing connection

After upgrade:
$ openssl s_client -brief -connect example.org:465
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:ECDSA+SHA1:RSA+SHA224:RSA+SHA1
Peer certificate: CN = example.org
Hash used: SHA384
Signature type: ECDSA
Verification: OK
Server Temp Key: ECDH, P-256, 256 bits
220 example.org ESMTP Sendmail 8.17.1/8.17.1; Thu, 14 Apr 2022 17:58:46 -0500 (CDT)
ehlo localhost
250-example.org Hello [IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-DELIVERBY
250 HELP
quit
221 2.0.0 example.org closing connection

Note the conspicuous absence of '250-AUTH LOGIN PLAIN' after the upgrade. No configurations were touched. Restoring the /usr/local/sbin/sendmail executable from a recent snapshot resolves the problem. Bug #262654 appears to be the the commit that introduced the problem.
Comment 1 Fabian Wenk 2022-04-17 00:39:45 UTC
I did run into the same problem, but with building it out of /usr/ports/.
I could re-create the sendmail-8.17.1_2.pkg out of a backup and I also did create the sendmail-8.17.1_3.pkg before reverting to the previous version.

I did then extract both .pkg into corresponding folders to be able to diff them. The following files have changes:
+COMPACT_MANIFEST (only version number and flatsize)
+MANIFEST (version number, flatsize, checksum of /usr/local/sbin/sendmail,  /usr/local/share/doc/sendmail/op.txt and /usr/local/share/licenses/sendmail-8.17.1_{2,3} directory)
/usr/local/sbin/sendmail (see hexdump diff below)
/usr/local/share/doc/sendmail/op.txt (some text formatting)
/usr/local/share/licenses/sendmail-8.17.1_{2,3} (directory names)

Hexdump diff of sendmail binary:
--- sendmail_8.17.1_2.hex	2022-04-17 01:33:01.780939000 +0200
+++ sendmail_8.17.1_3.hex	2022-04-17 01:33:08.236391000 +0200
@@ -11912,7 +11912,7 @@
 0031dc0 0e13 0000 1c75 c4bf 20fc 3100 e8c0 b1ee
 0031dd0 0007 80bf 2c2d e800 4a64 0000 058a 33fe
 0031de0 000e 0e3c 2d72 3d83 133b 000e 7500 bf24
-0031df0 e33d 0020 dfbe 1010 3110 e8c0 b1c0 0007
+0031df0 e33d 0020 efbe 1010 3110 e8c0 b1c0 0007
 0031e00 4be8 088b bf00 a160 0021 c689 c031 ade8
 0031e10 07b1 4800 1d8b e986 0008 8948 4f1d 0df9
 0031e20 4800 85c7 f5e8 ffff 0000 0000 8d48 e885
@@ -36772,7 +36772,7 @@
 0092f80 ffc2 00ff 0000 6600 ef0f 66c0 7f0f b085
 0092f90 ffc2 66ff 7f0f c085 ffc2 8bff c305 0802
 0092fa0 8900 b485 ffc2 c7ff b885 ffc2 00ff 0020
-0092fb0 0f00 05b6 1450 0008 8589 c2bc ffff 8b48
+0092fb0 0f00 05b7 1450 0008 8589 c2bc ffff 8b48
 0092fc0 e8bd ffc2 48ff 958d c2b0 ffff 65be 0000
 0092fd0 e800 844a 0002 c931 c085 940f 89c1 a08d
 0092fe0 ffc1 83ff a0bd ffc1 00ff 4b74 85c7 c2d4

The changes are very minimal, but with a huge impact regarding SMTP Auth.
Comment 2 Dirk Meyer freebsd_committer freebsd_triage 2022-04-17 09:52:07 UTC
Sorry, I can't reproduce the problem yet.

Verified to work with the home build packages on FreeBSD-12.3
on a few servers I can reach, SMTP-AUTH over port 465 still works here.

testing FreeBSD-12.3 packages, there is sendmail-8.17.1_3 yet.

testing FreeBSD-13 packages:

# fetch http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest/All/sendmail-8.17.1_3.pkg
# fetch http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly/All/sendmail-8.17.1_2.pkg
# mkdir sendmail-8.17.1_2
# mkdir sendmail-8.17.1_3
# tar -C sendmail-8.17.1_2 -xvf sendmail-8.17.1_2.pkg
# tar -C sendmail-8.17.1_3 -xvf sendmail-8.17.1_3.pkg
# diff -s sendmail-8.17.1_2/usr/local/sbin/sendmail sendmail-8.17.1_3/usr/local/sbin/sendmail
Files sendmail-8.17.1_2/usr/local/sbin/sendmail and sendmail-8.17.1_3/usr/local/sbin/sendmail are identical

# ls -l sendmail-8.17.1_2/usr/local/sbin/sendmail sendmail-8.17.1_3/usr/local/sbin/sendmail
-r-xr-sr-x  1 root  smmsp  796680  8 Apr. 23:41 sendmail-8.17.1_2/usr/local/sbin/sendmail
-r-xr-sr-x  1 root  smmsp  796680  9 Apr. 00:14 sendmail-8.17.1_3/usr/local/sbin/sendmail

# diff -qr sendmail-8.17.1_2/ sendmail-8.17.1_3/ | less
diff: sendmail-8.17.1_2/usr/local/bin/hoststat: No such file or directory
diff: sendmail-8.17.1_2/usr/local/bin/mailq: No such file or directory
diff: sendmail-8.17.1_2/usr/local/bin/newaliases: No such file or directory
diff: sendmail-8.17.1_2/usr/local/bin/purgestat: No such file or directory
Files sendmail-8.17.1_2/+COMPACT_MANIFEST and sendmail-8.17.1_3/+COMPACT_MANIFEST differ
Files sendmail-8.17.1_2/+MANIFEST and sendmail-8.17.1_3/+MANIFEST differ
Only in sendmail-8.17.1_2/usr/local/share/licenses: sendmail-8.17.1_2
Only in sendmail-8.17.1_3/usr/local/share/licenses: sendmail-8.17.1_3

Please check your options.

# pkg info sendmail
sendmail-8.17.1_3
[...]
Options        :
        BDB            : off
        BLACKLISTD     : on
        CYRUSLOOKUP    : off
        DANE           : on
        DOCS           : on
        GDBM           : off
        IPV6           : on
        LA             : on
        LDAP           : off
        MILTER         : on
        NIS            : on
        PICKY_HELO_CHECK: on
        SASL           : on
        SASLAUTHD      : on
        SEM            : on
        SHMEM          : on
        SMTPUTF8       : off
        SOCKETMAP      : off
        TLS            : on
[...]

Server here still offers:
250-AUTH PLAIN LOGIN CRAM-MD5

# sendmail -d0
Version 8.17.1
 Compiled with: DANE DNSMAP IPV6_FULL LOG MAP_REGEX MATCHGECOS MILTER
                MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6 NETUNIX NEWDB NIS
                PICKY_HELO_CHECK PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS
                TLS_EC TLS_VRFY_PER_CTX USERDB XDEBUG

Please check you logfile "/var/log/maillog" for problems when you restart sendmail.

You may try to check the differences in the binary file you see:
# objdump -x -D /usr/local/sbin/sendmail > objdump.1.txt

Beside the PORTREVISION the packages should be identical.
The dependency on icu is not used by default, only with option SMTPUTF8.
Comment 3 Greg Rivers 2022-04-17 21:51:40 UTC
Created attachment 233285 [details]
Working sendmail binary from October 30 2021 8.17.1_2 package
Comment 4 Greg Rivers 2022-04-17 22:18:51 UTC
(In reply to Dirk Meyer from comment #2)
Because I'm using the binary package from the FreeBSD pkg repo, my options are naturally the same as yours:

$ pkg info sendmail | sed -n -e /^Options/,/^Shared/p
Options        :
        BDB            : off
        BLACKLISTD     : on
        CYRUSLOOKUP    : off
        DANE           : on
        DOCS           : on
        GDBM           : off
        IPV6           : on
        LA             : on
        LDAP           : off
        MILTER         : on
        NIS            : on
        PICKY_HELO_CHECK: on
        SASL           : on
        SASLAUTHD      : on
        SEM            : on
        SHMEM          : on
        SMTPUTF8       : off
        SOCKETMAP      : off
        TLS            : on
Shared Libs required:

$ sendmail -d0 </dev/null
Version 8.17.1
 Compiled with: DANE DNSMAP IPV6_FULL LOG MAP_REGEX MATCHGECOS MILTER
                MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6 NETUNIX NEWDB NIS
                PICKY_HELO_CHECK PIPELINING SASLv2 SCANF STARTTLS TCPWRAPPERS
                TLS_EC TLS_VRFY_PER_CTX USERDB XDEBUG

I too can demonstrate that the _current_ quarterly/8.17.1_2 and latest/8.17.1_3 sendmail packages contain identical sendmail binaries:

$ mkdir -p sendmail/8.17.1_2 sendmail/8.17.1_3                                                                           
$ fetch -q -o - http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly/All/sendmail-8.17.1_2.pkg | tar -C sendmail/8.17.1_2 -xf -
tar: Removing leading '/' from member names
$ fetch -q -o - http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest/All/sendmail-8.17.1_3.pkg | tar -C sendmail/8.17.1_3 -xf -   
tar: Removing leading '/' from member names
$ sha256 -r sendmail/*/usr/local/sbin/sendmail                                                                              
f44c526fe616d0c2df349b5d40148d5fa9f201d6d945d62a259fcd9817c8bb21 sendmail/8.17.1_2/usr/local/sbin/sendmail
f44c526fe616d0c2df349b5d40148d5fa9f201d6d945d62a259fcd9817c8bb21 sendmail/8.17.1_3/usr/local/sbin/sendmail
$ ls -l sendmail/*/usr/local/sbin/sendmail                                                                                  
-r-xr-xr-x  1 gcr  wheel  796680 Apr  8 16:41 sendmail/8.17.1_2/usr/local/sbin/sendmail*
-r-xr-xr-x  1 gcr  wheel  796680 Apr  8 17:14 sendmail/8.17.1_3/usr/local/sbin/sendmail*

What that means is that (for me anyway) both of the currently published quarterly/8.17.1_2 and latest/8.17.1_3 packages are broken the same way. The original October 30 2021 8.17.1_2 package that was installed by this upgrade:

Nov  1 01:30:31 xxxxxxx pkg[34225]: sendmail upgraded: 8.17.1_1 -> 8.17.1_2

contains a different (working) binary with these properties:

$ sha256 -r /usr/.zfs/snapshot/zfs-auto-snap_daily-2022-04-13-00h07/local/sbin/sendmail
55d2df5b6b80a609d6d698d7c3bed2e2d8287ca929a24bda39205c7879916f59 /usr/.zfs/snapshot/zfs-auto-snap_daily-2022-04-13-00h07/local/sbin/sendmail
$ ls -l /usr/.zfs/snapshot/zfs-auto-snap_daily-2022-04-13-00h07/local/sbin/sendmail       
-r-xr-sr-x  1 root  smmsp  uarch 796680 Oct 30 17:58 /usr/.zfs/snapshot/zfs-auto-snap_daily-2022-04-13-00h07/local/sbin/sendmail*

Here are the server side logs with "define(`confLOG_LEVEL', `15')" set:

Working (with sendmail binary from  October 30 2021 8.17.1_2 package):
starting daemon (8.17.1): SMTP+queueing@00:30:00
STARTTLS: CRLFile missing
starting daemon (8.17.1): queueing@00:30:00
STARTTLS=server, Diffie-Hellman init, key=1024 bit (1)
STARTTLS=server, init=1
started as: /usr/sbin/sendmail -L sm-mta -bd -q30m
NOQUEUE: connect from [IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx]
AUTH: available mech=SCRAM-SHA-512 SCRAM-SHA-384 SCRAM-SHA-256 SCRAM-SHA-224 SCRAM-SHA-1 DIGEST-MD5 OTP NTLM CRAM-MD5, allowed mech=LOGIN PLAIN
23HInG88088306: Milter (greylist): init success to negotiate
23HInG88088306: Milter (dkim-filter): init success to negotiate
23HInG88088306: Milter: connect to filters
23HInG88088306: milter=greylist, action=connect, continue
23HInG88088306: milter=dkim-filter, action=connect, continue
tls_srv_features=(null), relay=[IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx] [IPv6:2001:470:1f11:107f:f918:9413:55d4:337f]
tls_srv_features=empty, stat=0, relay=[IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx] [IPv6:2001:470:1f11:107f:f918:9413:55d4:337f]
STARTTLS=server, info: fds=11/4, err=2
STARTTLS=server, get_verify: 0 get_peer: 0x0
STARTTLS=server, relay=[IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx], version=TLSv1.3, verify=NO, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
STARTTLS=server, cert-subject=, cert-issuer=, verifymsg=ok
AUTH: available mech=SCRAM-SHA-512 SCRAM-SHA-384 SCRAM-SHA-256 SCRAM-SHA-224 SCRAM-SHA-1 DIGEST-MD5 OTP NTLM CRAM-MD5 PLAIN LOGIN, allowed mech=LOGIN PLAIN
23HInG88088306: --- 220 example.org ESMTP Sendmail 8.17.1/8.17.1; Unauthorized use prohibited; Sun, 17 Apr 2022 13:49:16 -0500 (CDT)
STARTTLS=read, info: fds=11/4, err=2
23HInG88088306: <-- ehlo localhost
23HInG88088306: milter=greylist, action=helo, continue
23HInG88088306: --- 250-example.org Hello [IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx], pleased to meet you
23HInG88088306: --- 250-ENHANCEDSTATUSCODES
23HInG88088306: --- 250-PIPELINING
23HInG88088306: --- 250-8BITMIME
23HInG88088306: --- 250-SIZE
23HInG88088306: --- 250-DSN
23HInG88088306: --- 250-AUTH LOGIN PLAIN
23HInG88088306: --- 250-DELIVERBY
23HInG88088306: --- 250 HELP
STARTTLS=read, info: fds=11/4, err=2
23HInG88088306: <-- quit
23HInG88088306: --- 221 2.0.0 example.org closing connection
23HInG88088306: [IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx] did not issue MAIL/EXPN/VRFY/ETRN during connection to MSAv6
NOQUEUE: stopping daemon, reason=signal

Broken (with sendmail binary from April 8 2022 8.17.1_3 package):
starting daemon (8.17.1): SMTP+queueing@00:30:00
STARTTLS: CRLFile missing
starting daemon (8.17.1): queueing@00:30:00
STARTTLS=server, Diffie-Hellman init, key=1024 bit (1)
STARTTLS=server, init=1
started as: /usr/sbin/sendmail -L sm-mta -bd -q30m
NOQUEUE: connect from [IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx]
AUTH warning: no mechanisms
23HInmqN088377: Milter (greylist): init success to negotiate
23HInmqN088377: Milter (dkim-filter): init success to negotiate
23HInmqN088377: Milter: connect to filters
23HInmqN088377: milter=greylist, action=connect, continue
23HInmqN088377: milter=dkim-filter, action=connect, continue
tls_srv_features=(null), relay=[IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx] [IPv6:2001:470:1f11:107f:f918:9413:55d4:337f]
tls_srv_features=empty, stat=0, relay=[IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx] [IPv6:2001:470:1f11:107f:f918:9413:55d4:337f]
STARTTLS=server, info: fds=11/4, err=2
STARTTLS=server, get_verify: 0 get_peer: 0x0
STARTTLS=server, relay=[IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx], version=TLSv1.3, verify=NO, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
STARTTLS=server, cert-subject=, cert-issuer=, verifymsg=ok
AUTH warning: no mechanisms
23HInmqN088377: --- 220 example.org ESMTP Sendmail 8.17.1/8.17.1; Unauthorized use prohibited; Sun, 17 Apr 2022 13:49:48 -0500 (CDT)
STARTTLS=read, info: fds=11/4, err=2
23HInmqN088377: <-- ehlo localhost
23HInmqN088377: milter=greylist, action=helo, continue
23HInmqN088377: --- 250-example.org Hello [IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx], pleased to meet you
23HInmqN088377: --- 250-ENHANCEDSTATUSCODES
23HInmqN088377: --- 250-PIPELINING
23HInmqN088377: --- 250-8BITMIME
23HInmqN088377: --- 250-SIZE
23HInmqN088377: --- 250-DSN
23HInmqN088377: --- 250-DELIVERBY
23HInmqN088377: --- 250 HELP
STARTTLS=read, info: fds=11/4, err=2
23HInmqN088377: <-- quit
23HInmqN088377: --- 221 2.0.0 example.org closing connection
23HInmqN088377: [IPv6:xxxx:xxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx] did not issue MAIL/EXPN/VRFY/ETRN during connection to MSAv6

Note that in the working case we have:
AUTH: available mech=SCRAM-SHA-512 SCRAM-SHA-384 SCRAM-SHA-256 SCRAM-SHA-224 SCRAM-SHA-1 DIGEST-MD5 OTP NTLM CRAM-MD5, allowed mech=LOGIN PLAIN

whereas in the broken case we have:
AUTH warning: no mechanisms


It would appear that you are correct, the problem has nothing to do with the icu dependency. Both sendmail packages (quarterly/8.17.1_2 and latest/8.17.1_3) now are broken the same way by something else. Unfortunately, I have no clue what that might be.
Comment 5 Fabian Wenk 2022-04-18 00:06:04 UTC
Some info I did to provide initially and also some more tests I have done in the meantime.

Running on FreeBSD 12.3 with the following 
Options        :
	BDB            : off
	BLACKLISTD     : off
	CYRUSLOOKUP    : off
	DANE           : on
	DOCS           : on
	GDBM           : off
	IPV6           : on
	LA             : on
	LDAP           : off
	MILTER         : on
	NIS            : off
	PICKY_HELO_CHECK: off
	SASL           : on
	SASLAUTHD      : off
	SEM            : on
	SHMEM          : on
	SMTPUTF8       : off
	SOCKETMAP      : off
	TLS            : on

I am using cyrus-sasl-2.1.28 + cyrus-sasl-sql-2.1.28 (both build on 2nd March 2022) with mariadb103-client-10.3.34_3 (was also just upgraded from mariadb103-client-10.3.34_1). I also have a file /usr/local/lib/sasl2/Sendmail.conf with this:
pwcheck_method: auxprop
auxprop_plugin: sql
mech_list: login plain
sql_engine: mysql
sql_hostnames: localhost
sql_user: <username>
sql_passwd: <password>
sql_database: <database>
sql_verbose: no
sql_select: <SELECT statement to check password>
sasl_sql_usessl: 0
log_level: 1

And in the hostname.exmaple.com.mc file this:
define(`confAUTH_MECHANISMS', `PLAIN LOGIN')
TRUST_AUTH_MECH(`LOGIN PLAIN')

Diff of the two objdumps:
--- sendmail-8.17.1_2.objdump	2022-04-18 00:31:11.362943000 +0200
+++ sendmail-8.17.1_3.objdump	2022-04-18 00:30:17.000000000 +0200
@@ -79616,7 +79616,7 @@
   232de6:	83 3d 3b 13 0e 00 00 	cmpl   $0x0,0xe133b(%rip)        # 314128 <environ+0x52988>
   232ded:	75 24                	jne    232e13 <sleep-0x87e6d>
   232def:	bf 3d e3 20 00       	mov    $0x20e33d,%edi
-  232df4:	be df 10 10 10       	mov    $0x101010df,%esi
+  232df4:	be ef 10 10 10       	mov    $0x101010ef,%esi
   232df9:	31 c0                	xor    %eax,%eax
   232dfb:	e8 c0 b1 07 00       	callq  2adfc0 <sleep-0xccc0>
   232e00:	e8 4b 8b 08 00       	callq  2bb950 <OpenSSL_version_num@plt>
@@ -176036,7 +176036,7 @@
   293fa1:	89 85 b4 c2 ff ff    	mov    %eax,-0x3d4c(%rbp)
   293fa7:	c7 85 b8 c2 ff ff 00 	movl   $0x2000,-0x3d48(%rbp)
   293fae:	20 00 00 
-  293fb1:	0f b6 05 50 14 08 00 	movzbl 0x81450(%rip),%eax        # 315408 <environ+0x53c68>
+  293fb1:	0f b7 05 50 14 08 00 	movzwl 0x81450(%rip),%eax        # 315408 <environ+0x53c68>
   293fb8:	89 85 bc c2 ff ff    	mov    %eax,-0x3d44(%rbp)
   293fbe:	48 8b bd e8 c2 ff ff 	mov    -0x3d18(%rbp),%rdi
   293fc5:	48 8d 95 b0 c2 ff ff 	lea    -0x3d50(%rbp),%rdx

The build of sendmail-8.17.1_2 was done on 30th January 2022 out of the then synced Ports.

There is one ugly thing I do not fully understand yet.
I started to test sendmail on localhost on the system where I build the _3 pkg with 'openssl s_client -brief -starttls smtp -connect localhost:25' initial where I did not had the above mention Sendmail.conf in place or mention config in .mc. I got the 250-AUTH line but with CRAM-MD5 and something else, but not PLAIN and LOGIN. I then put the Sendmail.conf into place and did restart sendmail, and then the 250-AUTH line was gone from ehlo output.
I then checked / created backup (pkg) from the installed cyrus-sasl-2.1.28 + cyrus-sasl-sql-2.1.28 and sendmail-8.17.1_3. I then did a portupgrade -f for all of them. I restarted sendmail and the test with s_client and ehlo I had '250-AUTH PLAIN LOGIN'. To fully test, I migrated the complete SMTP Auth setup from the live server and also did revert to the previous versions (builds) of cyrus-sasl-2.1.28 + cyrus-sasl-sql-2.1.28 and sendmail-8.17.1_3 and it worked (sendmail still with the same above objdump diff to _2).
But when I upgraded to sendmail-8.17.1_3 on the live system from the same pkg from the build & test system, and it did not work and the 250-AUTH line was missing. Even building sendmail on the live server, the /usr/local/sbin/sendmail is identical to the one from the .pkg I created (with the small difference to _2).

The following files did differ after the rebuild of cyrus-sasl-2.1.28:
/usr/local/lib/libsasl2.a
/usr/local/lib/sasl2/libdigestmd5.a
/usr/local/lib/sasl2/libdigestmd5.so
/usr/local/lib/sasl2/libdigestmd5.so.3
/usr/local/lib/sasl2/libdigestmd5.so.3.0.0

And this from cyrus-sasl-sql-2.1.28:
/usr/local/lib/sasl2/libsql.a
/usr/local/lib/sasl2/libsql.so
/usr/local/lib/sasl2/libsql.so.3
/usr/local/lib/sasl2/libsql.so.3.0.0

According to ldd I did on libdigestmd5.so.3.0.0 and libsql.so.3.0.0 it looks like differences may be related to the openssl-1.1.1n,1 update I have done on 16th March 2022.

So I guess I may have to do some more checks on the live server. But hopefully the objdump diff above gives somebody a hint what could be causing this. Maybe it is caused from some leftover old version of a library in /usr/local/lib/compat/pkg/ maybe used from some build or run dependency? Or the forced portupgrade of the mention packages and/or re-install of the previous .pkg did some clean up on my build & test system, which was not done on the live server (yet).
In the past I already had build or run problems solved when I did clean out some specific files from /usr/local/lib/compat/pkg/. We probably just have to find the right ones to remove.
Comment 6 Greg Rivers 2022-04-18 01:48:02 UTC
(In reply to Fabian Wenk from comment #5)
I have the same sendmail AUTH configuration as you:

$ fgrep AUTH_MECH /etc/mail/example.org.mc
TRUST_AUTH_MECH(`LOGIN PLAIN')
define(`confAUTH_MECHANISMS', `LOGIN PLAIN')

I am running saslauthd 2.1.28 from the latest FreeBSD pkg repo since March 2022:
Mar  6 11:13:20 xxxxxx pkg[11907]: cyrus-sasl-saslauthd upgraded: 2.1.27_2 -> 2.1.28

As you said, it could very well be that there's nothing wrong with sendmail itself; the problem could lie elsewhere.

I'm running with the base system OpenSSL:

$ freebsd-version -uk
13.0-RELEASE-p11
13.0-RELEASE-p11
$ openssl version
OpenSSL 1.1.1k-freebsd  24 Aug 2021
Comment 7 Greg Rivers 2022-04-18 05:52:05 UTC
I spun up a pristine FreeBSD 13.0-RELEASE VM inside VirtualBox, brought it up to the current patch level with freebsd-update, and installed sendmail and cyrus-sasl from the "latest" FreeBSD pkg repo. I made all the necessary sendmail configurations and tested: sendmail AUTH works fine!

I went back to one of the hosts where the new sendmail binary does not present SMTP AUTH, and forced reinstallation of all packages (pkg upgrade -y -f). I restarted sendmail and saslauthd and tested again: sendmail AUTH is still broken.

So I have to conclude that the problem is somewhere in the base system. Both hosts where sendmail breaks have been carefully (and independently) maintained via pkg and freebsd-update (only) for years, so there must be some change that freebsd-update did not make properly. `freebsd-update IDS` does not show anything that I don't expect. Given that an older sendmail binary from last fall still works, it must be a library problem as Fabian suggested.

ldd output for both the old (working) and new (broken) binaries is identical:
$ ldd /usr/local/sbin/sendmail 
/usr/local/sbin/sendmail:
        libwrap.so.6 => /usr/lib/libwrap.so.6 (0x800304000)
        libsasl2.so.3 => /usr/local/lib/libsasl2.so.3 (0x800310000)
        libblacklist.so.0 => /usr/lib/libblacklist.so.0 (0x800331000)
        libssl.so.111 => /usr/lib/libssl.so.111 (0x800337000)
        libcrypto.so.111 => /lib/libcrypto.so.111 (0x8003cf000)
        libutil.so.9 => /lib/libutil.so.9 (0x8006c4000)
        libc.so.7 => /lib/libc.so.7 (0x8006dc000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x800aed000)
        libthr.so.3 => /lib/libthr.so.3 (0x800af1000)

The checksums of the above dependencies on both the pristine and broken systems are identical as well:
92bffeca892435de3bad30f9159478abd20958ce03ce7381c7e31b94eb83e641 /usr/lib/libwrap.so.6
c2e8401ed4332935ab5c536348a2b7360fbead5cefd3c8cb499ac12196c400be /usr/local/lib/libsasl2.so.3
885c2a2f48c6acc426f8f4f39a8b0abbc5e977d3f92773ae801d6cd6f448a3dd /usr/lib/libblacklist.so.0
2845f669d2e87828d196b75da41c8552d8500c69e5b0fd2338d7854614df539d /usr/lib/libssl.so.111
6bf3862b133ff89777ef7cf885f302577bb438e1208db03e66d7422c7af81521 /lib/libcrypto.so.111
4acf83f6feb7b0a415e84e82a827f5efd23e6d55e0e40448ae9ecf3774383d5d /lib/libutil.so.9
b448c6d5ea656755d07701d0e51fb0898a3d3db6d3c89205c1eabed3fb1ced71 /lib/libc.so.7
4b59dac32b6c3c2f7540faa7e6f56db08cb7807df905c7cf0dbee359b8d30973 /usr/lib/libdl.so.1
839f17275a3084a4ad21d4cae7b5a2e67deb1fb3b248c920bfb40055c72ca0e1 /lib/libthr.so.3

Do you have any suggestions about which other files to look at? It would be terribly inconvenient to have to rebuild both of the affected hosts from scratch. If you would like to play with the VM I created, contact me privately and I'll send you a download link.
Comment 8 Dirk Meyer freebsd_committer freebsd_triage 2022-04-18 07:12:37 UTC
This difference here is the OpenSSL version number at build time:

-  232df4:	be df 10 10 10       	mov    $0x101010df,%esi
+  232df4:	be ef 10 10 10       	mov    $0x101010ef,%esi

FreeBSD 12.3_
$ openssl version
OpenSSL 1.1.1l-freebsd  24 Aug 2021
$ grep OPENSSL_VERSION_NUMBER /usr/include/openssl/opensslv.h 
# define OPENSSL_VERSION_NUMBER  0x101010cfL

FreeBSD 13.1
$ openssl version
OpenSSL 1.1.1n-freebsd  15 Mar 2022
$ grep OPENSSL_VERSION_NUMBER /usr/include/openssl/opensslv.h 
# define OPENSSL_VERSION_NUMBER  0x101010efL

You can verify this with:
FreeBSD 13.1, sendmail-8.17.1_2 from quaterly
$ sendmail -d0.14 < /dev/null | grep OpenSSL
       OpenSSL: compiled 0x101010bf
       OpenSSL: linked   0x101010ef

plain freebsd.mc plus this lines:
DAEMON_OPTIONS(`Name=sm-smtps, Family=inet, Port=smtps, M=Esh')
DAEMON_OPTIONS(`Name=sm-smtps-v6, Family=inet6, Port=smtps, M=Esh')
define(`confAUTH_MECHANISMS', `PLAIN LOGIN CRAM-MD5')dnl
define(`confAUTH_OPTIONS', `y')
TRUST_AUTH_MECH(`PLAIN LOGIN CRAM-MD5')dnl
define(`confDONT_BLAME_SENDMAIL',`GroupReadableSASLDBFile')dnl

The package is offering AUTH.

Permissions may have effect too:

# ls -l /usr/local/lib/sasl2/Sendmail.conf
-rw-r--r--  1 root  wheel  26 Apr 18 08:22 /usr/local/lib/sasl2/Sendmail.conf

# ls -l /usr/local/etc/sasldb2.db 
-rw-r-----  1 cyrus  mail  16384 Mar 20 22:32 /usr/local/etc/sasldb2.db

If your permissions differ, please check if this line fixes the problem:
define(`confDONT_BLAME_SENDMAIL',`GroupReadableSASLDBFile')dnl

Thanks for the debug output.
sendmail calls: sasl_listmech() and gets an empty list on return.

Possible the ".so" files linked in to sasl2 are build with different OpenSSL versions than sendmail.
This can create a conflict (size of *ssl) and causes internal functions to fail.
Comment 9 Greg Rivers 2022-04-18 08:36:34 UTC
(In reply to Dirk Meyer from comment #8)
On one of my systems where the new sendmail binary fails I have:

$ uname -mrs
FreeBSD 13.0-RELEASE-p11 amd64

$ openssl version
OpenSSL 1.1.1k-freebsd  24 Aug 2021

$ fgrep OPENSSL_VERSION_NUMBER /usr/include/openssl/opensslv.h
# define OPENSSL_VERSION_NUMBER  0x101010bfL

$ sendmail -d0.14 </dev/null | fgrep OpenSSL
       OpenSSL: compiled 0x101010bf
       OpenSSL: linked   0x101010bf

$ ls -l /usr/local/lib/sasl2/Sendmail.conf /usr/local/etc/sasldb2.db
-rw-r-----  1 cyrus  mail   16384 Apr 17 23:33 /usr/local/etc/sasldb2.db
-rw-r--r--  1 root   wheel     26 Apr 17 23:34 /usr/local/lib/sasl2/Sendmail.conf

My .mc file includes:

define(`confAUTH_OPTIONS', `A p y')
TRUST_AUTH_MECH(`LOGIN PLAIN')
define(`confAUTH_MECHANISMS', `LOGIN PLAIN')
define(`confDONT_BLAME_SENDMAIL',`GroupReadableKeyFile GroupReadableSASLDBFile')
DAEMON_OPTIONS(`Family=inet, Name=MSAv4, Port=submission, M=Ea')dnl 'a' makes authentication mandatory.
DAEMON_OPTIONS(`Family=inet6, Name=MSAv6, Port=submission, M=Ea')
DAEMON_OPTIONS(`Family=inet, Name=MSAv4, Port=smtps, M=Eas')dnl 's' engages TLS upon connect.
DAEMON_OPTIONS(`Family=inet6, Name=MSAv6, Port=smtps, M=Eas')
DAEMON_OPTIONS(`Name=MTAv4, Family=inet')
DAEMON_OPTIONS(`Name=MTAv6, Family=inet6, Modifiers=O')

We now know there's nothing wrong with the new sendmail binary from the recently built packages because it works on a pristine system. Can you recommend further troubleshooting steps to determine why it fails to find authentication methods on my existing systems? Do you suspect a ports (sasl linking?) problem, or a base system problem?
Comment 10 Fabian Wenk 2022-04-18 16:07:35 UTC
(In reply to Greg Rivers from comment #6)

In my case I have the following key differences:

- FreeBSD 12.3-RELEASE-p5
- FreeBSD Base and all Ports are built on my system
- openssl-1.1.1n,1 from Ports (Base OpenSSL is also installed)
- not using saslauthd, but cyrus-sasl-2.1.28 with cyrus-sasl-sql-2.1.28 and directly connecting to a MariaDB database.

More details will follow in an other comment later, as I first would like to do some more comparing and testing.
Comment 11 Dirk Meyer freebsd_committer freebsd_triage 2022-04-18 21:01:26 UTC
(In reply to Greg Rivers from comment #9)

I suspect some ABI changes in OpenSSL.

sendmail links to /usr/lib/libssl.so.111
sendmail links to /lib/libcrypto.so.111
sendmail links to /usr/local/lib/libsasl2.so.3

libsasl2.so.3 links also to /lib/libcrypto.so.111
libsasl2.so.3 may load plugins that also use libcrypto or libssl.
But the already loaded libs from sendmail will be used.

All code must be build to the same OpenSSL version,
either with base only or from ports only.
Any mix will cause problems or even segfaults.

Even with when the port version is used,
all module have to be build with the same options set in the openssl port.


Diagnostics:

To compare the files from a working and non working system.
create a mtree file and drop the "time".

mtree -c -p / | sed -e 's| *time=[0-9.]*||' > all.system1.mtree

Then you can diff files and see if you can find something related changed in permissions or size.
Comment 12 Fabian Wenk 2022-04-18 21:32:41 UTC
(In reply to Dirk Meyer from comment #8)

Here some more details and chronology from my build & test and also live system.

on my build & test system
=========================

History of pkg build & install
pkg name		build date	(re-)install date
sendmail-8.17.1_2	2022-01-03	2022-01-03
cyrus-sasl-2.1.28	2022-03-02	2022-03-02
cyrus-sasl-sql-2.1.28	2022-03-02	2022-03-02
openssl-1.1.1n,1	2022-03-16	2022-03-16
sendmail-8.17.1_3	2022-04-17-a	2022-04-17 (failed when Sendmail.conf was present)
cyrus-sasl-2.1.28	2022-04-17	2022-04-17 (rebuild done)
cyrus-sasl-sql-2.1.28	2022-04-17	2022-04-17 (rebuild done)
sendmail-8.17.1_3	2022-04-17-b	2022-04-17 (rebuild done)
full setup of SMTP Auth, tested and worked, then did re-install previous pkg:
cyrus-sasl-2.1.28	2022-03-02	2022-04-17
cyrus-sasl-sql-2.1.28	2022-03-02	2022-04-17
sendmail-8.17.1_3	2022-04-17-a	2022-04-17
SMTP Auth still works

root@build+test:~ # freebsd-version -uk
12.3-RELEASE-p5
12.3-RELEASE-p5

root@build+test:~ # openssl version
OpenSSL 1.1.1l-freebsd  24 Aug 2021

root@build+test:~ # grep OPENSSL_VERSION_NUMBER /usr/include/openssl/opensslv.h
# define OPENSSL_VERSION_NUMBER  0x101010cfL

root@build+test:~ # /usr/local/bin/openssl version
OpenSSL 1.1.1n  15 Mar 2022

root@build+test:~ # grep OPENSSL_VERSION_NUMBER /usr/local/include/openssl/opensslv.h 
# define OPENSSL_VERSION_NUMBER  0x101010efL

root@build+test:~ # sendmail -d0.14 < /dev/null | grep OpenSSL
       OpenSSL: compiled 0x101010ef
       OpenSSL: linked   0x101010ef

root@build+test:~ # ls -l /usr/local/lib/sasl2/*so.3.0.0
-rwxr-xr-x  1 root  wheel  17600 Mar  2 16:13 /usr/local/lib/sasl2/libanonymous.so.3.0.0
-rwxr-xr-x  1 root  wheel  19944 Mar  2 16:13 /usr/local/lib/sasl2/libcrammd5.so.3.0.0
-rwxr-xr-x  1 root  wheel  57472 Mar  2 16:13 /usr/local/lib/sasl2/libdigestmd5.so.3.0.0
-rwxr-xr-x  1 root  wheel  18224 Mar  2 16:13 /usr/local/lib/sasl2/liblogin.so.3.0.0
-rwxr-xr-x  1 root  wheel  40008 Mar  2 16:13 /usr/local/lib/sasl2/libntlm.so.3.0.0
-rwxr-xr-x  1 root  wheel  27680 Mar  2 16:13 /usr/local/lib/sasl2/libotp.so.3.0.0
-rwxr-xr-x  1 root  wheel  18376 Mar  2 16:13 /usr/local/lib/sasl2/libplain.so.3.0.0
-rwxr-xr-x  1 root  wheel  40560 Mar  2 16:13 /usr/local/lib/sasl2/libscram.so.3.0.0
-rwxr-xr-x  1 root  wheel  30072 Mar  2 16:13 /usr/local/lib/sasl2/libsql.so.3.0.0
root@build+test:~ # ldd /usr/local/lib/libsasl2.so.3.0.0
/usr/local/lib/libsasl2.so.3.0.0:
	libdl.so.1 => /usr/lib/libdl.so.1 (0x800699000)
	libcrypto.so.11 => /usr/local/lib/libcrypto.so.11 (0x800e00000)
	libc.so.7 => /lib/libc.so.7 (0x80024e000)
	libthr.so.3 => /lib/libthr.so.3 (0x80069d000)
root@build+test:~ # ldd /usr/local/lib/sasl2/libsql.so.3.0.0
/usr/local/lib/sasl2/libsql.so.3.0.0:
	libmariadb.so.3 => /usr/local/lib/mysql/libmariadb.so.3 (0x80067b000)
	libc.so.7 => /lib/libc.so.7 (0x80024e000)
	libm.so.5 => /lib/libm.so.5 (0x8006d4000)
	libssl.so.11 => /usr/local/lib/libssl.so.11 (0x80070a000)
	libcrypto.so.11 => /usr/local/lib/libcrypto.so.11 (0x800e00000)
	libz.so.6 => /lib/libz.so.6 (0x8007a0000)
	libthr.so.3 => /lib/libthr.so.3 (0x8007bc000)
root@build+test:~ # ldd /usr/local/sbin/sendmail
/usr/local/sbin/sendmail:
	libwrap.so.6 => /usr/lib/libwrap.so.6 (0x800307000)
	libsasl2.so.3 => /usr/local/lib/libsasl2.so.3 (0x800313000)
	libssl.so.11 => /usr/local/lib/libssl.so.11 (0x800333000)
	libcrypto.so.11 => /usr/local/lib/libcrypto.so.11 (0x8003c9000)
	libutil.so.9 => /lib/libutil.so.9 (0x8006ba000)
	libc.so.7 => /lib/libc.so.7 (0x8006d2000)
	libdl.so.1 => /usr/lib/libdl.so.1 (0x800aca000)
	libthr.so.3 => /lib/libthr.so.3 (0x800ace000)
root@build+test:~ # ls -l /usr/local/lib/libcrypto.so.11
-rw-r--r--  1 root  wheel  3058832 Mar 16 21:50 /usr/local/lib/libcrypto.so.11
root@build+test:~ # ls -l /usr/local/lib/libssl.so.11
-rw-r--r--  1 root  wheel  600952 Mar 16 21:50 /usr/local/lib/libssl.so.11



on my live system
=================

History of pkg build & install (newer pkg where created on above build system
with taking care that the config options are the same):
pkg name		build date	(re-)install date
sendmail-8.17.1_2	2022-01-03	2022-01-03 (built on this system)
cyrus-sasl-2.1.28	2022-03-02	2022-03-02
cyrus-sasl-sql-2.1.28	2022-03-02	2022-03-02
openssl-1.1.1n,1	2022-03-16	2022-03-16
sendmail-8.17.1_3	2022-04-17	2022-04-17 (failed, removed)
sendmail-8.17.1_2	2022-01-03	2022-04-17

root@live:~ # freebsd-version -uk
12.3-RELEASE-p5
12.3-RELEASE-p5

root@live:~ # openssl version
OpenSSL 1.1.1l-freebsd  24 Aug 2021

root@live:~ # grep OPENSSL_VERSION_NUMBER /usr/include/openssl/opensslv.h
# define OPENSSL_VERSION_NUMBER  0x101010cfL

root@live:~ # /usr/local/bin/openssl version
OpenSSL 1.1.1n  15 Mar 2022

root@live:~ # grep OPENSSL_VERSION_NUMBER /usr/local/include/openssl/opensslv.h 
# define OPENSSL_VERSION_NUMBER  0x101010efL

root@live:~ # sendmail -d0.14 < /dev/null | grep OpenSSL
       OpenSSL: compiled 0x101010df
       OpenSSL: linked   0x101010ef

root@live:~ # pkg info | grep ^compat	# this are not present on build+test
compat10x-amd64-10.4.1004000.20181014 Convenience package to install the compat10x libraries
compat9x-amd64-9.3.903000.20170608 Convenience package to install the compat9x libraries

root@live:~ # ls -l /usr/local/lib/sasl2/*so.3.0.0
-rwxr-xr-x  1 root  wheel  17600 Mar  2 16:13 /usr/local/lib/sasl2/libanonymous.so.3.0.0
-rwxr-xr-x  1 root  wheel  19944 Mar  2 16:13 /usr/local/lib/sasl2/libcrammd5.so.3.0.0
-rwxr-xr-x  1 root  wheel  57472 Mar  2 16:13 /usr/local/lib/sasl2/libdigestmd5.so.3.0.0
-rwxr-xr-x  1 root  wheel  18224 Mar  2 16:13 /usr/local/lib/sasl2/liblogin.so.3.0.0
-rwxr-xr-x  1 root  wheel  40008 Mar  2 16:13 /usr/local/lib/sasl2/libntlm.so.3.0.0
-rwxr-xr-x  1 root  wheel  27680 Mar  2 16:13 /usr/local/lib/sasl2/libotp.so.3.0.0
-rwxr-xr-x  1 root  wheel  18376 Mar  2 16:13 /usr/local/lib/sasl2/libplain.so.3.0.0
-rwxr-xr-x  1 root  wheel  40560 Mar  2 16:13 /usr/local/lib/sasl2/libscram.so.3.0.0
-rwxr-xr-x  1 root  wheel  30072 Mar  2 16:13 /usr/local/lib/sasl2/libsql.so.3.0.0
root@live:~ # ldd /usr/local/lib/libsasl2.so.3.0.0 
/usr/local/lib/libsasl2.so.3.0.0:
	libdl.so.1 => /usr/lib/libdl.so.1 (0x800698000)
	libcrypto.so.11 => /usr/local/lib/libcrypto.so.11 (0x800e00000)
	libc.so.7 => /lib/libc.so.7 (0x80024e000)
	libthr.so.3 => /lib/libthr.so.3 (0x80069c000)
root@live:~ # ldd /usr/local/lib/sasl2/libsql.so.3.0.0 
/usr/local/lib/sasl2/libsql.so.3.0.0:
	libmariadb.so.3 => /usr/local/lib/mysql/libmariadb.so.3 (0x80067a000)
	libc.so.7 => /lib/libc.so.7 (0x80024e000)
	libm.so.5 => /lib/libm.so.5 (0x8006d3000)
	libssl.so.11 => /usr/local/lib/libssl.so.11 (0x800709000)
	libcrypto.so.11 => /usr/local/lib/libcrypto.so.11 (0x800e00000)
	libz.so.6 => /lib/libz.so.6 (0x80079f000)
	libthr.so.3 => /lib/libthr.so.3 (0x8007bb000)
root@live:~ # ldd /usr/local/sbin/sendmail 
/usr/local/sbin/sendmail:
	libwrap.so.6 => /usr/lib/libwrap.so.6 (0x800307000)
	libsasl2.so.3 => /usr/local/lib/libsasl2.so.3 (0x800313000)
	libssl.so.11 => /usr/local/lib/libssl.so.11 (0x800333000)
	libcrypto.so.11 => /usr/local/lib/libcrypto.so.11 (0x8003c9000)
	libutil.so.9 => /lib/libutil.so.9 (0x8006ba000)
	libc.so.7 => /lib/libc.so.7 (0x8006d2000)
	libdl.so.1 => /usr/lib/libdl.so.1 (0x800aca000)
	libthr.so.3 => /lib/libthr.so.3 (0x800ace000)
root@live:~ # ls -l /usr/local/lib/libcrypto.so.11
-rw-r--r--  1 root  wheel  3058832 Mar 16 21:50 /usr/local/lib/libcrypto.so.11
root@live:~ # ls -l /usr/local/lib/libssl.so.11
-rw-r--r--  1 root  wheel  600952 Mar 16 21:50 /usr/local/lib/libssl.so.11


After recording all the above details I did try something more.
As two files with the temporary rebuild done on 2022-04-17 of cyrus-sasl-2.1.28 and cyrus-sasl-sql-2.1.28 where different (libdigestmd5.so.3.0.0 and libsql.so.3.0.0), I again rebuilt this two Ports/packages and installed them on the build&test system, restarted sendmail and SMTP Auth was still available.
Then I also forced the upgrade of cyrus-sasl-2.1.28 and cyrus-sasl-sql-2.1.28 on the live system, cleaned out /usr/local/lib/compat/pkg/ of orphan cyrus-sasl* stuff, restarted sendmail (still 8.17.1_2) SMTP Auth was still working. Then I did also upgrade to sendmail-8.17.1_3 and SMTP Auth was missing. Restored only the old _2 /usr/local/sbin/sendmail, restarted sendmail and SMTP Auth was working again.

What I also found strange, that 'make -C /usr/ports/ search name=cyrus-sasl-sql' does list a lot of R-deps, but 'pkg info -d cyrus-sasl-sql' does only list openssl, cyrus-sasl and mariadb103-client, even if many of the mention R-deps are installed as well.
Comment 13 Greg Rivers 2022-04-19 19:27:18 UTC
It appears that this ticket is actually a duplicate of bug #262935. Bug #262935 explains what the problem is and how to fix it. Also see this thread: <https://lists.freebsd.org/archives/freebsd-ports/2022-April/001756.html>.

Can the sendmail port be upgraded to 8.17.1.9 soon?
Comment 14 Fabian Wenk 2022-04-19 19:40:18 UTC
(In reply to Greg Rivers from comment #13)

And still we both were able to create test systems where it works, with the same binaries where it fails on our live systems. And even we both have some key differences with OS version and how sasl is used.
Comment 15 Dirk Meyer freebsd_committer freebsd_triage 2022-04-19 20:58:16 UTC
Thanks,
the snapshot is already available sendmail-devel-8.17.1.9_1.

pkg install sendmail-devel

should help until the main port is patched.
Comment 16 Dirk Meyer freebsd_committer freebsd_triage 2022-04-19 21:13:44 UTC
Created attachment 233346 [details]
Suggested patch

Please test this patch for the mail/sendmail port
Comment 17 Fabian Wenk 2022-04-19 21:31:09 UTC
(In reply to Dirk Meyer from comment #16)

The patched mail/sendmail (sendmail-8.17.1_4) works on my build+test and also on my live systems.
Comment 18 Greg Rivers 2022-04-19 22:47:59 UTC
(In reply to Fabian Wenk from comment #14)
I'm right there with you Fabian. I don't understand why it worked on our test systems either. But since Dirk's patch works I think I'll just forget about it and move on. :)
Comment 19 Greg Rivers 2022-04-19 22:49:38 UTC
(In reply to Dirk Meyer from comment #16)
Thanks for all of your help Dirk. I'll watch for the updated mail/sendmail port to hit the pkg repos and report back after testing.
Comment 20 commit-hook freebsd_committer freebsd_triage 2022-04-20 05:35:27 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=0625156b336bacba1e5dc541680af85e97feb0e2

commit 0625156b336bacba1e5dc541680af85e97feb0e2
Author:     Dirk Meyer <dinoex@FreeBSD.org>
AuthorDate: 2022-04-20 05:34:44 +0000
Commit:     Dirk Meyer <dinoex@FreeBSD.org>
CommitDate: 2022-04-20 05:34:44 +0000

    mail/sendmail: Fix option SASL and SMTP AUTH

    PR: 263287

 mail/sendmail/Makefile                 |  2 +-
 mail/sendmail/files/patch-zsasl2 (new) | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)
Comment 21 commit-hook freebsd_committer freebsd_triage 2022-04-20 06:20:34 UTC
A commit in branch 2022Q2 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=02e50e4a661c036de9f4dbfb0b63642a05997b6a

commit 02e50e4a661c036de9f4dbfb0b63642a05997b6a
Author:     Dirk Meyer <dinoex@FreeBSD.org>
AuthorDate: 2022-04-20 05:34:44 +0000
Commit:     Dirk Meyer <dinoex@FreeBSD.org>
CommitDate: 2022-04-20 06:19:57 +0000

    mail/sendmail: Fix option SASL and SMTP AUTH

    PR: 263287
    (cherry picked from commit 0625156b336bacba1e5dc541680af85e97feb0e2)

 mail/sendmail/files/patch-zsasl2 (new) | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
Comment 22 Dirk Meyer freebsd_committer freebsd_triage 2022-04-20 06:25:31 UTC
Fixed in "latest" as sendmail-8.17.1_4.pkg
Fixed in "quarterly" as sendmail-8.17.1_3.pkg

Thanks for testing and debugging.
Comment 23 commit-hook freebsd_committer freebsd_triage 2022-04-22 05:28:57 UTC
A commit in branch 2022Q2 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=9a3d4adaa3013704507f256755d612af1e9dc4e9

commit 9a3d4adaa3013704507f256755d612af1e9dc4e9
Author:     Dirk Meyer <dinoex@FreeBSD.org>
AuthorDate: 2022-04-20 05:34:44 +0000
Commit:     Dirk Meyer <dinoex@FreeBSD.org>
CommitDate: 2022-04-22 05:28:09 +0000

    mail/sendmail: Fix option SASL and SMTP AUTH

    PR: 263287
    (cherry picked from commit 0625156b336bacba1e5dc541680af85e97feb0e2)

 mail/sendmail/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 24 Greg Rivers 2022-04-25 04:35:08 UTC
All is well after upgrading to 8.17.1_4 from the "latest" pkg repo.