Bug 272835 - kinit(8) segmentation fault with openssl-3.0 in CURRENT
Summary: kinit(8) segmentation fault with openssl-3.0 in CURRENT
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: Cy Schubert
URL:
Keywords:
: 275915 276257 (view as bug list)
Depends on:
Blocks: 14.0-erratas
  Show dependency treegraph
 
Reported: 2023-07-31 07:49 UTC by Joerg Pulz
Modified: 2024-04-03 19:25 UTC (History)
6 users (show)

See Also:
cy: mfc-stable14+
cy: mfc-stable13-
cy: mfc-stable12-


Attachments
patch to fix base heimdal with openssl-3.x (2.37 KB, patch)
2023-11-17 17:13 UTC, Joerg Pulz
no flags Details | Diff
patch to fix base heimdal and weak enctypes with base openssl-3.0.x v2 (2.38 KB, patch)
2023-11-21 09:24 UTC, Joerg Pulz
no flags Details | Diff
Enable legacy provider (8.17 KB, patch)
2023-12-08 07:17 UTC, Cy Schubert
no flags Details | Diff
Enable weak encrypton for heimdal (774 bytes, patch)
2023-12-08 14:25 UTC, Cy Schubert
no flags Details | Diff
Enable Legacy Provider (9.38 KB, patch)
2023-12-09 14:33 UTC, Cy Schubert
no flags Details | Diff
Heimdal - enable weak encryption, builds on 13 when building to upgrade to 14 or 15 (9.43 KB, patch)
2023-12-11 21:49 UTC, Cy Schubert
no flags Details | Diff
Heimdal - enable weak encryption, builds on 13 when building to upgrade to 14 or 15 (9.43 KB, patch)
2023-12-12 01:10 UTC, Cy Schubert
no flags Details | Diff
Also circumvent Linux and MacOS cross-build errors (9.01 KB, patch)
2024-01-15 17:09 UTC, Cy Schubert
no flags Details | Diff
Fix numerous Heimdal segfaults when using weak crypto (9.03 KB, patch)
2024-01-16 19:08 UTC, Cy Schubert
no flags Details | Diff
Final version of patch before commit. (8.44 KB, patch)
2024-01-17 18:47 UTC, Cy Schubert
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Joerg Pulz 2023-07-31 07:49:21 UTC
kinit(8) segfaults after my recent upgrade to CURRENT with OpenSSL-3.0 - FreeBSD 14.0-CURRENT amd64 1400093 #0 main-n264294-09e5d91069d1

Looking at the core dump it happens here:

   frame #1: 0x000032ea670715c8 libkrb5.so.11`ARCFOUR_string_to_key(context=0x000051189c638000, enctype=KRB5_ENCTYPE_ARCFOUR_HMAC_MD5, password=<unavailable>, salt=krb5_salt @ 0x000032ea64595c30, opaque=<unavailable>, key=0x000051189c6b7fc0) at salt-arcfour.c:84:2

Looking at crypto/heimdal/lib/krb5/salt-arcfour.c it happens when calling openssl's EVP_DigestUpdate():

    /* LE encoding */
    for (i = 0; i < len; i++) {
        unsigned char p;
        p = (s[i] & 0xff);
->      EVP_DigestUpdate (m, &p, 1);
        p = (s[i] >> 8) & 0xff;
        EVP_DigestUpdate (m, &p, 1);
    }

Running from lldb(1) it seems that I end in crypto/openssl/crypto/evp/digest.c in EVP_DigestUpdate() in line 412:

    410     /* Code below to be removed when legacy support is dropped. */
    411  legacy:
    412     return ctx->update(ctx, data, count);

Here ctx->update() is causing the segmentation fault.

I'm not an openssl expert and have no clue what ctx->update should be and where it is coming from, so I'm at the end of the road here.
Comment 1 Joerg Pulz 2023-11-17 17:13:06 UTC
Created attachment 246375 [details]
patch to fix base heimdal with openssl-3.x

Found some time to do further research.

rc4 and other weak crypto is covered by the OpenSSL legacy provider which is not loaded by default.

Adding the missing parts to explicitly load the legacy provider at the right places fixes this PR.

Changes in crypto/heimdal/lib/krb5/salt.c make kinit(8) work.
Changes in crypto/heimdal/lib/krb5/crypto.c make using the TGT and obtaining service tickets work.

Would be nice to see this in the tree and MFC'd to 14.
Comment 2 Joerg Pulz 2023-11-18 06:34:24 UTC
There is an easier solution without changing the code at all.

Just modify /etc/ssl/openssl.cnf to make the specific section look like this:

#####
[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1
#####

This will (re)enable the default disabled legacy provider in OpenSSL 3.x.

One could argue to either make this default or at least add a hint in UPDATING.
Comment 3 Joerg Pulz 2023-11-21 09:24:54 UTC
Created attachment 246458 [details]
patch to fix base heimdal and weak enctypes with base openssl-3.0.x v2

After thinking some more about this I propose we should fix this.

If we ship heimdal with weak enctypes enabled (what we do) it should work out of the box.

This can be achieved in two ways:
1)
Modify the code so heimdal is loading the required OpenSSL legacy providers by itself - see the attached patch.

2)
Modify the default OpenSSL configuration (/etc/ssl/openssl.cnf) we ship as mentioned in my previous comment.

New versions of heimdal make use of heimdal's own md4/rc4 implementations in lib/hcrypto. As this is not in our tree (see crypto/heimdal/FREEBSD-Xlist) it's not an option.
Comment 4 Joerg Pulz 2023-11-21 09:27:02 UTC
Re-open
Comment 5 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-06 08:22:30 UTC
Cy, I can confirm this bug (and have the same backtrace on kadmin -l add).

I will try the patch for src (which looks more sensible than enabling the legacy provider in openssl.cnf) and report back.

We have this issue in 14 and -current. It may justify an EN...
Comment 6 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-06 08:49:09 UTC
The patch is not enough to make kadmin work. A quick try with patching kadmin.c didn't help:
        modified:   crypto/heimdal/kadmin/kadmin.c
        modified:   crypto/heimdal/lib/krb5/crypto.c
        modified:   crypto/heimdal/lib/krb5/salt.c
        modified:   kerberos5/include/crypto-headers.h
A quick check with a change to openssl.cnf makes kadmin -l add work.

So this is the right track, but the patch is not a complete fix.
Comment 7 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-06 09:44:44 UTC
I didn't check if all the necessary includes are available in kadmin.c, adding the call without the #ifdef makes kadmin work.

I didn't check other programs (like kpasswd).
Comment 8 Joerg Pulz 2023-12-06 09:56:29 UTC
(In reply to Alexander Leidinger from comment #6)

You're completely right here. The patch is far from complete and only tested for kinit(8).
A fix for kadmin(8) probably has to be made somewhere in crypto/heimdal/lib/kadm5/ - I didn't looked deep into it.

Additionally, the patch definitely misses OSSL_PROVIDER_unload(). Without it we at least leak the providers reference counter.

Further modifications to our "ancient" base heimdal does not seem reasonable as the time spent with it could better be used to start updating base heimdal to a recent version.

As already mentioned in comment #3, heimdal itself switched to it's builtin hcrypto functions for 1DES/RC2/RC4 in the OpenSSL-3.x case. This should be taken into account for a future MFV as the whole hcrypto/ directory is currently excluded.

For now, the easiest way to solve our problem would be the modification of openssl.cnf.
Comment 9 Cy Schubert freebsd_committer freebsd_triage 2023-12-06 15:32:46 UTC
(In reply to Joerg Pulz from comment #2)

We should avoid this as it adds technical debt by encouraging the use of other old software.
Comment 10 Cy Schubert freebsd_committer freebsd_triage 2023-12-06 17:39:01 UTC
I think more than kadmin and kpasswd need to be looked at. I suspect kdc will also segfault with a KDC DB encoded using weak ciphers.

I will the jail used to fix the various bugs reported by rtm@mit.
Comment 11 Cy Schubert freebsd_committer freebsd_triage 2023-12-06 17:56:43 UTC
Just set up a clean jail previously used for rtm@mit bugs. kadmin segfaults.

rm -rf /var/heimdal/*
kstash --random-key
kadmin -l
kadmin> init EXAMPLE.ORG

I have a good dump. Will analyze.
Comment 12 Joerg Pulz 2023-12-06 18:14:21 UTC
Cy, thanks for taking care and looking into it.

If you want to fix this without touching openssl.cnf I think it has to be done on library level in heimdal and keeping track of the provider reference count and OSSL_PROVIDER_unload() in the right place for every OSSL_PROVIDER_load() - see OSSL_LIB_CTX(3ossl).

Let me know if I can be of any help.
Comment 13 Cy Schubert freebsd_committer freebsd_triage 2023-12-06 18:34:03 UTC
(In reply to Joerg Pulz from comment #12)

The best approach would be to add a function to libroken which can keep count of providers, loading them as needed. I don't think an unload function is needed.
Comment 14 Cy Schubert freebsd_committer freebsd_triage 2023-12-08 07:17:07 UTC
Created attachment 246893 [details]
Enable legacy provider

Can you give this patch a spin? I've tested this a little here but need to get some sleep.

Give it a good workout and let me know if it resolves all issues.
Comment 15 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-08 09:56:49 UTC
(In reply to Cy Schubert from comment #14)
"make clean obj depend all" in /usr/src/kerberos5/ results in:
---snip---
Building /space/system/usr_obj/space/system/usr_src/amd64.amd64/kerberos5/lib/libkadm5srv/libkadm5srv.so.11.full
building shared library libkadm5srv.so.11
ld: error: undefined symbol: fbsd_ossl_provider_load
>>> referenced by create_s.c:172 (/space/system/usr_src/crypto/heimdal/lib/kadm5/create_s.c:172)
>>>               create_s.pico:(kadm5_s_create_principal)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

Stop.
make[2]: stopped in /space/system/usr_src/kerberos5/lib/libkadm5srv
.ERROR_TARGET='libkadm5srv.so.11.full'
.ERROR_META_FILE='/space/system/usr_obj/space/system/usr_src/amd64.amd64/kerberos5/lib/libkadm5srv/libkadm5srv.so.11.full.meta'
.MAKE.LEVEL='2'
MAKEFILE=''
.MAKE.MODE='meta missing-filemon=yes missing-meta=yes silent=yes verbose'
_ERROR_CMD='@echo building shared library libkadm5srv.so.11; @rm -f libkadm5srv.so.11 libkadm5srv.so; cc -Wl,--no-undefined -Wl,-zrelro -Wl,-zretpolineplt -Wl,--version-script=/space/system/usr_src/crypto/heimdal/lib/kadm5/version-script.map -Wl,--undefined-version    -fstack-protector-strong -shared -Wl,-x -Wl,--fatal-warnings -Wl,--warn-shared-textrel  -o libkadm5srv.so.11.full -Wl,-soname,libkadm5srv.so.11 acl.pico bump_pw_expire.pico chpass_s.pico common_glue.pico context_s.pico create_s.pico delete_s.pico destroy_s.pico ent_setup.pico error.pico flush_s.pico free.pico get_princs_s.pico get_s.pico init_s.pico kadm5_err.pico keys.pico log.pico marshall.pico modify_s.pico password_quality.pico privs_s.pico randkey_s.pico rename_s.pico server_glue.pico set_keys.pico set_modifier.pico -L/space/system/usr_obj/space/system/usr_src/amd64.amd64/lib/libcom_err -lcom_err -L/space/system/usr_obj/space/system/usr_src/amd64.amd64/kerberos5/lib/libhdb -lhdb -L/space/system/usr_obj/space/system/usr_src/amd64.amd64/kerberos5/lib/libkrb5 -lkrb5 -L/space/system/usr_obj/space/system/usr_src/amd64.amd64/kerberos5/lib/libroken -lroken ;'
.CURDIR='/space/system/usr_src/kerberos5/lib/libkadm5srv'
.MAKE='make'
.OBJDIR='/space/system/usr_obj/space/system/usr_src/amd64.amd64/kerberos5/lib/libkadm5srv'
.TARGETS='all'
CPUTYPE=''
DESTDIR=''
LD_LIBRARY_PATH=''
MACHINE='amd64'
MACHINE_ARCH='amd64'
MACHINE_CPUARCH='amd64'
MAKEOBJDIRPREFIX=''
MAKESYSPATH='/space/system/usr_src/share/mk'
MAKE_VERSION='20230909'
PATH='/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin:/root/bin'
SRCTOP='/space/system/usr_src'
OBJTOP='/space/system/usr_obj/space/system/usr_src/amd64.amd64'
.MAKE.MAKEFILES='/space/system/usr_src/share/mk/sys.mk /space/system/usr_src/share/mk/local.sys.env.mk /space/system/usr_src/share/mk/src.sys.env.mk /etc/src-env.conf /space/system/usr_src/share/mk/bsd.mkopt.mk /space/system/usr_src/share/mk/src.sys.obj.mk /space/system/usr_src/share/mk/local.sys.machine.mk /space/system/usr_src/share/mk/meta.sys.mk /space/system/usr_src/share/mk/local.meta.sys.env.mk /space/system/usr_src/share/mk/auto.obj.mk /space/system/usr_src/share/mk/bsd.suffixes.mk /etc/make.conf /space/system/usr_src/share/mk/local.sys.mk /space/system/usr_src/share/mk/src.sys.mk /etc/src.conf /space/system/usr_src/kerberos5/lib/libkadm5srv/Makefile /space/system/usr_src/share/mk/bsd.lib.mk /space/system/usr_src/share/mk/bsd.init.mk /space/system/usr_src/share/mk/bsd.opts.mk /space/system/usr_src/share/mk/bsd.cpu.mk /space/system/usr_src/share/mk/local.init.mk /space/system/usr_src/share/mk/src.init.mk /space/system/usr_src/kerberos5/lib/libkadm5srv/../Makefile.inc /space/system/usr_src/kerberos5/lib/libkadm5srv/../../Makefile.inc /space/system/usr_src/share/mk/src.opts.mk /space/system/usr_src/share/mk/bsd.own.mk /space/system/usr_src/share/mk/bsd.compiler.mk /space/system/usr_src/share/mk/bsd.endian.mk /space/system/usr_src/share/mk/bsd.linker.mk /space/system/usr_src/share/mk/bsd.compat.pre.mk /space/system/usr_src/share/mk/bsd.sanitizer.mk /space/system/usr_src/share/mk/bsd.libnames.mk /space/system/usr_src/share/mk/src.libnames.mk /space/system/usr_src/share/mk/bsd.symver.mk /space/system/usr_src/share/mk/bsd.nls.mk /space/system/usr_src/share/mk/bsd.confs.mk /space/system/usr_src/share/mk/bsd.files.mk /space/system/usr_src/share/mk/bsd.dirs.mk /space/system/usr_src/share/mk/bsd.incs.mk /space/system/usr_src/share/mk/bsd.links.mk /space/system/usr_src/share/mk/bsd.man.mk /space/system/usr_src/share/mk/bsd.dep.mk /space/system/usr_src/share/mk/bs                                                                                    d.clang-analyze.mk /space/system/usr_src/share/mk/bsd.obj.mk /space/system/usr_src/share/mk/bsd.subdir.mk /space/system/usr_src/share/mk/bsd.sys.mk /dev/null'
.PATH='. /space/system/usr_src/kerberos5/lib/libkadm5srv /space/system/usr_src/crypto/heimdal/lib/kadm5'
*** Error code 1

Stop.
make[1]: stopped in /space/system/usr_src/kerberos5/lib
*** Error code 1
---snip---

To validate that I got all changes:
---snip---
# LC_ALL=C git status
On branch localchanges
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   crypto/heimdal/lib/kadm5/create_s.c
        modified:   crypto/heimdal/lib/kadm5/kadm5_locl.h
        modified:   crypto/heimdal/lib/krb5/context.c
        modified:   crypto/heimdal/lib/krb5/crypto.c
        modified:   crypto/heimdal/lib/krb5/salt.c
        modified:   crypto/heimdal/lib/roken/version-script.map
        modified:   kerberos5/include/crypto-headers.h
        modified:   kerberos5/lib/libroken/Makefile
        modified:   kerberos5/libexec/kdc/Makefile
        modified:   share/mk/src.libnames.mk

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        buildkernel.log
        buildworld.log
        installkernel.log
        installworld.log
        kerberos5/include/fbsd_ossl_provider.h
        kerberos5/lib/libroken/fbsd_ossl_provider_load.c
        krb5.diff
---snip---
Comment 16 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-08 10:15:59 UTC
Manually compiling libroken before the other libs makes the build work.

Quick check: I can kinit and kadmin -l add.

I also tried GSS auth via ssh, but this fails. I never tried that before, so not sure if this is a PEBKAC problem or not (enabling GSSAPIAuthentication for the client and the daemon + restart of sshd). When sshing to the remote, the client system get's the machine ticket, but the gssapi-with-mic authentication of ssh fails silently.
Comment 17 Cy Schubert freebsd_committer freebsd_triage 2023-12-08 14:25:07 UTC
Created attachment 246909 [details]
Enable weak encrypton for heimdal

(In reply to Alexander Leidinger from comment #16)

Then we will need to adjust /etc/ssl/openssl.cnf to enable weak encryption globally.
Comment 18 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-08 14:32:31 UTC
(In reply to Cy Schubert from comment #17)
Enabling the legacy (and default) provider on the remote system and the client system and restarting sshd on the target didn't help.

Would be good if someone could test this somewhere where it is known to work before...
Comment 19 Cy Schubert freebsd_committer freebsd_triage 2023-12-08 14:47:09 UTC
(In reply to Alexander Leidinger from comment #18)

This is certainly a bigger problem because one would probably need to downgrade those (including Linux) as well.

Which machine is your KDC? Is the KDC FreeBSD Heimdal 1.5?
Comment 20 Cy Schubert freebsd_committer freebsd_triage 2023-12-08 14:56:08 UTC
(In reply to Alexander Leidinger from comment #15)

This build failure could be due to meta mode. It built fine on my sandbox machine.

I will a) build again here and b) build on one of the universe15 machines to verify.
Comment 21 Joerg Pulz 2023-12-08 15:06:21 UTC
I've KDC's running on releng/13.2 and clients running releng/14.0.

The 14 clients run with sshd's
  GSSAPIAuthentication yes
  GSSAPICleanupCredentials yes

With my patch or legacy provider enabled in openssl.cnf those clients worked.

I will test your patch later.
Comment 22 Joerg Pulz 2023-12-08 15:06:32 UTC
I've KDC's running on releng/13.2 and clients running releng/14.0.

The 14 clients run with sshd's
  GSSAPIAuthentication yes
  GSSAPICleanupCredentials yes

With my patch or legacy provider enabled in openssl.cnf those clients worked.

I will test your patch later.
Comment 23 Cy Schubert freebsd_committer freebsd_triage 2023-12-08 15:29:10 UTC
Without getting into a long explanation, I suspect that Heimdal 1.5.2 KDC (the Heimdal in FreeBSD is 1.5.2) keyed with legacy ciphers is likely the cause of the problem. Just a hunch ATM.

Can both of you tell me what your KDC is? Is it a FreeBSD base KDC, Heimdal from ports KDC, or MIT KDC?
Comment 24 Joerg Pulz 2023-12-08 16:23:21 UTC
Ok, here are my results.
Besides from kdc/kpasswdd/kadmind (did not test those) everything seems to work with you patch.

KDC - base Heimdal (1.5.2) on releng/13.2
SSH server - base Heimdal (1.5.2) on releng/14.0
other servers - base Heimdal (1.5.2) on releng/13.2
client - base Heimdal (1.5.2) on releng/14.0

- kinit on client works
- ssh from client to SSH server works
####
debug1: Authentications that can continue: publickey,gssapi-with-mic,keyboard-interactive
debug3: start over, passed a different list publickey,gssapi-with-mic,keyboard-interactive
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 60
debug1: Delegating credentials
debug3: send packet: type 61
debug3: receive packet: type 61
debug1: Delegating credentials
debug3: send packet: type 66
debug3: receive packet: type 52
Authenticated to ...... ([.....]:22) using "gssapi-with-mic".
####

- ticket gets forwarded to SSH server
- client has ticket for SSH server in credentials cache
- on client and SSH server tried to use different kerberized services - all working
  - ldapwhoami to kerberized OpenLDAP works
  - reading (imap) and sending (smtp) email using a slightly modified version of mail/alpine (linked against base Heimdal on releng/14.0) works

 % klist
Credentials cache: FILE:/tmp/krb5cc_xxxxxx
        Principal: xxxxxx@XXXXXX.DE

  Issued                Expires               Principal
Dec  8 16:59:54 2023  Dec  9 02:59:54 2023  krbtgt/XXXXXX.DE@XXXXXX.DE
Dec  8 17:02:35 2023  Dec  9 02:59:54 2023  ldap/server01.xxxxxx.de@XXXXXX.DE
Dec  8 17:02:40 2023  Dec  9 02:59:54 2023  imap/imap.xxxxxx.de@XXXXXX.DE
Dec  8 17:03:02 2023  Dec  9 02:59:54 2023  smtp/smtp.xxxxxx.de@XXXXXX.DE
Comment 25 Joerg Pulz 2023-12-08 16:26:17 UTC
(In reply to Joerg Pulz from comment #24)

Forgot to mention:

- kadmin from client (releng/14.0) to remote kadmind (releng/13.2) works too
Comment 26 Cy Schubert freebsd_committer freebsd_triage 2023-12-08 16:59:56 UTC
(In reply to Joerg Pulz from comment #22)

This is the correct sshd setup. I use that same setup here with my FreeBSD base clients pointing to my MIT KDC.

(FreeBSD Heimdal 1.5.2 works well with MIT and has for the last 20-25 years.)
Comment 27 Cy Schubert freebsd_committer freebsd_triage 2023-12-09 01:25:31 UTC
This will still need a little work in Makefile.inc1 to set up bootstrap. Now that libroken needs libcrypto, libcrypto also needs to be bootstrapped. We're now into phabricator review territory. I will submit the patch for review once we, here, are happy with it.
Comment 28 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-09 13:15:31 UTC
In my case: krb from basesystem, -current as of 2-3 days ago.


I checked that my sshd has the options as specified by Jörg: OK.
I tried:
---snip---
debug1: Authentications that can continue: publickey,gssapi-with-mic,keyboard-interactive
debug1: Next authentication method: gssapi-with-mic
debug1:  Miscellaneous failure (see text)
Ticket expired


debug1: Next authentication method: publickey
---snip---

Ok, klist agrees -> kinit to get a new ticket.

klist:
---snip---
% klist
Credentials cache: FILE:/tmp/krb5cc_<uid>
        Principal: netchild@LEIDINGER.NET

  Issued                Expires               Principal
Dec  9 14:10:27 2023  Dec 10 00:10:27 2023  krbtgt/LEIDINGER.NET@LEIDINGER.NET
Dec  9 14:10:51 2023  Dec 10 00:10:27 2023  host/commit.leidinger.net@LEIDINGER.NET
---snip---

Second try:
---snip---
debug1: Authentications that can continue: publickey,gssapi-with-mic,keyboard-interactive
debug1: Next authentication method: gssapi-with-mic
debug1: Authentications that can continue: publickey,gssapi-with-mic,keyboard-interactive
debug1: Next authentication method: publickey
---snip---

No chance. Something is wrong...
Comment 29 Cy Schubert freebsd_committer freebsd_triage 2023-12-09 14:31:59 UTC
(In reply to Alexander Leidinger from comment #28)

Did you add host principals for your servers to your kdc? Did you save the principals' tickets in keytabs on each machine?
Comment 30 Cy Schubert freebsd_committer freebsd_triage 2023-12-09 14:33:16 UTC
Created attachment 246930 [details]
Enable Legacy Provider

This patch includes the Makefile.inc1 bits to build from start to finish.
Comment 31 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-09 14:43:12 UTC
(In reply to Cy Schubert from comment #29)

All hosts have this as /ertc/krb5.keytab:
kadmin -l add --random-key host/hostname.fqdn
kadmin -l ext_keytab --keytab=... host/hostname.fqdn
Comment 32 Cy Schubert freebsd_committer freebsd_triage 2023-12-09 15:23:36 UTC
(In reply to Alexander Leidinger from comment #31)

Unless Heimdal is different than MIT, your principals must be host/hostname.fqdn@REALM.FQDN, i.e. host/foobar.example.com@EXAMPLE.ORG.

If your the uname is the short name, you will need to do some /etc/hosts gymnastics or create principals with the short name. Been there done that at $((JOB-1)).
Comment 33 Joerg Pulz 2023-12-09 15:30:27 UTC
(In reply to Cy Schubert from comment #30)

Hmm, tried to build with the latest patch on the KDCs to make sure this part is working correctly too.
KDCs currently at releng/13.2.

Fresh checkout of releng/14.0 and clean world directory with latest patch applied results in this:

cc  -O2 -pipe -fno-common -I/usr/src/crypto/heimdal/lib/roken  -I/usr/src/kerberos5/include  -I/usr/src/crypto/heimdal/lib/krb5 -I.  -DHAVE_CONFIG_H -I/usr/src/kerberos5/include -g -gz=zlib -MD  -MF.depend.fbsd_ossl_provider_load.o -MTfbsd_ossl_provider_load.o -std=gnu99 -Wno-format-zero-length -Wsystem-headers -Wno-pointer-sign -Wdate-time -Wno-absolute-value -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-address-of-packed-member -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-parentheses  -Qunused-arguments    -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include -c /usr/src/kerberos5/lib/libroken/fbsd_ossl_provider_load.c -o fbsd_ossl_provider_load.o
/usr/src/kerberos5/lib/libroken/fbsd_ossl_provider_load.c:5:8: error: unknown type name 'OSSL_PROVIDER'
static OSSL_PROVIDER *legacy;
       ^
/usr/src/kerberos5/lib/libroken/fbsd_ossl_provider_load.c:6:8: error: unknown type name 'OSSL_PROVIDER'
static OSSL_PROVIDER *deflt;
       ^
2 errors generated.
*** Error code 1

Stop.
make[3]: stopped in /usr/src/kerberos5/lib/libroken

Missing #include ?
Comment 34 Cy Schubert freebsd_committer freebsd_triage 2023-12-09 15:48:47 UTC
(In reply to Joerg Pulz from comment #33)

I can try building it on releng/14.0. make universe built properly on universe15b; it builds on all architectures supported by FreeBSD. So, unless releng/14.0 is that much different from 15-CURRENT, there is probably something amiss with your build.

I'll try to MFC this to stable/14 locally and MFS it to releng/14.0 (locally) over the weekend and post patches here.
Comment 35 Joerg Pulz 2023-12-09 17:41:03 UTC
(In reply to Cy Schubert from comment #34)

Seems like it is related to the fact that I'm building on releng/13.2 and the wrong OpenSSL includes (those installed in /usr/include) are used - maybe due to building it early in the bootstrap phase?

Due to this, the OpenSSL version covered area in kerberos5/include/crypto-headers.h is never entered and openssl/provider.h not used.

After modifying CFLAGS in kerberos5/lib/libroken/Makefile to look like this:

CFLAGS+=-I${KRB5DIR}/lib/roken \
        -I${SRCTOP}/crypto/openssl/include \
        -I${SRCTOP}/kerberos5/include \
        -I${KRB5DIR}/lib/krb5 -I.

it builds just fine.

This didn't came up earlier as I was testing on a client that was already at releng/14.0 but now I'm building on my KDCs running releng/13.2.

Unrelated to this, there is another build error for the probably uncommon case building base Heimdal using WITH_OPENLDAP=yes - I use the LDAP hdb backend.
The following patch is required in this scenario:

diff --git a/crypto/heimdal/lib/hdb/hdb-ldap.c b/crypto/heimdal/lib/hdb/hdb-ldap.c
index 1b4024aa5..f92522a2f 100644
--- a/crypto/heimdal/lib/hdb/hdb-ldap.c
+++ b/crypto/heimdal/lib/hdb/hdb-ldap.c
@@ -816,7 +816,7 @@ escape_value(krb5_context context, const unsigned char *unquoted, char **quoted)
     }
 
     for (i = 0; unquoted[0] ; unquoted++) {
-       if (need_quote((unsigned char *)unquoted[0])) {
+       if (need_quote((unsigned char)unquoted[0])) {
            (*quoted)[i++] = '\\';
            (*quoted)[i++] = hexchar[(unquoted[0] >> 4) & 0xf];
            (*quoted)[i++] = hexchar[(unquoted[0]     ) & 0xf];

Would be nice if this would hit the tree too.
Comment 36 Cy Schubert freebsd_committer freebsd_triage 2023-12-10 03:16:43 UTC
(In reply to Joerg Pulz from comment #35)

Why apply this patch to 13.2? It doesn't need it. It's got OpenSSL 1.1.1.You do not need to patch 13.2 or anything 13 (or 12).
Comment 37 Joerg Pulz 2023-12-10 04:54:10 UTC
(In reply to Cy Schubert from comment #36)

Misunderstanding here.
I want to update my KDCs currently running 13.2-RELEASE-p7 to releng/14.0.
So on the KDCs I did a fresh checkout of releng/14.0 to /usr/src and applied your patch to the releng/14.0 sources.
Started a make buildworld buildkernel that ended in the error mentioned in comment #33.

So the build error occurs on a system currently running 13 while building 14.
Comment 38 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-10 09:26:45 UTC
(In reply to Cy Schubert from comment #32)

The destination system:
# kadmin -l list -l \*commit\*
            Principal: host/commit.leidinger.net@LEIDINGER.NET
    Principal expires: never
     Password expires: never

In the destination system:
% uname -n
commit.leidinger.net


The source system is similar.
Comment 39 Cy Schubert freebsd_committer freebsd_triage 2023-12-11 21:42:30 UTC
(In reply to Alexander Leidinger from comment #38)

What I did here in my Heimdal jail (I run MIT here) is,

kadmin -l
ank -r host/bob71@EXAMPLE.org     # I used short hostname in this jail
ext host/bob71@EXAMPLE.org
ank cy
^D

I logged in as myself in the jail,

kinit
ssh -K bob71
It used my TGT from the kinit. Without the -K it will use ssh keys and keyboard prompts. You can add GSSAPIAuthentication yes to your .ssh/config to default to GSSAPI thereby negating the need for -K

It works.
Comment 40 Cy Schubert freebsd_committer freebsd_triage 2023-12-11 21:49:53 UTC
Created attachment 246994 [details]
Heimdal - enable weak encryption, builds on 13 when building to upgrade to 14 or 15

This patch builds on stable/13 when upgrading to 14.

This is the version that was ssh tested as well.
Comment 41 Cy Schubert freebsd_committer freebsd_triage 2023-12-11 21:52:46 UTC
(In reply to Cy Schubert from comment #40)

BTW, if people are satisfied with this patch I will submit it for phabricator review.
Comment 42 Cy Schubert freebsd_committer freebsd_triage 2023-12-12 01:10:03 UTC
Created attachment 246998 [details]
Heimdal - enable weak encryption, builds on 13 when building to upgrade to 14 or 15

Fix a minor bug that flags the providers are loaded when they are not due to error.
Comment 43 Alexander Leidinger freebsd_committer freebsd_triage 2023-12-12 10:03:03 UTC
(In reply to Cy Schubert from comment #39)

I believe you that it works. :)
The question for me is why it doesn't work for me.

I rebuild inside src/kerberos5 everything with your latest patch. And I've rebuild sshd and ssh in the same way. All involved systems have the same binaries.

While I have the GSS setting for ssh, I also tried with -K. No success.

The sshd debug log shows this ("unknown mech-code 0"):
Dec 12 10:12:31 commit sshd[37533]: debug1: userauth-request for user netchild service ssh-connection method gssapi-with-mic [preauth]
Dec 12 10:12:31 commit sshd[37533]: debug1:  No credentials were supplied, or the credentials were unavailable or inaccessible.\nunknown mech-code 0 for mech { 1 2 840 113554 1 2 2 }\n\n

After a bit of debugging and searching, adding this to the sshd_config makes it work:
GSSAPIStrictAcceptorCheck no

The reason seems to be that my hostnames in DNS are differently cased as in the kerberos realm and libgssapi seems to be not case insensitive for the hostname. Attention: the setting does more than case insensitivity, it doesn't check the hostname at all but takes any kredential in the keytab as OK (at least to my reading).

It comes down to gss_import_name() from libgssapi in the end. A theoretical solution would be to lowercase all kerberos principals by default (and automatically on creation and/or on any lookup) and do the same in gss_import_name(). No idea if this is a feasible solution.
Comment 44 Joerg Pulz 2023-12-12 12:35:55 UTC
(In reply to Cy Schubert from comment #42)

Cy, I've updated my KDCs from 13.2 to releng/14.0 successfully using your latest patch.
Everything is working as expected and before.

Time to move on to phabricator, at least from my point of view.

Would you please take a look at the end of comment #35.
I ran into an build error using WITH_OPENLDAP=yes when building base Heimdal.
It's a very small change and it's the same change the Heimdal people have done in the Heimdal git repo - I already verified this.

It would be really helpful to get this fixed too.

If required I can open a separate PR for it.
Comment 45 Cy Schubert freebsd_committer freebsd_triage 2023-12-12 20:55:42 UTC
The patch is currently running on all my gear here.

It's been submitted for Phabricator review here: https://reviews.freebsd.org/D43009
Comment 46 commit-hook freebsd_committer freebsd_triage 2024-01-11 13:32:09 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=cb350ba7bf7ca7c4cb97ed2c20ab45af60382cfb

commit cb350ba7bf7ca7c4cb97ed2c20ab45af60382cfb
Author:     Cy Schubert <cy@FreeBSD.org>
AuthorDate: 2023-12-06 15:30:05 +0000
Commit:     Cy Schubert <cy@FreeBSD.org>
CommitDate: 2024-01-11 13:26:42 +0000

    kerberos: Fix numerous segfaults when using weak crypto

    Weak crypto is provided by the openssl legacy provider which is
    not load by default. Load the legacy providers as needed.

    When the legacy provider is loaded into the default context the default
    provider will no longer be automatically loaded. Without the default
    provider the various kerberos applicaions and functions will abort().

    PR:                     272835
    MFC after:              3 days
    Differential Revision:  https://reviews.freebsd.org/D43009
    Tested by:              netchild, Joerg Pulz <Joerg.Pulz@frm2.tum.de>

 Makefile.inc1                                      |  7 ++--
 crypto/heimdal/lib/kadm5/create_s.c                |  4 +++
 crypto/heimdal/lib/kadm5/kadm5_locl.h              |  1 +
 crypto/heimdal/lib/krb5/context.c                  |  4 +++
 crypto/heimdal/lib/krb5/crypto.c                   |  3 ++
 crypto/heimdal/lib/krb5/salt.c                     |  5 +++
 crypto/heimdal/lib/roken/version-script.map        |  1 +
 kerberos5/include/crypto-headers.h                 |  4 +++
 kerberos5/include/fbsd_ossl_provider.h (new)       |  4 +++
 kerberos5/lib/libroken/Makefile                    | 11 ++++--
 .../lib/libroken/fbsd_ossl_provider_load.c (new)   | 41 ++++++++++++++++++++++
 kerberos5/libexec/kdc/Makefile                     |  2 +-
 share/mk/src.libnames.mk                           |  2 +-
 13 files changed, 81 insertions(+), 8 deletions(-)
Comment 47 Cy Schubert freebsd_committer freebsd_triage 2024-01-13 14:23:14 UTC
(In reply to Joerg Pulz from comment #2)

This may be our only solution, because the source patch fails to cross build on Linux and MacOS. On those platforms security/libcrypto/Makefile.inc includes bsd.endian.mk which requires CPU type to be defined. CPU type is not defined on Linux and MacOS cross builds during pre-build phase (whereas it is on native FreeBSD builds of our sources). I think so@ should weigh in on this decision.
Comment 48 Cy Schubert freebsd_committer freebsd_triage 2024-01-15 17:09:12 UTC
Created attachment 247680 [details]
Also circumvent Linux and MacOS cross-build errors

so@ doesn't want us to globally enable weak crypto and the previous patch does not cross-build on Linux or MacOS, though this patch has not been tested with cross-builds yet, it should builds since it doesn't touch Makefile.inc1 nor src.libnames.mk.

This patch dynamically loads/links using dlopen()/dlsym() OpenSSL's libcrypto.so at runtime rather than at link (build) time. Can everyone please test this patch? I won't commit it unless it's tested by at least someone here.
Comment 49 Cy Schubert freebsd_committer freebsd_triage 2024-01-15 17:16:48 UTC
Comment on attachment 247680 [details]
Also circumvent Linux and MacOS cross-build errors

Hold off on applying the patch. It causes a different segfault now.
Comment 50 Cy Schubert freebsd_committer freebsd_triage 2024-01-16 19:08:12 UTC
Created attachment 247710 [details]
Fix numerous Heimdal segfaults when using weak crypto

Can people try this? I can't commit this without testing and sign-off here.
Comment 51 Alexander Leidinger freebsd_committer freebsd_triage 2024-01-16 20:35:07 UTC
I tried this with kinit and the ticket got renewed instead of a segfault.
Comment 52 Cy Schubert freebsd_committer freebsd_triage 2024-01-17 18:47:00 UTC
Created attachment 247730 [details]
Final version of patch before commit.

I removed the include of krb5_locl.h, replacing it with stdlib.h. This fixes the last of the Linux/MacOS build errors. It works well here. Please test and I will commit.
Comment 53 Joerg Pulz 2024-01-18 05:56:23 UTC
(In reply to Cy Schubert from comment #52)

Tested on my KDCs and on several clients - everything is working.

Please go ahead and commit.
Comment 54 commit-hook freebsd_committer freebsd_triage 2024-01-18 07:49:55 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=476d63e091c2e663b51d18acf6acb282e1f22bbc

commit 476d63e091c2e663b51d18acf6acb282e1f22bbc
Author:     Cy Schubert <cy@FreeBSD.org>
AuthorDate: 2023-12-06 15:30:05 +0000
Commit:     Cy Schubert <cy@FreeBSD.org>
CommitDate: 2024-01-18 07:46:57 +0000

    kerberos: Fix numerous segfaults when using weak crypto

    Weak crypto is provided by the openssl legacy provider which is
    not load by default. Load the legacy providers as needed.

    When the legacy provider is loaded into the default context the default
    provider will no longer be automatically loaded. Without the default
    provider the various kerberos applicaions and functions will abort().

    This is the second attempt at this patch. Instead of linking
    secure/lib/libcrypto at build time we now link it at runtime, avoiding
    buildworld failures under Linux and MacOS. This is because
    TARGET_ENDIANNESS is undefined at pre-build time.

    PR:             272835
    MFC after:      3 days
    X-MFC:          only to stable/14
    Tested by:      netchild
                    Joerg Pulz <Joerg.Pulz@frm2.tum.de> (previous version)

 crypto/heimdal/lib/kadm5/create_s.c                |  4 ++
 crypto/heimdal/lib/kadm5/kadm5_locl.h              |  1 +
 crypto/heimdal/lib/krb5/context.c                  |  4 ++
 crypto/heimdal/lib/krb5/crypto.c                   |  3 +
 crypto/heimdal/lib/krb5/salt.c                     |  5 ++
 crypto/heimdal/lib/roken/version-script.map        |  1 +
 kerberos5/include/crypto-headers.h                 |  4 ++
 kerberos5/include/fbsd_ossl_provider.h (new)       |  4 ++
 kerberos5/lib/libroken/Makefile                    |  8 ++-
 .../lib/libroken/fbsd_ossl_provider_load.c (new)   | 77 ++++++++++++++++++++++
 10 files changed, 109 insertions(+), 2 deletions(-)
Comment 55 commit-hook freebsd_committer freebsd_triage 2024-01-22 15:50:23 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=c7db2e15e4045e1daba939bb151fc5878f791c7b

commit c7db2e15e4045e1daba939bb151fc5878f791c7b
Author:     Cy Schubert <cy@FreeBSD.org>
AuthorDate: 2023-12-06 15:30:05 +0000
Commit:     Cy Schubert <cy@FreeBSD.org>
CommitDate: 2024-01-22 15:48:51 +0000

    kerberos: Fix numerous segfaults when using weak crypto

    Weak crypto is provided by the openssl legacy provider which is
    not load by default. Load the legacy providers as needed.

    When the legacy provider is loaded into the default context the default
    provider will no longer be automatically loaded. Without the default
    provider the various kerberos applicaions and functions will abort().

    This is the second attempt at this patch. Instead of linking
    secure/lib/libcrypto at build time we now link it at runtime, avoiding
    buildworld failures under Linux and MacOS. This is because
    TARGET_ENDIANNESS is undefined at pre-build time.

    PR:             272835
    Tested by:      netchild
                    Joerg Pulz <Joerg.Pulz@frm2.tum.de> (previous version)

    (cherry picked from commit 476d63e091c2e663b51d18acf6acb282e1f22bbc)

 crypto/heimdal/lib/kadm5/create_s.c                |  4 ++
 crypto/heimdal/lib/kadm5/kadm5_locl.h              |  1 +
 crypto/heimdal/lib/krb5/context.c                  |  4 ++
 crypto/heimdal/lib/krb5/crypto.c                   |  3 +
 crypto/heimdal/lib/krb5/salt.c                     |  5 ++
 crypto/heimdal/lib/roken/version-script.map        |  1 +
 kerberos5/include/crypto-headers.h                 |  4 ++
 kerberos5/include/fbsd_ossl_provider.h (new)       |  4 ++
 kerberos5/lib/libroken/Makefile                    |  8 ++-
 .../lib/libroken/fbsd_ossl_provider_load.c (new)   | 77 ++++++++++++++++++++++
 10 files changed, 109 insertions(+), 2 deletions(-)
Comment 56 Cy Schubert freebsd_committer freebsd_triage 2024-01-30 05:00:06 UTC
*** Bug 275915 has been marked as a duplicate of this bug. ***
Comment 57 Cy Schubert freebsd_committer freebsd_triage 2024-01-30 05:00:38 UTC
*** Bug 276257 has been marked as a duplicate of this bug. ***
Comment 58 commit-hook freebsd_committer freebsd_triage 2024-03-28 05:07:31 UTC
A commit in branch releng/14.0 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=aaf2c7fdb81a1dd9de9fc77c9313f4e60e68fa76

commit aaf2c7fdb81a1dd9de9fc77c9313f4e60e68fa76
Author:     Cy Schubert <cy@FreeBSD.org>
AuthorDate: 2023-12-06 15:30:05 +0000
Commit:     Gordon Tetlow <gordon@FreeBSD.org>
CommitDate: 2024-03-28 03:14:44 +0000

    kerberos: Fix numerous segfaults when using weak crypto

    Weak crypto is provided by the openssl legacy provider which is
    not load by default. Load the legacy providers as needed.

    When the legacy provider is loaded into the default context the default
    provider will no longer be automatically loaded. Without the default
    provider the various kerberos applicaions and functions will abort().

    This is the second attempt at this patch. Instead of linking
    secure/lib/libcrypto at build time we now link it at runtime, avoiding
    buildworld failures under Linux and MacOS. This is because
    TARGET_ENDIANNESS is undefined at pre-build time.

    PR:             272835
    Tested by:      netchild
                    Joerg Pulz <Joerg.Pulz@frm2.tum.de> (previous version)
    Approved by:    so
    Security:       FreeBSD-EN-24:08.kerberos

    (cherry picked from commit 476d63e091c2e663b51d18acf6acb282e1f22bbc)
    (cherry picked from commit c7db2e15e4045e1daba939bb151fc5878f791c7b)

 crypto/heimdal/lib/kadm5/create_s.c                |  4 ++
 crypto/heimdal/lib/kadm5/kadm5_locl.h              |  1 +
 crypto/heimdal/lib/krb5/context.c                  |  4 ++
 crypto/heimdal/lib/krb5/crypto.c                   |  3 +
 crypto/heimdal/lib/krb5/salt.c                     |  5 ++
 crypto/heimdal/lib/roken/version-script.map        |  1 +
 kerberos5/include/crypto-headers.h                 |  4 ++
 kerberos5/include/fbsd_ossl_provider.h (new)       |  4 ++
 kerberos5/lib/libroken/Makefile                    |  8 ++-
 .../lib/libroken/fbsd_ossl_provider_load.c (new)   | 77 ++++++++++++++++++++++
 10 files changed, 109 insertions(+), 2 deletions(-)