Bug 275241 - GSS-API aware nsupdate segfaults
Summary: GSS-API aware nsupdate segfaults
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 14.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: Mathieu Arnold
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-11-21 19:40 UTC by Michal Nowak
Modified: 2024-01-15 12:51 UTC (History)
4 users (show)

See Also:
linimon: maintainer-feedback? (mat)

Attachments
tarball with config files (5.86 KB, application/x-xz)
2023-11-21 19:40 UTC, Michal Nowak 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Michal Nowak 2023-11-21 19:40:19 UTC 
Created attachment 246472 [details]
tarball with config files

When built with `--with-gssapi`, the `nsupdate` command from BIND 9.18 and 9.16 segfaults in FreeBSD 14.0 gss, spnego, and kerberos library stack. BIND 9.19 (the `main` branch) is not affected. (First identified in https://gitlab.isc.org/isc-projects/bind9/-/issues/4436 when adding FreeBSD 14.0 to the BIND 9 CI.)

git clone https://gitlab.isc.org/isc-projects/bind9.git and check out the `bind-9.18` branch.

```
autoreconf -fi
./configure --enable-developer --with-gssapi
make -j5
cd bin/tests/system/
sudo ./ifconfig.sh up   # setup interfaces for named (i.e., 10.53.0.1)
```

Unpack the issue-4436.tar.xz tarball.

Start `named`:
```
~/bind9/bin/named/named -c named.conf -d 99 -g
```

Run the BIND 9.18 `nsupdate` command with the correct paths to the credential cache and file with `nsupdate` commands:
```
KRB5CCNAME="FILE:"/home/newman/issue-4436/administrator.ccache ~/bind9/bin/nsupdate/nsupdate -g ~/issue-4436/update.txt
```

I get: `Segmentation fault (core dumped)`.

Here's a sample GDB backtrace from the `tsiggss` system test (bin/tests/system/tsiggss/):
```
Core was generated by `/root/bind9/bin/nsupdate/.libs/nsupdate -g -d ns1/update.txt'.
Program terminated with signal SIGSEGV, Segmentation fault.
Address not mapped to object.
#0  0x00000008316a1a0f in EVP_Cipher () from /lib/libcrypto.so.30
[Current thread is 1 (LWP 188477)]
#0  0x00000008316a1a0f in EVP_Cipher () from /lib/libcrypto.so.30
#1  0x000000082e96f4b6 in ?? () from /usr/lib/libkrb5.so.11
#2  0x000000082e973ac8 in krb5_encrypt_ivec () from /usr/lib/libkrb5.so.11
#3  0x000000082e973de5 in krb5_encrypt () from /usr/lib/libkrb5.so.11
#4  0x000000082e9675bf in _krb5_build_authenticator () from /usr/lib/libkrb5.so.11
#5  0x000000082dcff3f6 in ?? () from /usr/lib/libgssapi_krb5.so.10
#6  0x000000082dcfed0b in _gsskrb5_init_sec_context () from /usr/lib/libgssapi_krb5.so.10
#7  0x000000082d95bd4f in gss_init_sec_context () from /usr/lib/libgssapi.so.10
#8  0x000000083ed613b6 in ?? () from /usr/lib/libgssapi_spnego.so.10
#9  0x000000083ed5f5c0 in _gss_spnego_indicate_mechtypelist () from /usr/lib/libgssapi_spnego.so.10
#10 0x000000083ed607ee in _gss_spnego_init_sec_context () from /usr/lib/libgssapi_spnego.so.10
#11 0x000000082d95bd4f in gss_init_sec_context () from /usr/lib/libgssapi.so.10
#12 0x0000000822a308e5 in dst_gssapi_initctx (name=<optimized out>, intoken=intoken@entry=0x0, outtoken=outtoken@entry=0x83d56d700, gssctx=0x83d56e218, mctx=0x1aef866b3000, err_message=0x83d56e200) at gssapictx.c
#13 0x0000000822b0c9af in dns_tkey_buildgssquery (msg=0x1aef87203a80, name=0x2130e0 <fkname>, gname=0x1aef87234300, gname@entry=0x83d56d7a0, intoken=0x1aef872700f0, intoken@entry=0x0, lifetime=lifetime@entry=0, context=0xcf, context@entry=0x83d56e218, win2k=<optimized out>, mctx=0x1aef866b3000, err_message=0x83d56e200) at tkey.c
#14 0x000000000020e790 in start_gssrequest (primary=primary@entry=0x83d56e730) at nsupdate.c
#15 0x000000000020e33c in recvsoa (task=<optimized out>, event=0x0) at nsupdate.c
#16 0x0000000821c68370 in task_run (task=0x1aef8665c140) at task.c
#17 isc_task_run (task=0x1aef8665c140) at task.c
#18 0x0000000821c38689 in isc__nm_async_task (worker=worker@entry=0x1aef866d0000, ev0=0x1aef872700f0, ev0@entry=0x1aef8721c480) at netmgr/netmgr.c
#19 0x0000000821c32ec6 in process_netievent (worker=worker@entry=0x1aef866d0000, ievent=ievent@entry=0x1aef8721c480) at netmgr/netmgr.c
#20 0x0000000821c384f2 in process_queue (worker=worker@entry=0x1aef866d0000, type=type@entry=NETIEVENT_TASK) at netmgr/netmgr.c
#21 0x0000000821c2e6bd in process_all_queues (worker=0x1aef866d0000) at netmgr/netmgr.c
#22 async_cb (handle=0x1aef866d02d8) at netmgr/netmgr.c
#23 0x0000000829b3c871 in ?? () from /usr/local/lib/libuv.so.1
#24 0x0000000829b4e0fd in ?? () from /usr/local/lib/libuv.so.1
#25 0x0000000829b3ce60 in uv_run () from /usr/local/lib/libuv.so.1
#26 0x0000000821c2e7ab in nm_thread (worker0=0x1aef866d0000) at netmgr/netmgr.c
#27 0x0000000821c70e46 in isc__trampoline_run (arg=0x1aef8662bb90) at trampoline.c
#28 0x00000008376e0a75 in ?? () from /lib/libthr.so.3
#29 0x0000000000000000 in ?? ()
```

The crash seems to happen not in the BIND 9 code but deep in the FreeBSD 14.0 stack.
Comment 1 Michael Osipov freebsd_committer freebsd_triage 2023-11-22 11:14:23 UTC 
This seems to affect Heimdal in base. Is MIT Kerberos or Heimdal from ports also affected? I wonder wether this is a problem with old Heimdal not properly working with OpenSSL 3.x...
Comment 2 Michal Nowak 2023-11-22 14:43:10 UTC 
krb5: 1.21.2 from ports: BIND 9 configured with --with-gssapi=/usr/local/bin/krb5-config
libraries... -I/usr/local/include -L/usr/local/lib -lkrb5 -lk5crypto -lcom_err

`nsupdate` does not crash; all tests pass.

---

heimdal 7.8.0_6 from ports: BIND 9 configured with --with-gssapi=/usr/local/bin/krb5-config
checking for krb5 libraries... -I/usr/local/include/heimdal -Wl,--enable-new-dtags -Wl,-rpath -Wl,/usr/local/lib/heimdal -L/usr/local/lib/heimdal -lkrb5 -lhx509 -lcom_err -lhcrypto -lasn1 -lwind -lheimbase -lroken -lcrypt -lpthread -lpthread -L/usr/local/lib

`nsupdate` does not crash, but clearly does not work with BIND 9 as the nsupdate and tsiggss system tests fail; `nsupdate` reports the following on the UPDATE command: "tkey query failed: GSSAPI error: Major =  An unsupported mechanism was requested, Minor = unknown mech-code 0 for mech unknown."
Comment 3 Michael Osipov freebsd_committer freebsd_triage 2023-11-22 15:00:53 UTC 
(In reply to Michal Nowak from comment #2)

This is totally weird, isn't it? The mechanism is plain Kerberos...

Do yo have a chance to use MIT Kerberos from packages for the BIND CI setup? I have been using nsupdate with GSS-TSIG and MIT for past 5+ years w/o any issues...
Comment 4 Michal Nowak 2023-11-22 15:33:35 UTC 
(In reply to Michael Osipov from comment #3)

Yes, we could install MIT Kerberos from Ports to our CI image and configure it for the FreeBSD 14 CI build, but currently, we lean towards disabling Kerberos support in the FreeBSD 14 CI build until it is resolved in the base OS as we prefer to use whatever the OS provides by default.
Comment 5 Michael Osipov freebsd_committer freebsd_triage 2023-11-22 15:36:57 UTC 
(In reply to Michal Nowak from comment #4)

I see and this makes sense. Ideally, you would test all the three because as we have seen Heimdal is broken altogether.
Comment 6 Cy Schubert freebsd_committer freebsd_triage 2023-11-22 16:35:41 UTC 
(In reply to Michal Nowak from comment #4)

I am working on the labour intensive task of replacing Heimdal 1.5.2 in base (7.9.0 is latest) with MIT 1.21.2. It is suggested you use MIT instead of Heimdal in base because even though Heimdal is a drop-in replacement for MIT, they have incompatible kadmin protocols. (The only reason FreeBSD chose to use Heimdal at the time is the same reason OpenBSD chose to base themselves in Calgary AB, Canada. This no longer applies for many years.)

Heimdal 1.5.2 is ancient. While replacing it in base is a labour intensive task of reverse engineering the MIT GNU build process to create dozens of Makfiles by hand for the bespoke BSD based FreeBSD build system. It's a very slow process.

Go with MIT. It will replace Heimdal in base and you will avoid some of the pain of having to convert Heimdal KDC to MIT when time comes.

BTW, I use MIT here. I started using it here about 25 years ago as a POC (proof of concept) when I worked at $((JOB-1)). I implemented it there using FreeBSD infrastructure servers. After I left there it was subsequently replaced by Microsoft's Active Directory, which itself is based on MIT KRB5 and LDAP. Another reason to switch to MIT.
Comment 7 Michael Osipov freebsd_committer freebsd_triage 2023-11-22 19:54:43 UTC 
(In reply to Cy Schubert from comment #6)

Cy, would it make sense only to import MIT's client side libs into base? A KDC shouldn't be in base, no?
Comment 8 Cy Schubert freebsd_committer freebsd_triage 2023-11-22 20:07:09 UTC 
(In reply to Michael Osipov from comment #7)

That would not solve the Heimdal KDC problem. Heimdal users would need to install security/heimdal regardless.
Comment 9 Michael Osipov freebsd_committer freebsd_triage 2023-11-23 07:29:05 UTC 
(In reply to Cy Schubert from comment #8)

While this is correct I consider that only little need a KDC in base, just line the issue with BIND. This is special software and shouldn't be bundled, IMHO.
Comment 10 Michal Nowak 2023-11-27 10:45:55 UTC 
For the bind918 and bind916 Port maintainer, nsupdate from binary Ports is not affected because, by default, it's built with GSSAPI_NONE. Should someone rebuild it with GSSAPI_BASE=on, it would segfault. GSSAPI_HEIMDAL=on won't work, but neither will crash. GSSAPI_MIT=on will work. Just in case you want to limit GSSAPI choices for FreeBSD 14.0 bind918 and bind916 Ports.
Comment 11 Michael Osipov freebsd_committer freebsd_triage 2023-11-27 12:16:10 UTC 
(In reply to Michal Nowak from comment #10)

So both OPTIONs need to be removed (Heimdal base + ports).
Comment 12 commit-hook freebsd_committer freebsd_triage 2024-01-14 08:51:25 UTC 
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=0646643ef51ba3867d5dfaabc13fcec111e74abe

commit 0646643ef51ba3867d5dfaabc13fcec111e74abe
Author:     Mathieu Arnold <mat@FreeBSD.org>
AuthorDate: 2024-01-14 08:49:03 +0000
Commit:     Mathieu Arnold <mat@FreeBSD.org>
CommitDate: 2024-01-14 08:50:10 +0000

    dns/bind9*: add a comment about nsupdate being broken with heimdal

    PR:     275241

 dns/bind9-devel/Makefile | 4 ++--
 dns/bind916/Makefile     | 4 ++--
 dns/bind918/Makefile     | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)
Comment 13 Michal Nowak 2024-01-15 12:51:25 UTC 
BIND 9.19 is not affected, so the bind-devel port Makefile comment (https://cgit.freebsd.org/ports/diff/dns/bind9-devel/Makefile?id=0646643ef51ba3867d5dfaabc13fcec111e74abe) is unnecessary.