I upgraded two servers to FreeBSD 12.4-RELEASE using freebsd-update as soon as it became available. After rebooting, ntpd fails to start: +FreeBSD 12.4-RELEASE r372781 GENERIC amd64 +FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) +CPU: Intel Xeon E312xx (Sandy Bridge) (4245.58-MHz K8-class CPU) +Launching APs: 2 1 3 +pid 695 (ntpd), jid 0, uid 123: exited on signal 11 I ran 'freebsd-update install' again as is mandatory and then tried to start the daemon myself: $ /usr/sbin/ntpd -n 6 Dec 10:41:24 ntpd[39682]: ntpd 4.2.8p15-a (1): Starting 6 Dec 10:41:24 ntpd[39682]: Command line: /usr/sbin/ntpd -n 6 Dec 10:41:24 ntpd[39682]: ---------------------------------------------------- 6 Dec 10:41:24 ntpd[39682]: ntp-4 is maintained by Network Time Foundation, 6 Dec 10:41:24 ntpd[39682]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 6 Dec 10:41:24 ntpd[39682]: corporation. Support and training for ntp-4 are 6 Dec 10:41:24 ntpd[39682]: available at https://www.nwtime.org/support 6 Dec 10:41:24 ntpd[39682]: ---------------------------------------------------- Segmentation fault This happens on both (identical) systems. Please let me know what additional information to provide in order to solve this!
Thanks for the report. Were both servers updated prior to upgrade?
Yes, I keep them up to date. It appears that ntpd broke down when we went to 12.3-RELEASE-p10 last week (22/11/30). Silly that I didn't notice before.
I was unable to reproduce the crash on a 12.4-RELEASE upgrade from 12.0. Do you happen have a core file from ntpd?
Absolutely. Please download it from here: https://www.netraam.nl/od/ntpd.core.tbz
(In reply to Maarten de Vries from comment #4) Ah, the core file doesn't have symbols (of course, it's a release build). That makes it difficult to figure out what's going on. If you're up for it, you can compile ntpd with debugging symbols and execute it on your 12.4-RELEASE system. This should produce a core file that may point to what's failing. Here's a simple procedure: If you didn't install src, download src.txz from: https://download.freebsd.org/ftp/releases/amd64/amd64/12.4-RELEASE/src.txz untar it in /usr/src: tar -xvC / -f src.txz Check /usr/src/UPDATING and verify the first entry says: 20221205: 12.4-RELEASE. compile ntpd: cd /usr/src/usr.sbin/ntp make execute ntpd with debugging symbols: /usr/obj/usr/src/amd64.amd64/usr.sbin/ntp/ntpd/ntpd -n debug the core file: lldb --core ntpd.core /usr/obj/usr/src/amd64.amd64/usr.sbin/ntp/ntpd/ntpd
Created attachment 238583 [details] Disable ASLR stack gap. Can you apply this patch and rebuild ntp, please. If this is not possible (prefer you use the patch), add the following to rc.conf. ntpd_prepend="/usr/bin/proccontrol -m aslr -s disable" Applying the patch is preferred. I haven't been watching the stable/12 branch but if this fixes the problem, then someone has merged a prior version of ASLR stack gap code into the kernel, as this is no longer needed on 13 or 14 since ASLR is on something like version 3 or 4 of the code, and wasn't needed on 12 since it didn't have ASLR. If either of the above fixes this, it will tell us that some earlier version of ASLR is in the 12.4 kernel.
(In reply to John Grafton from comment #5) Love to help. Been a while since I compiled stuff, but here is the core dump from a freshly compiled ntp daemon: https://www.netraam.nl/od/new_ntpd.core.tbz
(In reply to Maarten de Vries from comment #7) Unfortunately a 12.4-RELEASE core dump will display symbols properly. Fortunately, we don't need symbols since the stack is corrupted. A corrupted stack is an ASLR stack gap issue. Use the patch or the rc.conf workaround.
(In reply to Cy Schubert from comment #6) The patch applied successfully but then the binary still crashed. However, adding that line to rc.conf and then starting ntpd did work!
This suggests that PIE has been MFCed to stable/12 (kernel) but not the fix to PIE that avoids this problem. Without looking at the 12.4 kernel more closely I doubt the additional work to ASLR/PIE can be merged into stable/12 meaning the only fix is the rc.conf workaround.