freebsd-update upgrade -r 12.0-BETA[12] does not execute pwd_mkdb. If we are using ntpd on 11.2-RELEASE or before, freebsd-update upgrade adds ntpd/ntpd to user/group but does not execute pwd_mkdb. So we must execute 'pwd_mkdb -p /etc/master.passwd' by hands.
This bug makes upgrading the system with source impossible since `make installworld’ now checks for the presence of the ntpd user (with uid tool). Running the command given in comment #1 prior to installworld fixes it.
This still seems to be an issue with upgrades using freebsd-update. I updated to, if I recall correctly, 12.0-RC1 several weeks ago and got an error on boot "pid file /var/db/ntp/ntpd.pid: Permission denied". The directions above fixed it. I skipped RC2 and updated to RC3 today from RC1 and got the error again. I have another system I've been updating via source, and I haven't had this issue on the source-based system.
I also encountered this issue doing a binary upgrade and it can cause a bit of trouble: If pwd_mkdb is not run, ntpd will still run as root since the user ntpd is unknown. On the very first start, the rc.d script will move the drift file to /var/db/ntp and also place the .pid file there. Both files will be owned by root at this point. If you now run pwd_mkdb to create user ntpd, the daemon won't be properly started next time since user ntpd cannot write to the drift and the .pid file. This can be solved by simply removing the files or fixing the permissions but it may be worth noting this in the release notes or upgrade instructions.
Google first led me to the same bug reported in bug #234014 - providing a back reference to make it easier to resolve both at the same time. This is still an issue in 12.0-RELEASE when upgrading an 11.x system with freebsd-update. The proposed solution (pwd_mkdb) resolved the issue for me. I wanted to add a few odd coincidences caused by this bug, maybe it helps someone. I use openntpd instead of the system ntpd and I have _ntp user on my system. The above bug manifested when I tried to setup synth (ports-mgmt/synth) with the following mtree error: raised REPLICANT.SCENARIO_UNEXPECTED : /usr/sbin/mtree -p /usr/obj/synth-live/SL09/var -f /etc/mtree/BSD.var.dist -deqU mtree file requires /var/db/ntp to have the following ownership: ntp uname=ntpd gname=ntpd However, my system had _ntp:ntpd owning /var/db/ntp. I don't know if this is an earlier change (though openntpd doesn't seem to be using /var/db/ntp), something done during the upgrade (maybe due to finding _ntp user and using that user instead of ntpd) or the default permissions for the directory created by the upgrade. Issuing an errata since this is present in -RELEASE and freebsd-update is a supported upgrade option seems like a good idea.
A commit references this bug: Author: emaste Date: Mon Jan 21 19:04:38 UTC 2019 New revision: 343270 URL: https://svnweb.freebsd.org/changeset/base/343270 Log: freebsd-update: Update /etc/passwd after password db changes Add -p to pwd_mkdb in order to ensure password db changes are also included in /etc/passwd. PR: 165954, 232921, 229487 Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com> Reviewed by: jilles MFC after: 1 month Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D18827 Changes: head/usr.sbin/freebsd-update/freebsd-update.sh
A commit references this bug: Author: emaste Date: Tue Mar 5 18:29:26 UTC 2019 New revision: 344809 URL: https://svnweb.freebsd.org/changeset/base/344809 Log: MFC r343270: freebsd-update: Update /etc/passwd after password db changes Add -p to pwd_mkdb in order to ensure password db changes are also included in /etc/passwd. PR: 165954, 232921, 229487 Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com> Reviewed by: jilles Sponsored by: The FreeBSD Foundation Changes: _U stable/12/ stable/12/usr.sbin/freebsd-update/freebsd-update.sh
A commit references this bug: Author: emaste Date: Tue Mar 5 18:30:43 UTC 2019 New revision: 344810 URL: https://svnweb.freebsd.org/changeset/base/344810 Log: MFC r343270: freebsd-update: Update /etc/passwd after password db changes Add -p to pwd_mkdb in order to ensure password db changes are also included in /etc/passwd. PR: 165954, 232921, 229487 Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com> Reviewed by: jilles Sponsored by: The FreeBSD Foundation Changes: _U stable/11/ stable/11/usr.sbin/freebsd-update/freebsd-update.sh
I'm don't think I understand why it was still necessary, but I had to recover from this after a freebsd-update from 11.3-RELEASE-p3 to 12.1-RELEASE-p0. I had to A) manually run pwd_mkdb, and B) forcibly delete /var/db/ntp/ntpd.pid.
This is still an issue with 12.1-RELEASE-p7 and -p8: I too upgraded several systems from 11.3 to 12.1 and on each of them I ran into the problem that ntpd was broken after the upgrade: # freebsd-version 12.1-RELEASE-p7 # finger ntpd finger: ntpd: no such user # id ntpd id: ntpd: no such user mail2:/etc# grep ntpd /etc/passwd ntpd:*:123:123:NTP Daemon:/var/db/ntp:/usr/sbin/nologin mail2:/etc# ll /etc/passwd /etc/master.passwd /etc/spwd.db /etc/pwd.db -rw------- 1 root wheel 3256 Aug 5 14:37 /etc/master.passwd -rw-r--r-- 1 root wheel 2862 Aug 5 14:37 /etc/passwd -rw-r--r-- 1 root wheel 40960 Aug 5 14:37 /etc/pwd.db -rw------- 1 root wheel 40960 Aug 5 14:37 /etc/spwd.db # pwd_mkdb -p /etc/master.passwd # ll /etc/passwd /etc/master.passwd /etc/spwd.db /etc/pwd.db -rw------- 1 root wheel 3256 Aug 5 14:37 /etc/master.passwd -rw-r--r-- 1 root wheel 2862 Aug 6 11:23 /etc/passwd -rw-r--r-- 1 root wheel 40960 Aug 6 11:23 /etc/pwd.db -rw------- 1 root wheel 40960 Aug 6 11:23 /etc/spwd.db # id ntpd uid=123(ntpd) gid=123(ntpd) groups=123(ntpd) # finger ntpd Login: ntpd Name: NTP Daemon Directory: /var/db/ntp Shell: /usr/sbin/nologin No Mail. No Plan.
FWIW this also happens when upgrading an 11.4-RELEASE system to 12.1-RELEASE.
I noticed this from the the "user disappeared during update" behavior mentioned in bug #235766. I had this problem on three systems, one free, 12.1-RELEASE -> 12.2-RELEASE with freebsd-update.
I also can confirm this. A quick run of vipw or pwd_mkdb fixes the issue.
A commit references this bug: Author: kevans Date: Thu Dec 17 03:42:54 UTC 2020 New revision: 368713 URL: https://svnweb.freebsd.org/changeset/base/368713 Log: freebsd-update: unconditionally regenerate passwd/login.conf files The existing logic is nice in theory, but in practice freebsd-update will not preserve the timestamps on these files. When doing a major upgrade, e.g. from 12.1-RELEASE -> 12.2-RELEASE, pwd.mkdb et al. appear in the INDEX and we clobber the timestamp several times in the process of packaging up the existing system into /var/db/freebsd-update/files and extracting for comparisons. This leads to these files not getting regenerated when they're most likely to be needed. Measures could be taken to preserve timestamps, but it's unclear whether the complexity and overhead of doing so is really outweighed by the marginal benefit. I observed this issue when pkg subsequently failed to install a package that wanted to add a user, claiming that the user was removed in the process. bapt@ pointed to this pre-existing bug with freebsd-update as the cause. PR: 234014, 232921 Reviewed by: bapt, emaste MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D27635 Changes: head/usr.sbin/freebsd-update/freebsd-update.sh
*** Bug 235766 has been marked as a duplicate of this bug. ***
*** Bug 234014 has been marked as a duplicate of this bug. ***
Sorry, I'd completely dropped the ball here -- this was MFC'd to stable/11 and stable/12 back in December, and I've just now filled out the EN template to bring it to 11.4 and 12.2.
Yesterday's <https://cgit.freebsd.org/src/commit/?id=9b6591109e8> noted with thanks. (In reply to Kyle Evans from comment #16) > … to bring it to 11.4 and 12.2. For now: if a user performs a minor upgrade from non-patched 12.1-RELEASE to 12.2-RELEASE-p3 _with the intention of upgrading to_ 13.0-⋯ … … will be it necessary/appropriate to manually run: /usr/sbin/pwd_mkdb -p /etc/master.passwd – BEFORE the major upgrade? freebsd-update upgrade -r 13.0-BETA3 ---- Put another way: will the benefit of 9b6591109e8 not reach 12.2 until 12.2-RELEASE-p4 (patch level 4)? Apologies for not getting my head around these things.
(In reply to Graham Perrin from comment #17) You run it after the upgrade, which installs the generated files. The commit you noted is from over a year ago, and appeared in both 12.1 and 12.2.
(In reply to Kyle Evans from comment #18) > … The commit you noted is from over a year ago, … Sorry! (Truly, I looked at 2019-01-21 umpteen times and misread it as 21st February 2021. How embarrassing. Something that I read yesterday linked to it … I must have become word-blind.) Recently I have ebebc41e4cfe44b8e8fd881badf2fa2c4be65aa4 <https://cgit.freebsd.org/src/commit/?id=ebebc41e4cfe44b8e8fd881badf2fa2c4be65aa4> on my mind, also for pwd_mkdb
A commit in branch releng/11.4 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=2ca137b4306dea2dbe1db31c44102060caedb19a commit 2ca137b4306dea2dbe1db31c44102060caedb19a Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2020-12-17 03:42:54 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2021-02-24 01:34:51 +0000 MFC freebsd-update: unconditionally regenerate passwd/login.conf files The existing logic is nice in theory, but in practice freebsd-update will not preserve the timestamps on these files. When doing a major upgrade, e.g. from 12.1-RELEASE -> 12.2-RELEASE, pwd.mkdb et al. appear in the INDEX and we clobber the timestamp several times in the process of packaging up the existing system into /var/db/freebsd-update/files and extracting for comparisons. This leads to these files not getting regenerated when they're most likely to be needed. Measures could be taken to preserve timestamps, but it's unclear whether the complexity and overhead of doing so is really outweighed by the marginal benefit. I observed this issue when pkg subsequently failed to install a package that wanted to add a user, claiming that the user was removed in the process. bapt@ pointed to this pre-existing bug with freebsd-update as the cause. PR: 234014, 232921 Approved by: so Security: FreeBSD-EN-21:08.freebsd-update (cherry picked from commit ebebc41e4cfe44b8e8fd881badf2fa2c4be65aa4) (cherry picked from commit cd7da1deb581122c94c3735b78fafdd04ce77b67) usr.sbin/freebsd-update/freebsd-update.sh | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-)
A commit in branch releng/12.2 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=df378912ebc2fe2603fcbcedf72af9d7b0e94db7 commit df378912ebc2fe2603fcbcedf72af9d7b0e94db7 Author: Kyle Evans <kevans@FreeBSD.org> AuthorDate: 2020-12-17 03:42:54 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2021-02-24 01:42:01 +0000 MFC freebsd-update: unconditionally regenerate passwd/login.conf files The existing logic is nice in theory, but in practice freebsd-update will not preserve the timestamps on these files. When doing a major upgrade, e.g. from 12.1-RELEASE -> 12.2-RELEASE, pwd.mkdb et al. appear in the INDEX and we clobber the timestamp several times in the process of packaging up the existing system into /var/db/freebsd-update/files and extracting for comparisons. This leads to these files not getting regenerated when they're most likely to be needed. Measures could be taken to preserve timestamps, but it's unclear whether the complexity and overhead of doing so is really outweighed by the marginal benefit. I observed this issue when pkg subsequently failed to install a package that wanted to add a user, claiming that the user was removed in the process. bapt@ pointed to this pre-existing bug with freebsd-update as the cause. PR: 234014, 232921 Approved by: so Security: FreeBSD-EN-21:08.freebsd-update (cherry picked from commit ebebc41e4cfe44b8e8fd881badf2fa2c4be65aa4) (cherry picked from commit 5cf1ba21eebaf89cf4d8639a18345dc49116d1d5) usr.sbin/freebsd-update/freebsd-update.sh | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-)
@Kyle Anything left in order resolve/close this issue?
(In reply to Kubilay Kocak from comment #22) Resolve # /usr/sbin/pwd_mkdb -p /etc/master.passwd # pkill ntpd # rm /var/db/ntp/ntpd.pid # service ntpd start