when rebooting to 13.1-RC4 for the first time and trying to connect via ssh I get on the client a "kex_exchange_identification: read: Connection reset by peer" - this appears after a few hours of running, not immediately. On the server I see: sshd[xxx]: fatal: recv_rexec_state: parse config: incomplete message
> rebooting to 13.1-RC4 for the first time Was this system running a different version of FreeBSD previously? The error message makes me wonder if a configuration file wasn't updated/merged properly when upgrading.
I went from 13.0-RELEASE p11 to RC4 ... this happened on both my test systems (different hardware). Will try to reproduce that today/tomorrow
had another instance today, again different hardware. Now with some more details: *) my sshd_config only uses ed25519, having the line "HostKey /etc/ssh/ssh_host_ed25519_key" in my config *) reconnection after upgrade to RC4 works, after the finishing process of executing "freebsd-update install" it seems I cannot connect any more. *) dmesg: Apr 26 13:56:16 hpnb2 sshd[32523]: fatal: recv_rexec_state: parse config: incomplete message Apr 26 13:56:20 hpnb2 sshd[33082]: fatal: recv_rexec_state: parse config: incomplete message *) before rebooting I switched loglevel to DEBUG in sshd_config, and I got: Apr 26 13:56:16 hpnb2 sshd[37588]: debug1: fd 5 clearing O_NONBLOCK Apr 26 13:56:16 hpnb2 sshd[37588]: debug1: Forked child 32523. Apr 26 13:56:16 hpnb2 sshd[32523]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8 This just happens once, so a restart of the sshd service solves the problem, however, imho this should be done automatically when running freebsd-update after the reboot, otherwise I guess quite some users will forget to manually restart the service and will be locked out if they only have remote access.
I did a stock installation of RELEASE in bhyve VM (booted via uefi; NOT bhyveload), upgraded to RC4, had just a standard install with nothing changed, sshd_config left as it was after install. Bug was again triggered after issuing "freebsd-update install" after the reboot!
out of interest I tried to find out if I missed the bug with the recent upgrades, so I did the following upgrades: 12.2 -> 12.3 -> 13.0: they all went fine. 13.0 -> 13.0-BETA1 is the first time I encountered this issue.
Can you confirm that this occurs only once after freebsd-update, and `service sshd restart` is sufficient to recover?
yes, I can confirm
sorry, comment 5 is wrong ... first appearance I did realize in 13.1-BETA1 not 13.0-BETA1. Connection on all my 7 test machines was fine once I did `service sshd restart` after the (reboot + `freebsd-update install`). As a side note, it just appears on upgrades from FreeBSD <= 13.0 to 13.1-XX - so a update from e.g. 13.1-RC3 to RC4 does not trigger the bug
Here's a report of this issue from Arch Linux https://bugs.archlinux.org/task/65517 The issue is upgrading across OpenSSH 8.2, so indeed it will appear the first time upgrading to any 13.1 beta/RC, and not again from there to any later 13.1 beta/RC/release.
any chance we get a fix for RC5?
just tested RC5 - same unwanted defect
in case this is not being worked on: please include a message in the installation instructions
I upgrade using source from 13.0 to 13.1 : gitup release cd /usr/src make -j1 buildworld make -j1 buildkernel make installkernel etcupdate -p make installworld etcupdate make -DBATCH_DELETE_OLD_FILES delete-old make -DBATCH_DELETE_OLD_FILES delete-old-libs rm -fr /usr/obj/usr /usr/src After these commands I can't connect with a new ssh connection but my current ssh connection still works. If I reboot then ssh works again. Are we talking about the same issue?
(In reply to Christos Chatzaras from comment #13) Yes this is the same issue. After upgrading from OpenSSH < 8.2 to >= 8.2 it is necessary to run `service sshd restart` (or reboot).
Review adding a caution to the release notes: https://reviews.freebsd.org/D35108
Also note that UPDATING specifies a reboot after installing the new userland: make buildworld make buildkernel KERNCONF=YOUR_KERNEL_HERE make installkernel KERNCONF=YOUR_KERNEL_HERE [1] <reboot in single user> [3] etcupdate -p [5] make installworld etcupdate -B [4] make delete-old [6] <reboot> I thought this was also the case for FreeBSD-update, but it does not appear to be so. Review to have freebsd-update automatically restart sshd: https://reviews.freebsd.org/D35109
(In reply to Ed Maste from comment #16) I know but always I do it with one reboot since the 7.x days. So far no issues.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/doc/commit/?id=d335263b28a62ce379baf871918873e1ea038a12 commit d335263b28a62ce379baf871918873e1ea038a12 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-05-02 17:00:35 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2022-05-02 18:23:39 +0000 relnotes: Mention sshd restart issue PR: 263489 Reviewed by: gjb (earlier) Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D35108 website/content/en/releases/13.1R/relnotes.adoc | 6 ++++++ 1 file changed, 6 insertions(+)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=6cd1bc53160973fc421c59f66aaa7e4b37a8cebe commit 6cd1bc53160973fc421c59f66aaa7e4b37a8cebe Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-05-02 17:16:19 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2022-05-02 18:38:19 +0000 freebsd-update: restart sshd after upgrade Sometimes the parent-child sshd protocol changes during an upgrade, and when this happens sshd will not accept new connections until it is restarted. PR: 263489 Reviewed by: kevans, gjb MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D35109 usr.sbin/freebsd-update/freebsd-update.sh | 8 ++++++++ 1 file changed, 8 insertions(+)
please include in RC6
(In reply to rob2g2 from comment #20) For 13.1 the manual restart will be required.
> Blocks: 264030 There's nothing more to do about this in 13.1: the release notes document that restarting sshd is necessary, and it is not possible to retroactively change freebsd-update (in 12.x/13.0) to automatically restart sshd.
(In reply to Ed Maste from comment #22) FYI, this just happened to me on an update using 12/stable - to 12.3-stable as of 20220804 from 12.3-stable as of 20211104. In 12/stable, openssh was updated to 8.8p1 on 20211219, so that makes sense. As with the OP's problem, 'sudo service sshd restart' fixed it. When 12.4-RELEASE happens, those updating from 12.3-RELEASE or earlier will need to reboot or restart sshd. It might be nice to get that sshd restart in freebsd-update, I suppose.
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b296e60ed35ef9936dbe673e4966da439841b320 commit b296e60ed35ef9936dbe673e4966da439841b320 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2022-05-02 17:16:19 +0000 Commit: Eugene Grosbein <eugen@FreeBSD.org> CommitDate: 2022-12-10 04:56:24 +0000 freebsd-update: restart sshd after upgrade Sometimes the parent-child sshd protocol changes during an upgrade, and when this happens sshd will not accept new connections until it is restarted. PR: 263489 Reviewed by: kevans, gjb MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D35109 (cherry picked from commit 6cd1bc53160973fc421c59f66aaa7e4b37a8cebe) usr.sbin/freebsd-update/freebsd-update.sh | 8 ++++++++ 1 file changed, 8 insertions(+)
Note that the same issue happens when upgrading from FreeBSD 12.3 to 12.4 (when you do not restart ssd or reboot).
I have checked /usr/sbin/freebsd-update in 12.4 and the fix to restart sshd is not there. I believe it should be backported to 12.4 (e.g. 12.4-p1).
(In reply to Marián Černý from comment #26) There are no more 12.x releases to come, and the update from an SSH version prior to 8.8p1 to 8.8p1 or later has already happened in 12.4, so this issue is no longer relevant for 12.x.
(In reply to commit-hook from comment #18) Please also add that notice to the 12.4 release notes. I just got bit by this upgrading 12.3 to 12.4.
(In reply to Andrew Daugherity from comment #28) Bug 268199 was for the 12.4 case. Please follow the link from there, through to review D37669, which received no feedback, so it was abandoned.
(In reply to Graham Perrin from comment #29) > … no feedback, so it was abandoned. More accurately: * on one hand, it made sense to abandon the planned change to release notes after an item (not an errata notice) was added to release errata, which is separate from the errata subsection of release notes * on the other hand, I think we might improve the ways in which things such as errata are represented/organised. In any case: * there's now (again) a plan to change the notes for 12.4-RELEASE.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=8ee97b1928e75f7f81a870ffb063010778e0a129 commit 8ee97b1928e75f7f81a870ffb063010778e0a129 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-09-16 20:46:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-18 09:47:01 +0000 freebsd-update: do not restart sshd when updating jail/basedir In 6cd1bc531609 for PR 263489 I changed freebsd-update to restart sshd after upgrade, to avoid an upgrade-related incompatibility that made it impossible to login. This is intended to avoid losing access to remote hosts, and ought not apply to upgrading jails (from outside). PR: 263489, 272282 Reported by: otis Reviewed by: otis, kevans MFC after: 3 days Sponsored by: The FreeBSD Foundation Fixes: 6cd1bc531609 ("freebsd-update: restart sshd after upgrade") Differential Revision: https://reviews.freebsd.org/D41890 usr.sbin/freebsd-update/freebsd-update.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=cce342e21357833892083e865710c4a05603b7b5 commit cce342e21357833892083e865710c4a05603b7b5 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-09-16 20:46:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-21 14:38:48 +0000 freebsd-update: do not restart sshd when updating jail/basedir In 6cd1bc531609 for PR 263489 I changed freebsd-update to restart sshd after upgrade, to avoid an upgrade-related incompatibility that made it impossible to login. This is intended to avoid losing access to remote hosts, and ought not apply to upgrading jails (from outside). PR: 263489, 272282 Reported by: otis Reviewed by: otis, kevans MFC after: 3 days Sponsored by: The FreeBSD Foundation Fixes: 6cd1bc531609 ("freebsd-update: restart sshd after upgrade") Differential Revision: https://reviews.freebsd.org/D41890 (cherry picked from commit 8ee97b1928e75f7f81a870ffb063010778e0a129) usr.sbin/freebsd-update/freebsd-update.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=877d15d57931350dcde264de1dabe7fccf769388 commit 877d15d57931350dcde264de1dabe7fccf769388 Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-09-16 20:46:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-21 14:40:03 +0000 freebsd-update: do not restart sshd when updating jail/basedir In 6cd1bc531609 for PR 263489 I changed freebsd-update to restart sshd after upgrade, to avoid an upgrade-related incompatibility that made it impossible to login. This is intended to avoid losing access to remote hosts, and ought not apply to upgrading jails (from outside). PR: 263489, 272282 Reported by: otis Reviewed by: otis, kevans MFC after: 3 days Sponsored by: The FreeBSD Foundation Fixes: 6cd1bc531609 ("freebsd-update: restart sshd after upgrade") Differential Revision: https://reviews.freebsd.org/D41890 (cherry picked from commit 8ee97b1928e75f7f81a870ffb063010778e0a129) (cherry picked from commit cce342e21357833892083e865710c4a05603b7b5) usr.sbin/freebsd-update/freebsd-update.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
A commit in branch releng/14.0 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=78534afbcab6e97565c06041fecfe28587e13f2b commit 78534afbcab6e97565c06041fecfe28587e13f2b Author: Ed Maste <emaste@FreeBSD.org> AuthorDate: 2023-09-16 20:46:16 +0000 Commit: Ed Maste <emaste@FreeBSD.org> CommitDate: 2023-09-23 13:08:37 +0000 freebsd-update: do not restart sshd when updating jail/basedir In 6cd1bc531609 for PR 263489 I changed freebsd-update to restart sshd after upgrade, to avoid an upgrade-related incompatibility that made it impossible to login. This is intended to avoid losing access to remote hosts, and ought not apply to upgrading jails (from outside). PR: 263489, 272282 Reported by: otis Reviewed by: otis, kevans Sponsored by: The FreeBSD Foundation Fixes: 6cd1bc531609 ("freebsd-update: restart sshd after upgrade") Differential Revision: https://reviews.freebsd.org/D41890 (cherry picked from commit 8ee97b1928e75f7f81a870ffb063010778e0a129) (cherry picked from commit cce342e21357833892083e865710c4a05603b7b5) Approved by: re (gjb) usr.sbin/freebsd-update/freebsd-update.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)