Bug 264600 - openssh: sshd spamming logs: error: Fssh_kex_exchange_identification: Connection closed by remote host
Summary: openssh: sshd spamming logs: error: Fssh_kex_exchange_identification: Connect...
Status: Closed Not A Bug
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 13.1-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2022-06-10 17:26 UTC by Jose Luis Duran
Modified: 2022-12-25 09:39 UTC (History)
7 users (show)

See Also:
koobs: maintainer-feedback? (emaste)
koobs: mfc-stable13?


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jose Luis Duran 2022-06-10 17:26:26 UTC
After upgrading a public-facing SSH server to OpenSSH 8.8 (13.1-RELEASE), it has started spamming dmesg logs with:

    error: Fssh_kex_exchange_identification: Connection closed by remote host

To replicate it, just `nc ssh-server 22`.

Per conversation with @emaste, opening this PR to track it.
Comment 1 Jose Luis Duran 2022-06-11 13:36:57 UTC
This issue is also present in -CURRENT... I "noticed" it on 13.1, or after the bump to OpenSSH 8.8, but it is still present on -CURRENT's OpenSSH 9.0.

If you have SSH enabled locally, it's easy to replicate:

nc localhost 22
(CRTL+C)
(Error message on stdout)
Comment 2 Larry Rosenman freebsd_committer freebsd_triage 2022-06-11 13:55:06 UTC
it also spams the console.
Comment 3 Jose Luis Duran 2022-06-12 15:06:30 UTC
(In reply to Larry Rosenman from comment #2)

Yes, it's almost as if LogVerbose is always enabled.
Comment 4 Ed Maste freebsd_committer freebsd_triage 2022-07-29 18:12:15 UTC
9.0p1 is in the tree now and has been for a little while; has anyone noticed if this issue is no longer reproducible there?
Comment 5 Jose Luis Duran 2022-07-29 18:39:51 UTC
(In reply to Ed Maste from comment #4)

It is still present:

# uname -a
FreeBSD fbsd 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n256882-8f733dabcc3: Fri Jul 22 08:31:37 UTC 2022     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

# ssh -V
OpenSSH_9.0p1, OpenSSL 1.1.1q-freebsd  5 Jul 2022

# nc localhost 22
SSH-2.0-OpenSSH_9.0 FreeBSD-20220415
^C

# dmesg -a
...
error: Fssh_kex_exchange_identification: Connection closed by remote host
Comment 6 nerozero 2022-08-01 15:30:50 UTC
Here as well, after updating to 13.1
Comment 7 parv 2022-08-02 10:06:40 UTC
Via "nc localhost ... ^C" as listed in comment #5, I saw only one(1) message in ~8 minutes in /var/log/{console.log,messages} on 14 (c 20220810, commit d1f5aa56) & 13 (c 20220729, commit 158071c51a). No spam yet.
Comment 8 parv 2022-08-02 10:13:57 UTC
Wait ... I take it that "Fssh..." message appears (or has started appearing) after closing of a connection. So login failures could certainly create a flood for internet-facing machines.
Comment 9 Jose Luis Duran 2022-08-02 15:52:43 UTC
(In reply to parv from comment #8)
Yes, the spam is due to the fact that port TCP/22 might be publicly exposed (i.e. a public SSH or Git server), and every failed connection triggers a message.

Sorry if this information was left out, probably because I opened bug #264599 at the same time.
Comment 10 Morgan Davis 2022-08-07 21:32:02 UTC
In my case, these messages are appearing as a result of ssh port probes from my network monitoring script.

Before upgrading from 12.3 to 13.1, logs would show one entry per test:

Aug  7 13:54:19 dtl-02 sshd[54100]: Did not receive identification string from xx.xx.xx.xx port 54996

Now in 13.1 they show two lines:

Aug  7 14:20:05 dtl-02 sshd[34173]: error: Fssh_kex_exchange_identification: Connection closed by remote host
Aug  7 14:20:05 dtl-02 sshd[34173]: Connection closed by xx.xx.xx.xx port 51499
Comment 11 Cristian Cardoso 2022-09-06 14:08:14 UTC
In my case, using FreeBSD 13.1 and OpenSSH_8.8p1, the error "Fssh_kex_exchange_identification:" makes me no longer able to log into the machine remotely.
Comment 12 smars 2022-09-21 02:42:27 UTC
same problem
Comment 13 Jose Luis Duran 2022-09-22 02:14:46 UTC
I believe we should change the title to:

"openssh: spamming error logs to the console and /var/log/messages"

Or something more general like that, since other errors can also be easily triggered:

echo "\0" | nc ssh-server 22
error: Fssh_kex_exchange_identification: banner line contains invalid characters

echo "SSH-XXX\n" | nc ssh-server 22
error: Bad remote protocol version identification: 'SSH-XXX'

You can get more ideas from kex_exchange_identification.
Comment 14 Jose Luis Duran 2022-09-24 01:01:31 UTC
Just for reference, here is the same bug report for Debian:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959206
Comment 15 Ed Maste freebsd_committer freebsd_triage 2022-10-05 20:41:40 UTC
(In reply to Jose Luis Duran from comment #0)
> After upgrading a public-facing SSH server to OpenSSH 8.8

What version were you running before?

This error comes from:
                        len = atomicio(read, ssh_packet_get_connection_in(ssh),
                            &c, 1);
                        if (len != 1 && errno == EPIPE) {
                                error_f("Connection closed by remote host");
                                r = SSH_ERR_CONN_CLOSED;
                                goto out;

This commit in OpenSSH-portable is probably relevant:

commit 0a843d9a0e805f14653a555f5c7a8ba99d62c12d
Author: djm@openbsd.org <djm@openbsd.org>
Date:   Thu Dec 27 03:25:24 2018 +0000

    upstream: move client/server SSH-* banners to buffers under
    
    ssh->kex and factor out the banner exchange. This eliminates some common code
    from the client and server.
    
    Also be more strict about handling \r characters - these should only
    be accepted immediately before \n (pointed out by Jann Horn).
    
    Inspired by a patch from Markus Schmidt.
    (lots of) feedback and ok markus@
    
    OpenBSD-Commit-ID: 1cc7885487a6754f63641d7d3279b0941890275b

which came between 7.9 and 8.0
Comment 16 Jose Luis Duran 2022-10-05 21:00:39 UTC
(In reply to Ed Maste from comment #15)
> What version were you running before?

I was running FreeBSD 13.0, so OpenSSH 7.9.
Comment 17 Jose Luis Duran 2022-10-29 04:35:10 UTC
I guess I will close this one as not a bug.

This was a deliberate change in OpenSSH (https://github.com/openssh/openssh-portable/commit/5becbec023f2037394987f85ed7f74b9a28699e0 and a few other commits).

Let's use the non-portable version (OpenBSD 7.2, OpenSSH 9.1):

Edit /etc/syslog.conf and uncomment the line:

    *.err;auth.notice;authpriv.none;kern.debug;mail.crit	/dev/console

As the comment says: "be aware that this could create lots of output".

At this point it will exhibit the same behavior.

FreeBSD ships with this line uncommented, that's why it is more perceptible.