Bug 207783 - OpenSSH client segfaults in 9.3-RELEASE-p37
Summary: OpenSSH client segfaults in 9.3-RELEASE-p37
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 9.3-RELEASE
Hardware: amd64 Any
: --- Affects Many People
Assignee: Xin LI
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-03-07 18:32 UTC by Ryan Frederick
Modified: 2016-03-11 07:43 UTC (History)
14 users (show)

See Also:


Attachments
gdb Backtrace (1001 bytes, text/plain)
2016-03-07 18:41 UTC, Ryan Frederick
no flags Details
Workaround (11.26 KB, patch)
2016-03-09 09:14 UTC, Xin LI
no flags Details | Diff
Fix bug caused by r296462 (627 bytes, patch)
2016-03-09 22:02 UTC, Dimitry Andric
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Ryan Frederick 2016-03-07 18:32:05 UTC
After updating one of my systems from 9.3-RELEASE-p36 to 9.3-RELEASE-p37 the OpenSSH client segfaults when attempting to connect:

[root@dc ~]# ssh -vvvv c3560ryan
OpenSSH_6.6.1, OpenSSL 0.9.8zh-freebsd 3 Dec 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to c3560ryan [10.193.66.197] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/root/.ssh/id_dsa" as a RSA1 public key
debug1: identity file /root/.ssh/id_dsa type 2
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420
debug1: Remote protocol version 1.99, remote software version Cisco-1.25
debug1: no match: Cisco-1.25
debug2: fd 3 setting O_NONBLOCK
debug3: ssh_load_hostkeys: loading entries for host "c3560ryan" from file "/root/.ssh/known_hosts"
debug3: ssh_load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss,ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_setup: setup hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Segmentation fault: 11 (core dumped)
Comment 1 Ryan Frederick 2016-03-07 18:41:28 UTC
Created attachment 167819 [details]
gdb Backtrace
Comment 2 William F. Dudley Jr. 2016-03-07 21:56:54 UTC
imapd is constantly dying too, from SSL ?
Comment 3 Ryan Frederick 2016-03-07 22:04:08 UTC
(In reply to William F. Dudley Jr. from comment #2)

Yes. On one system running 9.3-RELEASE-p37 where mail/cyrus-imapd25 is installed the process is terminating when a client attempts an SSL connection:

Mar  7 15:06:30 ncklug imap[53452]: inittls: Loading hard-coded DH parameters
Mar  7 15:06:30 ncklug master[53426]: process type:SERVICE name:imap path:/usr/local/cyrus/bin/imapd age:0.044s pid:53452 signaled to death by signal 11 (Segmentation fault: 11)
Comment 4 William F. Dudley Jr. 2016-03-07 22:44:58 UTC
Oh, and sshd dies with signal 11 when someone tries to ssh into the system.
Comment 5 Ryan Frederick 2016-03-07 22:45:53 UTC
(In reply to William F. Dudley Jr. from comment #4)

Yes, I'm observing that behavior as well after updating to -p37.
Comment 6 Athanasios Douitsis 2016-03-08 13:15:25 UTC
Same here, freebsd-update rollback or replacing libcrypto.so.6 with the previous one fixes the problem.

As already reported, backtrace shows the problem being in BN_mod_exp_mont_consttime () from /lib/libcrypto.so.6. Maybe this has something to do with r296462 (https://svnweb.freebsd.org/base?view=revision&revision=296462).
Comment 7 Athanasios Douitsis 2016-03-08 15:44:21 UTC
Some more info, in case it helps:

- Everything using libcrypto.so.6 can be affected. I've noticed problems with ssh server (ssh child dies with segfault), cyrus-imapd and check-imaps from nagios plugins. 
- Ssh'ing from another FreeBSD to a system with the problematic sshd is successful, but ssh'ing using openssh client e.g. from a linux box is not. Some clients do work, some do not. So some care must be taken to reproduce the problem, as not all clients trigger the problem.
Comment 8 Alexandros Kosiaris 2016-03-08 21:53:23 UTC
Just reporting I 've been having the same issue. ssh to the 9.3 box from a recently installed 10.2 succeeded, ssh from a Debian Jessie failed. Version is:

$ ssh -V
OpenSSH_6.7p1 Debian-5+deb8u1, OpenSSL 1.0.1k 8 Jan 2015

A freebsd-update rollback fixed the issue for me.
Comment 9 Xin LI freebsd_committer 2016-03-09 00:50:29 UTC
Take.
Comment 10 Xin LI freebsd_committer 2016-03-09 09:14:39 UTC
Created attachment 167918 [details]
Workaround

An update: This is a workaround (would reintroduce CVE-2016-0702) which is known to fix RSA for me.  We are still investigating the issue.
Comment 11 Dimitry Andric freebsd_committer 2016-03-09 22:02:50 UTC
Created attachment 167941 [details]
Fix bug caused by r296462

If you ask me, this is caused by r296462 [1], specifically the part:

> constant-time MOD_EXP_CTIME_COPY_FROM_PREBUF.
> [CVE-2016-0702, upstream d6482a8. 5ea08bd, d6d422e,
> 8fc8f48 317be63 skipped intentionally as we are not
> using the code on FreeBSD.  Backport done by jkim@.

The problem is that all calls of MOD_EXP_CTIME_COPY_TO_PREBUF() are adjusted to use the 'window' parameter, but it appears the one call to MOD_EXP_CTIME_COPY_FROM_PREBUF() was forgotten: it still uses 'numPowers', which is actually 1 << window!

Now MOD_EXP_CTIME_COPY_FROM_PREBUF() itself calls this input parameter 'window', and then proceeds to calculate the xstride as 1 << (window - 2), which in some cases can end up being 2^30.  The loop which then goes through 'table' (the buffer) will almost certainly hit bad memory.

The fix is to call MOD_EXP_CTIME_COPY_FROM_PREBUF() with 'window' instead.

[1] https://svnweb.freebsd.org/changeset/base/296462
Comment 12 commit-hook freebsd_committer 2016-03-10 03:57:41 UTC
A commit references this bug:

Author: delphij
Date: Thu Mar 10 03:57:37 UTC 2016
New revision: 296597
URL: https://svnweb.freebsd.org/changeset/base/296597

Log:
  Fix a regression introduced in r296462 that causes out-of-bound access in
  the BN code and have slipped my review.

  PR:		207783
  Submitted by:	dim

Changes:
  stable/9/crypto/openssl/crypto/bn/bn_exp.c
Comment 13 Xin LI freebsd_committer 2016-03-11 07:43:03 UTC
Resolved with 9.3-RELEASE-p38.