After updating one of my systems from 9.3-RELEASE-p36 to 9.3-RELEASE-p37 the OpenSSH client segfaults when attempting to connect: [root@dc ~]# ssh -vvvv c3560ryan OpenSSH_6.6.1, OpenSSL 0.9.8zh-freebsd 3 Dec 2015 debug1: Reading configuration data /etc/ssh/ssh_config debug2: ssh_connect: needpriv 0 debug1: Connecting to c3560ryan [10.193.66.197] port 22. debug1: Connection established. debug1: permanently_set_uid: 0/0 debug1: identity file /root/.ssh/id_rsa type -1 debug1: identity file /root/.ssh/id_rsa-cert type -1 debug3: Incorrect RSA1 identifier debug3: Could not load "/root/.ssh/id_dsa" as a RSA1 public key debug1: identity file /root/.ssh/id_dsa type 2 debug1: identity file /root/.ssh/id_dsa-cert type -1 debug1: identity file /root/.ssh/id_ecdsa type -1 debug1: identity file /root/.ssh/id_ecdsa-cert type -1 debug1: identity file /root/.ssh/id_ed25519 type -1 debug1: identity file /root/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420 debug1: Remote protocol version 1.99, remote software version Cisco-1.25 debug1: no match: Cisco-1.25 debug2: fd 3 setting O_NONBLOCK debug3: ssh_load_hostkeys: loading entries for host "c3560ryan" from file "/root/.ssh/known_hosts" debug3: ssh_load_hostkeys: loaded 0 keys debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss,ssh-ed25519-cert-v01@openssh.com,ssh-ed25519 debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96 debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96 debug2: kex_parse_kexinit: none debug2: kex_parse_kexinit: none debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_setup: setup hmac-md5 debug1: kex: server->client aes128-cbc hmac-md5 none debug2: mac_setup: setup hmac-md5 debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP Segmentation fault: 11 (core dumped)
Created attachment 167819 [details] gdb Backtrace
imapd is constantly dying too, from SSL ?
(In reply to William F. Dudley Jr. from comment #2) Yes. On one system running 9.3-RELEASE-p37 where mail/cyrus-imapd25 is installed the process is terminating when a client attempts an SSL connection: Mar 7 15:06:30 ncklug imap[53452]: inittls: Loading hard-coded DH parameters Mar 7 15:06:30 ncklug master[53426]: process type:SERVICE name:imap path:/usr/local/cyrus/bin/imapd age:0.044s pid:53452 signaled to death by signal 11 (Segmentation fault: 11)
Oh, and sshd dies with signal 11 when someone tries to ssh into the system.
(In reply to William F. Dudley Jr. from comment #4) Yes, I'm observing that behavior as well after updating to -p37.
Same here, freebsd-update rollback or replacing libcrypto.so.6 with the previous one fixes the problem. As already reported, backtrace shows the problem being in BN_mod_exp_mont_consttime () from /lib/libcrypto.so.6. Maybe this has something to do with r296462 (https://svnweb.freebsd.org/base?view=revision&revision=296462).
Some more info, in case it helps: - Everything using libcrypto.so.6 can be affected. I've noticed problems with ssh server (ssh child dies with segfault), cyrus-imapd and check-imaps from nagios plugins. - Ssh'ing from another FreeBSD to a system with the problematic sshd is successful, but ssh'ing using openssh client e.g. from a linux box is not. Some clients do work, some do not. So some care must be taken to reproduce the problem, as not all clients trigger the problem.
Just reporting I 've been having the same issue. ssh to the 9.3 box from a recently installed 10.2 succeeded, ssh from a Debian Jessie failed. Version is: $ ssh -V OpenSSH_6.7p1 Debian-5+deb8u1, OpenSSL 1.0.1k 8 Jan 2015 A freebsd-update rollback fixed the issue for me.
Take.
Created attachment 167918 [details] Workaround An update: This is a workaround (would reintroduce CVE-2016-0702) which is known to fix RSA for me. We are still investigating the issue.
Created attachment 167941 [details] Fix bug caused by r296462 If you ask me, this is caused by r296462 [1], specifically the part: > constant-time MOD_EXP_CTIME_COPY_FROM_PREBUF. > [CVE-2016-0702, upstream d6482a8. 5ea08bd, d6d422e, > 8fc8f48 317be63 skipped intentionally as we are not > using the code on FreeBSD. Backport done by jkim@. The problem is that all calls of MOD_EXP_CTIME_COPY_TO_PREBUF() are adjusted to use the 'window' parameter, but it appears the one call to MOD_EXP_CTIME_COPY_FROM_PREBUF() was forgotten: it still uses 'numPowers', which is actually 1 << window! Now MOD_EXP_CTIME_COPY_FROM_PREBUF() itself calls this input parameter 'window', and then proceeds to calculate the xstride as 1 << (window - 2), which in some cases can end up being 2^30. The loop which then goes through 'table' (the buffer) will almost certainly hit bad memory. The fix is to call MOD_EXP_CTIME_COPY_FROM_PREBUF() with 'window' instead. [1] https://svnweb.freebsd.org/changeset/base/296462
A commit references this bug: Author: delphij Date: Thu Mar 10 03:57:37 UTC 2016 New revision: 296597 URL: https://svnweb.freebsd.org/changeset/base/296597 Log: Fix a regression introduced in r296462 that causes out-of-bound access in the BN code and have slipped my review. PR: 207783 Submitted by: dim Changes: stable/9/crypto/openssl/crypto/bn/bn_exp.c
Resolved with 9.3-RELEASE-p38.