Bug 196771 - /usr/bin/ssh: Undefined symbol "ssh_lowercase"
Summary: /usr/bin/ssh: Undefined symbol "ssh_lowercase"
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 9.3-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs mailing list
Depends on:
Reported: 2015-01-15 17:40 UTC by Dan Busarow
Modified: 2015-01-24 02:20 UTC (History)
3 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Dan Busarow 2015-01-15 17:40:52 UTC
After running freebsd-update install in 1/15/2015, 

The following files will be updated as part of updating to 9.3-RELEASE-p8:

I can no longer ssh into or out of one of my servers.

9 other servers updated just fine.  This one gives me this when sshing out

OpenSSH_6.6.1, OpenSSL 0.9.8za-freebsd 5 Jun 2014
debug 1: Reading configuration data /etc/ssh/ssh_config
/usr/bin/ssh: Undefined symbol "ssh_lowercase"

ssh into this box goves 

OpenSSH_6.6.1, OpenSSL 0.9.8za-freebsd 5 Jun 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to doheny.buildingonline.net [2607:fc50:1001:8f00::10] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/dan/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/dan/.ssh/id_rsa type 1
debug1: identity file /home/dan/.ssh/id_rsa-cert type -1
debug1: identity file /home/dan/.ssh/id_dsa type -1
debug1: identity file /home/dan/.ssh/id_dsa-cert type -1
debug1: identity file /home/dan/.ssh/id_ecdsa type -1
debug1: identity file /home/dan/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/dan/.ssh/id_ed25519 type -1
debug1: identity file /home/dan/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420
ssh_exchange_identification: Connection closed by remote host

Comment 1 Glen Barber freebsd_committer 2015-01-15 17:45:33 UTC
What architecture is this?

This appears to be related to this PR:

I'm setting up environments to reproduce and debug this now.
Comment 2 Glen Barber freebsd_committer 2015-01-15 17:47:33 UTC
Also, can you provide the previous FreeBSD version, if possible?
Comment 3 Dan Busarow 2015-01-15 18:17:23 UTC
Old release was 9.3-RELEASE-p5

Problem has been resolved.

there was no /usr/lib/private and therefore none of the libssh* files were in /usr/lib/private

I copied those from another box and all is good now.

My question now is where did /usr/lib/private go???

I ssh'd into the box in order to run freebsd-update so ssh was working before the freebsd-update install


not used in p5 or should I be looking for a flaky drive?

Comment 4 Glen Barber freebsd_committer 2015-01-15 18:21:43 UTC
Can you show output of 'file /bin/sh' ?

It was discovered in another report that a prior upgrade to 9.3-RELEASE from 9.2-RELEASE was incomplete.

9.3-RELEASE (and all subsequent releases) should have /usr/lib/private, so if that is missing on your system, you will run into issues in the future.  It's important to know what FreeBSD version sh(1) is.
Comment 5 Dan Busarow 2015-01-15 18:50:10 UTC
This was an upgrade from 9.2-RELEASE to 9.3-RELEASE done via freebsd-update -r 9.3-RELEASE upgrade

I don't remember any problem with the upgrade but I guess I may have missed something in the install reboot install cycle

Here's the file /bin/sh output

doheny:/home/dan $ file /bin/sh
/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), for FreeBSD 9.2, stripped

Ahh, one of my others shows 

/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), for FreeBSD 9.3, stripped


Comment 6 Glen Barber freebsd_committer 2015-01-15 18:56:08 UTC
Ok, so this is the third of these exhibiting the exact same behavior.

I'm doing an update on a VM from 9.2-RELEASE -> 9.3-RELEASE, skipping the userland update to try to identify a proper fix.

In the meantime, I've CC'd cperciva@ for any input he could provide.

Colin, it may be worth evaluating 'uname -K' and 'uname -U' upon freebsd-update(8) invocation and yell loudly when they do not match, so people do not shoot their foots off like this.  Thoughts?
Comment 7 Dan Langille freebsd_committer 2015-01-15 21:18:43 UTC
Dan B: I am trying an upgrade method tested by others.  I will get back with the results.
Comment 8 Dan Langille freebsd_committer 2015-01-15 22:11:39 UTC
Dan B: the ticket Glen references now has the update I did.  Worked for me.  Let us know.
Comment 9 Dan Busarow 2015-01-16 01:42:19 UTC
Dan L,

You mean

env UNAME_r=9.2-RELEASE freebsd-update upgrade -r 9.3-RELEASE
freebsd-update install
freebsd-update install

and perhaps a third:

freebsd-update install

worked for you?  Upgrade the upgrade? Force it to run by specifying UNAME?

Thanks for following up.

Comment 10 Dan Langille freebsd_committer 2015-01-16 01:45:41 UTC
Yes.  That's what I did.  Exactly.
Comment 11 Dan Busarow 2015-01-16 01:47:39 UTC
Cool.  I'll test it out on a less critical machine and then go ahead and re-upgrade this one.

Comment 12 Dan Busarow 2015-01-24 02:20:36 UTC
Just to really close this ticket out I wanted to note that I did follow Dan L's advice and ran

env UNAME_r=9.2-RELEASE freebsd-update upgrade -r 9.3-RELEASE
freebsd-update install

I tried it a couple of times on virtualbox and everything went fine.  So I ran it on this production server and it ran fine here too.

Didn't even need a reboot.

Thanks all.