Bug 196771

Summary:	/usr/bin/ssh: Undefined symbol "ssh_lowercase"
Product:	Base System	Reporter:	Dan Busarow <dan>
Component:	bin	Assignee:	freebsd-bugs (Nobody) <bugs>
Status:	Closed FIXED
Severity:	Affects Only Me	CC:	cperciva, dvl, emaste
Priority:	---
Version:	9.3-RELEASE
Hardware:	Any
OS:	Any

Description Dan Busarow 2015-01-15 17:40:52 UTC

After running freebsd-update install in 1/15/2015, 

The following files will be updated as part of updating to 9.3-RELEASE-p8:

I can no longer ssh into or out of one of my servers.

9 other servers updated just fine.  This one gives me this when sshing out

OpenSSH_6.6.1, OpenSSL 0.9.8za-freebsd 5 Jun 2014
debug 1: Reading configuration data /etc/ssh/ssh_config
/usr/bin/ssh: Undefined symbol "ssh_lowercase"


ssh into this box goves 

OpenSSH_6.6.1, OpenSSL 0.9.8za-freebsd 5 Jun 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to doheny.buildingonline.net [2607:fc50:1001:8f00::10] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/dan/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/dan/.ssh/id_rsa type 1
debug1: identity file /home/dan/.ssh/id_rsa-cert type -1
debug1: identity file /home/dan/.ssh/id_dsa type -1
debug1: identity file /home/dan/.ssh/id_dsa-cert type -1
debug1: identity file /home/dan/.ssh/id_ecdsa type -1
debug1: identity file /home/dan/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/dan/.ssh/id_ed25519 type -1
debug1: identity file /home/dan/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1_hpn13v11 FreeBSD-20140420
ssh_exchange_identification: Connection closed by remote host


Dan

Comment 1 Glen Barber freebsd_committer

2015-01-15 17:45:33 UTC

What architecture is this?

This appears to be related to this PR:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196760

I'm setting up environments to reproduce and debug this now.

Comment 2 Glen Barber freebsd_committer

2015-01-15 17:47:33 UTC

Also, can you provide the previous FreeBSD version, if possible?

Comment 3 Dan Busarow 2015-01-15 18:17:23 UTC

Old release was 9.3-RELEASE-p5

Problem has been resolved.

there was no /usr/lib/private and therefore none of the libssh* files were in /usr/lib/private

I copied those from another box and all is good now.

My question now is where did /usr/lib/private go???

I ssh'd into the box in order to run freebsd-update so ssh was working before the freebsd-update install

Were
/usr/lib/private/libssh.a
/usr/lib/private/libssh.so.5
/usr/lib/private/libssh_p.a

not used in p5 or should I be looking for a flaky drive?

Thanks,
Dan

Comment 4 Glen Barber freebsd_committer

2015-01-15 18:21:43 UTC

Can you show output of 'file /bin/sh' ?

It was discovered in another report that a prior upgrade to 9.3-RELEASE from 9.2-RELEASE was incomplete.

9.3-RELEASE (and all subsequent releases) should have /usr/lib/private, so if that is missing on your system, you will run into issues in the future.  It's important to know what FreeBSD version sh(1) is.

Comment 5 Dan Busarow 2015-01-15 18:50:10 UTC

This was an upgrade from 9.2-RELEASE to 9.3-RELEASE done via freebsd-update -r 9.3-RELEASE upgrade

I don't remember any problem with the upgrade but I guess I may have missed something in the install reboot install cycle

Here's the file /bin/sh output

doheny:/home/dan $ file /bin/sh
/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), for FreeBSD 9.2, stripped

Ahh, one of my others shows 


/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), for FreeBSD 9.3, stripped


Suggestions?

Thanks,
Dan

Comment 6 Glen Barber freebsd_committer

2015-01-15 18:56:08 UTC

Ok, so this is the third of these exhibiting the exact same behavior.

I'm doing an update on a VM from 9.2-RELEASE -> 9.3-RELEASE, skipping the userland update to try to identify a proper fix.

In the meantime, I've CC'd cperciva@ for any input he could provide.

Colin, it may be worth evaluating 'uname -K' and 'uname -U' upon freebsd-update(8) invocation and yell loudly when they do not match, so people do not shoot their foots off like this.  Thoughts?

Comment 7 Dan Langille freebsd_committer

2015-01-15 21:18:43 UTC

Dan B: I am trying an upgrade method tested by others.  I will get back with the results.

Comment 8 Dan Langille freebsd_committer

2015-01-15 22:11:39 UTC

Dan B: the ticket Glen references now has the update I did.  Worked for me.  Let us know.

Comment 9 Dan Busarow 2015-01-16 01:42:19 UTC

Dan L,

You mean

env UNAME_r=9.2-RELEASE freebsd-update upgrade -r 9.3-RELEASE
freebsd-update install
freebsd-update install

and perhaps a third:

freebsd-update install

worked for you?  Upgrade the upgrade? Force it to run by specifying UNAME?

Thanks for following up.

Dan

Comment 10 Dan Langille freebsd_committer

2015-01-16 01:45:41 UTC

Yes.  That's what I did.  Exactly.

Comment 11 Dan Busarow 2015-01-16 01:47:39 UTC

Cool.  I'll test it out on a less critical machine and then go ahead and re-upgrade this one.

Thanks,
Dan

Comment 12 Dan Busarow 2015-01-24 02:20:36 UTC

Just to really close this ticket out I wanted to note that I did follow Dan L's advice and ran

env UNAME_r=9.2-RELEASE freebsd-update upgrade -r 9.3-RELEASE
freebsd-update install

I tried it a couple of times on virtualbox and everything went fine.  So I ran it on this production server and it ran fine here too.

Didn't even need a reboot.

Thanks all.

Dan