Bug 196760 - freebsd-update on 9.3R leaves sshd with Undefined symbol "ssh_explicit_bzero"
Summary: freebsd-update on 9.3R leaves sshd with Undefined symbol "ssh_explicit_bzero"
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: misc (show other bugs)
Version: 9.3-RELEASE
Hardware: Any Any
: --- Affects Some People
Assignee: FreeBSD Release Engineering
URL:
Keywords:
: 195568 199669 (view as bug list)
Depends on:
Blocks:
 
Reported: 2015-01-15 13:50 UTC by Dan Langille
Modified: 2015-12-30 01:39 UTC (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dan Langille freebsd_committer 2015-01-15 13:50:23 UTC
I upgraded four systems last night. Three of them have problems with sshd after running freebsd-update on 9.3R.

The i386 host did not start sshd:

# service sshd start
Performing sanity check on sshd configuration.
/usr/sbin/sshd: Undefined symbol "ssh_explicit_bzero"
/etc/rc.d/sshd: WARNING: failed precmd routine for sshd

All jails on two amd64 hosts failed to start sshd.  The fourth host (amd64) was not affected.

There is no /usr/lib/private in any jail nor on the i386 host which failed to start.

Both amd64 hosts have that directory.

Erwin Lansing reports same problem

see also: https://lists.freebsd.org/pipermail/freebsd-stable/2014-July/079334.html
Comment 1 Erwin Lansing freebsd_committer 2015-01-15 13:54:05 UTC
Similar experience here. Simple "sudo freebsd-update fetch" "sudo freebsd-update install" on a amd64 VMware virtual machine, no jails involved, going from (probably) 9.3-p5 to 9.3-p8. No /usr/lib/private either.
Comment 2 Dan Langille freebsd_committer 2015-01-15 14:55:02 UTC
I have a KVM for the busted i386 system if forensics will help.
Comment 3 Glen Barber freebsd_committer 2015-01-15 15:33:43 UTC
The PR states the version is 9.2-STABLE.  Please confirm what the FreeBSD OS version was prior to the upgrade.
Comment 4 Glen Barber freebsd_committer 2015-01-15 15:35:12 UTC
Dan, if you can provide access, I would like to investigate.
Comment 5 Dan Langille freebsd_committer 2015-01-15 15:35:46 UTC
Sorry, it was 9.3-RELEASE prior to running freebsd-update.  It was most likely on 9.3-RELEASE-p5
Comment 6 Dan Langille freebsd_committer 2015-01-15 15:36:10 UTC
Glen: I will find you on IRC.
Comment 7 Dan Langille freebsd_committer 2015-01-15 15:49:18 UTC
FYI, the jails were running 9.2 based on what I find here:

# mount -t zfs system/usr/jails.KEEP/basejail.OLD /mnt
# cd /mnt/bin
# file sh
sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked (uses shared libs), for FreeBSD 9.2, stripped
Comment 8 Dan Langille freebsd_committer 2015-01-15 17:37:18 UTC
From the i386 system:

/usr/sbin/sshd:
	libssh.so.5 => /usr/lib/libssh.so.5 (0x280b2000)
	libutil.so.9 => /lib/libutil.so.9 (0x28106000)
	libwrap.so.6 => /usr/lib/libwrap.so.6 (0x28119000)
	libpam.so.5 => /usr/lib/libpam.so.5 (0x28121000)
	libbsm.so.3 => /usr/lib/libbsm.so.3 (0x2812c000)
	libgssapi_krb5.so.10 => /usr/lib/libgssapi_krb5.so.10 (0x28144000)
	libgssapi.so.10 => /usr/lib/libgssapi.so.10 (0x2815b000)
	libkrb5.so.10 => /usr/lib/libkrb5.so.10 (0x28164000)
	libhx509.so.10 => /usr/lib/libhx509.so.10 (0x281c4000)
	libasn1.so.10 => /usr/lib/libasn1.so.10 (0x281fa000)
	libcom_err.so.5 => /usr/lib/libcom_err.so.5 (0x28271000)
	libroken.so.10 => /usr/lib/libroken.so.10 (0x28273000)
	libcrypt.so.5 => /lib/libcrypt.so.5 (0x28283000)
	libcrypto.so.6 => /lib/libcrypto.so.6 (0x282a8000)
	libz.so.6 => /lib/libz.so.6 (0x2840b000)
	libthr.so.3 => /lib/libthr.so.3 (0x2841f000)
	libc.so.7 => /lib/libc.so.7 (0x28440000)
	libmd.so.5 => /lib/libmd.so.5 (0x28572000)
Comment 9 Dan Langille freebsd_committer 2015-01-15 17:40:29 UTC
From the i386 system:

$ ls -l /usr/lib/libssh.so
lrwxr-xr-x  1 root  wheel      11 Feb 18  2011 /usr/lib/libssh.so -> libssh.so.5
-r--r--r--  1 root  wheel  341544 Apr  9  2014 /usr/lib/libssh.so.5
Comment 10 Dan Langille freebsd_committer 2015-01-15 18:08:54 UTC
From a reboot email, the i386 host was on:

Jan 12 16:24:11 gelt kernel: FreeBSD 9.3-RELEASE-p5 #0: Mon Nov  3 22:02:57 UTC 2014

before these commands were run:

freebsd-update fetch install
shutdown -r now
Comment 11 Allan Jude freebsd_committer 2015-01-15 18:21:59 UTC
The problem seems to stem from a previous minor upgrade (9.2 -> 9.3) being incomplete. The kernel was upgraded but the userland was not, so /usr/lib/private/ didn't exist (as there wasn't one in 9.2)
Comment 12 Dan Langille freebsd_committer 2015-01-15 22:10:51 UTC
This properly upgraded my host:

env UNAME_r=9.2-RELEASE freebsd-update upgrade -r 9.3-RELEASE
freebsd-update install
freebsd-update install

and perhaps a third:

freebsd-update install
Comment 13 Erwin Lansing freebsd_committer 2015-01-16 10:15:04 UTC
That workaround indeed fixed the broken host, and none of the other machines had a 9.2 /bin/sh, and furthermore none had any issues with the upgrade to 9.3R-p8.  Thanks to all for finding the cause!  I'd say this ticket can be closed.
Comment 14 Allan Jude freebsd_committer 2015-02-13 05:12:15 UTC
*** Bug 195568 has been marked as a duplicate of this bug. ***
Comment 15 Allan Jude freebsd_committer 2015-03-03 20:09:54 UTC
A fix to prevent this issue has been committed to -CURRENT

https://reviews.freebsd.org/D1550
Comment 16 Xin LI freebsd_committer 2015-03-03 20:13:38 UTC
Do we need an EN for this?
Comment 17 Allan Jude freebsd_committer 2015-03-03 22:13:53 UTC
(In reply to Xin LI from comment #16)

This is a result of user error, so I am not sure it needs an EN. The fix is just a safety belt to keep you from shooting off your own foot.
Comment 18 Glen Barber freebsd_committer 2015-03-03 22:35:25 UTC
(In reply to Xin LI from comment #16)
> Do we need an EN for this?

IMHO, yes, and EN should be issued for this.

I'll handle the "paperwork" for it.
Comment 19 Glen Barber freebsd_committer 2015-03-03 22:57:33 UTC
(In reply to Allan Jude from comment #15)
> A fix to prevent this issue has been committed to -CURRENT
> 

Which svn revision resolves this?
Comment 20 commit-hook freebsd_committer 2015-03-03 23:20:23 UTC
A commit references this bug:

Author: allanjude
Date: Tue Mar  3 23:20:19 UTC 2015
New revision: 279571
URL: https://svnweb.freebsd.org/changeset/base/279571

Log:
  Add a new safetly belt to freebsd-update to prevent a user doing a minor update (-pX) while having an unfinished major upgrade (9.x to 9.y)
  Safetly belt can be disabled with the -F flag

  Additionally, add the --not-running-from-cron flag they bypasses the TTY requirement, and allows freebsd-update to be invoked by orchestration frameworks, scripts, or otherwise.

  PR:		196760
  Differential Revision:	https://reviews.freebsd.org/D1550
  Reviewed by:	cperciva, delphij
  Approved by:	bcr (mentor), rodrigc (src)
  MFC after:	1 month
  Relnotes:	yes
  Sponsored by:	ScaleEngine Inc.

Changes:
  head/usr.sbin/freebsd-update/freebsd-update.8
  head/usr.sbin/freebsd-update/freebsd-update.sh
Comment 21 Allan Jude freebsd_committer 2015-03-03 23:21:26 UTC
(In reply to Glen Barber from comment #19)

r279571
Comment 22 Glen Barber freebsd_committer 2015-03-04 00:06:24 UTC
Reassign, pending EN.
Comment 23 commit-hook freebsd_committer 2015-05-13 22:36:20 UTC
A commit references this bug:

Author: delphij
Date: Wed May 13 22:36:01 UTC 2015
New revision: 282870
URL: https://svnweb.freebsd.org/changeset/base/282870

Log:
  MFC r279571,281563 (allanjude):

  Add a new safetly belt to freebsd-update to prevent a user doing a minor
  update (-pX) while having an unfinished major upgrade (9.x to 9.y)
  Safetly belt can be disabled with the -F flag

  Additionally, add the --not-running-from-cron flag they bypasses the TTY
  requirement, and allows freebsd-update to be invoked by orchestration
  frameworks, scripts, or otherwise.

  PR:		196760
  Relnotes:	yes
  Sponsored by:	ScaleEngine Inc.

Changes:
_U  stable/10/
  stable/10/usr.sbin/freebsd-update/freebsd-update.8
  stable/10/usr.sbin/freebsd-update/freebsd-update.sh
Comment 24 commit-hook freebsd_committer 2015-05-13 22:37:23 UTC
A commit references this bug:

Author: delphij
Date: Wed May 13 22:36:53 UTC 2015
New revision: 282871
URL: https://svnweb.freebsd.org/changeset/base/282871

Log:
  MFC r279571,281563 (allanjude):

  Add a new safetly belt to freebsd-update to prevent a user doing a minor
  update (-pX) while having an unfinished major upgrade (9.x to 9.y)
  Safetly belt can be disabled with the -F flag

  Additionally, add the --not-running-from-cron flag they bypasses the TTY
  requirement, and allows freebsd-update to be invoked by orchestration
  frameworks, scripts, or otherwise.

  PR:		196760

Changes:
_U  stable/9/usr.sbin/freebsd-update/
  stable/9/usr.sbin/freebsd-update/freebsd-update.8
  stable/9/usr.sbin/freebsd-update/freebsd-update.sh
Comment 25 Xin LI freebsd_committer 2015-05-13 23:29:31 UTC
This is fixed in FreeBSD-EN-15:04.freebsd-update.
Comment 26 Jason Unovitch freebsd_committer 2015-12-30 01:39:20 UTC
*** Bug 199669 has been marked as a duplicate of this bug. ***