Bug 209743 - www/apache24: Cannot load /usr/local/libexec/apache24/mod_ssl.so, after upgrade from 2.4.18
Summary: www/apache24: Cannot load /usr/local/libexec/apache24/mod_ssl.so, after upgra...
Status: Closed Not A Bug
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Many People
Assignee: freebsd-ports-bugs (Nobody)
URL:
Keywords: needs-qa
Depends on:
Blocks:
 
Reported: 2016-05-24 23:30 UTC by Ron Wingfield
Modified: 2016-08-15 05:36 UTC (History)
6 users (show)

See Also:
vlad-fbsd: maintainer-feedback? (apache)


Attachments
2 log files, 1 PDF (483.25 KB, application/pdf)
2016-05-25 21:09 UTC, Ron Wingfield
no flags Details
Plain text vi readable. (626.03 KB, text/plain)
2016-05-26 11:12 UTC, Ron Wingfield
no flags Details
diff of logs (16.79 KB, text/plain)
2016-05-26 15:56 UTC, Ron Wingfield
no flags Details
diffs of the httpd.conf files (34.46 KB, text/plain)
2016-05-26 17:20 UTC, Ron Wingfield
no flags Details
Hopefully in plain text format (16.79 KB, text/plain)
2016-05-26 17:25 UTC, Ron Wingfield
no flags Details
Details of solution. (107.91 KB, application/pdf)
2016-05-29 16:33 UTC, Ron Wingfield
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ron Wingfield 2016-05-24 23:30:32 UTC
Just upgraded from Apache 2.4.18 to 2.4.20. Now mod_ssl.so cannot be loaded.
The object was installed today as 178K May 24 13:01 mod_ssl.so

Error msg:
/usr/local/libexec/apache24/mod_ssl.so: Undefined symbol "SSL_get_srp_userinfo"
Comment 1 VK freebsd_triage 2016-05-25 00:02:53 UTC
Thanks for the report. Please provide the output of `pkg info apache24`, if you built it from ports.
Comment 2 Ron Wingfield 2016-05-25 00:34:05 UTC
(In reply to Vladimir Krstulja from comment #1)

Certainly, here 'tis:

apache24-2.4.20_1
Name           : apache24
Version        : 2.4.20_1
Installed on   : Tue May 24 13:03:12 2016 CDT
Origin         : www/apache24
Architecture   : freebsd:10:x86:32
Prefix         : /usr/local
Categories     : ipv6 www
Licenses       :
Maintainer     : apache@FreeBSD.org
WWW            : http://httpd.apache.org/
Comment        : Version 2.4.x of Apache web server
Options        :
        ACCESS_COMPAT  : on
        ACTIONS        : on
        ALIAS          : on
        ALLOWMETHODS   : on
        ASIS           : on
        AUTHNZ_FCGI    : on
        AUTHNZ_LDAP    : off
        AUTHN_ANON     : on
        AUTHN_CORE     : on
        AUTHN_DBD      : on
        AUTHN_DBM      : on
        AUTHN_FILE     : on
        AUTHN_SOCACHE  : on
        AUTHZ_CORE     : on
        AUTHZ_DBD      : on
        AUTHZ_DBM      : on
        AUTHZ_GROUPFILE: on
        AUTHZ_HOST     : on
        AUTHZ_OWNER    : on
        AUTHZ_USER     : on
        AUTH_BASIC     : on
        AUTH_DIGEST    : on
        AUTH_FORM      : on
        AUTOINDEX      : on
        BUCKETEER      : off
        BUFFER         : on
        CACHE          : on
        CACHE_DISK     : on
        CACHE_SOCACHE  : on
        CASE_FILTER    : off
        CASE_FILTER_IN : off
        CERN_META      : on
        CGI            : on
        CGID           : on
        CHARSET_LITE   : on
        DATA           : on
DAV            : on
        DAV_FS         : on
        DAV_LOCK       : on
        DBD            : on
        DEFLATE        : on
        DIALUP         : on
        DIR            : on
        DUMPIO         : on
        ECHO           : off
        ENV            : on
        EXAMPLE_HOOKS  : off
        EXAMPLE_IPC    : off
        EXPIRES        : on
        EXT_FILTER     : on
        FILE_CACHE     : on
        FILTER         : on
        HEADERS        : on
        HEARTBEAT      : off
        HEARTMONITOR   : off
        HTTP2          : off
        IDENT          : off
        IMAGEMAP       : on
        INCLUDE        : on
        INFO           : on
        IPV4_MAPPED    : off
        LBMETHOD_BYBUSYNESS: on
        LBMETHOD_BYREQUESTS: on
        LBMETHOD_BYTRAFFIC: on
        LBMETHOD_HEARTBEAT: off
        LDAP           : off
        LOGIO          : on
        LOG_DEBUG      : on
        LOG_FORENSIC   : on
        LUA            : off
        LUAJIT         : off
        MACRO          : on
        MIME           : on
        MIME_MAGIC     : on
        MPM_EVENT      : off
        MPM_PREFORK    : on
        MPM_SHARED     : on
        MPM_WORKER     : off
        NEGOTIATION    : on
        OPTIONAL_FN_EXPORT: off
        OPTIONAL_FN_IMPORT: off
        OPTIONAL_HOOK_EXPORT: off
        OPTIONAL_HOOK_IMPORT: off
        PROXY          : on
PROXY_AJP      : on
        PROXY_BALANCER : on
        PROXY_CONNECT  : on
        PROXY_EXPRESS  : on
        PROXY_FCGI     : on
        PROXY_FDPASS   : on
        PROXY_FTP      : on
        PROXY_HTML     : on
        PROXY_HTTP     : on
        PROXY_SCGI     : on
        PROXY_WSTUNNEL : on
        RATELIMIT      : on
        REFLECTOR      : on
        REMOTEIP       : on
        REQTIMEOUT     : on
        REQUEST        : on
        REWRITE        : on
        SED            : on
        SESSION        : on
        SESSION_COOKIE : on
        SESSION_CRYPTO : on
        SESSION_DBD    : on
        SETENVIF       : on
        SLOTMEM_PLAIN  : on
        SLOTMEM_SHM    : on
        SOCACHE_DBM    : on
        SOCACHE_DC     : off
        SOCACHE_MEMCACHE: on
        SOCACHE_SHMCB  : on
        SPELING        : on
        SSL            : on
        STATUS         : off
        SUBSTITUTE     : on
        SUEXEC         : off
        UNIQUE_ID      : on
        USERDIR        : on
        USERTRACK      : on
        VERSION        : on
        VHOST_ALIAS    : on
        WATCHDOG       : off
        XML2ENC        : off
Shared Libs required:
        libpcre.so.1
        libgdbm.so.4
        libexpat.so.1
        libaprutil-1.so.0
        libapr-1.so.0
        libdb-5.3.so.0
        libxml2.so.2
Annotations    :
        cpe            : cpe:2.3:a:apache:http_server:2.4.20:::::freebsd10:x86:1
        repo_type      : binary
        repository     : Synth
Flat size      : 23.7MiB
Description    :
The Apache HTTP Server Project is an effort to develop and maintain an
open-source HTTP server for various modern desktop and server operating
systems, such as UNIX and Windows NT. The goal of this project is to
provide a secure, efficient and extensible server which provides HTTP
services in sync with the current HTTP standards.
The 2.x branch of Apache Web Server includes several improvements like
threading, use of APR, native IPv6 and SSL support, and many more.

WWW: http://httpd.apache.org/
Comment 3 Olli Hauer freebsd_committer 2016-05-25 03:40:58 UTC
This looks like an issue in the LD_LIBRARY_PATH.

Was there an alternative SSL (openssl from ports/libressl) installed during build time and later removed?

Can you run the following commands and post the output?

$ ldd /usr/local/libexec/apache24/mod_ssl.so
$ ldconfig -r | grep -e lssl -e lcrypto
$ apr-1-config --ldflags
$ apr-1-config --libs
Comment 4 Ron Wingfield 2016-05-25 08:02:13 UTC
(In reply to Olli Hauer from comment #3)

RE:  "Was there an alternative SSL (openssl from ports/libressl) installed during build time and later removed?"  --  Not to my knowledge; certainly would not have been my intention.  I used Synth to rebuild the app.

BTW, the reason for rebuilding was because something(?) totally destroyed/deleted the installation . . .I was using Synth to rebuild things MySQL server and client, p5-DBI and p5-DBD-mysql.  Next morning, I realized that apache24 was broken.

The output of the following commands as requested:

# ldd /usr/local/libexec/apache24/mod_ssl.so
/usr/local/libexec/apache24/mod_ssl.so:
        libssl.so.7 => /usr/lib/libssl.so.7 (0x28224000)
        libcrypto.so.7 => /lib/libcrypto.so.7 (0x28c00000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x28281000)
        libthr.so.3 => /lib/libthr.so.3 (0x282a5000)
        libc.so.7 => /lib/libc.so.7 (0x2806f000)


# ldconfig -r | grep -e lssl -e lcrypto
        4:-lcrypto.5 => /lib/libcrypto.so.5
        13:-lcrypto.7 => /lib/libcrypto.so.7
        52:-lssl.5 => /usr/lib/libssl.so.5
        86:-lssl.7 => /usr/lib/libssl.so.7
        170:-lcrypto.8 => /usr/local/lib/libcrypto.so.8
        428:-lcrypto.7 => /usr/local/lib/libcrypto.so.7
        429:-lssl.7 => /usr/local/lib/libssl.so.7
        447:-lssl.8 => /usr/local/lib/libssl.so.8
        563:-lcrypto.4 => /usr/local/lib/compat/libcrypto.so.4
        616:-lssl.4 => /usr/local/lib/compat/libssl.so.4
        652:-lssl3.1 => /usr/local/lib/nss/libssl3.so.1


# apr-1-config --ldflags
<nothing>


# apr-1-config --libs
 -lcrypt  -lpthread
Comment 5 Olli Hauer freebsd_committer 2016-05-25 18:35:12 UTC
The ldd output looks OK for apache, but only if no other third party module loads one of the other ssl/crypto libs.

The ldconfig -r output shows there are at last three different ssl versions on the system.
- openssl from base
- openssl from ports
- openssl from a previous FreeBSD release (freebsd-compat or saved by a port build tool)

Are there any third party module enabled in apache?
Do you have the build logs, if yes I would be interested to get a copy of the devel/apr1 and www/apache24 build logs.
Comment 6 Ron Wingfield 2016-05-25 21:09:44 UTC
Created attachment 170664 [details]
2 log files, 1 PDF
Comment 7 Ron Wingfield 2016-05-25 21:12:33 UTC
Hope this helps, Thanks!
Comment 8 Olli Hauer freebsd_committer 2016-05-26 05:47:07 UTC
Urg, please no pdf's (they are not readable with vi)

Please send them as simple plain text, if you like also per PM and compressed (gzip).
Comment 9 Ron Wingfield 2016-05-26 11:12:22 UTC
Created attachment 170673 [details]
Plain text vi readable.
Comment 10 Ron Wingfield 2016-05-26 11:19:10 UTC
. . .still learning this forum software.  Regarding the previous revised attachment, lost my comment to find the apache24 build log data begins at ~line #2271.
Comment 11 Olli Hauer freebsd_committer 2016-05-26 12:51:48 UTC
Hi Ron,

that's not a forum, it is a bug tracking tool :)

First thanks for the logs, unluckily I cannot see any issues in them.

Can you run the following command.
 $ nm /usr/lib/libssl.a | grep SSL_get_srp_userinfo
Comment 12 Olli Hauer freebsd_committer 2016-05-26 12:58:56 UTC
Ah, stop, I think I see whats going on.
 In the ldd output there are two ssl libs in /usr/lib
    52:-lssl.5 => /usr/lib/libssl.so.5
    86:-lssl.7 => /usr/lib/libssl.so.7

Can you temporary move /usr/lib/libssl.so.5 out to a different dir 
$ mkdir /usr/lib.old
$ mv /usr/lib/libssl.so.5 /usr/lib.old/
$ mv /lib/libcrypto.so.5 /usr/lib.old/

and try to start apache
Comment 13 Ron Wingfield 2016-05-26 13:40:01 UTC
Unfortunately, no joy per your suggestion:

# mkdir /usr/lib.old
# mv /usr/lib/libssl.so.5 /usr/lib.old/
# mv /lib/libcrypto.so.5 /usr/lib.old/

# apachectl -T
httpd: Syntax error on line 143 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: /usr/local/libexec/apache24/mod_ssl.so: Undefined symbol "SSL_get_srp_userinfo"



I've looked at the dates of the two versions of libssl.so.n
I can't explain the versions.  I will say that portmaster has done some ugly things in the past.
 

# ls -lh /usr/lib/libssl.so.*
-r--r--r--  1 root  wheel   367K Dec 25 23:06 libssl.so.7
# ls -lh /usr/lib.old/libssl.so.*
-r--r--r--  1 root  wheel   258K Jun  3  2015 /usr/lib.old/libssl.so.5

Regarding this "Undefined symbol "SSL_get_srp_userinfo"  . . .how and where would this code be introduced into the system?  Is there a missing ".h" header file?  . . .somewhere?
Comment 14 Ron Wingfield 2016-05-26 14:15:10 UTC
You've peaked my curiosity, and I've searched for variants: 

The following "finds" surprised me; where did this stuff come from?  (rhetorical question)

root@bravo:/usr/lib # find / -name 'libssl.so.*' -print

/usr/lib/lib.bu/libssl.so.5.bu   . . .probably could/should delete these 3 by now.
/usr/lib/lib.bu/libssl.so.6.bu   . . .
/usr/lib/lib.bu/libssl.so.bu     . . .

/usr/lib/libssl.so.7

/usr/local/lib/compat/libssl.so.4   ???  What made these 3?
/usr/local/lib/libssl.so.7          ???  What for and why?
/usr/local/lib/libssl.so.8          ???

/usr/lib.old/libssl.so.5    . . .we just made this one.
Comment 15 Ron Wingfield 2016-05-26 14:25:23 UTC
. . .and FYI:

# nm /usr/lib/libssl.a | grep SSL_get_srp_userinfo
00000eb0 T SSL_get_srp_userinfo
Comment 16 Olli Hauer freebsd_committer 2016-05-26 14:36:02 UTC
Only /usr/lib/libssl.so.7 comes from the OS, I can only suspect the lib.bu folders are backups.

/usr/lib/libssl.so.7    => FreeBSD 10.x

/usr/local/lib/compat/libssl.so.4   ???  possible the compat6 package
/usr/local/lib/libssl.so.7          ???  possible older openssl-1.0.x from ports
/usr/local/lib/libssl.so.8          ???  possible current openssl-1.1.x from ports
/usr/lib.old/libssl.so.5    . . .   ???  that one should be from FreeBSD 7.x

if you have the sources installed try the commands (but answer NO) to gat an overview (do not delete them)

$ cd /usr/src
$ make delete-old
$ make delete-old-libs

Would you mind to rebuild apache again (without the old lib in /usr/lib)
Comment 17 Miroslav Lachman 2016-05-26 14:56:02 UTC
(In reply to Olli Hauer from comment #16)

Right commands for checks are:

cd /usr/src/
make check-old           - List obsolete directories/files/libraries.

make check-old-dirs      - List obsolete directories.
make check-old-files     - List obsolete files.
make check-old-libs      - List obsolete libraries.
Comment 18 Olli Hauer freebsd_committer 2016-05-26 15:13:03 UTC
@Miroslav 
Thanks, for the correct check command!
Comment 19 Ron Wingfield 2016-05-26 15:54:51 UTC
(In reply to Miroslav Lachman from comment #17)

Gentlemen:  These make check commands fail, eg.

# make check-old-dirs
make: don't know how to make check-old-dirs. Stop


Regardless, I have used Synth to rebuild apache24:

# synth force www/apache24
Scanning existing packages.
 progress: 76.90%


The task is complete.  Final tally:
Initial queue size: 1
    packages built: 1
           ignored: 0
           skipped: 0
            failed: 0

Duration: 00:04:10
The build logs can be found at: /var/log/synth
Would you like to rebuild the local repository (Y/N)? y
Stand by, prescanning existing packages.
Stand by, recursively scanning 70 ports serially.
Scanning existing packages.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt
Would you like to upgrade your system with the new packages now (Y/N)? y
Updating Synth repository catalogue...
Fetching meta.txz: 100%    260 B   0.3kB/s    00:01
Fetching packagesite.txz: 100%   21 KiB  21.5kB/s    00:01
Processing entries: 100%
Synth repository update completed. 70 packages processed.
Checking integrity... done (0 conflicting)
The most recent version of packages are already installed


. . .not sure why Synth reported "The most recent version of packages are already installed".  

Still same error:   # apachectl -t
httpd: Syntax error on line 143 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: /usr/local/libexec/apache24/mod_ssl.so: Undefined symbol "SSL_get_srp_userinfo"


I will attach the  
# diff www___apache24.log-1 www___apache24.log-2 > diff-apache24.logs

with a follow-up comment
Comment 20 Ron Wingfield 2016-05-26 15:56:47 UTC
Created attachment 170686 [details]
diff of logs

diff of log as promised.
Comment 21 Olli Hauer freebsd_committer 2016-05-26 16:27:23 UTC
>  not sure why Synth reported "The most recent version of packages are already installed".

Because the installed version match the new version, therefore pkg will not update / reinstall the package without forcing it to do so.

In case you don't have any packages depending on apache installed try to remove the package 
$ pkg delete apache24 
and install apache again or 
$ pkg upgrade -f /var/synth/live_packages/All/apache24-2.4.20_1.txz
Comment 22 Ron Wingfield 2016-05-26 16:36:51 UTC

Summary of command line options - see synth.1 man page for more details
===============================================================================
synth status              Dry-run: Shows what 'upgrade-system' would build
~
synth build [ports]       Incrementally build ports based on given list, but
                          asks before updating repository and system
~
synth force [ports]       Like 'build', but deletes existing packages first


. . .you know, "force" sounds like it should have "replaced" the package.

I'll try the "pkg upgrade -f /var/synth/live_packages/All/apache24-2.4.20_1.txz" option.
Comment 23 Ron Wingfield 2016-05-26 17:20:17 UTC
Created attachment 170693 [details]
diffs of the httpd.conf files

Completed the rebuild/install.  There is this caveat:  (but I think applied to older upgrade, for example, from v22 to v24.)

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

- apache24 default build changed from static MPM to modular MPM
- more modules are now enabled per default in the port
- icons and error pages moved from WWWDIR to DATADIR

   If build with modular MPM and no MPM is activated in
   httpd.conf, then mpm_prefork will be activated as default
   MPM in etc/apache24/modules.d to keep compatibility with
   existing php/perl/python modules!

Please compare the existing httpd.conf with httpd.conf.sample
and merge missing modules/instructions into httpd.conf!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

I have diff'ed the two httpd.conf files (attached) and I need to take a few minutes to sort them out.  

BTW, running/testing utilizing the custom/production httpd.conf vs the new httpd.conf.sample (without merging the two) fails as per the following . . .same "Undefined symbol 'SSL_get_srp_userinfo'"

# apachectl -t
httpd: Syntax error on line 143 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: /usr/local/libexec/apache24/mod_ssl.so: Undefined symbol "SSL_get_srp_userinfo"
Comment 24 Ron Wingfield 2016-05-26 17:25:07 UTC
Created attachment 170694 [details]
Hopefully in plain text format

Oops . . .had converted the previous to WordPerfect format.
Comment 25 Olli Hauer freebsd_committer 2016-05-26 17:43:02 UTC
I think we will get nothing new without inspecting the base OS and finding out what is broken (e.g. make.conf vars old leftovers cleanup and so on)

As a stated per PM I'm offering to look at the system via a shared tmux session.
Comment 26 Markus Kohlmeyer 2016-05-26 17:50:01 UTC
IMO your system is kind of broken by leftovers from old base and ports. To prove this, you could for example have a look at the so-versions from

ldconfig -r | sort -b -d -f -k 2 -t '-'

and compare them to the coresponding base/port versions.

This might come from broken or incomplete binary updates/installs done with tools like freebsd-update or pkg. Another cause can be misconfigured, misused or incompatible/outdated pkg/port-management tools like synth, specifically when used in conjunction with ccache or out-of-date ports tree.



An other thing that has to be figured out is, where these *.bu directories come from. You have a lot of them:

find / -type d -name \*\.bu



As long as the system is not a mostly clean state, there is no real chance to get your current problem(s) solved.
Comment 27 Ron Wingfield 2016-05-26 18:59:45 UTC
(In reply to Olli Hauer from comment #25)

Olli, I tried to respond to your PM; however, the mail was rejected, complaining that my server has no PTR record in DNS.  Very odd.  This occurred on Tuesday, too, when I was trying to eMail to a client that I regularly mail to . . .an SBC-Global (AT&T) account.  Why this is happening, may have something to do with AT&T's U-verse.  I've had trouble with them before.  I think something has changed with their upstream Prodigy.net relay.  When I first converted from their legacy ADSL commercial account to a commercial U-verse account, they (AT&T) did not like the idea that I was operating our own DNS server, and were refusing to forward/relay any queries below their DNS server.  All this associated with a commercial 8-block of static IP's.  After a lot of screaming, they relinquished, but now the problem seems to have magically reappeared.  But this has nothing to do with the Apache problem  --  DNS and the mail server are running on a totally separate platform.

Back on topic, I am in the process of cleaning out the dead-wood -- *.bu's etc.  I still insist that anything renamed as *.bu is just as isolated as anything renamed *.foo or *.bar but I'll start cleaning.

Thanks to all!
Comment 28 Ron Wingfield 2016-05-26 19:21:55 UTC
(In reply to Markus Kohlmeyer from comment #26)

Markus, respectfully (and I mean respectfully) may I ask, would you feel better if I renamed a backup directory/file as "something.foobar" as opposed to "something.bu"?
Comment 29 Ron Wingfield 2016-05-29 16:33:38 UTC
Created attachment 170802 [details]
Details of solution.

Good morning Gentlemen:

I have analyzed, identified and solved the immediate problem, and I have attached a PDF copy of my solution.  I have discovered that the FreeBSD base (/usr/lib) instance of libssl.so.7 does NOT contain ANY symbols; however, libssl.a DOES.  This before and after upgrading the OS from v10.1 to v10.3.  The solution as noted in the attached PDF was to delete and reinstall Apache24, i.e., # make build install  Apache24 . . .with following caveat in /etc/make.conf:  WITH_OPENSSL_PORT=YES.  This directs Apache2.4.20_1 to incorporate the Ports version of OPENSSL.

Can this be a "bug" in the base version of OPENSSL?
Comment 30 Kurt Jaeger freebsd_committer 2016-06-27 19:17:53 UTC
(In reply to Ron Wingfield from comment #29)

/usr/lib/lib*.so* files are stripped in the general case on FreeBSD,
so that's not the cause of your troubles.

Comment 26, some leftovers from older installs/packages, seems to be
the cause. Difficult to clean up without a second system.