Just upgraded from Apache 2.4.18 to 2.4.20. Now mod_ssl.so cannot be loaded. The object was installed today as 178K May 24 13:01 mod_ssl.so Error msg: /usr/local/libexec/apache24/mod_ssl.so: Undefined symbol "SSL_get_srp_userinfo"
Thanks for the report. Please provide the output of `pkg info apache24`, if you built it from ports.
(In reply to Vladimir Krstulja from comment #1) Certainly, here 'tis: apache24-2.4.20_1 Name : apache24 Version : 2.4.20_1 Installed on : Tue May 24 13:03:12 2016 CDT Origin : www/apache24 Architecture : freebsd:10:x86:32 Prefix : /usr/local Categories : ipv6 www Licenses : Maintainer : apache@FreeBSD.org WWW : http://httpd.apache.org/ Comment : Version 2.4.x of Apache web server Options : ACCESS_COMPAT : on ACTIONS : on ALIAS : on ALLOWMETHODS : on ASIS : on AUTHNZ_FCGI : on AUTHNZ_LDAP : off AUTHN_ANON : on AUTHN_CORE : on AUTHN_DBD : on AUTHN_DBM : on AUTHN_FILE : on AUTHN_SOCACHE : on AUTHZ_CORE : on AUTHZ_DBD : on AUTHZ_DBM : on AUTHZ_GROUPFILE: on AUTHZ_HOST : on AUTHZ_OWNER : on AUTHZ_USER : on AUTH_BASIC : on AUTH_DIGEST : on AUTH_FORM : on AUTOINDEX : on BUCKETEER : off BUFFER : on CACHE : on CACHE_DISK : on CACHE_SOCACHE : on CASE_FILTER : off CASE_FILTER_IN : off CERN_META : on CGI : on CGID : on CHARSET_LITE : on DATA : on DAV : on DAV_FS : on DAV_LOCK : on DBD : on DEFLATE : on DIALUP : on DIR : on DUMPIO : on ECHO : off ENV : on EXAMPLE_HOOKS : off EXAMPLE_IPC : off EXPIRES : on EXT_FILTER : on FILE_CACHE : on FILTER : on HEADERS : on HEARTBEAT : off HEARTMONITOR : off HTTP2 : off IDENT : off IMAGEMAP : on INCLUDE : on INFO : on IPV4_MAPPED : off LBMETHOD_BYBUSYNESS: on LBMETHOD_BYREQUESTS: on LBMETHOD_BYTRAFFIC: on LBMETHOD_HEARTBEAT: off LDAP : off LOGIO : on LOG_DEBUG : on LOG_FORENSIC : on LUA : off LUAJIT : off MACRO : on MIME : on MIME_MAGIC : on MPM_EVENT : off MPM_PREFORK : on MPM_SHARED : on MPM_WORKER : off NEGOTIATION : on OPTIONAL_FN_EXPORT: off OPTIONAL_FN_IMPORT: off OPTIONAL_HOOK_EXPORT: off OPTIONAL_HOOK_IMPORT: off PROXY : on PROXY_AJP : on PROXY_BALANCER : on PROXY_CONNECT : on PROXY_EXPRESS : on PROXY_FCGI : on PROXY_FDPASS : on PROXY_FTP : on PROXY_HTML : on PROXY_HTTP : on PROXY_SCGI : on PROXY_WSTUNNEL : on RATELIMIT : on REFLECTOR : on REMOTEIP : on REQTIMEOUT : on REQUEST : on REWRITE : on SED : on SESSION : on SESSION_COOKIE : on SESSION_CRYPTO : on SESSION_DBD : on SETENVIF : on SLOTMEM_PLAIN : on SLOTMEM_SHM : on SOCACHE_DBM : on SOCACHE_DC : off SOCACHE_MEMCACHE: on SOCACHE_SHMCB : on SPELING : on SSL : on STATUS : off SUBSTITUTE : on SUEXEC : off UNIQUE_ID : on USERDIR : on USERTRACK : on VERSION : on VHOST_ALIAS : on WATCHDOG : off XML2ENC : off Shared Libs required: libpcre.so.1 libgdbm.so.4 libexpat.so.1 libaprutil-1.so.0 libapr-1.so.0 libdb-5.3.so.0 libxml2.so.2 Annotations : cpe : cpe:2.3:a:apache:http_server:2.4.20:::::freebsd10:x86:1 repo_type : binary repository : Synth Flat size : 23.7MiB Description : The Apache HTTP Server Project is an effort to develop and maintain an open-source HTTP server for various modern desktop and server operating systems, such as UNIX and Windows NT. The goal of this project is to provide a secure, efficient and extensible server which provides HTTP services in sync with the current HTTP standards. The 2.x branch of Apache Web Server includes several improvements like threading, use of APR, native IPv6 and SSL support, and many more. WWW: http://httpd.apache.org/
This looks like an issue in the LD_LIBRARY_PATH. Was there an alternative SSL (openssl from ports/libressl) installed during build time and later removed? Can you run the following commands and post the output? $ ldd /usr/local/libexec/apache24/mod_ssl.so $ ldconfig -r | grep -e lssl -e lcrypto $ apr-1-config --ldflags $ apr-1-config --libs
(In reply to Olli Hauer from comment #3) RE: "Was there an alternative SSL (openssl from ports/libressl) installed during build time and later removed?" -- Not to my knowledge; certainly would not have been my intention. I used Synth to rebuild the app. BTW, the reason for rebuilding was because something(?) totally destroyed/deleted the installation . . .I was using Synth to rebuild things MySQL server and client, p5-DBI and p5-DBD-mysql. Next morning, I realized that apache24 was broken. The output of the following commands as requested: # ldd /usr/local/libexec/apache24/mod_ssl.so /usr/local/libexec/apache24/mod_ssl.so: libssl.so.7 => /usr/lib/libssl.so.7 (0x28224000) libcrypto.so.7 => /lib/libcrypto.so.7 (0x28c00000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x28281000) libthr.so.3 => /lib/libthr.so.3 (0x282a5000) libc.so.7 => /lib/libc.so.7 (0x2806f000) # ldconfig -r | grep -e lssl -e lcrypto 4:-lcrypto.5 => /lib/libcrypto.so.5 13:-lcrypto.7 => /lib/libcrypto.so.7 52:-lssl.5 => /usr/lib/libssl.so.5 86:-lssl.7 => /usr/lib/libssl.so.7 170:-lcrypto.8 => /usr/local/lib/libcrypto.so.8 428:-lcrypto.7 => /usr/local/lib/libcrypto.so.7 429:-lssl.7 => /usr/local/lib/libssl.so.7 447:-lssl.8 => /usr/local/lib/libssl.so.8 563:-lcrypto.4 => /usr/local/lib/compat/libcrypto.so.4 616:-lssl.4 => /usr/local/lib/compat/libssl.so.4 652:-lssl3.1 => /usr/local/lib/nss/libssl3.so.1 # apr-1-config --ldflags <nothing> # apr-1-config --libs -lcrypt -lpthread
The ldd output looks OK for apache, but only if no other third party module loads one of the other ssl/crypto libs. The ldconfig -r output shows there are at last three different ssl versions on the system. - openssl from base - openssl from ports - openssl from a previous FreeBSD release (freebsd-compat or saved by a port build tool) Are there any third party module enabled in apache? Do you have the build logs, if yes I would be interested to get a copy of the devel/apr1 and www/apache24 build logs.
Created attachment 170664 [details] 2 log files, 1 PDF
Hope this helps, Thanks!
Urg, please no pdf's (they are not readable with vi) Please send them as simple plain text, if you like also per PM and compressed (gzip).
Created attachment 170673 [details] Plain text vi readable.
. . .still learning this forum software. Regarding the previous revised attachment, lost my comment to find the apache24 build log data begins at ~line #2271.
Hi Ron, that's not a forum, it is a bug tracking tool :) First thanks for the logs, unluckily I cannot see any issues in them. Can you run the following command. $ nm /usr/lib/libssl.a | grep SSL_get_srp_userinfo
Ah, stop, I think I see whats going on. In the ldd output there are two ssl libs in /usr/lib 52:-lssl.5 => /usr/lib/libssl.so.5 86:-lssl.7 => /usr/lib/libssl.so.7 Can you temporary move /usr/lib/libssl.so.5 out to a different dir $ mkdir /usr/lib.old $ mv /usr/lib/libssl.so.5 /usr/lib.old/ $ mv /lib/libcrypto.so.5 /usr/lib.old/ and try to start apache
Unfortunately, no joy per your suggestion: # mkdir /usr/lib.old # mv /usr/lib/libssl.so.5 /usr/lib.old/ # mv /lib/libcrypto.so.5 /usr/lib.old/ # apachectl -T httpd: Syntax error on line 143 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: /usr/local/libexec/apache24/mod_ssl.so: Undefined symbol "SSL_get_srp_userinfo" I've looked at the dates of the two versions of libssl.so.n I can't explain the versions. I will say that portmaster has done some ugly things in the past. # ls -lh /usr/lib/libssl.so.* -r--r--r-- 1 root wheel 367K Dec 25 23:06 libssl.so.7 # ls -lh /usr/lib.old/libssl.so.* -r--r--r-- 1 root wheel 258K Jun 3 2015 /usr/lib.old/libssl.so.5 Regarding this "Undefined symbol "SSL_get_srp_userinfo" . . .how and where would this code be introduced into the system? Is there a missing ".h" header file? . . .somewhere?
You've peaked my curiosity, and I've searched for variants: The following "finds" surprised me; where did this stuff come from? (rhetorical question) root@bravo:/usr/lib # find / -name 'libssl.so.*' -print /usr/lib/lib.bu/libssl.so.5.bu . . .probably could/should delete these 3 by now. /usr/lib/lib.bu/libssl.so.6.bu . . . /usr/lib/lib.bu/libssl.so.bu . . . /usr/lib/libssl.so.7 /usr/local/lib/compat/libssl.so.4 ??? What made these 3? /usr/local/lib/libssl.so.7 ??? What for and why? /usr/local/lib/libssl.so.8 ??? /usr/lib.old/libssl.so.5 . . .we just made this one.
. . .and FYI: # nm /usr/lib/libssl.a | grep SSL_get_srp_userinfo 00000eb0 T SSL_get_srp_userinfo
Only /usr/lib/libssl.so.7 comes from the OS, I can only suspect the lib.bu folders are backups. /usr/lib/libssl.so.7 => FreeBSD 10.x /usr/local/lib/compat/libssl.so.4 ??? possible the compat6 package /usr/local/lib/libssl.so.7 ??? possible older openssl-1.0.x from ports /usr/local/lib/libssl.so.8 ??? possible current openssl-1.1.x from ports /usr/lib.old/libssl.so.5 . . . ??? that one should be from FreeBSD 7.x if you have the sources installed try the commands (but answer NO) to gat an overview (do not delete them) $ cd /usr/src $ make delete-old $ make delete-old-libs Would you mind to rebuild apache again (without the old lib in /usr/lib)
(In reply to Olli Hauer from comment #16) Right commands for checks are: cd /usr/src/ make check-old - List obsolete directories/files/libraries. make check-old-dirs - List obsolete directories. make check-old-files - List obsolete files. make check-old-libs - List obsolete libraries.
@Miroslav Thanks, for the correct check command!
(In reply to Miroslav Lachman from comment #17) Gentlemen: These make check commands fail, eg. # make check-old-dirs make: don't know how to make check-old-dirs. Stop Regardless, I have used Synth to rebuild apache24: # synth force www/apache24 Scanning existing packages. progress: 76.90% The task is complete. Final tally: Initial queue size: 1 packages built: 1 ignored: 0 skipped: 0 failed: 0 Duration: 00:04:10 The build logs can be found at: /var/log/synth Would you like to rebuild the local repository (Y/N)? y Stand by, prescanning existing packages. Stand by, recursively scanning 70 ports serially. Scanning existing packages. Packages validated, rebuilding local repository. Local repository successfully rebuilt Would you like to upgrade your system with the new packages now (Y/N)? y Updating Synth repository catalogue... Fetching meta.txz: 100% 260 B 0.3kB/s 00:01 Fetching packagesite.txz: 100% 21 KiB 21.5kB/s 00:01 Processing entries: 100% Synth repository update completed. 70 packages processed. Checking integrity... done (0 conflicting) The most recent version of packages are already installed . . .not sure why Synth reported "The most recent version of packages are already installed". Still same error: # apachectl -t httpd: Syntax error on line 143 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: /usr/local/libexec/apache24/mod_ssl.so: Undefined symbol "SSL_get_srp_userinfo" I will attach the # diff www___apache24.log-1 www___apache24.log-2 > diff-apache24.logs with a follow-up comment
Created attachment 170686 [details] diff of logs diff of log as promised.
> not sure why Synth reported "The most recent version of packages are already installed". Because the installed version match the new version, therefore pkg will not update / reinstall the package without forcing it to do so. In case you don't have any packages depending on apache installed try to remove the package $ pkg delete apache24 and install apache again or $ pkg upgrade -f /var/synth/live_packages/All/apache24-2.4.20_1.txz
Summary of command line options - see synth.1 man page for more details =============================================================================== synth status Dry-run: Shows what 'upgrade-system' would build ~ synth build [ports] Incrementally build ports based on given list, but asks before updating repository and system ~ synth force [ports] Like 'build', but deletes existing packages first . . .you know, "force" sounds like it should have "replaced" the package. I'll try the "pkg upgrade -f /var/synth/live_packages/All/apache24-2.4.20_1.txz" option.
Created attachment 170693 [details] diffs of the httpd.conf files Completed the rebuild/install. There is this caveat: (but I think applied to older upgrade, for example, from v22 to v24.) !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - apache24 default build changed from static MPM to modular MPM - more modules are now enabled per default in the port - icons and error pages moved from WWWDIR to DATADIR If build with modular MPM and no MPM is activated in httpd.conf, then mpm_prefork will be activated as default MPM in etc/apache24/modules.d to keep compatibility with existing php/perl/python modules! Please compare the existing httpd.conf with httpd.conf.sample and merge missing modules/instructions into httpd.conf! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! I have diff'ed the two httpd.conf files (attached) and I need to take a few minutes to sort them out. BTW, running/testing utilizing the custom/production httpd.conf vs the new httpd.conf.sample (without merging the two) fails as per the following . . .same "Undefined symbol 'SSL_get_srp_userinfo'" # apachectl -t httpd: Syntax error on line 143 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: /usr/local/libexec/apache24/mod_ssl.so: Undefined symbol "SSL_get_srp_userinfo"
Created attachment 170694 [details] Hopefully in plain text format Oops . . .had converted the previous to WordPerfect format.
I think we will get nothing new without inspecting the base OS and finding out what is broken (e.g. make.conf vars old leftovers cleanup and so on) As a stated per PM I'm offering to look at the system via a shared tmux session.
IMO your system is kind of broken by leftovers from old base and ports. To prove this, you could for example have a look at the so-versions from ldconfig -r | sort -b -d -f -k 2 -t '-' and compare them to the coresponding base/port versions. This might come from broken or incomplete binary updates/installs done with tools like freebsd-update or pkg. Another cause can be misconfigured, misused or incompatible/outdated pkg/port-management tools like synth, specifically when used in conjunction with ccache or out-of-date ports tree. An other thing that has to be figured out is, where these *.bu directories come from. You have a lot of them: find / -type d -name \*\.bu As long as the system is not a mostly clean state, there is no real chance to get your current problem(s) solved.
(In reply to Olli Hauer from comment #25) Olli, I tried to respond to your PM; however, the mail was rejected, complaining that my server has no PTR record in DNS. Very odd. This occurred on Tuesday, too, when I was trying to eMail to a client that I regularly mail to . . .an SBC-Global (AT&T) account. Why this is happening, may have something to do with AT&T's U-verse. I've had trouble with them before. I think something has changed with their upstream Prodigy.net relay. When I first converted from their legacy ADSL commercial account to a commercial U-verse account, they (AT&T) did not like the idea that I was operating our own DNS server, and were refusing to forward/relay any queries below their DNS server. All this associated with a commercial 8-block of static IP's. After a lot of screaming, they relinquished, but now the problem seems to have magically reappeared. But this has nothing to do with the Apache problem -- DNS and the mail server are running on a totally separate platform. Back on topic, I am in the process of cleaning out the dead-wood -- *.bu's etc. I still insist that anything renamed as *.bu is just as isolated as anything renamed *.foo or *.bar but I'll start cleaning. Thanks to all!
(In reply to Markus Kohlmeyer from comment #26) Markus, respectfully (and I mean respectfully) may I ask, would you feel better if I renamed a backup directory/file as "something.foobar" as opposed to "something.bu"?
Created attachment 170802 [details] Details of solution. Good morning Gentlemen: I have analyzed, identified and solved the immediate problem, and I have attached a PDF copy of my solution. I have discovered that the FreeBSD base (/usr/lib) instance of libssl.so.7 does NOT contain ANY symbols; however, libssl.a DOES. This before and after upgrading the OS from v10.1 to v10.3. The solution as noted in the attached PDF was to delete and reinstall Apache24, i.e., # make build install Apache24 . . .with following caveat in /etc/make.conf: WITH_OPENSSL_PORT=YES. This directs Apache2.4.20_1 to incorporate the Ports version of OPENSSL. Can this be a "bug" in the base version of OPENSSL?
(In reply to Ron Wingfield from comment #29) /usr/lib/lib*.so* files are stripped in the general case on FreeBSD, so that's not the cause of your troubles. Comment 26, some leftovers from older installs/packages, seems to be the cause. Difficult to clean up without a second system.