Bug 225499

Summary: www/apache24: child pid exit signal Segmentation fault
Product: Ports & Packages Reporter: O. Hartmann <ohartmann>
Component: Individual Port(s)Assignee: freebsd-apache (Nobody) <apache>
Status: Closed Not A Bug    
Severity: Affects Many People CC: brnrd, greenreaper, joneum, tz, w.schwarzenfeld
Priority: ---    
Version: Latest   
Hardware: Any   
OS: Any   

Description O. Hartmann 2018-01-28 09:30:59 UTC
On most recent CURRENT (FreeBSD 12.0-CURRENT #77 r328474: Sat Jan 27 14:00:13 CET 2018 amd64) with CLANG 6.0.0, Apache server apache24-2.4.29 seems to crasch with:

[...]
[Sun Jan 28 08:23:37.411040 2018] [core:notice] [pid 86932:tid 34376777728] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
[Sun Jan 28 08:24:16.503091 2018] [core:notice] [pid 86932:tid 34376777728] AH00052: child pid 1639 exit signal Segmentation fault (11)
[Sun Jan 28 08:24:16.503193 2018] [core:notice] [pid 86932:tid 34376777728] AH00052: child pid 99579 exit signal Segmentation fault (11)
[Sun Jan 28 08:24:16.503209 2018] [core:notice] [pid 86932:tid 34376777728] AH00052: child pid 99174 exit signal Segmentation fault (11)
[Sun Jan 28 08:24:17.516989 2018] [core:notice] [pid 86932:tid 34376777728] AH00052: child pid 8563 exit signal Segmentation fault (11)
[Sun Jan 28 08:24:18.524138 2018] [core:notice] [pid 86932:tid 34376777728] AH00052: child pid 40873 exit signal Segmentation fault (11)
[Sun Jan 28 08:24:18.524238 2018] [core:notice] [pid 86932:tid 34376777728] AH00052: child pid 38477 exit signal Segmentation fault (11)
[Sun Jan 28 08:24:19.551776 2018] [core:notice] [pid 86932:tid 34376777728] AH00052: child pid 4649 exit signal Segmentation fault (11)
[...]

whenever configurations where hit, where webdav (mod_dav_svn-1.9.7) has been configured. But this is just a hunch, since www/nextcloud crashes, and the webinterface of textproc/refdb trigger these crashes with a resulting non-accessibility of the service. All other apache24 driven/related services, like remote subversion via "URL: https://..." work properly or the web view of poudriere, locally as well as remotely from other sites.

The problem arose suddenly! Prior to CURRENT rr328400 it seems to work, but I can not aure, since I did several CURRENT updates AND ports-tree updates since then.

Below, you'll find the "showconfig" output.


 # make showconfig
===> The following configuration options are available for apache24-2.4.29:
     ACCESS_COMPAT=on: mod_access compatibility
     ACTIONS=on: Action triggering on requests
     ALIAS=on: Mapping of requests to different filesystem parts
     ALLOWMETHODS=on: Restrict allowed HTTP methods
     ASIS=on: Sends files that contain their own HTTP headers
     AUTHNZ_FCGI=on: FastCGI authorizer-based authentication and authorization
     AUTHNZ_LDAP=on: LDAP based authentication
     AUTHN_ANON=on: Anonymous user authentication control
     AUTHN_CORE=on: Core authentication module
     AUTHN_DBD=on: SQL-based authentication control
     AUTHN_DBM=on: DBM-based authentication control
     AUTHN_FILE=on: File-based authentication control
     AUTHN_SOCACHE=on: Cached authentication control
     AUTHZ_CORE=on: Core authorization provider vector module
     AUTHZ_DBD=on: SQL based authorization and Login/Session support
     AUTHZ_DBM=on: DBM-based authorization control
     AUTHZ_GROUPFILE=on: "require group" authorization control
     AUTHZ_HOST=on: Host-based authorization control
     AUTHZ_OWNER=on: "require file-owner" authorization control
     AUTHZ_USER=on: "require user" authorization control
     AUTH_BASIC=on: Basic authentication
     AUTH_DIGEST=on: RFC2617 Digest authentication
     AUTH_FORM=on: Form authentication
     AUTOINDEX=on: Directory listing
     BROTLI=off: Brotli compression support
     BUFFER=on: Filter Buffering
     CACHE=on: Dynamic file caching
     CACHE_DISK=on: Disk caching module
     CACHE_SOCACHE=on: Shared object cacheing module
     CERN_META=on: CERN-type meta files
     CGI=on: CGI scripts (non-threaded MPMs)
     CGID=on: CGI scripts (threaded MPMs)
     CHARSET_LITE=on: Character set translation. Enabled by default only on EBCDIC systems
     DATA=on: RFC2397 data encoder
     DAV=on: WebDAV protocol handling. --enable-dav also enables mod_dav_fs
     DAV_FS=on: DAV provider for the filesystem. --enable-dav also enables mod_dav_fs
     DAV_LOCK=on: DAV provider for generic locking
     DBD=on: Apache DBD Framework
     DEFLATE=on: Deflate transfer encoding support
     DIALUP=on: Rate limits static files to dialup modem speeds
     DIR=on: Directory request handling
     DUMPIO=on: I/O dump filter
     ENV=on: Clearing/setting of ENV vars
     EXPIRES=on: Expires header control
     EXT_FILTER=on: External filter module
     FILE_CACHE=on: File cache
     FILTER=on: Smart Filtering
     HEADERS=on: HTTP header control
     HEARTBEAT=on: Generates Heartbeats
     HEARTMONITOR=on: Collects Heartbeats
     HTTP2=off: HTTP/2 (RFC 7540) support
     IDENT=off: RFC 1413 ident lookups
     IMAGEMAP=on: Server-side imagemaps
     INCLUDE=on: Server-side includes
     INFO=on: Server information
     IPV4_MAPPED=on: Allow IPv6 sockets to handle IPv4 connections
     LBMETHOD_BYBUSYNESS=on: Apache proxy Load balancing by busyness
     LBMETHOD_BYREQUESTS=on: Apache proxy Load balancing by request counting
     LBMETHOD_BYTRAFFIC=on: Apache proxy Load balancing by traffic counting
     LBMETHOD_HEARTBEAT=on: Apache proxy Load balancing from Heartbeats
     LDAP=on: LDAP caching and connection pooling services
     LOGIO=on: Input and output logging
     LOG_DEBUG=on: Configurable debug logging
     LOG_FORENSIC=on: Forensic logging
     LUA=off: Apache Lua Framework
     LUAJIT=off: LuaJit Support
     MACRO=on: Define and use macros in configuration files
     MIME=on: Mapp file-ext. to MIME (recommended)
     MIME_MAGIC=on: Automagically determining MIME type
     NEGOTIATION=on: Content negotiation
     PROXY=on: Build enabled PROXY modules
     RATELIMIT=on: Output Bandwidth Limiting
     REFLECTOR=on: Reflect request through the output filter stack
     REMOTEIP=on: Translate header contents to an apparent client remote_ip
     REQTIMEOUT=on: Limit time waiting for request from client
     REQUEST=on: Request Body Filtering
     REWRITE=on: Rule based URL manipulation
     SED=on: Filter request and/or response bodies through sed
     SESSION=on: Build enabled SESSION modules
     SETENVIF=on: Modify ENV vars based on characteristics of the request
     SLOTMEM_PLAIN=on: Slotmem provider that uses plain memory
     SLOTMEM_SHM=on: Slotmem provider that uses shared memory
     SOCACHE_DBM=on: dbm small object cache provider
     SOCACHE_DC=off: distcache small object cache provider
     SOCACHE_MEMCACHE=on: memcache small object cache provider
     SOCACHE_SHMCB=on: shmcb small object cache provider
     SPELING=on: Correct common URL misspellings
     SSL=on: SSL/TLS support (mod_ssl)
     STATUS=on: Process/thread monitoring
     SUBSTITUTE=on: Response content rewrite-like filtering
     SUEXEC=off: Set uid and gid for spawned processes
     UNIQUE_ID=on: Per-request unique ids
     USERDIR=on: Mapping of requests to user-specific directories
     USERTRACK=on: User-session tracking
     VERSION=on: Determining httpd version in config files
     VHOST_ALIAS=on: Mass virtual hosting
     WATCHDOG=on: Watchdog module
     XML2ENC=on: i18n support for markup filters
====> Build enabled PROXY modules: you have to choose at least one of them
     PROXY_AJP=on: AJP support module for mod_proxy
     PROXY_BALANCER=on: mod_proxy extension for load balancing
     PROXY_CONNECT=on: mod_proxy extension for CONNECT request handling
     PROXY_EXPRESS=on: Dynamic mass reverse proxy extension for mod_proxy
     PROXY_FCGI=on: FastCGI support module for mod_proxy
     PROXY_HTTP2=off: HTTP/2 support module for h2 and h2c
     PROXY_FDPASS=on: fdpass external process support module for mod_proxy
     PROXY_FTP=on: FTP support module for mod_proxy
     PROXY_HCHECK=on: Dynamic health check of Balancer members (workers) for mod_proxy
     PROXY_HTML=on: Fix HTML Links in a Reverse Proxy
     PROXY_HTTP=on: HTTP support module for mod_proxy
     PROXY_SCGI=on: SCGI gateway module for mod_proxy
     PROXY_WSTUNNEL=on: Websockets Tunnel module for mod_proxy
====> Build enabled SESSION modules: you have to choose at least one of them
     SESSION_COOKIE=on: Session cookie module
     SESSION_CRYPTO=on: Session crypto module
     SESSION_DBD=on: Session dbd module
====> Example and devel modules (do not use in prod)
     BUCKETEER=off: (dev) buckets manipulation filter
     CASE_FILTER=off: (dev) example uppercase conversion filter
     CASE_FILTER_IN=off: (dev) example uppercase conversion input filter
     ECHO=off: (dev) example echo server
     EXAMPLE_HOOKS=off: (dev) example hook callback handler module
     EXAMPLE_IPC=off: (dev) example of shared memory and mutex usage
     OPTIONAL_FN_EXPORT=off: (dev) example optional function exporter
     OPTIONAL_FN_IMPORT=off: (dev) example optional function importer
     OPTIONAL_HOOK_EXPORT=off: (dev) example optional hook exporter
     OPTIONAL_HOOK_IMPORT=off: (dev) example optional hook importer
====> The default MPM module: you have to select exactly one of them
     MPM_PREFORK=off: non-threaded, pre-forking web server
     MPM_WORKER=on: hybrid multi-threaded multi-process web server
     MPM_EVENT=off: MPM worker variant with the goal of consuming threads only for connections with active processing
====> Build all MPMs as shared Module: you can only select none or one of them
     MPM_SHARED=on: all MPMs as loadable module
===> Use 'make config' to modify these settings
Comment 1 O. Hartmann 2018-01-28 09:32:06 UTC
*** Bug 225483 has been marked as a duplicate of this bug. ***
Comment 2 O. Hartmann 2018-02-03 11:00:59 UTC
I followed an advice checking the debugging output on apache24 when it crashes. I can provide the following (minimalistic) information:

[ from starting recent www/apache24 with gdb ]

Starting program: /usr/local/sbin/httpd -X
[Sat Feb 03 11:41:27.704031 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such file
or directory: AH00075: Failed to enable the 'httpready' Accept Filter [Sat Feb 03
11:41:27.704359 2018] [core:warn] [pid 59320:tid 34376777728] (2)No such file or
directory: AH00075: Failed to enable the 'dataready' Accept Filter [New LWP 101096 of
process 59320] [New LWP 101105 of process 59320] [New LWP 101109 of process 59320]
[New LWP 101103 of process 59320]
[New LWP 101102 of process 59320]
[New LWP 101110 of process 59320]
[New LWP 101107 of process 59320]
[New LWP 101098 of process 59320]
[New LWP 101099 of process 59320]
[New LWP 101108 of process 59320]
[New LWP 101112 of process 59320]
[New LWP 101101 of process 59320]
[New LWP 101104 of process 59320]
[New LWP 101097 of process 59320]
[New LWP 101106 of process 59320]
[New LWP 101111 of process 59320]
[New LWP 101100 of process 59320]
[LWP 101096 of process 59320 exited]

Thread 11 received signal SIGSEGV, Segmentation fault.
[Switching to LWP 101108 of process 59320]
0x0000000802300382 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so

[...]

lang/php56 is compiled with option ZTS enabled.

www/mod_php56 has the following options set:

===> The following configuration options are available for mod_php56-5.6.33:
     AP2FILTER=off: Use Apache 2.x filter interface (experimental)
     PHPDBG=off
     DEBUG=off: Enable debug
     DTRACE=off: Enable DTrace support
     IPV6=on: IPv6 protocol support
     MAILHEAD=off: Enable mail header patch
     LINKTHR=on: Link thread lib (for threaded extensions)
     ZTS=on: Force Zend Thread Safety (ZTS) build
===> Use 'make config' to modify these settings

Option ZTS is strongly recommende, even required, when having a threaded Apache server.
Comment 3 Walter Schwarzenfeld 2018-02-03 12:08:52 UTC
 Failed to enable the 'httpready' Accept Filter or Failed to enable the 'dataready' Accept Filter
are typical errors if
accf_http_load="YES"
accf_data_load="YES"

is not set in boot/loader.conf.
Comment 4 Bernard Spil freebsd_committer freebsd_triage 2018-02-12 14:36:27 UTC
I've seen many of these errors when modules load in an incorrect order. Do you have the new modules layout?
Comment 5 O. Hartmann 2018-02-12 19:15:23 UTC
(In reply to w.schwarzenfeld from comment #3)


I haven't changed anything critical in the configuration and the best indication for that is, that when I first reported this bug, it has been shown itself on the first boxes I've updated. By the end of last week I updated several other boxes running Apache - updating CURRENT, updating ports tree, never touched the configuration. They all show the very same error now.
Comment 6 Bernard Spil freebsd_committer freebsd_triage 2018-02-12 19:55:25 UTC
Does php-cli work OK? (invoke as php) and please post output of `php -M` using the same config as your mod_php56 uses.

It looks like you're using apache with worker MPM, have you tried the new (and faster) event MPM?
Comment 7 O. Hartmann 2018-02-12 20:12:49 UTC
The problems occured when updating php56-5.6.32 -> php56-5.6.33 last week.

php-cli seems to work, invoked via "php -a":
Interactive mode enabled

php > echo function_exists("foo") ? "yes" : "no";
no
php > 

Is this right? really php -M?
# php -M
Usage: php [options] [-f] <file> [--] [args...]
   php [options] -r <code> [--] [args...]
   php [options] [-B <begin_code>] -R <code> [-E <end_code>] [--] [args...]
   php [options] [-B <begin_code>] -F <file> [-E <end_code>] [--] [args...]
   php [options] -S <addr>:<port> [-t docroot]
   php [options] -- [args...]
   php [options] -a

  -a               Run interactively
  -c <path>|<file> Look for php.ini file in this directory
  -n               No php.ini file will be used
  -d foo[=bar]     Define INI entry foo with value 'bar'
  -e               Generate extended information for debugger/profiler
  -f <file>        Parse and execute <file>.
  -h               This help
  -i               PHP information
  -l               Syntax check only (lint)
  -m               Show compiled in modules
  -r <code>        Run PHP <code> without using script tags <?..?>
  -B <begin_code>  Run PHP <begin_code> before processing input lines
  -R <code>        Run PHP <code> for every input line
  -F <file>        Parse and execute <file> for every input line
  -E <end_code>    Run PHP <end_code> after processing all input lines
  -H               Hide any passed arguments from external tools.
  -S <addr>:<port> Run with built-in web server.
  -t <docroot>     Specify document root <docroot> for built-in web server.
  -s               Output HTML syntax highlighted source.
  -v               Version number
  -w               Output source with stripped comments and whitespace.
  -z <file>        Load Zend extension <file>.

  args...          Arguments passed to script. Use -- args when first argument
                   starts with - or script is read from stdin

  --ini            Show configuration file names

  --rf <name>      Show information about function <name>.
  --rc <name>      Show information about class <name>.
  --re <name>      Show information about extension <name>.
  --rz <name>      Show information about Zend extension <name>.
  --ri <name>      Show configuration for extension <name>.
Comment 8 Bernard Spil freebsd_committer freebsd_triage 2018-02-13 10:25:15 UTC
(In reply to O. Hartmann from comment #7)

Added tz@ hoping he can chip in from the PHP side.

So we've established that PHP actually can interpret things. Had't read the complete thread previously. It also segfaults when running occ upgrade on the cli.

I meant output of `php -m` so you can check if all your modules are loading OK. Alternatively generate phpinfo() to see what loads and with what params.

I don't really have a clue what triggers this (but I have a port MariaDB 5.5 that doesn't build on CURRENT...). Were any Spectre/Meltdown mitigations added in the newer CURRENT you use?

If your objective is running Nextcloud, you can run it with PHP 7.2 (after update to Nextcloud 13, see PR #223778). Works fine for me. Running PHP 5.6 (ancient) on FreeBSD CURRENT (bleeding edge) is kind of weird... You will need to upgrade PHP by end of this year anyway. You will have more flexibility if you run PHP-FPM in jails (either unix domain sockets or TCP) as you can run multiple PHP versions in parallel.

Your apache is built with all modules, I hope you're not loading them all at runtime. Can you provide output of httpd -M (this one capital M) so we can see what you load?

If I were to try and fix this for myself, I'd go back to bare minimum config and enable modules one-by-one to figure out what makes it crash. You now have both Apache and PHP with dynamically loaded modules, the number of combinations is pretty much unlimited.
Comment 9 O. Hartmann 2018-02-14 08:36:09 UTC
(In reply to Bernard Spil from comment #8)

Sorry for being so dumb on "php -m"! I'm a bit overstressed right now, so I wasn't even capable of getting the fact that it mus have been lowercase "em" insteda of its uppercase sibbling, even when the fact was sitting in front of me!

At first: yes, I build "everything" (modules) as it is the standard of the port and no, I do not load everything at runtime, since this is also considered a security risc.

Below, you'll find the modules I load a t the moment. I'm quite sure that I do not need all of them, but I stopped  tweaking. Authentication is done via our OpenLDAP backend!

Furthermore, I'll provide the "php -m" output, also attached below.

# httpd -M
Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 mpm_event_module (shared)
 authn_file_module (shared)
 authn_core_module (shared)
 authz_host_module (shared)
 authz_groupfile_module (shared)
 authz_user_module (shared)
 authz_core_module (shared)
 authnz_ldap_module (shared)
 access_compat_module (shared)
 auth_basic_module (shared)
 socache_shmcb_module (shared)
 socache_memcache_module (shared)
 reqtimeout_module (shared)
 filter_module (shared)
 mime_module (shared)
 ldap_module (shared)
 log_config_module (shared)
 logio_module (shared)
 env_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 headers_module (shared)
 setenvif_module (shared)
 version_module (shared)
 session_module (shared)
 session_cookie_module (shared)
 session_crypto_module (shared)
 ssl_module (shared)
 unixd_module (shared)
 status_module (shared)
 autoindex_module (shared)
 vhost_alias_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 speling_module (shared)
 userdir_module (shared)
 alias_module (shared)
 rewrite_module (shared)
 php5_module (shared)
 dav_module (shared)
 dav_svn_module (shared)
 dav_fs_module (shared)
 dav_lock_module (shared)
 authz_svn_module (shared)
 dontdothat_module (shared)


And here we go with "php -m":

# php -m
[PHP Modules]
bcmath
bz2
Core
ctype
curl
date
dom
ereg
exif
fileinfo
filter
gd
gettext
hash
iconv
imagick
intl
json
ldap
libxml
mbstring
mcrypt
memcache
mhash
mysql
mysqlnd
openssl
pcre
PDFlib
PDO
pdo_mysql
pdo_pgsql
pdo_sqlite
pgsql
Phar
posix
readline
Reflection
session
SimpleXML
snmp
soap
sockets
SPL
sqlite3
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
Comment 10 Torsten Zuehlsdorff freebsd_committer freebsd_triage 2018-02-16 08:41:17 UTC
(In reply to O. Hartmann from comment #2)

> 0x0000000802300382 in ?? () from /usr/local/lib/php/20131226-zts/opcache.so

Mh, there are plenty of bug reports regarding opcache. The only segfault issues are for 5.6.

There problem here is: there is no support for PHP 5.6 anymore. Even the extended security support ends this year - but there are no bug fixes.

Even if we can pin down the problem to PHP there won't be any fix at all.

You should at least upgrade to PHP 7.1 and check if the error still happens. If so we can analyze further.
Comment 11 O. Hartmann 2018-02-17 08:52:24 UTC
(In reply to Torsten Zuehlsdorff from comment #10)

Checking /usr/ports/Mk/bsd.default-versions.mk reports:

[...]
# Possible values: 5.6, 7.0, 7.1, 7.2
PHP_DEFAULT?=           5.6
[...]

So I guess there is a reason, why php=5.6 is still the default? If 5.6 is still the default, we can consider this problem with opcache a serious issue that needs to be solved in 5.6!

Most modules I watched being present on the systems in question are reeled in by some ports's default settings. I do not care much about this issue, but now, it became a serious problem because its paralysing our Apache sites, which we use for SVN, RefDB, Nextcloud, bareos and other maintenance stuff. I'd rather like to isolate the opcache module from being build than starting rebuilding all PHP dependend ports according to switching towards php=7.1.

Since I have no idea which ports relying on php 5.6 are reluctant to play with 7.1, I do not want to figure that out by myself, since the boxes in question performing the build do not have lots of performance. Well, it would be another story if php 7.1 is becoming the new standard soon and there are few, clearly identifyable ports not playing well we do not use.

I checked which ports are relying on www/php56-opcache:

 # pkg info -r www/php56-opcache
php56-opcache-5.6.33:
        ZendFramework-2.3.3_2
        php56-extensions-1.0

So I consider this issue a serious one and it could not be ignored in favour of 7.1. We do not use "special cases" so I fear other will suffer from the same issue sooner or later when this crap gets common in the package tree.
Comment 12 Torsten Zuehlsdorff freebsd_committer freebsd_triage 2018-02-20 16:36:51 UTC
(In reply to O. Hartmann from comment #11)

> So I consider this issue a serious one and it could not be ignored in favour of 
> 7.1. We do not use "special cases" so I fear other will suffer from the same 
> issue sooner or later when this crap gets common in the package tree.

Its not ignored in favor of 7.1. There is *no* support for PHP 5.6 by the PHP group it self. Same for 7.0. 

That PHP 5.6 is currently the default PHP version has a little back story. I already prepared the ports-tree to get to 7.1. But it was stopped by mat in favor of PHP flavors. This will allow you to install every software with the PHP version you want - so without a change of default versions.

The current work is done by mat here:
https://reviews.freebsd.org/D14208

Since this will most likely land in Q1 2018 there isn't currently any plan to change the default version.
Comment 13 O. Hartmann 2018-02-21 19:09:51 UTC
(In reply to Torsten Zuehlsdorff from comment #12)

Well, I can follow your reasoning. In the meanwhile, I set the default version of php to 7.1 and recompiled/rebuild every port necessary.

The bug reported herein is gone! There are some bumpy portions of that upgrade road, but one can get rid of such obstacles easily.

I guess, the bugreport can be closed. but need to have a notice that the problem is solved with upcoming php 7.1 - I guess?
Comment 14 Laurence 'GreenReaper' Parry 2018-03-25 23:36:51 UTC
For what it's worth, I ran into this after a rebuild, and found that setting:

[opcache]
opcache.optimization_level=0x2ef

resolved the issue for me on a MediaWiki and Drupal installation (FreeBSD 10.4).

This removes passes 5 and 9 which perform block optimization and register allocation, while leaving other optimizations intact. If this doesn't work for you you could try setting it to 0 and then flipping bits like I did.

Obviously, this is not ideal, but as others have noted, 5.6 is kind of on life support and the best solution is likely to be an orderly migration to 7.x (ideally 7.1 or above) which will require testing of your application as there are likely to be some issues. On the plus side, the performance gain is incredible.