Bug 256844 - www/nextcloud: apache/occ/php causes segmentation fault error
Summary: www/nextcloud: apache/occ/php causes segmentation fault error
Status: Closed Not A Bug
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Some People
Assignee: Bernard Spil
URL:
Keywords:
Depends on: 257194
Blocks:
  Show dependency treegraph
 
Reported: 2021-06-26 14:46 UTC by O. Hartmann
Modified: 2021-07-14 21:13 UTC (History)
3 users (show)

See Also:
bugzilla: maintainer-feedback? (brnrd)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description O. Hartmann 2021-06-26 14:46:11 UTC
Running latest port www/nextcloud (see below) on CURRENT (recently updated, problem is present a couple of weeks for now: FreeBSD 14.0-CURRENT #18 main-n247572-9e4d9e4c4d7: Fri Jun 25 15:48:12 CEST 2021 amd64), the nextcloud instance on that server is broken and inaccessible now for a while. When the apache24 webserver (port info see below) is accessed, Mozilla Firefox as well as any other browser reiceive "Error: Secure connection failed".
The error log of Apache 2.4 shows only this line:

[Sat Jun 26 14:41:11.684675 2021] [core:notice] [pid 60531:tid 34374492160] AH00052: child pid 77545 exit signal Segmentation fault (11)

Other PHP based services on the very same server (refdb, phpldapadmin, icinga2, webalizer for instance) do not show up with this seg fault.

Trying to use /usr/local/www/nextcloud/occ via
su -m www- c "php ./occ upgrade"
for instance also results in a segmentation fault.

I have already reinstalled www/apache24, www/nextcloud as well as lang/php74 with all php74-*packages via "portmaster -dfR" in the hope there is something out of synchronisation so far.

The problem does occur on CURRENT and recent 13-STABLE hosts, it doesn't occur on 12.2-RELENG-p8, 13.0-RELENG-p2. 


[...]
nextcloud-php74-21.0.2
Name           : nextcloud-php74
Version        : 21.0.2
Installed on   : Mon Jun  7 04:43:40 2021 CEST
Origin         : www/nextcloud
Architecture   : FreeBSD:14:*
Prefix         : /usr/local
Categories     : www
Licenses       : AGPLv3
Maintainer     : brnrd@FreeBSD.org
WWW            : https://nextcloud.com
Comment        : Personal cloud which runs on your own server
Options        :
        APCU           : on
        EXIF           : on
        IMAGICK        : off
        INTL           : on
        LDAP           : on
        MYSQL          : off
        OPCACHE        : on
        PCNTL          : off
        PGSQL          : on
        REDIS          : off
        SMB            : off
        SQLITE         : on
        SSL            : on
Annotations    :
        cpe            : cpe:2.3:a:nextcloud:nextcloud:21.0.2:::::freebsd14:x64
        flavor         : php74
Flat size      : 377MiB
Description    :
Nextcloud is a system for the creation and management of personal
cloud resources. It is a fork of Owncloud.

WWW: https://nextcloud.com



[...]
apache24-2.4.48
Name           : apache24
Version        : 2.4.48
Installed on   : Sat Jun 19 19:25:48 2021 CEST
Origin         : www/apache24
Architecture   : FreeBSD:14:amd64
Prefix         : /usr/local
Categories     : www
Licenses       : APACHE20
Maintainer     : apache@FreeBSD.org
WWW            : https://httpd.apache.org/
Comment        : Version 2.4.x of Apache web server
Options        :
        ACCESS_COMPAT  : on
        ACTIONS        : on
        ALIAS          : on
        ALLOWMETHODS   : on
        ASIS           : on
        AUTHNZ_FCGI    : on
        AUTHNZ_LDAP    : on
        AUTHN_ANON     : on
        AUTHN_CORE     : on
        AUTHN_DBD      : on
        AUTHN_DBM      : on
        AUTHN_FILE     : on
        AUTHN_SOCACHE  : on
        AUTHZ_CORE     : on
        AUTHZ_DBD      : on
        AUTHZ_DBM      : on
        AUTHZ_GROUPFILE: on
        AUTHZ_HOST     : on
        AUTHZ_OWNER    : on
        AUTHZ_USER     : on
        AUTH_BASIC     : on
        AUTH_DIGEST    : on
        AUTH_FORM      : on
        AUTOINDEX      : on
        BROTLI         : off
        BUCKETEER      : off
        BUFFER         : on
        CACHE          : on
        CACHE_DISK     : on
        CACHE_SOCACHE  : on
        CASE_FILTER    : off
        CASE_FILTER_IN : off
        CERN_META      : on
        CGI            : on
        CGID           : on
        CHARSET_LITE   : on
        DATA           : on
        DAV            : on
        DAV_FS         : on
        DAV_LOCK       : on
        DBD            : on
        DEFLATE        : on
        DIALUP         : on
        DIR            : on
        DOCS           : on
        DUMPIO         : on
        ECHO           : off
        ENV            : on
        EXAMPLE_HOOKS  : off
        EXAMPLE_IPC    : off
        EXPIRES        : on
        EXT_FILTER     : on
        FILE_CACHE     : on
        FILTER         : on
        HEADERS        : on
        HEARTBEAT      : on
        HEARTMONITOR   : on
        HTTP2          : on
        IDENT          : off
        IMAGEMAP       : on
        INCLUDE        : on
        INFO           : on
        IPV4_MAPPED    : off
        LBMETHOD_BYBUSYNESS: on
        LBMETHOD_BYREQUESTS: on
        LBMETHOD_BYTRAFFIC: on
        LBMETHOD_HEARTBEAT: on
        LDAP           : on
        LOGIO          : on
        LOG_DEBUG      : on
        LOG_FORENSIC   : on
        LUA            : off
        LUAJIT         : off
        MACRO          : on
        MD             : on
        MIME           : on
        MIME_MAGIC     : on
        MPM_EVENT      : off
        MPM_PREFORK    : on
        MPM_SHARED     : on
        MPM_WORKER     : off
        NEGOTIATION    : on
        OPTIONAL_FN_EXPORT: off        OPTIONAL_FN_IMPORT: off
        OPTIONAL_HOOK_EXPORT: off
        OPTIONAL_HOOK_IMPORT: off
        PROXY          : on
        PROXY_AJP      : on
        PROXY_BALANCER : on
        PROXY_CONNECT  : on
        PROXY_EXPRESS  : on
        PROXY_FCGI     : on
        PROXY_FDPASS   : on
        PROXY_FTP      : on
        PROXY_HCHECK   : on
        PROXY_HTML     : on
        PROXY_HTTP     : on
        PROXY_HTTP2    : on
        PROXY_SCGI     : on
        PROXY_UWSGI    : on
        PROXY_WSTUNNEL : on
        RATELIMIT      : on
        REFLECTOR      : on
        REMOTEIP       : on
        REQTIMEOUT     : on
        REQUEST        : on
        REWRITE        : on
        SED            : on
        SESSION        : on
        SESSION_COOKIE : on
        SESSION_CRYPTO : on
        SESSION_DBD    : on
        SETENVIF       : on
        SLOTMEM_PLAIN  : on
        SLOTMEM_SHM    : on
        SOCACHE_DBM    : on
        SOCACHE_DC     : off
        SOCACHE_MEMCACHE: on
        SOCACHE_REDIS  : off
        SOCACHE_SHMCB  : on
        SPELING        : on
        SSL            : on
        STATUS         : on
        SUBSTITUTE     : on
        SUEXEC         : off
        SUEXEC_SYSLOG  : off
        UNIQUE_ID      : on
        USERDIR        : on
        USERTRACK      : on
        VERSION        : on
        VHOST_ALIAS    : on
        WATCHDOG       : on
        XML2ENC        : onShared Libs required:
        libcurl.so.4
        libpcre.so.1
        libnghttp2.so.14
        libgdbm.so.6
        libldap-2.4.so.2
        libexpat.so.1
        libaprutil-1.so.0
        libapr-1.so.0
        liblber-2.4.so.2
        libdb-5.3.so.0
        libxml2.so.2
        libjansson.so.4
Annotations    :
        FreeBSD_version: 1400023
        cpe            : cpe:2.3:a:apache:http_server:2.4.48:::::freebsd14:x64
Flat size      : 27.2MiB
Description    :
The Apache HTTP Server Project is an effort to develop and maintain an
open-source HTTP server for various modern desktop and server operating
systems, such as UNIX and Windows NT. The goal of this project is to
provide a secure, efficient and extensible server which provides HTTP
services in sync with the current HTTP standards.
The 2.x branch of Apache Web Server includes several improvements like
threading, use of APR, native IPv6 and SSL support, and many more.

WWW: https://httpd.apache.org/
Comment 1 Michael Gmelin freebsd_committer 2021-06-30 18:36:40 UTC
As written on the mailing list:

I found this issue on the nextcloud issue tracker:
https://github.com/nextcloud/server/issues/25761

Based on that, you should enable apcu for php CLI by adding this
to your php.ini:

  apc.enable_cli=1

which should fix `occ upgrade'.

No idea if it works, but it sounds similar enough to give it a shot.

See also https://www.php.net/manual/en/apcu.configuration.php
Comment 2 Bernard Spil freebsd_committer 2021-07-01 07:17:08 UTC
Hi,

Sounds like an issue with PHP to me. Reordering loading of PHP extensions has saved me in the past (not recently).

Can you separate the Apache and PHP runtimes using fastcgi instead of mod_php?

Bernard.
Comment 3 O. Hartmann 2021-07-10 17:21:08 UTC
(In reply to Michael Gmelin from comment #1)


I put the statement 

apc.enable_cli=1

just before the section tagged

apc.enable_cli=1

;;;;;;;;;;;;;;;;;;;
; Module Settings ;
;;;;;;;;;;;;;;;;;;;


Restarting apache24 ... it has no effect!

php itself doesn't show any effect so far, at least I could easily trigger. Other webserver based services utilizing php also do not show any strange effect.
Comment 4 Michael Gmelin freebsd_committer 2021-07-10 17:24:23 UTC
(In reply to O. Hartmann from comment #3)

> Restarting apache24 ... it has no effect!

Like I wrote:

>> which should fix `occ upgrade'.

This is a setting for cli only, so it certainly won't fix apache.

So, did it fix `occ upgrade'?
Comment 5 Michael Gmelin freebsd_committer 2021-07-10 17:30:41 UTC
(In reply to Michael Gmelin from comment #4)
p.s. Please double check the output of `php -i | grep apc' to make sure the setting caught on.
Comment 6 O. Hartmann 2021-07-10 19:03:20 UTC
(In reply to Michael Gmelin from comment #5)

[...]
:/usr/local/www/nextcloud # php -i | grep apc
/usr/local/etc/php/ext-20-apcu.ini,
apcu
apc.coredump_unmap => Off => Off
apc.enable_cli => On => On
apc.enabled => On => On
apc.entries_hint => 4096 => 4096
apc.gc_ttl => 3600 => 3600
apc.mmap_file_mask => no value => no value
apc.preload_path => no value => no value
apc.serializer => php => php
apc.shm_segments => 1 => 1
apc.shm_size => 32M => 32M
apc.slam_defense => Off => Off
apc.smart => 0 => 0
apc.ttl => 0 => 0
apc.use_request_time => Off => Off
Comment 7 O. Hartmann 2021-07-10 19:07:05 UTC
It does NOT do anything mitigating the problem of immediate segmentation faulting when performing

su -m www -c 'php ./occ upgrade'

Anything done with the php script "occ" segemntation faults immediately.

Since my capabilties of  debugging the host (remote site, CURRENT) are limited, I can see into it what is going wrong. Apache acts as expected on other services provided by the same server, using the same SSL/TLS facilities et cetera. KTLS is enabled, OpenSSL is provided with the base system.
Comment 8 Michael Gmelin freebsd_committer 2021-07-10 19:13:38 UTC
(In reply to O. Hartmann from comment #7)

Hm, so this is basically running CURRENT with KTLS enabled with packages built/updated locally using portmaster? Hard to debug/reproduce remotely.

You could try to create copies of the packages installed with "pkg create -a" and use them on another host to experiment/share them with others so they can try to reproduce the problem in a vm.
Comment 9 O. Hartmann 2021-07-14 21:12:41 UTC
(In reply to Michael Gmelin from comment #8)

I found the issue:

Jul 14 22:53:54 <16.6> host postgres[60832]: [2-1] 2021-07-14 22:54:21.477 CEST [60832] LOG:  \
   starting PostgreSQL 13.3 on amd64-portbld-freebsd14.0, compiled by FreeBSD clang version 12.0.1 \
   (git@github.com:llvm/llvm-project.git llvmorg-12.0.1-rc2-0-ge7dac564cd0e), 64-bit

[...]
Jul 14 22:54:22 <16.6> host postgres[75483]: [16-1] 2021-07-14 22:54:49.591 CEST [75483] LOG:  redo done at 1/827D0370
Jul 14 22:54:22 <16.6> host postgres[60832]: [12-1] 2021-07-14 22:54:49.852 CEST [60832] LOG:  database system is ready to accept connections
Jul 14 22:54:51 <16.6> host postgres[60832]: [13-1] 2021-07-14 22:55:18.170 CEST [60832] LOG:  server process (PID 93689) was terminated by signal 11: Segmentation fault
Jul 14 22:54:51 <16.6> host postgres[60832]: [14-1] 2021-07-14 22:55:18.170 CEST [60832] LOG:  terminating any other active server processes
Jul 14 22:54:51 <16.5> host postgres[85999]: [12-1] 2021-07-14 22:55:18.170 CEST [85999] WARNING:  terminating connection because of crash of another server process
Jul 14 22:54:51 <16.5> host postgres[85999]: [12-2] 2021-07-14 22:55:18.170 CEST [85999] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
Jul 14 22:54:51 <16.5> host postgres[85999]: [12-3] 2021-07-14 22:55:18.170 CEST [85999] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
Jul 14 22:54:51 <16.6> host postgres[60832]: [15-1] 2021-07-14 22:55:18.172 CEST [60832] LOG:  all server processes terminated; reinitializing
Jul 14 22:54:51 <16.6> host postgres[94255]: [16-1] 2021-07-14 22:55:18.185 CEST [94255] LOG:  database system was interrupted; last known up at 2021-07-14 22:54:49 CEST
Jul 14 22:54:51 <16.6> host postgres[94255]: [17-1] 2021-07-14 22:55:18.579 CEST [94255] LOG:  database system was not properly shut down; automatic recovery in progress
Jul 14 22:54:51 <16.6> host postgres[94255]: [18-1] 2021-07-14 22:55:18.597 CEST [94255] LOG:  redo starts at 1/827D0420
Jul 14 22:54:51 <16.6> host postgres[94255]: [19-1] 2021-07-14 22:55:18.597 CEST [94255] LOG:  invalid record length at 1/827D0458: wanted 24, got 0
Jul 14 22:54:51 <16.6> host postgres[94255]: [20-1] 2021-07-14 22:55:18.597 CEST [94255] LOG:  redo done at 1/827D0420
Jul 14 22:54:51 <16.6> host postgres[60832]: [16-1] 2021-07-14 22:55:18.649 CEST [60832] LOG:  database system is ready to accept connections
[...]

The host is running recent FreeBSD 14-CURRENT, and databases/postgresql13-server as the  DB backend.
In postgresql.conf, SSL is enabled as well as encryption via scram-sha-256 (if this is of use, just for the record).

The database is mainly used for nextcloud at the moment, so the issue - obviously existing since May, 26th, hasn't been revealed. The database is seg faulting periodically, I do  not see why.

I tried to recompile the port databases/postgresql13-server without option LLVM avoiding usage of JIT, but that did not have any effect.
Comment 10 O. Hartmann 2021-07-14 21:13:15 UTC
Not  a bug of www/nextcloud.