Bug 267435 - www/apache24 core dumps when mod_perl, mod_php8, mod_dbd, and GD active
Summary: www/apache24 core dumps when mod_perl, mod_php8, mod_dbd, and GD active
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-apache (Nobody)
URL:
Keywords: crash, needs-qa
Depends on:
Blocks:
 
Reported: 2022-10-30 13:52 UTC by papowell
Modified: 2024-03-13 16:05 UTC (History)
12 users (show)

See Also:
bugzilla: maintainer-feedback? (apache)


Attachments
details and how to reproduce problem (35.50 KB, application/x-tar)
2022-10-30 13:52 UTC, papowell
no flags Details
steps taken, output of pkg version -v, and error messaes (6.45 KB, text/plain)
2022-11-01 01:53 UTC, papowell
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description papowell 2022-10-30 13:52:20 UTC
Created attachment 237724 [details]
details and how to reproduce problem

Apache core dumps when mod_perl, mod_php and mod_dbd with MySQL support
are active.

The Perl CGI scripts which use GD to generate images now fail with a core dump.
But only if the mod_php or mod_dbd modules are active.

It appears that there are some problems with the Apache DSO support for
mod_dbd, mod_perl and/or mod_php.

See description.txt and SIM95.tar files for details and how to reproduce this problem.

Details:

FreeBSD 13.1-RELEASE
FreeBSD 12.3-RELEASE
ports Oct 26, 2022
apache Apache/2.4.54 (FreeBSD)
mod_perl ap24-mod_perl2-2.0.12,3
mod_php  mod_php74-7.4.32_1

See attached file SIM95_WEB.tar for details and how to reproduce problem

I have a web based application which uses:
  mod_perl   - to run perl based CGI scripts.
               Some of these use the Perl GD module to generate graphics.
  mod_php7   - to run PHP74 based php scripts
  mod_dbd    -  Authentication and password are done by the mod_dbd
              using a MySQL database (details below).

I have been downloading binaries from the PKG server:
FreeBSD: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly", }

I last downloaded a set of packages on Sept 14 2022.
This was done with a set of scripts and downloads and installs
the packages and package dependencies.
They were basically mod_perl, mod_php and their dependencies.
I also compiled devel/apr1, setting MYSQL with the 'make config'.
I installed the compiled devel/apr1 and the installed application
passed my tests.  This was installed on server sim97 and there
were no problems.

On Oct 10 2022 I set up a sim95, a new server (identical hardware to sim97),
installed FreeBSD 12.3-RELEASE, and repeated the package downloads and installs.
Much to my surprise there was a problem with the Apache server.

As noted above,  the application uses mod_perl, mod_php, and mod_dbd.
The httpd.conf file has the following entries which load these modules
into the Apache server. (There is also a startup.pl file used to
set up mod_perl)

The Perl CGI scripts which use GD to generate images now fail with a core dump.
But only if the mod_php or mod_dbd modules are active.
httpd.conf pertinent entries:

   LoadModule authn_dbd_module libexec/apache24/mod_authn_dbd.so
   LoadModule authz_dbd_module libexec/apache24/mod_authz_dbd.so
   # TEST
   LoadModule dbd_module libexec/apache24/mod_dbd.so

   Virtual Host Entry
     <IfModule mod_dbd.c>
       DBDriver mysql     # Line 920 in httpd.conf file
       DBDParams "host=127.0.0.1 dbname=password user=pwd pass=pwd"
     </IfModule>

  # TEST
  LoadModule perl_module        libexec/apache24/mod_perl.so
  LoadModule php7_module        libexec/apache24/libphp7.so

If the mod_dbd, perl_module, and php7_module lines are all present
THEN a web page which uses GD will cause apache to segment fault.
This message will be present in the /var/log/apache/httpd-error.log file:

  [Tue Oct 25 05:37:28.327954 2022] [core:notice] [pid 27255]
  AH00052: child pid 27371 exit signal Segmentation fault (11)

This message will be present in the /var/log/messages file:
Oct 25 06:02:28 sim95 kernel: pid 27583 (httpd), jid 0, uid 80:
  exited on signal 11
Comment 1 Kubilay Kocak freebsd_committer freebsd_triage 2022-10-30 21:50:28 UTC
Thank you for your report.

 - Could you please include a full `pkg version -v` output file as an attachment

 - Is the issue reproducible without any one or more of the perl/php/dbd{,mysql} combinations? It would be great to isolate the minimum reproduction requirements

 - Reproduction test against packages:latest (versions) would be useful.

 - Are you able to provide a backtrace as an attachment here (via gdb or similar) of the resulting core file? Rebuilding one or more or all of the implicated apache modules (and apache, and gd) may provide better details.
Comment 2 Tatsuki Makino 2022-10-31 07:17:47 UTC
My apache24 httpd had a similar bomb so I'll watch here too :)

In my case, originally there was no problem, but after making the following changes, it occurs rarely.

+ LoadModule http2_module
+ switch to LoadModule mpm_event_module instead mpm_prefork_module for http2_module
+ reinstalled mod_php74 with turn on ZTS for mpm_event_module

Since I only recently noticed it, I have added CoreDumpDirectory /tmp to see what happens, but so far that hasn't happened yet.
Comment 3 papowell 2022-11-01 01:53:56 UTC
Created attachment 237776 [details]
steps taken, output of pkg version -v, and error messaes
Comment 4 papowell 2022-11-01 02:42:32 UTC
 
MPORTANT UPDATE - See the end of this comment.

#  - Could you please include a full `pkg version -v` output file as an
# attachment
See the attached error file
#
#  - Is the issue reproducible without any one or more of the
# perl/php/dbd{,mysql} combinations? It would be great to isolate the minimum
# reproduction requirements

I seem to have found the minimum configuration.
You need mod_perl (with GD), mod_php7, and mod_dbd (apr1 with MySQL)

mod_perl (no mod_php) mod_dbd  is OK
mod_perl  mod_php (no mod_dbd)  is OK

#
#  - Reproduction test against packages:latest (versions) would be useful.
#
I did this - see comments at end
#  - Are you able to provide a backtrace as an attachment here (via gdb or
# similar) of the resulting core file? Rebuilding one or more or all of the
# implicated apache modules (and apache, and gd) may provide better details.
Tell me how to get the dump/corefile and I will send it.

UPDATE Mon Oct 31 19:17:26 PDT 2022
I started with a new/clean 12-3 machine, downloaded the packages
from 12-3 Latest repository, portsnap fetch extract (latest ports???)

Ran apacchetl restart - no errors, able to fetch gd.cgi, Info.php

I compiled devel/apr1 with MySQL support and enabled
the LoadModule dbd_module line.

# TEST
LoadModule dbd_module libexec/apache24/mod_dbd.so
# TEST
LoadModule perl_module        libexec/apache24/mod_perl.so
LoadModule pWhen I ran apachectl restart I got this message - it is the
first time I have seen this.
   # apachectl restart
   Performing sanity check on apache24 configuration:
   ld-elf.so.1: /usr/local/sbin/httpd: Undefined symbol "apr_crypto_block_cleanup"
   root@sim96:/install/SIM95_WEB-1.3 # apachectl start
   Performing sanity check on apache24 configuration:
   ld-elf.so.1: /usr/local/sbin/httpd: Undefined symbol "apr_crypto_block_cleanup"
   /usr/local/etc/rc.d/apache24: WARNING: failed to start apache24


Here is the apache log files:

root@sim96:/install/SIM95_WEB-1.3 # tail /var/log/apache/*
==> /var/log/apache/httpd-access.log <==

==> /var/log/apache/httpd-error.log <==
[Mon Oct 31 17:47:45.918582 2022] [mpm_prefork:notice] [pid 88032] AH00163: Apache/2.4.54 (
FreeBSD) PHP/7.4.32 mod_perl/2.0.12 Perl/v5.32.1 configured -- resuming normal operations
[Mon Oct 31 17:47:45.918672 2022] [core:notice] [pid 88032] AH00094: Command line: '/usr/lo
cal/sbin/httpd -D NOHTTPACCEPT'

==> /var/log/apache/sim96-access.log <==
128.1.1.5 - - [31/Oct/2022:18:09:44 -0700] "GET /gd.cgi HTTP/1.1" 200 136
128.1.1.5 - - [31/Oct/2022:18:09:45 -0700] "GET /image.png HTTP/1.1" 200 631

==> /var/log/apache/sim96-error.log <==

I update the devel/apr1 configuration, removed the MySQl.
did make clean deinstall reinstall and the restarted apache:
I got the same error.

I went to /var/cache/pkg and did:
pkg add -f apr-1.7.0.1.6.1_2.pkg

This eliminated the error message.

I suspect that the apr1 code may be the culprit.
hp7_module        libexec/apache24/libphp7.so
Comment 5 Tatsuki Makino 2022-11-01 21:34:22 UTC
I got
[Wed Nov 02 06:01:58.484944 2022] [core:notice] [pid 39094:tid 34371067904] AH00052: child pid 39098 exit signal Segmentation fault (11)

However, no core dump was obtained because the following was forgotten :)

sysctl kern.sugid_coredump=1

(In reply to papowell from comment #4)
> Tell me how to get the dump/corefile and I will send it.

The core dump configuration on the Apache httpd side is to set CoreDumpDirectory directive.
The following sysctl settings are available for core dumping on the FreeBSD side.

kern.coredump
kern.corefile
kern.sugid_coredump
...such as.

Since a non-privileged user will be dumping core, the write destination must be a directory that anyone can write to.
Although not found in the Japanese documentation, it also seems necessary to set kern.sugid_coredump to 1.
Comment 6 Tatsuki Makino 2022-11-05 03:57:32 UTC
I will paste the backtrace of the core dump I just obtained.
The result of uname -mrs is FreeBSD 12.4-STABLE amd64. The source is git stable/12-n235813-a43afddd4325-dirty.

The backtrace is the standard output of the following command.
Maybe we should build a debug version since there are many unnamed symbols?

lldb -c /tmp/httpd.core -f /usr/local/sbin/httpd -o "bt all" -o q

(lldb) target create "/usr/local/sbin/httpd" --core "/tmp/httpd.core"
Core file '/tmp/httpd.core' (x86_64) was loaded.
(lldb) bt all
* thread #1, name = 'httpd', stop reason = signal SIGSEGV
  * frame #0: 0x000000080081623a libc.so.7`__sys_read + 10
    frame #1: 0x0000000800637be6 libthr.so.3`___lldb_unnamed_symbol577 + 54
    frame #2: 0x0000000000275690 httpd`ap_mpm_podx_check + 48
    frame #3: 0x0000000800b5218b mod_mpm_event.so`___lldb_unnamed_symbol375 + 1291
    frame #4: 0x0000000800b51b5a mod_mpm_event.so`___lldb_unnamed_symbol373 + 362
    frame #5: 0x0000000800b5026c mod_mpm_event.so`___lldb_unnamed_symbol364 + 668
    frame #6: 0x0000000000270b9b httpd`ap_run_mpm + 75
    frame #7: 0x000000000025e0f0 httpd`main + 2480
    frame #8: 0x000000000025d530 httpd`_start + 256
  thread #2, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301
    frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #3, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301
    frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #4, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301
    frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #5, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301
    frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #6, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301
    frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #7, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301
    frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #8, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301
    frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #9, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301
    frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #10, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134
    frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300
    frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #11, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134
    frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300
    frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #12, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134
    frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300
    frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #13, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134
    frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300
    frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #14, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134
    frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300
    frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #15, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134
    frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300
    frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #16, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008007a6f2a libc.so.7`__sys_kill + 10
    frame #1: 0x000000080063ac72 libthr.so.3`___lldb_unnamed_symbol638 + 210
    frame #2: 0x000000080063a16e libthr.so.3`___lldb_unnamed_symbol619 + 318
    frame #3: 0x00007ffffffff003
    frame #4: 0x000000080298eb4f libgd.so.6`gdImageStringFTEx + 399
    frame #5: 0x0000000802944ad1 gd.so`___lldb_unnamed_symbol499 + 849
    frame #6: 0x0000000801d02cca libphp7.so`___lldb_unnamed_symbol7884 + 122
    frame #7: 0x0000000801ccb388 libphp7.so`execute_ex + 72
    frame #8: 0x0000000801ccb66f libphp7.so`zend_execute + 575
    frame #9: 0x0000000801c7d38c libphp7.so`zend_execute_scripts + 348
    frame #10: 0x0000000801c01f1b libphp7.so`php_execute_script + 667
    frame #11: 0x0000000801d1eec6 libphp7.so`___lldb_unnamed_symbol8299 + 1526
    frame #12: 0x00000000002632d7 httpd`ap_invoke_handler + 279
    frame #13: 0x000000000029e0c3 httpd`ap_process_async_request + 979
    frame #14: 0x000000000029ac62 httpd`___lldb_unnamed_symbol4046 + 210
    frame #15: 0x00000000002746a7 httpd`ap_run_process_connection + 55
    frame #16: 0x0000000800b5374e mod_mpm_event.so`___lldb_unnamed_symbol385 + 1214
    frame #17: 0x0000000800b53189 mod_mpm_event.so`___lldb_unnamed_symbol384 + 985
    frame #18: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #17, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92
    frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623
    frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134
    frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300
    frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
  thread #18, name = 'httpd', stop reason = signal SIGSEGV
    frame #0: 0x00000008007fd1ba libc.so.7`__sys_kevent + 10
    frame #1: 0x00000008006380a3 libthr.so.3`___lldb_unnamed_symbol589 + 83
    frame #2: 0x00000008005f9d62 libapr-1.so.0`___lldb_unnamed_symbol1233 + 130
    frame #3: 0x0000000800b54a50 mod_mpm_event.so`___lldb_unnamed_symbol390 + 1760
    frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324
(lldb) q
Comment 7 Tatsuki Makino 2022-11-05 04:11:47 UTC
  thread #16, name = 'httpd', stop reason = signal SIGSEGV
︙
    frame #3: 0x00007ffffffff003
    frame #4: 0x000000080298eb4f libgd.so.6`gdImageStringFTEx + 399

This 0x7fff... is? :)
If this part is relevant, it may be a problem on graphics/gd side.
Comment 8 Marek Zarychta 2022-11-05 08:21:45 UTC
I have had a similar problem lately, so upgraded PHP from 8.0 to 8.2, but the new version of PHP was built with ZTS support. Everything seems to be working fine since then. Enabling ZTS usually helps with such breakages of mod_php.
Comment 9 Marek Zarychta 2022-12-11 21:21:34 UTC
*** Bug 268318 has been marked as a duplicate of this bug. ***
Comment 10 Muhammad Moinur Rahman freebsd_committer freebsd_triage 2022-12-11 21:28:28 UTC
(In reply to Marek Zarychta from comment #8)
Can you please confirm whether if this affects php80 or php81? And did it just automagically fix itself when you enabled ZTS for php?

Requesting further info as the maintainer of php[78].
Comment 11 Marek Zarychta 2022-12-12 19:18:55 UTC
(In reply to Muhammad Moinur Rahman from comment #10)
I am sorry but that's really not true what I have written in comment #8. We see fewer (almost no) issues with the recent www/mod_php82 but it's still crashing the Apache during log rotation (see bug 268318). Without mod_php loaded, logs are rotated fine.
Comment 12 Muhammad Moinur Rahman freebsd_committer freebsd_triage 2022-12-12 19:24:31 UTC
(In reply to Marek Zarychta from comment #11)
With or without ZTS?
Comment 13 Marek Zarychta 2022-12-12 19:27:13 UTC
(In reply to Muhammad Moinur Rahman from comment #12)
With ZTS
Comment 14 Tatsuki Makino 2022-12-12 20:48:46 UTC
I am here because I was able to obtain a core dump in a similar environment, which may be related to comment #6.
However, SIGSEGV can occur and dump core even where mod_php and other modules are not involved.

I have switched httpd and apr to debug versions so that the location is clear, but they don't SIGSEGV very well with the way I use my poudriere server :)
Comment 15 Wout Decré 2023-04-18 14:04:31 UTC
After upgrading to FreeBSD 13.2, I also notice Apache crashes on reload when using the mod_php (with OPCache) module.

It seems related with ASLR as disabling this prevents the crashes for me.
Comment 16 Matt 2023-05-18 01:57:41 UTC
I can confirm that setting kern.elf.aslr.enable=0 while keeping the opcache php extension in version mod_php82 enabled prevents a graceful restart of apache24 from core dumping.
Comment 17 Tatsuki Makino 2023-06-12 21:49:25 UTC
(In reply to Tatsuki Makino from comment #6)
> * thread #1, name = 'httpd', stop reason = signal SIGSEGV
>   * frame #0: 0x000000080081623a libc.so.7`__sys_read + 10
>     frame #1: 0x0000000800637be6 libthr.so.3`___lldb_unnamed_symbol577 + 54
>     frame #2: 0x0000000000275690 httpd`ap_mpm_podx_check + 48
>     frame #3: 0x0000000800b5218b mod_mpm_event.so`___lldb_unnamed_symbol375 + 1291

>   thread #16, name = 'httpd', stop reason = signal SIGSEGV
>     frame #0: 0x00000008007a6f2a libc.so.7`__sys_kill + 10
>     frame #1: 0x000000080063ac72 libthr.so.3`___lldb_unnamed_symbol638 + 210
>     frame #2: 0x000000080063a16e libthr.so.3`___lldb_unnamed_symbol619 + 318
>     frame #3: 0x00007ffffffff003
>     frame #4: 0x000000080298eb4f libgd.so.6`gdImageStringFTEx + 399
>     frame #5: 0x0000000802944ad1 gd.so`___lldb_unnamed_symbol499 + 849
>     frame #6: 0x0000000801d02cca libphp7.so`___lldb_unnamed_symbol7884 + 122

It seems that one of the unnamed symbols around here is _flockfile of /usr/src/lib/libc/stdio/_flock_stub.c.
For those who use php, I feel that the way fp->_fl_count counts like an access counter looks strange :)
Does it matter at all? :)
Comment 18 Tatsuki Makino 2023-06-14 04:51:59 UTC
I have started using 12.4-STABLE fedfe274225a amd64 reverted bug252579.
This seems to have caused some sort of prolonged hang, but so far there is no signs of httpd coredumping.
Could it be just a placebo, so who is willing to do the same experiment? :)
Comment 19 Hiroo Ono 2023-06-15 19:56:52 UTC
There is a report that apache24 + mod_php81 dumps core on i386.
He says that compiling (only) mod_php81 with gcc12 could avoid the problem.
It may be worth trying.

(The mail is here, but it is in Japanese.)
https://lists.freebsd.org/archives/freebsd-users-jp/2023-April/000196.html
Comment 20 Hiroo Ono 2023-06-15 20:30:43 UTC
Another issue which may be related (but it is on OpenBSD 7.0):
https://github.com/php/php-src/issues/8159
Comment 21 Miroslav Lachman 2023-06-15 21:10:05 UTC
(In reply to Matt from comment #16)
You don't need to disable ASLR globally, it can be disabled for Apache (httpd) only as described in bug 268318 
"elfctl -e +noaslr /usr/local/sbin/httpd" or with patched rc.d/apache24 using 
command="/usr/bin/proccontrol -m aslr -s disable ${command}"
Comment 22 Tatsuki Makino 2023-06-15 21:32:19 UTC
Even if the current workaround is to disable ASLR, I don't think we want to make that the solution, given the reasons for doing ASLR :)


Regarding the experimentation of the side effects of solving bug252579.
It seems that there is unknown memory leakage in it and that only when that memory leakage is accumulating does it receive a signal to do a core dump. At least 10GB :)

Oh, and SIGSEGV is probably caused by multiple different reasons.
Comment 23 Mina Galić freebsd_triage 2023-06-17 17:25:10 UTC
(In reply to Tatsuki Makino from comment #6)
you are running Apache with mpm_event. that's not supported with mod_php, at least not with all possible php extensions.

if you want the benefits of mpm_event without random crashes you will need to move away from mod_php and towards fpm+mod_proxy_fcgi

i don't know if the same is true for mod_perl. either way, it would make sense to run multiple instances of Apache Httpd, so that unrelated applications aren't crashing each other
Comment 24 Tatsuki Makino 2023-06-18 08:58:52 UTC
> Bug 267435 - www/apache24 core dumps ... GD active

This may be correct on graphics/gd side.

(lldb) bt                                                                       
* thread #3, name = 'httpd', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x0000000802dd008b libgd.so.6`gdCacheGet(head=0x0000000000000000, keydata=0x00007fffdf7fadf0) at gdcache.c:107:15
    frame #1: 0x0000000802dd0633 libgd.so.6`gdImageStringFTEx(im=0x0000000000000000, brect=0x00007fffdf7fafc0, fg=-1, fontlist="/home/tatsuki/.fonts/somefont.ttf", ptsize=12.705882352941178, angle=0, x=0, y=0, string="-0.18", strex=0x0000000000000000) at gdft.c:1163:20
    frame #2: 0x0000000802dd0347 libgd.so.6`gdImageStringFT(im=0x0000000000000000, brect=0x00007fffdf7fafc0, fg=-1, fontlist="/home/tatsuki/.fonts/somefont.ttf", ptsize=12.705882352941178, angle=0, x=0, y=0, string="-0.18") at gdft.c:932:9
    frame #3: 0x0000000802d797b2 gd.so`___lldb_unnamed_symbol535 + 818
    frame #4: 0x0000000801cf00ca libphp.so`___lldb_unnamed_symbol9102 + 122

This argument with a 0x000000000... is the variable fontCache in src/gdft.c of libgd-2.3.3.
There are 2 places to assign to fontCache (gdFontCacheShutdown and gdFontCacheSetup), is there something wrong with Mutex around them?
Comment 25 Tatsuki Makino 2023-06-19 05:50:51 UTC
(In reply to Tatsuki Makino from comment #24)

As for this font issue, it seems to be possible to work around it by making sure to use the bundled libgd on graphics/php??-gd (-> lang/php??) side.
The bundled version of libgd for PHP is older than the version it is based on, but it seems to have improvements related to thread mutex.
Comment 26 Tatsuki Makino 2023-06-21 03:17:53 UTC
In php, it seems to be possible to work around this by enclosing the 4 functions (imageftbbox, imagefttext, imagettfbbox and imagettftext) in semaphores, etc.
However, it would seem that if different languages are running on the same process, they must use the same semaphore.
The gd bundled with php seems to have a workaround for the problem when font cache is used in multi-threading.
But if mod_php uses it, I don't know what happens when a different language (mod_perl2) tries to use gd (p5-GD) in the same process.

So I prepared something like bug 272091.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272091
It patches libgd to work equivalent to the one bundled with php.
Could it be a workaround? Someone please try it out ahead of time :)
Comment 27 Tatsuki Makino 2023-07-06 23:29:12 UTC
My earlier written comment bug 252579 is not relevant.
bug 272091 is committed (thanks).
By using the added option, httpd will not die instantly when gd uses fonts.
However, mod_php will continue to leak memory, so it is imperative that MaxConnectionsPerChild be set to something other than 0.