Summary: | www/apache24 core dumps when mod_perl, mod_php8, mod_dbd, and GD active | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Ports & Packages | Reporter: | papowell | ||||||
Component: | Individual Port(s) | Assignee: | freebsd-apache (Nobody) <apache> | ||||||
Status: | Open --- | ||||||||
Severity: | Affects Some People | CC: | 000.fbsd, bofh, dpetrov67, freebsd, freebsd, hiroo.ono+freebsd, matt, michael.osipov, pi, tatsuki_makino, wout, zarychtam | ||||||
Priority: | --- | Keywords: | crash, needs-qa | ||||||
Version: | Latest | Flags: | bugzilla:
maintainer-feedback?
(apache) |
||||||
Hardware: | amd64 | ||||||||
OS: | Any | ||||||||
Attachments: |
|
Description
papowell
2022-10-30 13:52:20 UTC
Thank you for your report. - Could you please include a full `pkg version -v` output file as an attachment - Is the issue reproducible without any one or more of the perl/php/dbd{,mysql} combinations? It would be great to isolate the minimum reproduction requirements - Reproduction test against packages:latest (versions) would be useful. - Are you able to provide a backtrace as an attachment here (via gdb or similar) of the resulting core file? Rebuilding one or more or all of the implicated apache modules (and apache, and gd) may provide better details. My apache24 httpd had a similar bomb so I'll watch here too :) In my case, originally there was no problem, but after making the following changes, it occurs rarely. + LoadModule http2_module + switch to LoadModule mpm_event_module instead mpm_prefork_module for http2_module + reinstalled mod_php74 with turn on ZTS for mpm_event_module Since I only recently noticed it, I have added CoreDumpDirectory /tmp to see what happens, but so far that hasn't happened yet. Created attachment 237776 [details]
steps taken, output of pkg version -v, and error messaes
MPORTANT UPDATE - See the end of this comment. # - Could you please include a full `pkg version -v` output file as an # attachment See the attached error file # # - Is the issue reproducible without any one or more of the # perl/php/dbd{,mysql} combinations? It would be great to isolate the minimum # reproduction requirements I seem to have found the minimum configuration. You need mod_perl (with GD), mod_php7, and mod_dbd (apr1 with MySQL) mod_perl (no mod_php) mod_dbd is OK mod_perl mod_php (no mod_dbd) is OK # # - Reproduction test against packages:latest (versions) would be useful. # I did this - see comments at end # - Are you able to provide a backtrace as an attachment here (via gdb or # similar) of the resulting core file? Rebuilding one or more or all of the # implicated apache modules (and apache, and gd) may provide better details. Tell me how to get the dump/corefile and I will send it. UPDATE Mon Oct 31 19:17:26 PDT 2022 I started with a new/clean 12-3 machine, downloaded the packages from 12-3 Latest repository, portsnap fetch extract (latest ports???) Ran apacchetl restart - no errors, able to fetch gd.cgi, Info.php I compiled devel/apr1 with MySQL support and enabled the LoadModule dbd_module line. # TEST LoadModule dbd_module libexec/apache24/mod_dbd.so # TEST LoadModule perl_module libexec/apache24/mod_perl.so LoadModule pWhen I ran apachectl restart I got this message - it is the first time I have seen this. # apachectl restart Performing sanity check on apache24 configuration: ld-elf.so.1: /usr/local/sbin/httpd: Undefined symbol "apr_crypto_block_cleanup" root@sim96:/install/SIM95_WEB-1.3 # apachectl start Performing sanity check on apache24 configuration: ld-elf.so.1: /usr/local/sbin/httpd: Undefined symbol "apr_crypto_block_cleanup" /usr/local/etc/rc.d/apache24: WARNING: failed to start apache24 Here is the apache log files: root@sim96:/install/SIM95_WEB-1.3 # tail /var/log/apache/* ==> /var/log/apache/httpd-access.log <== ==> /var/log/apache/httpd-error.log <== [Mon Oct 31 17:47:45.918582 2022] [mpm_prefork:notice] [pid 88032] AH00163: Apache/2.4.54 ( FreeBSD) PHP/7.4.32 mod_perl/2.0.12 Perl/v5.32.1 configured -- resuming normal operations [Mon Oct 31 17:47:45.918672 2022] [core:notice] [pid 88032] AH00094: Command line: '/usr/lo cal/sbin/httpd -D NOHTTPACCEPT' ==> /var/log/apache/sim96-access.log <== 128.1.1.5 - - [31/Oct/2022:18:09:44 -0700] "GET /gd.cgi HTTP/1.1" 200 136 128.1.1.5 - - [31/Oct/2022:18:09:45 -0700] "GET /image.png HTTP/1.1" 200 631 ==> /var/log/apache/sim96-error.log <== I update the devel/apr1 configuration, removed the MySQl. did make clean deinstall reinstall and the restarted apache: I got the same error. I went to /var/cache/pkg and did: pkg add -f apr-1.7.0.1.6.1_2.pkg This eliminated the error message. I suspect that the apr1 code may be the culprit. hp7_module libexec/apache24/libphp7.so I got [Wed Nov 02 06:01:58.484944 2022] [core:notice] [pid 39094:tid 34371067904] AH00052: child pid 39098 exit signal Segmentation fault (11) However, no core dump was obtained because the following was forgotten :) sysctl kern.sugid_coredump=1 (In reply to papowell from comment #4) > Tell me how to get the dump/corefile and I will send it. The core dump configuration on the Apache httpd side is to set CoreDumpDirectory directive. The following sysctl settings are available for core dumping on the FreeBSD side. kern.coredump kern.corefile kern.sugid_coredump ...such as. Since a non-privileged user will be dumping core, the write destination must be a directory that anyone can write to. Although not found in the Japanese documentation, it also seems necessary to set kern.sugid_coredump to 1. I will paste the backtrace of the core dump I just obtained. The result of uname -mrs is FreeBSD 12.4-STABLE amd64. The source is git stable/12-n235813-a43afddd4325-dirty. The backtrace is the standard output of the following command. Maybe we should build a debug version since there are many unnamed symbols? lldb -c /tmp/httpd.core -f /usr/local/sbin/httpd -o "bt all" -o q (lldb) target create "/usr/local/sbin/httpd" --core "/tmp/httpd.core" Core file '/tmp/httpd.core' (x86_64) was loaded. (lldb) bt all * thread #1, name = 'httpd', stop reason = signal SIGSEGV * frame #0: 0x000000080081623a libc.so.7`__sys_read + 10 frame #1: 0x0000000800637be6 libthr.so.3`___lldb_unnamed_symbol577 + 54 frame #2: 0x0000000000275690 httpd`ap_mpm_podx_check + 48 frame #3: 0x0000000800b5218b mod_mpm_event.so`___lldb_unnamed_symbol375 + 1291 frame #4: 0x0000000800b51b5a mod_mpm_event.so`___lldb_unnamed_symbol373 + 362 frame #5: 0x0000000800b5026c mod_mpm_event.so`___lldb_unnamed_symbol364 + 668 frame #6: 0x0000000000270b9b httpd`ap_run_mpm + 75 frame #7: 0x000000000025e0f0 httpd`main + 2480 frame #8: 0x000000000025d530 httpd`_start + 256 thread #2, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301 frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #3, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301 frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #4, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301 frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #5, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301 frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #6, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301 frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #7, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301 frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #8, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301 frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #9, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x00000008012376ed mod_http2.so`___lldb_unnamed_symbol930 + 301 frame #3: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #10, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134 frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300 frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #11, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134 frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300 frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #12, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134 frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300 frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #13, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134 frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300 frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #14, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134 frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300 frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #15, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134 frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300 frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #16, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008007a6f2a libc.so.7`__sys_kill + 10 frame #1: 0x000000080063ac72 libthr.so.3`___lldb_unnamed_symbol638 + 210 frame #2: 0x000000080063a16e libthr.so.3`___lldb_unnamed_symbol619 + 318 frame #3: 0x00007ffffffff003 frame #4: 0x000000080298eb4f libgd.so.6`gdImageStringFTEx + 399 frame #5: 0x0000000802944ad1 gd.so`___lldb_unnamed_symbol499 + 849 frame #6: 0x0000000801d02cca libphp7.so`___lldb_unnamed_symbol7884 + 122 frame #7: 0x0000000801ccb388 libphp7.so`execute_ex + 72 frame #8: 0x0000000801ccb66f libphp7.so`zend_execute + 575 frame #9: 0x0000000801c7d38c libphp7.so`zend_execute_scripts + 348 frame #10: 0x0000000801c01f1b libphp7.so`php_execute_script + 667 frame #11: 0x0000000801d1eec6 libphp7.so`___lldb_unnamed_symbol8299 + 1526 frame #12: 0x00000000002632d7 httpd`ap_invoke_handler + 279 frame #13: 0x000000000029e0c3 httpd`ap_process_async_request + 979 frame #14: 0x000000000029ac62 httpd`___lldb_unnamed_symbol4046 + 210 frame #15: 0x00000000002746a7 httpd`ap_run_process_connection + 55 frame #16: 0x0000000800b5374e mod_mpm_event.so`___lldb_unnamed_symbol385 + 1214 frame #17: 0x0000000800b53189 mod_mpm_event.so`___lldb_unnamed_symbol384 + 985 frame #18: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #17, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008006442bc libthr.so.3`___lldb_unnamed_symbol732 + 92 frame #1: 0x000000080064191f libthr.so.3`___lldb_unnamed_symbol701 + 623 frame #2: 0x000000000027b166 httpd`ap_queue_pop_something + 134 frame #3: 0x0000000800b52edc mod_mpm_event.so`___lldb_unnamed_symbol384 + 300 frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 thread #18, name = 'httpd', stop reason = signal SIGSEGV frame #0: 0x00000008007fd1ba libc.so.7`__sys_kevent + 10 frame #1: 0x00000008006380a3 libthr.so.3`___lldb_unnamed_symbol589 + 83 frame #2: 0x00000008005f9d62 libapr-1.so.0`___lldb_unnamed_symbol1233 + 130 frame #3: 0x0000000800b54a50 mod_mpm_event.so`___lldb_unnamed_symbol390 + 1760 frame #4: 0x0000000800634fd4 libthr.so.3`___lldb_unnamed_symbol538 + 324 (lldb) q thread #16, name = 'httpd', stop reason = signal SIGSEGV ︙ frame #3: 0x00007ffffffff003 frame #4: 0x000000080298eb4f libgd.so.6`gdImageStringFTEx + 399 This 0x7fff... is? :) If this part is relevant, it may be a problem on graphics/gd side. I have had a similar problem lately, so upgraded PHP from 8.0 to 8.2, but the new version of PHP was built with ZTS support. Everything seems to be working fine since then. Enabling ZTS usually helps with such breakages of mod_php. *** Bug 268318 has been marked as a duplicate of this bug. *** (In reply to Marek Zarychta from comment #8) Can you please confirm whether if this affects php80 or php81? And did it just automagically fix itself when you enabled ZTS for php? Requesting further info as the maintainer of php[78]. (In reply to Muhammad Moinur Rahman from comment #10) I am sorry but that's really not true what I have written in comment #8. We see fewer (almost no) issues with the recent www/mod_php82 but it's still crashing the Apache during log rotation (see bug 268318). Without mod_php loaded, logs are rotated fine. (In reply to Marek Zarychta from comment #11) With or without ZTS? (In reply to Muhammad Moinur Rahman from comment #12) With ZTS I am here because I was able to obtain a core dump in a similar environment, which may be related to comment #6. However, SIGSEGV can occur and dump core even where mod_php and other modules are not involved. I have switched httpd and apr to debug versions so that the location is clear, but they don't SIGSEGV very well with the way I use my poudriere server :) After upgrading to FreeBSD 13.2, I also notice Apache crashes on reload when using the mod_php (with OPCache) module. It seems related with ASLR as disabling this prevents the crashes for me. I can confirm that setting kern.elf.aslr.enable=0 while keeping the opcache php extension in version mod_php82 enabled prevents a graceful restart of apache24 from core dumping. (In reply to Tatsuki Makino from comment #6) > * thread #1, name = 'httpd', stop reason = signal SIGSEGV > * frame #0: 0x000000080081623a libc.so.7`__sys_read + 10 > frame #1: 0x0000000800637be6 libthr.so.3`___lldb_unnamed_symbol577 + 54 > frame #2: 0x0000000000275690 httpd`ap_mpm_podx_check + 48 > frame #3: 0x0000000800b5218b mod_mpm_event.so`___lldb_unnamed_symbol375 + 1291 > thread #16, name = 'httpd', stop reason = signal SIGSEGV > frame #0: 0x00000008007a6f2a libc.so.7`__sys_kill + 10 > frame #1: 0x000000080063ac72 libthr.so.3`___lldb_unnamed_symbol638 + 210 > frame #2: 0x000000080063a16e libthr.so.3`___lldb_unnamed_symbol619 + 318 > frame #3: 0x00007ffffffff003 > frame #4: 0x000000080298eb4f libgd.so.6`gdImageStringFTEx + 399 > frame #5: 0x0000000802944ad1 gd.so`___lldb_unnamed_symbol499 + 849 > frame #6: 0x0000000801d02cca libphp7.so`___lldb_unnamed_symbol7884 + 122 It seems that one of the unnamed symbols around here is _flockfile of /usr/src/lib/libc/stdio/_flock_stub.c. For those who use php, I feel that the way fp->_fl_count counts like an access counter looks strange :) Does it matter at all? :) I have started using 12.4-STABLE fedfe274225a amd64 reverted bug252579. This seems to have caused some sort of prolonged hang, but so far there is no signs of httpd coredumping. Could it be just a placebo, so who is willing to do the same experiment? :) There is a report that apache24 + mod_php81 dumps core on i386. He says that compiling (only) mod_php81 with gcc12 could avoid the problem. It may be worth trying. (The mail is here, but it is in Japanese.) https://lists.freebsd.org/archives/freebsd-users-jp/2023-April/000196.html Another issue which may be related (but it is on OpenBSD 7.0): https://github.com/php/php-src/issues/8159 (In reply to Matt from comment #16) You don't need to disable ASLR globally, it can be disabled for Apache (httpd) only as described in bug 268318 "elfctl -e +noaslr /usr/local/sbin/httpd" or with patched rc.d/apache24 using command="/usr/bin/proccontrol -m aslr -s disable ${command}" Even if the current workaround is to disable ASLR, I don't think we want to make that the solution, given the reasons for doing ASLR :) Regarding the experimentation of the side effects of solving bug252579. It seems that there is unknown memory leakage in it and that only when that memory leakage is accumulating does it receive a signal to do a core dump. At least 10GB :) Oh, and SIGSEGV is probably caused by multiple different reasons. (In reply to Tatsuki Makino from comment #6) you are running Apache with mpm_event. that's not supported with mod_php, at least not with all possible php extensions. if you want the benefits of mpm_event without random crashes you will need to move away from mod_php and towards fpm+mod_proxy_fcgi i don't know if the same is true for mod_perl. either way, it would make sense to run multiple instances of Apache Httpd, so that unrelated applications aren't crashing each other > Bug 267435 - www/apache24 core dumps ... GD active
This may be correct on graphics/gd side.
(lldb) bt
* thread #3, name = 'httpd', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
* frame #0: 0x0000000802dd008b libgd.so.6`gdCacheGet(head=0x0000000000000000, keydata=0x00007fffdf7fadf0) at gdcache.c:107:15
frame #1: 0x0000000802dd0633 libgd.so.6`gdImageStringFTEx(im=0x0000000000000000, brect=0x00007fffdf7fafc0, fg=-1, fontlist="/home/tatsuki/.fonts/somefont.ttf", ptsize=12.705882352941178, angle=0, x=0, y=0, string="-0.18", strex=0x0000000000000000) at gdft.c:1163:20
frame #2: 0x0000000802dd0347 libgd.so.6`gdImageStringFT(im=0x0000000000000000, brect=0x00007fffdf7fafc0, fg=-1, fontlist="/home/tatsuki/.fonts/somefont.ttf", ptsize=12.705882352941178, angle=0, x=0, y=0, string="-0.18") at gdft.c:932:9
frame #3: 0x0000000802d797b2 gd.so`___lldb_unnamed_symbol535 + 818
frame #4: 0x0000000801cf00ca libphp.so`___lldb_unnamed_symbol9102 + 122
This argument with a 0x000000000... is the variable fontCache in src/gdft.c of libgd-2.3.3.
There are 2 places to assign to fontCache (gdFontCacheShutdown and gdFontCacheSetup), is there something wrong with Mutex around them?
(In reply to Tatsuki Makino from comment #24) As for this font issue, it seems to be possible to work around it by making sure to use the bundled libgd on graphics/php??-gd (-> lang/php??) side. The bundled version of libgd for PHP is older than the version it is based on, but it seems to have improvements related to thread mutex. In php, it seems to be possible to work around this by enclosing the 4 functions (imageftbbox, imagefttext, imagettfbbox and imagettftext) in semaphores, etc. However, it would seem that if different languages are running on the same process, they must use the same semaphore. The gd bundled with php seems to have a workaround for the problem when font cache is used in multi-threading. But if mod_php uses it, I don't know what happens when a different language (mod_perl2) tries to use gd (p5-GD) in the same process. So I prepared something like bug 272091. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272091 It patches libgd to work equivalent to the one bundled with php. Could it be a workaround? Someone please try it out ahead of time :) My earlier written comment bug 252579 is not relevant. bug 272091 is committed (thanks). By using the added option, httpd will not die instantly when gd uses fonts. However, mod_php will continue to leak memory, so it is imperative that MaxConnectionsPerChild be set to something other than 0. |