The change "lang/php81: Update version 8.1.25=>8.1.26" https://cgit.freebsd.org/ports/commit/?id=06be0d5391f475771145a7a9d68d34b0ad5a9b85 is causing apache with mod_php and a bunch of php extensions to crash on reload (apachectl graceful or signal 30 - SIGUSR1) It crashes reliably on apachectl reload with extensions1 (see comments) with the following back trace: (lldb) target create "httpd" --core "httpd-9988.core" Core file '/tmp/httpd-9988.core' (x86_64) was loaded. (lldb) bt * thread #1, name = 'httpd', stop reason = signal SIGSEGV * frame #0: 0x000000080141a5d4 libzstd.so.1`___lldb_unnamed_symbol756 + 740 frame #1: 0x00000008027ce1c6 libphp.so`zend_extension_startup(extension=0x0000000801b18510) at zend_extensions.c:196:7 frame #2: 0x00000008027b0ba1 libphp.so`zend_llist_apply_with_del(l=0x0000000802954100, func=(libphp.so`zend_extension_startup at zend_extensions.c:193)) at zend_llist.c:171:7 frame #3: 0x00000008027ce1a8 libphp.so`zend_startup_extensions at zend_extensions.c:217:2 [artificial] frame #4: 0x000000080274f8c7 libphp.so`php_module_startup(sf=<unavailable>, additional_modules=<unavailable>, num_additional_modules=1) at main.c:2263:2 frame #5: 0x00000008028ad385 libphp.so`php_apache2_startup(sapi_module=<unavailable>) at sapi_apache2.c:381:6 frame #6: 0x00000008028acbd6 libphp.so`php_apache_server_startup(pconf=0x0000000800907028, plog=<unavailable>, ptemp=<unavailable>, s=<unavailable>) at sapi_apache2.c:495:6 frame #7: 0x0000000000263d6f httpd`ap_run_post_config + 79 frame #8: 0x000000000025f4cc httpd`main + 2476 frame #9: 0x000000000025e910 httpd`_start + 256 It crashes sporadically on another server with different set of extensions2 (see comments) with the following back trace: (lldb) target create "httpd" --core "httpd-99468.core" Core file '/tmp/httpd-99468.core' (x86_64) was loaded. (lldb) bt * thread #1, name = 'httpd', stop reason = signal SIGSEGV * frame #0: 0x00000008020f5124 libphp.so`zend_get_callable_name_ex + 164 frame #1: 0x00000008020f9f06 libphp.so`___lldb_unnamed_symbol7929 + 22 frame #2: 0x00000008020dc5d1 libphp.so`zend_llist_apply_with_del + 81 frame #3: 0x000000080207b107 libphp.so`php_module_startup + 2359 frame #4: 0x00000008021da2c5 libphp.so`___lldb_unnamed_symbol9598 + 21 frame #5: 0x00000008021d9aa6 libphp.so`___lldb_unnamed_symbol9593 + 118 frame #6: 0x00000000002627cf httpd`ap_run_post_config + 79 frame #7: 0x000000000025de8c httpd`main + 2476 frame #8: 0x000000000025d2d0 httpd`_start + 256 When I revert external-pcre (the following "lines") from the Makefile: BUILD_DEPENDS= re2c>0:devel/re2c CONFIGURE_ARGS+= --with-external-pcre=${LOCALBASE} and revert pkg-plist so that include/php/ext/pcre/pcre2lib/ header files are installed, the crash on reload does not happen. See also my bug report to PHP project, where I originally thought it is a bug of PHP 8.1.26: mod_php 8.1.26 crashes on apache graceful restart https://github.com/php/php-src/issues/12844 Note the following comment by nielsdos (PHP contributor): > If I had to take a guess, one possibility is that a conflicting version of pcre2 is used. e.g. when one (transitive) library dependency of an extension depends on pcre2 and php also uses pcre2 for ext/pcre, this can cause problems when different versions are used because of conflicting symbols. > Using the bundled pcre2 would fix this because we prefix some symbol names with the name php to avoid that exact issue.
# extensions1 Portbuild server. Clean build of "just" pkg, apache24, php81, mod_php81 and php81-extensions. Simple apache configuration. php81-bcmath-8.1.26 php81-calendar-8.1.26 php81-ctype-8.1.26 php81-curl-8.1.26 php81-dom-8.1.26 php81-extensions-1.1 php81-fileinfo-8.1.26 php81-filter-8.1.26 php81-gd-8.1.26 php81-gettext-8.1.26 php81-iconv-8.1.26 php81-intl-8.1.26_1 php81-mbstring-8.1.26 php81-mysqli-8.1.26 php81-opcache-8.1.26 php81-pdo-8.1.26 php81-pdo_mysql-8.1.26 php81-pdo_sqlite-8.1.26 php81-phar-8.1.26 php81-session-8.1.26 php81-simplexml-8.1.26 php81-soap-8.1.26 php81-sqlite3-8.1.26 php81-tokenizer-8.1.26 php81-xml-8.1.26 php81-xmlreader-8.1.26
# extensions2 Production server with similar set of php extensions. There are few more packages installed and apache has different (real) configuration. mod_php81-8.1.26 php81-8.1.26 php81-calendar-8.1.26 php81-ctype-8.1.26 php81-curl-8.1.26 php81-dom-8.1.26 php81-extensions-1.1 php81-fileinfo-8.1.26 php81-filter-8.1.26 php81-gd-8.1.26 php81-iconv-8.1.26 php81-intl-8.1.26_1 php81-mbstring-8.1.26 php81-mysqli-8.1.26 php81-opcache-8.1.26 php81-pdo-8.1.26 php81-pdo_mysql-8.1.26 php81-pdo_sqlite-8.1.26 php81-pecl-imagick-3.7.0_2 php81-pecl-memcached-3.2.0 php81-pecl-ssh2-1.3.1 php81-phar-8.1.26 php81-session-8.1.26 php81-simplexml-8.1.26 php81-soap-8.1.26 php81-sqlite3-8.1.26 php81-tokenizer-8.1.26 php81-xml-8.1.26 php81-xmlreader-8.1.26 php81-zip-8.1.26
I will look into it. One major reason I actually moved into system pcre is that the pcre library is not at all updated in the lifetime of a major php versions and except php8.3 all are vulnerable versions which is a security risk from my perspective. If the system pcre creates a regression and it's known to them I am still unsure why do they have those dependencies mentioned in the documentation or even have that configure knob.
Ok, the reason to use system pcre make sense. I don't know how the internals work and the backtraces look strange, e.g. not related to pcre. So maybe there is some "pointer" bug not related to pcre that manifests in some conditions (e.g. when system pcre is used and some specific extensions are used). I was able to replicate the crash with "just" the following php82-extensions options: OPTIONS_FILE_SET+=CURL OPTIONS_FILE_SET+=FILEINFO OPTIONS_FILE_SET+=OPCACHE OPTIONS_FILE_SET+=ZLIB # service apache24 start # apachectl graceful # lldb --core httpd-72229.core httpd (lldb) target create "httpd" --core "httpd-72229.core" Core file '/tmp/httpd-72229.core' (x86_64) was loaded. (lldb) bt * thread #1, name = 'httpd', stop reason = signal SIGSEGV * frame #0: 0x0000000801ecdf66 opcache.so`___lldb_unnamed_symbol1013 + 838 frame #1: 0x00000008025ce1c6 libphp.so`zend_extension_startup(extension=0x0000000801b18010) at zend_extensions.c:196:7 frame #2: 0x00000008025b0ba1 libphp.so`zend_llist_apply_with_del(l=0x0000000802754100, func=(libphp.so`zend_extension_startup at zend_extensions.c:193)) at zend_llist.c:171:7 frame #3: 0x00000008025ce1a8 libphp.so`zend_startup_extensions at zend_extensions.c:217:2 [artificial] frame #4: 0x000000080254f8c7 libphp.so`php_module_startup(sf=<unavailable>, additional_modules=<unavailable>, num_additional_modules=1) at main.c:2263:2 frame #5: 0x00000008026ad385 libphp.so`php_apache2_startup(sapi_module=<unavailable>) at sapi_apache2.c:381:6 frame #6: 0x00000008026acbd6 libphp.so`php_apache_server_startup(pconf=0x0000000800907028, plog=<unavailable>, ptemp=<unavailable>, s=<unavailable>) at sapi_apache2.c:495:6 frame #7: 0x0000000000263d6f httpd`ap_run_post_config + 79 frame #8: 0x000000000025f4cc httpd`main + 2476 frame #9: 0x000000000025e910 httpd`_start + 256 So maybe try to reproduce it, if possible. It did not crash when I was using: OPTIONS_FILE_SET+=CURL OPTIONS_FILE_SET+=FILEINFO OPTIONS_FILE_SET+=OPCACHE OPTIONS_FILE_SET+=ZLIB So maybe just OPCACHE would be enough (will try next). The crash now ends in opcache.so instead of libzstd.so.
(In reply to Marián Černý from comment #4) typo: * php81-extensions
From my experience, Apache2 recently crashes almost on any module including its own. See bug 268318. Try building with GCC or workaround it.
I can confirm that it crashes also just with only: OPTIONS_FILE_SET+=OPCACHE I have seen bug 268318. I have skimmed it before posting my own. I thought they are not related. But now I see this backtrace in this comment by Miroslav Lachman is very similar to my own with opcache: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268318#c18 I have tried: elfctl -e +noaslr /usr/local/sbin/httpd but it still crashes. (Migrating to php-fpm is on my TODO list for next year.)
Merry Christmas and Happy New Year. :D We are running a php setup with huge hits using nginx and php_fpm. We have tried these ports with that setup even before committing and never seen anything like these. So cannot confirm with apache. I am closing this for now. But if you have any additional information or changes that might help you let me know and reopen this ticket.
(In reply to Marián Černý from comment #7) I can also confirm that reverting --with-external-pcre fixes the crash for me even with just: OPTIONS_FILE_SET+=OPCACHE
(In reply to Marián Černý from comment #9) Okiz. I will look into it and revert back.
Badly affected by this issue. Is it enough to just revert the commit?
(In reply to void from comment #11) Sorry to hear that. But as said by some Apache members they have been discouraging this for a while and some people are still using it in production. I will work on it and revert back with a commit by tomorrow night.
After some additional info by Muhammad Moinur Rahman at https://github.com/php/php-src/issues/12844#issuecomment-1840780257 I think it is one of the ASLR related bugs. I am using FreeBSD 12.4 where ASLR is not enabled by default. I guess there is a pointer bug somewhere in apache or php that is causing the crash. When ASLR is not used, the pointer points to some valid memory. However when ASLR is used, the memory becomes invalid. Similar thing probably happens with external pcre (php code image is smaller in memory) and gcc (it compiles the code differently so the pointer might point to some valid memory, unless it's llvm bug, which I guess it's much less probable). The core dumps I have seen crash when going from libphp.so to some other place, so I guess the bug is in PHP. I would suspect the list that is accessed by zend_llist_apply_with_del to be causing the problem. If the crash is reproducible, somebody with source code knowledge of php and apache should be able to track it down. So I think pcre is not directly involved, but when using external pcre on FreeBSD 12.4 the memory layout changes that manifests this bug.
(In reply to Marián Černý from comment #13) Thanks for the additional info. I have made a patch to OPTIONIZE the PCRE usage for now with php81(non-default). But if you are using 12.4-RELEASE then you should plan on updating ASAP as the EOL reaches on 31st December. I will look into the possibilities of adding the same for php82 and php83 but cannot confirm now whether if it will be DEFAULT or non-DEFAULT. But even in the worse case users will need to chose and build there own pkgs. And as said mod_php is the legacy and should be let go while moving into php-fpm. The commit will take place sometimes tomorrow once I test all packages dependent on php81.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=5acc30bdc24452e17fff0127122096bda42b1d18 commit 5acc30bdc24452e17fff0127122096bda42b1d18 Author: Muhammad Moinur Rahman <bofh@FreeBSD.org> AuthorDate: 2023-12-05 17:20:34 +0000 Commit: Muhammad Moinur Rahman <bofh@FreeBSD.org> CommitDate: 2023-12-05 17:29:32 +0000 lang/php81: Runtime fix - php modules while build with system pcre fails to run when using apache httpd and mod_php. So OPTIONIZE and add as a non-DEFAULT option. However to my knowledge and experience I haven't faced this with nginx and php-fpm yet. [1] - There is another corner case with apache httpd and mod_php when run on 13.2-RELEASE or newer specially on versions where ASLR is by default turned on. All modules specialy opcache crashes httpd. So add a warning message to advise users to move into php-fpm. [2] Thanks to brnrd@ for the patch. [2] PR: 275522 [1], 268318 [2] Reported by: majo-bugs.freebsd.org@cerny.sk [1], zarychtam@plan-b.pwste.edu.pl [2] lang/php81/Makefile | 15 ++++++++++----- lang/php81/pkg-message.mod | 15 ++++++++++----- lang/php81/pkg-plist | 7 +++++++ 3 files changed, 27 insertions(+), 10 deletions(-)
(In reply to Muhammad Moinur Rahman from comment #14) Did you manage to look into adding the same patch for PHP 8.2 and 8.3? Unfortunately I am also affected by this issue and had to revert the last commit as a workaround. Switching to php-fpm is being evaluated but not yet ready (as we have a pretty big infrastructure), so an option to switch back to the bundled pcre would be nice.
The refactor also broke php80. There are also other bugs because of this issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275809 These changes caused multiple setups to break and there wasn't even an UPDATING note.. This should seriously get fixed for all versions asap or reverted..
(In reply to vincent.jancso from comment #16) The patch will go on the next update of php82 and php83 in this week hopefully. I am already testing 8.2.14RC and 8.3.1RC.
By the way: Why did you remove a whole set of patches for PHP 8.0 and 8.3 (e.g. https://cgit.freebsd.org/ports/commit/?id=4f698c44ee8be9017a07ba9a2b14c9d766ea005b) but not for PHP 8.1 and 8.2? This seems to be inconsistent too...
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=e216badd22213c1654b46954576b85887a0a92b7 commit e216badd22213c1654b46954576b85887a0a92b7 Author: Muhammad Moinur Rahman <bofh@FreeBSD.org> AuthorDate: 2023-12-22 00:33:13 +0000 Commit: Muhammad Moinur Rahman <bofh@FreeBSD.org> CommitDate: 2023-12-22 01:24:36 +0000 lang/php82: Update version 8.2.13=>8.2.14 - php modules while build with system pcre fails to run when using apache httpd and mod_php. So OPTIONIZE and add as a non-DEFAULT option. However to my knowledge and experience I haven't faced this with nginx and php-fpm yet. [1] - There is another corner case with apache httpd and mod_php when run on 13.2-RELEASE or newer specially on versions where ASLR is by default turned on. All modules specialy opcache crashes httpd. So add a warning message to advise users to move into php-fpm. [2] - Remove unnecesary patches from devel/php82-readline [3] Changelog: https://www.php.net/ChangeLog-8.php#8.2.14 PR: 275522 [1], 268318 [2], 275795 [3] Reported by: majo-bugs.freebsd.org@cerny.sk [1], zarychtam@plan-b.pwste.edu.pl [2], x0r+freebsd@x0r.fr [3] devel/php82-readline/files/patch-config.m4 (gone) | 33 ------------------- .../files/patch-readline_cli.c (gone) | 13 -------- lang/php82/Makefile | 16 +++++---- lang/php82/distinfo | 6 ++-- lang/php82/files/patch-TSRM_TSRM.c | 4 +-- lang/php82/files/patch-configure.ac | 10 +++--- lang/php82/pkg-message.mod | 15 ++++++--- lang/php82/pkg-plist | 7 ++++ sysutils/php82-posix/files/patch-posix.c (gone) | 38 ---------------------- www/php82-opcache/files/patch-config.m4 (gone) | 20 ------------ 10 files changed, 37 insertions(+), 125 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=58de78405733ce66390af3d1db0e40e35a0d4355 commit 58de78405733ce66390af3d1db0e40e35a0d4355 Author: Muhammad Moinur Rahman <bofh@FreeBSD.org> AuthorDate: 2023-12-21 23:46:58 +0000 Commit: Muhammad Moinur Rahman <bofh@FreeBSD.org> CommitDate: 2023-12-22 01:24:35 +0000 lang/php83: Update version 8.3.0=>8.3.1 - php modules while build with system pcre fails to run when using apache httpd and mod_php. So OPTIONIZE and add as a non-DEFAULT option. However to my knowledge and experience I haven't faced this with nginx and php-fpm yet. [1] - There is another corner case with apache httpd and mod_php when run on 13.2-RELEASE or newer specially on versions where ASLR is by default turned on. All modules specialy opcache crashes httpd. So add a warning message to advise users to move into php-fpm. [2] Thanks to brnrd@ for the patch. [2] Changelog: https://www.php.net/ChangeLog-8.php#8.3.1 PR: 275522 [1], 268318 [2] Reported by: majo-bugs.freebsd.org@cerny.sk [1], zarychtam@plan-b.pwste.edu.pl [2] lang/php83/Makefile | 17 +++++++++++------ lang/php83/distinfo | 6 +++--- lang/php83/files/patch-configure.ac | 12 ++++++------ ...tch-ext_pcre_pcre2lib_sljit_sljitConfigInternal.h | 10 +++++----- lang/php83/pkg-message.mod | 15 ++++++++++----- lang/php83/pkg-plist | 7 +++++++ www/php83-opcache/files/patch-config.m4 (gone) | 20 -------------------- 7 files changed, 42 insertions(+), 45 deletions(-)
(In reply to Muhammad Moinur Rahman from comment #18) Aren't the config.m4-patches you removed needed when restoring the option to use the bundled pcre? E.g. https://cgit.freebsd.org/ports/diff/mail/php83-imap/files/patch-config.m4?id=8667a82724a9c7d2560c438d98f23990d08a08eb
(In reply to Dani I. from comment #22) No. In previous if you look at the entire php ports framework what used to happen was these modules/extensions had --with-pcre-dir=${LOCALBASE} which means that they were built using System pcre even though php itself was built using bundled pcre. Now they are chosen automatically based on the --with-external-pcre flag.
(In reply to Muhammad Moinur Rahman from comment #23) But then whats the reason why they still exist for PHP 8.1 (e. g. https://github.com/freebsd/freebsd-ports/blob/main/archivers/php81-phar/files/patch-config.m4 ) and PHP 8.2 (e. g. https://github.com/freebsd/freebsd-ports/blob/main/mail/php82-imap/files/patch-config.m4 ) and have only been removed for PHP 8.0 and 8.3?
(In reply to Dani I. from comment #24) At the time when I have explicitly started working on this part 8.1 was the DEFAULT and 8.2 was the next default. And we were already using php83 in production without these patches. Hence I did not have the chance to check these in php81 and php82. Although I do build test all php consumers every night with all versions of php but it's not runtime tested. And there are too many combinations for runtime combinations which are not at all practical for me to do but wait for end users feedback. And I did made the changes in php80 because it was already the legacy version and we had one application(Customer is not willing to pay for upgrade) to test php80 in production. And in all our cases we use nginx/php-fpm combo. During this upgrade I have removed the readline patches as someone mentioned that the patch was trouble. I have also upstreamed the opcache patches. The posix patches are no longer required and it looks like we have carried over the POSIX patches for so long.