For a few months, www/apache24 doesn't respect SIGUSR1 from newsyslog(8), but segfaults[1]. To reproduce add[2] to newsyslog.conf(5) and run newsyslog(8). I am certain that the issue was introduced by the upgrade to apache24-2.4.54. By the way, sending SIGUSR1 by hand doesn't confuse the www/apache24 daemon. [1] [40067] pid 3446 (httpd), jid 0, uid 80: exited on signal 11 [40067] pid 2656 (httpd), jid 0, uid 80: exited on signal 11 [40067] pid 2650 (httpd), jid 0, uid 0: exited on signal 11 [2] /var/log/httpd-*.log 640 9 1 * XG /var/run/httpd.pid SIGUSR1
Maybe I haven't explained it clearly. What crashes apache24 is log rotation, not SIGUSR1 received, since it's probably already dead when notifications are sent[1]. Nevertheless, the error[1] occurs only with some rotations. [1] # newsyslog newsyslog: can't notify daemon, pid 2650 = /var/run/httpd.pid: No such process
Created attachment 238714 [details] core A bit of insight into the core file.
Without module libphp.so loaded log rotation goes just fine. Please see also bug 267435. Maybe a duplicate of this bug, but triggered in a different way. *** This bug has been marked as a duplicate of bug 267435 ***
Still not resolved.
Disabling the opcache module solves the issue. Maybe not a bug, but a feature; mod_php seems to be very fragile and vicious.
(In reply to Marek Zarychta from comment #5) Unfortunately we do not use apache at all in our environment. So cannot comment on anything about it. But indeed we switched to nginx due to the various problems of apache. Although apache httpd has been a widely used webserver and a classic application however my experience is apache really works well with other products from Apache family. Apart from that the overhead is too much for normal users like me.
After upgrading to 13.2, I also notice Apache crashes on reload when using the mod_php (with OPCache) module. It seems related with ASLR as disabling this prevents the crashes for me.
(In reply to Wout Decré from comment #7) Looks like opcache and ASLR actually do not play well. Although it was mentioned for php7 but it might be that it still exists for now. Rather than disabling ASLR entirely can you check whether if this works: elfctl -e +noaslr /usr/local/sbin/httpd
(In reply to Wout Decré from comment #7) Could you please specify which ASLR-related sysctl knobs have you turned off Wout? I am reopening this since it affects some people.
(In reply to Marek Zarychta from comment #9) I set kern.elf64.aslr.enable to 0 (setting kern.elf64.aslr.stack or kern.elf64.aslr.pie_enable only do not prevent the crashing). The following also works for me to prevent Apache from crashing: elfctl -e +noaslr /usr/local/sbin/httpd
(In reply to Muhammad Moinur Rahman from comment #8) >Rather than disabling ASLR entirely can you check whether if this works: >elfctl -e +noaslr /usr/local/sbin/httpd It helps a bit, some Apache processes crash, but not all of them. (In reply to Wout Decré from comment #7) >It seems related with ASLR as disabling this prevents the crashes for me. Yes, it solves the problem. Now no single process of Apache crashes. Opcache module can be enabled again.
(In reply to Marek Zarychta from comment #11) > It helps a bit, some Apache processes crash, but not all of them. Are you sure you are not just seeing the effect of the reload? For me, setting this option on the executable also prevents it from crashing on reload.
(In reply to Wout Decré from comment #12) I am updating Apache and PHP to the most recent version, and will run it for some time with ASLR turned off this way: elfctl -e +noaslr /usr/local/sbin/httpd I will give you feedback in a few days.
Confirmed same behavior. Work fine on 13.1 amd64, but crashed on 13.2 after kill -30 (newsyslog). php 8.1. gdb and lldb show opcache.so as point of crash. After "elfctl -e +noaslr /usr/local/sbin/httpd" no more crash.
(In reply to Vladimir Druzenko from comment #14) > After "elfctl -e +noaslr /usr/local/sbin/httpd" no more crash. Didn't help. Test was incorrect - with opcache disabled. So after it was enabled - apache crashed after kill -30 again…
(In reply to Vladimir Druzenko from comment #15) That is strange, I have been running Apache + PHP OPCache with +noaslr flag for about 3 weeks now without any crash on reload. Just double checked by removing it again and than I can reproduce the issue.
As an original reporter I have to confirm that it still crashes with opcache despite the fact of marking this binary noasrl (elfctl -e +noaslr /usr/local/sbin/httpd). I am still testing, as promised in comment 13. Let me check how it would cope with ASLR being disabled completely on the host. I will provide more feedback in a few days.
(In reply to Vladimir Druzenko from comment #15) As many of you I see Apache httpd crashes after OS upgrade to 13.2-RELEASE. According to gdb it crashed in opcache.so (PHP extension Opcache) #0 0x000000083e878950 in ?? () (gdb) bt #0 0x000000083e878950 in ?? () #1 0x0000000841c37c52 in ?? () from /usr/local/lib/php/20210902/opcache.so #2 0x000000083f27ec46 in ?? () from /usr/local/libexec/apache24/libphp.so #3 0x000000083f260a12 in zend_llist_apply_with_del () from /usr/local/libexec/apache24/libphp.so #4 0x000000083f1fedd8 in php_module_startup () from /usr/local/libexec/apache24/libphp.so #5 0x000000083f35c995 in ?? () from /usr/local/libexec/apache24/libphp.so #6 0x000000083f35c1e6 in ?? () from /usr/local/libexec/apache24/libphp.so #7 0x00000000002620cf in ap_run_post_config () #8 0x000000000025d889 in main () Setting opcache.enable=0 in /usr/local/etc/php.ini helped as a quick workaround. I tried to disable ASLR for httpd with opcache enabled and in my case "elfctl -e +noaslr /usr/local/sbin/httpd" works. No more crashes on SIGUSR1. To disable ASLR for httpd Apache must not be running, otherwise you get an error like this: elfctl: error opening file /usr/local/sbin/httpd: Text file busy The whole setup is something like this: # service apache24 stop # elfctl -e +noaslr /usr/local/sbin/httpd # sed -i '' -E 's/^(opcache.enable *)=.*/\1=1/' /usr/local/etc/php.ini # service apache24 start # kill -USR1 `cat /var/run/httpd.pid` # service apache24 status apache24 is running as pid 50219. # php -m [PHP Modules] bcmath bz2 calendar Core ctype curl date dom exif fileinfo filter ftp gd hash iconv imap intl json ldap libxml mbstring mcrypt memcached mysqli mysqlnd openssl pcre PDO pdo_dblib pdo_mysql pdo_sqlite Phar posix Reflection session SimpleXML soap sockets SPL sqlite3 ssh2 standard tokenizer xml xmlreader xmlwriter xsl Zend OPcache zip zlib [Zend Modules] Zend OPcache The only downside of this is invalidated package checksum on /usr/local/sbin/httpd # pkg check -sa Checking all packages: 1% apache24-2.4.56: checksum mismatch for /usr/local/sbin/httpd Checking all packages: 100% Checksum can be recalculated by pkg check -r or pkg check --recompute. And "elfctl -e +noaslr /usr/local/sbin/httpd" must be executed after each Apache upgrade / reinstall.
(In reply to Miroslav Lachman from comment #18) That's not as simple as it looks in your comment Miroslav. Apparently, after disabling ASLR for the Apache binary everything is fine with SIGUSR1, but the real problem is when the log is rotated. With ASRL disabled for the binary, Apache sometimes dies, and sometimes survives log rotation. It doesn't make much difference, since the surviving process is not functional. Perhaps during the log rotation not only SIGUSR1 is sent, but other factors come to play.
(In reply to Marek Zarychta from comment #19) I tried to rotate some logs (manually) and then send SIGUSER1 - no crash. Can the crash be more related to newsyslog's handling of log files and sending signals to processes?
(In reply to Miroslav Lachman from comment #18) Did all your steps, but still got crush: # kill -USR1 `cat /var/run/httpd.pid` # service apache24 status apache24 is not running. # tail -n1 /var/log/messages May 5 14:38:20 HOST-NAME kernel: pid 48881 (httpd), jid 0, uid 0: exited on signal 4 (core dumped) And no real core dump. With default apache signal 11 and core dump at /httpd.core. And gdb look same as in your comment.
(In reply to Vladimir Druzenko from comment #21) It is very strange behavior. I would expect it to be consistent across all machines, but this means we have to delay the deployment of 13.2 into production until this is resolved.
(In reply to Miroslav Lachman from comment #22) It's one more bug: graphics/pecl-imagick-im7 + graphics/ImageMagick7 with OPENMP=on. After rebuild graphics/ImageMagick7 with OPENMP=off and "elfctl -e +noaslr /usr/local/sbin/httpd" - no more crashes after "kill -USR1 `cat /var/run/httpd.pid`".
(In reply to Vladimir Druzenko from comment #23) Good to know about pecl-imagick, thank you! Luckily we only use pecl-imagick-im7 on a few machines, I checked ImageMagick's config and we have OpenMP=off. I upgraded another machine to 13.2 (this one has pecl-imagick-im7) and all machines with "elfctl -e +noaslr /usr/local/sbin/httpd" survived the midnight log rotation with SIGUSR1 a few minutes ago.
I am not able to solve it in any way but completely disabling the opcache. Turning ASLR on the host doesn't help. The host is really a bhyve guest. Could that affect Apache running in this bhyve guest ?
(In reply to Marek Zarychta from comment #25) > Turning ASLR on the host doesn't help. The host is really a bhyve guest. I meant of course "turning ASLR off".
(In reply to Marek Zarychta from comment #25) Can you test it with Opcache as the only enabled PHP extension? (disable or uninstall everything else). I guess if it can be coincidence between Opcache and some other PHP extension.
(In reply to Miroslav Lachman from comment #27) Thanks for the tip Miroslav. TBH I don't really have to use opcache there. For me the investigation is over, but to make this PR proficient we need some outcome. Probably adding post install messages in pkg-message.mod for www/mod_php8{0,1,2} suggesting disabling ASLR for www/apache24 with the command "elfctl -e +noaslr /usr/local/sbin/httpd" if any issues occur for the users, could solve some issues would be helpful and the only outcome we can provide. What do you think about it Muhammad ?
Adding pkg-message is alright with me. But we cannot ask the end users to do that on executable. We have to optionize apache24 with ASLR/ASLR_OFF and do that in post-install target. And add the pkg message like Build apache24 with ASLR off when you need opcache. Apart from that as I do not have the infrastructure to test apache now I am more of a blind person here.
Created attachment 242126 [details] www/apache24: Disable ASLR for compiled binaries (In reply to Muhammad Moinur Rahman from comment #29) This patch adds new NOASLR option to the Apache port. Please test if it helps. It is also possible to create a new flavor of the port, but I'd rather avoid it. Please test if www/apache24 built with NOASLR option works fine with mod_php and opcache enabled. Perhaps we want only httpd to be run without ASLR, now the patch marks all binaries this way.
Comment on attachment 242126 [details] www/apache24: Disable ASLR for compiled binaries Noops. This is not the way of doing this. Check the file Mk/Uses/elfctl.mk.
i will discuss it with the rest of the apache team to see if we approve it
Comment on attachment 242126 [details] www/apache24: Disable ASLR for compiled binaries >diff --git a/www/apache24/Makefile b/www/apache24/Makefile >index 96526dddad..b96cb3d4ba 100644 >--- a/www/apache24/Makefile >+++ b/www/apache24/Makefile >@@ -1,5 +1,6 @@ > PORTNAME= apache24 > PORTVERSION= 2.4.57 >+PORTREVISION= 1 > CATEGORIES= www > MASTER_SITES= APACHE_HTTPD > DISTNAME= httpd-${PORTVERSION} >@@ -226,6 +227,15 @@ post-install: > # Remove files left behind by strip > ${RM} ${STAGEDIR}${DATADIR}/build/ecp.???????? 2>/dev/null > >+.if ${PORT_OPTIONS:MNOASLR} >+. for f in checkgid fcgistarter htcacheclean httpd rotatelogs >+ ${ELFCTL} -e +noaslr ${STAGEDIR}${PREFIX}/sbin/${f} >+. endfor >+. for f in ab htdbm htdigest htpasswd httxt2dbm logresolve >+ ${ELFCTL} -e +noaslr ${STAGEDIR}${PREFIX}/bin/${f} >+. endfor >+.endif >+ > post-install-LOG_FORENSIC-on: > ${INSTALL_SCRIPT} ${WRKSRC}/support/check_forensic ${STAGEDIR}${PREFIX}/sbin > >diff --git a/www/apache24/Makefile.options b/www/apache24/Makefile.options >index c0e6a60b8c..8f059ec85a 100644 >--- a/www/apache24/Makefile.options >+++ b/www/apache24/Makefile.options >@@ -59,7 +59,7 @@ MOST_DISABLED_MODULES= \ > # enable/disable additional PROXY/SESSION modules > META_MODULES= PROXY SESSION > >-ADDITIONAL_OPT= LUAJIT IPV4_MAPPED >+ADDITIONAL_OPT= LUAJIT IPV4_MAPPED NOASLR > > # =================================================================================== > >diff --git a/www/apache24/Makefile.options.desc b/www/apache24/Makefile.options.desc >index 353658c36b..9df7e91a32 100644 >--- a/www/apache24/Makefile.options.desc >+++ b/www/apache24/Makefile.options.desc >@@ -18,6 +18,7 @@ MPM_EVENT_DESC= MPM worker variant with the goal of consuming threads only for > # additional (no modules) > IPV4_MAPPED_DESC= Allow IPv6 sockets to handle IPv4 connections > LUAJIT_DESC= LuaJit Support >+NOASLR_DESC= Disable ASLR for compiled binaries > > # ===================================== > # modules
I don't think disabling ASLR on the apache port is the way to go to address this issue. This requires a user to create their own pkg repo when the PHP port comes with a solution for this: php-fpm.
(In reply to Bernard Spil from comment #34) I think that is why we are adding a non-default option which will allow users to create the custom port in their pkg repo.
(In reply to Bernard Spil from comment #34) I don't think php-fpm is "the solution". It is completely different case of setup, much different than mod_php and if somebody uses mod_php it has probably it's reason.
I see the same with www/mod_php81. It also has prevented py39-certbot-apache from renewing my server's certificate, as it uses apachectl graceful. This problem was not present under FreeBSD 13.1, but stated with the upgrade to 13.2.
(In reply to Marius Schamschula from comment #37) I went ahead with a more drastic solution: I dumped my last instance of apache for nginx/php-fpm. After 28 years I no longer have a production server running apache.
Maybe add an option (apache24_aslr?) to /usr/local/etc/rc.d/apache24 to start apache with: proccontrol -m aslr -s disable /usr/local/sbin/httpd As example apache24_fib option: command="/usr/sbin/setfib -F ${apache24_fib} ${command}" For apache24_aslr it can be: command="/usr/bin/proccontrol -m aslr -s disable ${command}" Need to test compatibility with apache24_fib.
(In reply to Vladimir Druzenko from comment #39) I never heard about proccontrol but if it works as it looks I think it would be the best workaround as it can be changed at runtime with one line modification in rc.conf.
Created attachment 242619 [details] Use apache24_aslr_disable="YES" in rc.conf as a workaround This patch work for me. Test it please.
(In reply to Vladimir Druzenko from comment #41) Just tested. It works as expected: no more craches w/ php8.2.6 + opcache enabled. Renewed a few ssl certs by certbot w/ apache plugin. Thank you!
Waiting maintainer approval.
can we test this some more to reduce where the bug is? does this bug occur in php only setups like fpm, or is it an interaction with the Apache Httpd API? I'm asking because no one associated with the Apache Httpd project has endorsed running PHP applications with mod_php for thirteen years. The recommendation has been fpm + mod_proxy_fcgi: https://cwiki.apache.org/confluence/display/HTTPD/PHP-FPM
Quite a few weeks passed and the bug still hits users. I've got another report of this problem today from local user. Any plans to apply some kind of fix to the port?
(In reply to Eugene Grosbein from comment #45) Try my patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268318#c41 P.S. Agreed - time to commit a workaround if we can't fix it quickly.
(In reply to Vladimir Druzenko from comment #46) The patch looks good for me. I already recommended ASLR disable to the user and this patch and it works.
(In reply to Eugene Grosbein from comment #47) use this patch since its announcement here. I can confirm its effectiveness and vote to apply it to www/apache24 port tree
We are also running with apache24_aslr_disable="YES" and patched apache24 port for more than month on many machines. Please commit it.
Maintainer timeout?
Hi, I ran into this as well with apache24, after upgrading to 13.2, but I see a slightly different behaviour than what's described in the comments above. Just adding this for completeness. To start with, disabling aslr via elfctl(1) seems to work, however I saw one crash on SIGHUP even with it disabled since I did that change (unfortunately I am unable to repro it now, and I didn't keep that coredump... :-/ might have been user error on my side). That said, disabling php opcache makes no difference for me, when ASLR is on. What I see doesn't seem to be influenced by any php option, actually. Also, in my case, with ASLR on, apache not only crashes absolutely reliably on every USR1 and HUP signal, but also every once in a while directly after just starting the process (initial start, not restart). So I had times where I tried starting it 3-4 times in a row before it kept running. Thanks for the work on this, this PR was very helpful.
Seeing the same issue, graceful restarts initiated via log rotation or certbot renewal/registration still cause Apache24 to become unresponsive. Never had an issue until upgrading to FreeBSD 13.2-RELEASE a few months ago. This fixes it for me: sudo elfctl -e +noaslr /usr/local/sbin/httpd The above command needed to be re-applied after upgrading to apache24-2.4.58 Currently running 13.2-RELEASE-p3 and php80-8.0.30 I'd be happy to provide anymore info required if needed, thx.
Moin moin lgtm as mentor, feel free commit with the approval from the maintainer. mfg Tobias
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=688bfd0c38351f28306d7bca5a8ceb280bee53d9 commit 688bfd0c38351f28306d7bca5a8ceb280bee53d9 Author: Vladimir Druzenko <vvd@FreeBSD.org> AuthorDate: 2023-10-26 10:55:24 +0000 Commit: Vladimir Druzenko <vvd@FreeBSD.org> CommitDate: 2023-10-26 10:55:24 +0000 www/apache24: workaround for bug "opcache + ASLR turned on crashes Apache if used www/mod_php8{0,1,2,3}" PR: 268318 Approved by: joneum (apache@), tcberner (mentor) MFH: 2023Q4 www/apache24/Makefile | 1 + www/apache24/files/apache24.in | 12 ++++++++++++ 2 files changed, 13 insertions(+)
A commit in branch 2023Q4 references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=deef0db3f36352c1fa22fd9eb168cb4621623a42 commit deef0db3f36352c1fa22fd9eb168cb4621623a42 Author: Vladimir Druzenko <vvd@FreeBSD.org> AuthorDate: 2023-10-26 10:55:24 +0000 Commit: Vladimir Druzenko <vvd@FreeBSD.org> CommitDate: 2023-10-26 11:02:16 +0000 www/apache24: workaround for bug "opcache + ASLR turned on crashes Apache if used www/mod_php8{0,1,2,3}" PR: 268318 Approved by: joneum (apache@), tcberner (mentor) MFH: 2023Q4 (cherry picked from commit 688bfd0c38351f28306d7bca5a8ceb280bee53d9) www/apache24/Makefile | 1 + www/apache24/files/apache24.in | 12 ++++++++++++ 2 files changed, 13 insertions(+)
Commited workaround. If someone makes the real fix, they can replace this workaround with it. Thanks!
In case someone has time to check and can help me out it would be highly appreciable if someone can actually disable ASLR on php rather than apache and see if the problem still persists. I have a plan to add ASLR option for all php ports after the EOL of 12. There is already an option NOASLR in lang/php83 which was introduced for a different issue but I am open to hear more cases as php has always had problems with ASLR. :/
(In reply to Vladimir Druzenko from comment #56) Thank you for coming up with and committing the workaround!
(In reply to Muhammad Moinur Rahman from comment #57) I expect: 1. A (security) warning for the user when they disable ASLR in Apache (with every rc command). 2. A warning in all mod_php ports telling the user to migrate to php-fpm as per Apache httpd project guidance. With a link to https://cwiki.apache.org/confluence/display/HTTPD/PHP-FPM I think we should trust Mina Galić's guidance, an ASF member Apache httpd.
Upgrading to apache24-2.4.58_1 and FreeBSD 13.4-RELEASE-p4 did not resolve the issue. Still had to use the " elfctl -e +noaslr /usr/local/sbin/httpd " workaround. Does anyone know if this bug persists in FreeBSD 14.0? Thx.
(In reply to Charles from comment #60) > 13.4-RELEASE-p4 What's the future like? :-D Try to use apache24_aslr_disable="YES" in rc.conf as a workaround and restart the apache: service apache24 restart
I'm testing this fix now. Found this ticket after finding httpd semi-randomly crashing. By semi-randomly I mean the timestamp on the core was at the exact hour. One was created when freebsd-update cron ran at 0500 Another was created when an internal cronjob within the website (php81) ran Recent upgrade from 12.4-p7 to 13.2-p4. So far, freebsd-update seems to have not triggered it
Problem is still happening even with apache24_aslr_disable="YES"
ASLR is obviously problematic. How can it be turned off system-wide?
(In reply to void from comment #63) What version of the www/apache24 do you have installed - 2.4.58_1?
(In reply to Vladimir Druzenko from comment #65) 2.4.58_1 built in poudriere-devel and installed on Thu Nov 30 00:47:28 2023 GMT
(In reply to void from comment #66) "elfctl -e +noaslr /usr/local/sbin/httpd" didn't help too? Do you have /httpd.core - can you show trace?
(In reply to Vladimir Druzenko from comment #67) I have the core but gdb shows nothing because it's not a debug build. I'll build with debug and reinstall, but can't do that right this minute because it's production.
(In reply to Vladimir Druzenko from comment #67) This is what I've had so far: (gdb) core /httpd.core [New LWP 101030] Core was generated by `/usr/local/sbin/httpd -DNOHTTPACCEPT'. Program terminated with signal SIGSEGV, Segmentation fault. Address not mapped to object. #0 0x000000083f237930 in ?? () (gdb) bt full #0 0x000000083f237930 in ?? () No symbol table info available. (goes on for another 10 or so lines with the same information)
Unfortunately, www/apache24 doesn't build with debug configured in poudriere for the port as per https://wiki.freebsd.org/DebuggingPorts and it fails here, with WITH_DEBUG_PORTS=www/apache24 DEBUG_FLAGS=-g -O0 also tried with just the WITH_DEBUG_PORTS=www/apache24 line [00:00:45] /usr/local/share/apr/build-1/libtool --silent --mode=compile cc -std=c89 -Werror -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wdeclaration-after-statement -Wpointer-arith -Wformat -Wformat-security -Wunused -g -g -O0 -fstack-protector-strong -fno-strict-aliasing -DAP_DEBUG -DLIBICONV_PLUG -I. -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/os/unix -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/include -I/usr/local/include/apr-1 -I/usr/include -I/usr/local/include -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/aaa -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/cache -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/core -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/database -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/filters -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/ldap -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/server -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/loggers -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/lua -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/proxy -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/http2 -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/session -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/ssl -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/test -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/server -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/md -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/arch/unix -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/dav/main -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/generators -I/wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/modules/mappers -prefer-non-pic -static -c util_expr_parse.c && touch util_expr_parse.lo [00:00:45] --- listen.lo --- [00:00:45] listen.c:416:20: error: unused variable 'next' [-Werror,-Wunused-variable] [00:00:45] ap_listen_rec *next; [00:00:45] ^ [00:00:45] listen.c:417:20: error: variable 'previous' set but not used [-Werror,-Wunused-but-set-variable] [00:00:45] ap_listen_rec *previous; [00:00:45] ^ [00:00:45] 2 errors generated. [00:00:45] *** [listen.lo] Error code 1 [00:00:45] [00:00:45] make[3]: stopped in /wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/server [00:00:46] 1 error [00:00:46] [00:00:46] make[3]: stopped in /wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/server [00:00:46] *** [all-recursive] Error code 1 [00:00:46] [00:00:46] make[2]: stopped in /wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/server [00:00:46] 1 error [00:00:46] [00:00:46] make[2]: stopped in /wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58/server [00:00:46] *** [all-recursive] Error code 1 [00:00:46] [00:00:46] make[1]: stopped in /wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58 [00:00:46] 1 error [00:00:46] [00:00:46] make[1]: stopped in /wrkdirs/usr/ports/www/apache24/work/httpd-2.4.58 [00:00:46] ===> Compilation failed unexpectedly. [00:00:46] Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to [00:00:46] the maintainer. [00:00:46] *** Error code 1 [00:00:46]
(In reply to void from comment #70) please report this upstream. And add the reported issues to CFLAGS, for example CFLAGS+=-Wno-unused-but-set-variable
Perhaps, ASLR problem in Apache is related to https://www.freebsd.org/security/advisories/FreeBSD-EN-23:15.sanitizer.asc ?
Created attachment 246722 [details] git diff for lang/php8 ``` lang/php81: Add guidance on mod_php usage * Users of mod_php should switch to php-fpm PR: 268318 ``` Without these additions to the lang/php8* ports, all changes related to www/apache24 for mod_php being broken will be reverted. with hat: Apache
(In reply to Mina Galić from comment #71) To make sure the builder was ok, I created a brand new poudriere jail instance for 13.2 with the same make.conf (with debug enabled) in order to compare, and told it to build www/apache24. The debug version built. The instance that didn't work: 13.2-RELEASE-p4 1302001 d20ece445 (2023-10-13 07:19:51) The one that did: 13.2-RELEASE-p6 1302001 08225c103 (2023-12-02 12:13:38) Maybe the issue Eugene https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268318#c72 raises is in play here? Right now, rebuilding the older instance.
I see this thread became popular recently. From my experience, the issue got even worse after upgrading to apache24-2.4.58_1 (build locally from prots tree). Now, signal 30 crashes Apache even without mod_php loaded, but the daemon dies now on signal 4. I think it's a new bug but I had no time to take a closer look at it. It all happens on quite a recent stable/13: FreeBSD 13.2-STABLE #87 stable/13test-n256818-d84eb20b8f9b: Fri Dec 1 18:47:08 CET 2023, already containing mentioned Clang sanitizer errata, with ASLR globally enabled and without mod_php loaded. Perhaps SIGUSR1 is no longer accepted by the daemon for some reason?
Multiple bugs are involved. I am closing since the original one is solved.
(In reply to Bernard Spil from comment #73) I will commit these in next set of php updates as there is actually no underlying changes that effect the PORTS building.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=5acc30bdc24452e17fff0127122096bda42b1d18 commit 5acc30bdc24452e17fff0127122096bda42b1d18 Author: Muhammad Moinur Rahman <bofh@FreeBSD.org> AuthorDate: 2023-12-05 17:20:34 +0000 Commit: Muhammad Moinur Rahman <bofh@FreeBSD.org> CommitDate: 2023-12-05 17:29:32 +0000 lang/php81: Runtime fix - php modules while build with system pcre fails to run when using apache httpd and mod_php. So OPTIONIZE and add as a non-DEFAULT option. However to my knowledge and experience I haven't faced this with nginx and php-fpm yet. [1] - There is another corner case with apache httpd and mod_php when run on 13.2-RELEASE or newer specially on versions where ASLR is by default turned on. All modules specialy opcache crashes httpd. So add a warning message to advise users to move into php-fpm. [2] Thanks to brnrd@ for the patch. [2] PR: 275522 [1], 268318 [2] Reported by: majo-bugs.freebsd.org@cerny.sk [1], zarychtam@plan-b.pwste.edu.pl [2] lang/php81/Makefile | 15 ++++++++++----- lang/php81/pkg-message.mod | 15 ++++++++++----- lang/php81/pkg-plist | 7 +++++++ 3 files changed, 27 insertions(+), 10 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=e216badd22213c1654b46954576b85887a0a92b7 commit e216badd22213c1654b46954576b85887a0a92b7 Author: Muhammad Moinur Rahman <bofh@FreeBSD.org> AuthorDate: 2023-12-22 00:33:13 +0000 Commit: Muhammad Moinur Rahman <bofh@FreeBSD.org> CommitDate: 2023-12-22 01:24:36 +0000 lang/php82: Update version 8.2.13=>8.2.14 - php modules while build with system pcre fails to run when using apache httpd and mod_php. So OPTIONIZE and add as a non-DEFAULT option. However to my knowledge and experience I haven't faced this with nginx and php-fpm yet. [1] - There is another corner case with apache httpd and mod_php when run on 13.2-RELEASE or newer specially on versions where ASLR is by default turned on. All modules specialy opcache crashes httpd. So add a warning message to advise users to move into php-fpm. [2] - Remove unnecesary patches from devel/php82-readline [3] Changelog: https://www.php.net/ChangeLog-8.php#8.2.14 PR: 275522 [1], 268318 [2], 275795 [3] Reported by: majo-bugs.freebsd.org@cerny.sk [1], zarychtam@plan-b.pwste.edu.pl [2], x0r+freebsd@x0r.fr [3] devel/php82-readline/files/patch-config.m4 (gone) | 33 ------------------- .../files/patch-readline_cli.c (gone) | 13 -------- lang/php82/Makefile | 16 +++++---- lang/php82/distinfo | 6 ++-- lang/php82/files/patch-TSRM_TSRM.c | 4 +-- lang/php82/files/patch-configure.ac | 10 +++--- lang/php82/pkg-message.mod | 15 ++++++--- lang/php82/pkg-plist | 7 ++++ sysutils/php82-posix/files/patch-posix.c (gone) | 38 ---------------------- www/php82-opcache/files/patch-config.m4 (gone) | 20 ------------ 10 files changed, 37 insertions(+), 125 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=58de78405733ce66390af3d1db0e40e35a0d4355 commit 58de78405733ce66390af3d1db0e40e35a0d4355 Author: Muhammad Moinur Rahman <bofh@FreeBSD.org> AuthorDate: 2023-12-21 23:46:58 +0000 Commit: Muhammad Moinur Rahman <bofh@FreeBSD.org> CommitDate: 2023-12-22 01:24:35 +0000 lang/php83: Update version 8.3.0=>8.3.1 - php modules while build with system pcre fails to run when using apache httpd and mod_php. So OPTIONIZE and add as a non-DEFAULT option. However to my knowledge and experience I haven't faced this with nginx and php-fpm yet. [1] - There is another corner case with apache httpd and mod_php when run on 13.2-RELEASE or newer specially on versions where ASLR is by default turned on. All modules specialy opcache crashes httpd. So add a warning message to advise users to move into php-fpm. [2] Thanks to brnrd@ for the patch. [2] Changelog: https://www.php.net/ChangeLog-8.php#8.3.1 PR: 275522 [1], 268318 [2] Reported by: majo-bugs.freebsd.org@cerny.sk [1], zarychtam@plan-b.pwste.edu.pl [2] lang/php83/Makefile | 17 +++++++++++------ lang/php83/distinfo | 6 +++--- lang/php83/files/patch-configure.ac | 12 ++++++------ ...tch-ext_pcre_pcre2lib_sljit_sljitConfigInternal.h | 10 +++++----- lang/php83/pkg-message.mod | 15 ++++++++++----- lang/php83/pkg-plist | 7 +++++++ www/php83-opcache/files/patch-config.m4 (gone) | 20 -------------------- 7 files changed, 42 insertions(+), 45 deletions(-)
(In reply to commit-hook from comment #79) > lang/php82: Update version 8.2.13=>8.2.14 > .../files/patch-readline_cli.c (gone) | 13 -------- cc -I. -I/opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline -I/opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline/include -I/opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline/main -I/opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -I/usr/local/include -I/usr/local/include/editline -isystem /usr/local/include -DHAVE_CONFIG_H -O2 -pipe -march=nehalem -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -D_GNU_SOURCE -Wno-strict-prototypes -DZEND_COMPILE_DL_EXT=1 -c /opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline/readline.c -MMD -MF readline.dep -MT readline.lo -fPIC -DPIC -o .libs/readline.o /opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline/readline.c:163:38: error: use of undeclared identifier 'rl_mark' add_assoc_long(return_value,"mark",rl_mark); ^ /opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline/readline.c:165:47: error: use of undeclared identifier 'rl_pending_input' add_assoc_long(return_value,"pending_input",rl_pending_input); ^ /opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline/readline.c:201:16: error: use of undeclared identifier 'rl_mark' RETVAL_LONG(rl_mark); ^ /opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline/readline.c:209:13: error: use of undeclared identifier 'rl_pending_input' oldval = rl_pending_input; ^ /opt/obj/usr/ports/devel/php82-readline/work/php-8.2.14/ext/readline/readline.c:214:5: error: use of undeclared identifier 'rl_pending_input' rl_pending_input = Z_STRVAL_P(value)[0]; ^ 5 errors generated. *** [readline.lo] Error code 1
Restoring files from: https://cgit.freebsd.org/ports/tree/devel/php82-readline/files?id=665b149d2692ff6926a092568304091995e07d39 fixed build.
(In reply to Charles from comment #60) Yes. Workaround is 'sysctl kern.elf64.aslr.enable=0' or other methods of disabling ASLR discussed here.
(In reply to Miroslav Lachman from comment #36) Gold words: "if somebody uses mod_php it has probably it's reason"! Those who force users to use php-fpm are crippled.
I am running apache24-2.4.59 with mod_php82-8.2.18 on FreeBSD 13.2-RELEASE-p10 and even when I add apache24_aslr_disable="YES" into /etc/rc.conf and restart apache24 it still does crash on 'apachectl graceful'. Is there any way I can help in further debugging this? Initially as a workaround I have just removed /usr/local/etc/php/ext-10-opcache.ini, but that gets re-created after a php update, so I have now removed www/php82-opcache completely.
I suppose, the root of problem is in opcache extension of PHP, when it run under mod_php and ASLR is activated. This extension boosts code execution. Disabling opcache extension is bad idea.
(In reply to Alexey Koscheev from comment #86) I am aware that opcache can be useful and there are reasons to have it enabled. For the installations I currently have, it is not really needed. So for me it is a doable workaround, but may not be for others. This really depends on the use case.
(In reply to Fabian Wenk from comment #85) This problem is more complicated with almost each release of Apache or PHP. About a month ago I tried to debug the issue on 2 of our machines (just the 2 crashing, more than a dozen of others don't) As a workaround Opcache can be disabled by opcache.enable=0 in php.ini but on some workload the disabled Opcache is too bad for performance. I was curious why it crashes on those 2 machines and not on the others, I compared configuration in php.ini and the list of installed PHP extensions. I thought there are some extra extensions which can cause the problem but it was the other way around - 2 machines had fewer PHP extensions than other machines. Once I installed more extensions it stoped crashing the Apache with Opcache enabled. Then I tried to isolate minimalistic example and it was more complicated again. Even only 2 extensions (one is opcache) can crash Apache on reload. The minimal set of extensions installed to have Apache without crashes was this: ext-10-opcache.ini ext-18-session.ini ext-20-bcmath.ini ext-20-bz2.ini ext-20-ctype.ini ext-20-curl.ini ext-20-dom.ini ext-20-exif.ini ext-20-fileinfo.ini ext-20-filter.ini ext-20-ftp.ini ext-20-gd.ini ext-20-iconv.ini ext-20-intl.ini Once I remove any of them the Apache will crash on reload with Opcache enabled. The ftp PHP extension was one missing on the 2 mentioned machines. But when you add bunch of PHP extensions then you play this game again because it can start crashing again until you add even more extensions to stop it. This problem is back again after the latest update of Apache to 2.4.59 on different machines than before. 3 machines where Apache crashed on reload with this version and unmodified list of PHP extensions and unmodified php.ini. The more interesting is that it crashes Apache with PHP 8.1 on 2 machines and PHP 8.3 on third machine. The list of extensions is the same. Again, it can be worked around by opcache.enable=0 but it is not a solution. As the configuration of PHP is the same as bunch of other machines which do not crash I started to dig deeper - the list of Apache modules. And there are some differences. The main difference is that Apache with PHP with Opcache crashes on machines where proxy_wstunnel_module is loaded! Once I commented out the LoadModule proxy_wstunnel_module in httpd.conf Apache survived reload with PHP Opcache enabled. But we cannot disable wstunnel because it is required by some web apps. And things are still not clear enough. Another machine with proxy_wstunnel_module loaded and with PHP Opcache enabled does not crashes Apache on reload. So ... there are too many things together which must match to make Apache (don't) crash. All tests were with this in rc.conf: apache24_enable="YES" apache24_aslr_disable="YES" apache24_http_accept_enable="YES" And in case of PHP 8.3 both PHP and mod_php were built with NOASLR = On.
(In reply to Miroslav Lachman from comment #88) Try to change your httpd.conf: move "LoadModule php_module libexec/apache24/libphp.so" to the top of list LoadModule directives. If libphp.so is not loaded first - it is another reason to crash httpd on reload (since FreeBSD 12.2 or 12.3).
(In reply to Alexey Koscheev from comment #89) I never had this issue on FreeBSD 12.3 / 12.4, but to my knowledge ASLR was not enabled (on default). I did upgrade from FreeBSD 12.4 to 13.2 at the end of 2023, and after that this issue occurred. I have now just tried on my two systems where it did crash on 'apachectl graceful' even with apache24_aslr_disable="YES" in /etc/rc.conf. I did re-install www/php82-opcache (had it removed as workaround) and also have put the "LoadModule php_module libexec/apache24/libphp.so" line as first LoadModule line in httpd.conf. It still needs apache24_aslr_disable="YES" in /etc/rc.conf, but now it works just fine with 'apachectl graceful' on both of this FreeBSD 13.2 systems.
It looks like an endless bug. I have added to /etc/crontab this line as a workaround: 10 6 * * 1 root /usr/local/sbin/apachectl start It was solved, but broke again, then solved again and in future will probably break. But currently, after removing the PCRE2 PHP extension, signal 30 seems to be intercepted fine by Apache.
(In reply to Fabian Wenk from comment #90) ASLR was enabled by default in 13.2, but if you not had crashes on 12.4 you just lucky.
(In reply to Marek Zarychta from comment #91) pcre PHP extension is compiled in PHP core...
(In reply to Alexey Koscheev from comment #89) I can confirm that moving php_module to the top of the LoadModule list in httpd.conf solved my issue. It would be useful to have this noted in UPDATING or in pkg message for mod_php.
Don't confirm that moving php_module to the top of the LoadModule list in httpd.conf solved my issue.
(In reply to Aleksander Gda from comment #95) Even with 'sysctl kern.elf64.aslr.enable=0'?
I just want to confirm, that disabling ASLR works for me. I am migrating our servers from 12.4 to 13.3 (with mod_php81). As expected, apache24 was crashing on graceful reload even with bundled pcre (see Bug 275522). After disabling ASLR (kern.elf64.aslr.enable=0, kern.elf64.aslr.pie_enable=0) it does not crash. (So I will run without ASLR for now.)