Bug 258767

Summary: www/apache24: 2.4.49 stops accepting connections
Product: Ports & Packages Reporter: Morgan Davis <freebsd-bugs>
Component: Individual Port(s)Assignee: freebsd-apache (Nobody) <apache>
Status: Closed FIXED    
Severity: Affects Many People CC: apache, brnrd, joneum, lwhsu
Priority: Normal Flags: linimon: maintainer-feedback? (apache)
Version: Latest   
Hardware: Any   
OS: Any   
URL: https://bz.apache.org/bugzilla/show_bug.cgi?id=65596
Attachments:
Description Flags
patch file for Apache PR 65592 none

Description Morgan Davis 2021-09-28 19:03:04 UTC
Upgraded apache24 via pkg from 2.4.48 to 2.4.49 on a moderately heavy web server that has been running solidly for 30+ days, and within about 6 hours it stopped responding to requests.

The httpd process was still running, memory usage was fine, CPU load was fine, and no errors in logs.  Running "apachectl restart" fixed the issue, but would likely only be temporary.

This appears to be a common problem in Apache HTTPd 2.4.49 as also reported here:

https://forum.directadmin.com/threads/httpd-2-4-49.64548/

https://forum.directadmin.com/threads/apache-2-4-49-strange-unreachable-after-pass-few-hours.64559/

Solution is to revert to previous 2.4.48 version.

Sadly, 2.4.49 has patches for several CVEs.
Comment 1 Morgan Davis 2021-09-28 19:31:09 UTC
Additional related bug report here:

https://bz.apache.org/bugzilla/show_bug.cgi?id=65596
Comment 2 Bernard Spil freebsd_committer freebsd_triage 2021-09-29 07:41:00 UTC
Hi Morgan,

Have you tried to revert the change linked by Yann Ylavic in the Apache bugzilla post?

http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/server/mpm/event/event.c?r1=1893659&r2=1893660&pathrev=1893660&view=patch

I've been unable to replicate the issue sofar.
Comment 3 Morgan Davis 2021-09-29 08:40:33 UTC
@Bernard, no I've not tried applying patches and recompiling the port. For now, reverting to 2.4.48 across all my servers is a handful.

Just discovered that if you issue "apachectl graceful" to do a graceful restart, the 2.4.49 version would shutdown and exit with no restart. (All the servers I had yet to revert to 2.4.48 does this at midnight and I found httpd not running at all on those).

Details: all my servers are pretty vanilla running 12.2-RELEASE, GENERIC kernel, with standard packages (nothing custom, no ports).

FreeBSD xxxxxxxxxx 12.2-RELEASE-p7 FreeBSD 12.2-RELEASE-p7 GENERIC  amd64

# freebsd-version -u
12.2-RELEASE-p10
# freebsd-version -k
12.2-RELEASE-p7

They are also running php-fpm (just in case there is some odd interaction occurring now).

Hope this helps!
Comment 4 Morgan Davis 2021-09-29 08:46:35 UTC
Also, from everything I've read elsewhere, this seems to be an issue more with servers with relatively high traffic (over 250,000 requests in a 24 hour period in my case). So reproducing this in any lighter traffic environment might be difficult.
Comment 5 Morgan Davis 2021-09-29 08:56:46 UTC
More:

Apache not stable after update to 2.4.49-1.1.1.cpanel.x86_64
https://support.cpanel.net/hc/en-us/articles/4409287913111

"It has been reported that after upgrading to the RPM ea-apache24-2.4.49-1.1.1, Apache will restart and crash regularly."
Comment 6 Morgan Davis 2021-09-29 18:54:23 UTC
(In reply to Bernard Spil from comment #2)

Bernard, I found another reference to this same patch in this posting which also claims it fixes the problem: https://news.ycombinator.com/item?id=28645786

Assuming this patch truly resolves the issue, would you take on the work to incorporate the patch in the FreeBSD port and release an updated package?

If so, when might this appear in "pkg upgrade"? Considering the need to use "pkg lock" on the 2.4.48 version for now.

Thanks for your help on this.
Comment 7 Bernard Spil freebsd_committer freebsd_triage 2021-10-03 17:43:59 UTC
Created attachment 228408 [details]
patch file for Apache PR 65592

Currently testing this on my machine.

You can drop this file in www/apache24/files and rebuild Apache.
Comment 8 Bernard Spil freebsd_committer freebsd_triage 2021-10-05 07:16:03 UTC
Prepping to release 2.4.50 which should also address this bug.
Comment 9 commit-hook freebsd_committer freebsd_triage 2021-10-05 07:39:14 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=17acc171db1e4fd164df884e49072c91178b4831

commit 17acc171db1e4fd164df884e49072c91178b4831
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-10-05 07:26:21 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-10-05 07:26:21 +0000

    www/apache24: Bugfix update to 2.4.50

     * Fixes hang with event MPM

    PR:             258767

 www/apache24/Makefile | 2 +-
 www/apache24/distinfo | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)
Comment 10 Bernard Spil freebsd_committer freebsd_triage 2021-10-05 07:39:58 UTC
Closing this PR. I'm sure you'll let me know if there's still an issue.
Comment 11 commit-hook freebsd_committer freebsd_triage 2021-10-05 08:52:30 UTC
A commit in branch 2021Q4 references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=ba7ccc09c2978db59fc6e87a38745202cf0e6bfa

commit ba7ccc09c2978db59fc6e87a38745202cf0e6bfa
Author:     Bernard Spil <brnrd@FreeBSD.org>
AuthorDate: 2021-10-05 07:26:21 +0000
Commit:     Bernard Spil <brnrd@FreeBSD.org>
CommitDate: 2021-10-05 08:50:53 +0000

    www/apache24: Security update to 2.4.50

     * Fixes hang with event MPM

    PR:             258767
    security:       25b78bdd-25b8-11ec-a341-d4c9ef517024
    (cherry picked from commit 17acc171db1e4fd164df884e49072c91178b4831)

 www/apache24/Makefile | 2 +-
 www/apache24/distinfo | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)
Comment 12 Morgan Davis 2021-10-05 09:22:00 UTC
(In reply to Bernard Spil from comment #10)

Thank you, Bernard. I will let you know if there are any issues after testing this from the package on our dev systems first. They don't get anywhere near the same high traffic, but if they seem to be OK with some simulated traffic, I'll roll this out for production testing (and a fingers poised to revert to 2.4.48 if needed). Your efforts are much appreciated.