Created attachment 246018 [details] my squid.conf After the update to 6.4 on my freeBSD 13.2-RELEASE-p4 on amd64 squid dumps core. In /var/log/messages I get: Oct 31 01:23:43 belisama squid[47596]: Squid Parent: squid-1 process 47654 exited due to signal 6 with status 0 Oct 31 01:23:43 belisama kernel: pid 47654 (squid), jid 0, uid 100: exited on signal 6 (core dumped) Oct 31 01:23:43 belisama squid[47596]: Squid Parent: squid-1 process 47654 will not be restarted for 3600 seconds due to repeated, frequent failures Oct 31 01:23:43 belisama squid[47596]: Exiting due to repeated, frequent failures gdb reports: root@belisama:/var/cache/pkg # gdb /usr/local/sbin/squid /var/cache/squid/squid.core GNU gdb (GDB) 13.2 [GDB v13.2 for FreeBSD] Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd13.2". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/squid... (No debugging symbols found in /usr/local/sbin/squid) [New LWP 100607] warning: Corrupted shared library list: 0x0 != 0x6e75466d654d6243 warning: Corrupted shared library list: 0x0 != 0x853280 Core was generated by `(squid-1) --kid squid-1 -f /usr/local/etc/squid/squid.conf'. Program terminated with signal SIGABRT, Aborted. Sent by thr_kill() from pid 47654 and user 100. #0 0x0000000830279c5a in ?? () (gdb) bt #0 0x0000000830279c5a in ?? () #1 0x00000008301f26d4 in ?? () #2 0x00000000000188ff in ?? () #3 0x0173bcf36f64ed8a in ?? () #4 0x00000008210a5874 in ?? () #5 0x0000000000399c93 in ?? () #6 0x00000008210a5890 in ?? () #7 0x00000008302a3a59 in ?? () #8 0x00000000009b1f60 in ?? () #9 0xfffffffffffffff3 in ?? () #10 0xffffffdf6f64ed8a in ?? () #11 0xffffffffffffffff in ?? () #12 0x00000008ffffffff in ?? () #13 0x0000000832ec98d8 in ?? () #14 0x00000008210a58c0 in ?? () #15 0x00000000007d407d in Adaptation::AccessRule::finalize() () Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) q I attach my squid.conf.
It seems there is a bug in Squid 6.4 https://www.mail-archive.com/squid-users@lists.squid-cache.org/msg25032.html
I have seen this too. Upstream submitted the following pull request: https://github.com/squid-cache/squid/pull/1561
(In reply to Dimitry Andric from comment #2) Has anybody tried this?
Ok, it's merged and included into upcoming 6.5 https://github.com/squid-cache/squid/pull/1565
Created attachment 246064 [details] Patch with upstream fix Would you try this update? It applies the patch from upstream.
(In reply to Dimitry Andric from comment #2) Hm I tried manually applying the #1561 fix, but it still asserted and dumped core for me. Maybe I made a mistake, I will try the newer fix in this bug.
(In reply to Dimitry Andric from comment #6) Nope, still getting the assertion:: 2023/11/04 19:12:00 kid1| storeLateRelease: released 72 objects 2023/11/04 19:12:02 kid1| FATAL: assertion failed: stmem.cc:98: "lowestOffset () <= target_offset" current master transaction: master65 2023/11/04 19:12:02| Removing PID file (/var/run/squid/squid.pid) Unfortunately I don't have debug symbols here: #0 thr_kill () at thr_kill.S:4 #1 0x000000082cc1a304 in __raise (s=s@entry=6) at /usr/src/lib/libc/gen/raise.c:52 #2 0x000000082cccb449 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 #3 0x00000000007c6a4d in xassert () #4 0x00000000005e1338 in mem_hdr::freeDataUpto(long) () #5 0x000000000050b295 in MemObject::trimUnSwappable() () #6 0x00000000005ebe82 in StoreEntry::trimMemory(bool) () #7 0x000000000081e545 in Store::Controller::memoryOut(StoreEntry&, bool) () #8 0x00000000005f9ada in StoreEntry::swapOut() () #9 0x00000000005f322e in StoreEntry::invokeHandlers() () #10 0x00000000005e7f05 in StoreEntry::write(StoreIOBuffer) () #11 0x0000000000666d93 in Client::addVirginReplyBody(char const*, long) () #12 0x00000000005a3ce8 in HttpStateData::writeReplyBody() () #13 0x00000000005a4163 in HttpStateData::processReplyBody() () #14 0x00000000005a1822 in HttpStateData::processReply() () #15 0x00000000005a2b72 in HttpStateData::readReply(CommIoCbParams const&) () #16 0x00000000005ab194 in JobDialer<HttpStateData>::dial(AsyncCall&) () #17 0x00000000006b84bd in AsyncCall::make() () #18 0x00000000006b981d in AsyncCallQueue::fire() () #19 0x00000000004d69f2 in EventLoop::runOnce() () #20 0x00000000004d6918 in EventLoop::run() () #21 0x00000000005b735c in SquidMain(int, char**) () #22 0x00000000005b62f2 in main () I'll rebuild with debug symbols and try it again.
(In reply to Dimitry Andric from comment #7) #0 thr_kill () at thr_kill.S:4 #1 0x000000083078d304 in __raise (s=s@entry=6) at /usr/src/lib/libc/gen/raise.c:52 #2 0x000000083083e449 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 #3 0x0000000000b95b2a in xassert (msg=0x4654ce "lowestOffset () <= target_offset", file=0x471471 "stmem.cc", line=98) at debug.cc:1269 #4 0x0000000000895dfa in mem_hdr::freeDataUpto (this=0x859d94628, target_offset=857) at stmem.cc:98 #5 0x000000000072f5b9 in MemObject::trimUnSwappable (this=0x859d94600) at MemObject.cc:400 #6 0x00000000008a3800 in StoreEntry::trimMemory (this=0x859c08d90, preserveSwappable=false) at store.cc:1816 #7 0x0000000000c1eaca in Store::Controller::memoryOut (this=0x832d8b4c0, e=..., preserveSwappable=false) at Controller.cc:600 #8 0x00000000008b9cc5 in StoreEntry::swapOut (this=0x859c08d90) at store_swapout.cc:168 #9 0x00000000008b00de in StoreEntry::invokeHandlers (this=0x859c08d90) at store_client.cc:847 #10 0x000000000089e4bf in StoreEntry::write (this=0x859c08d90, writeBuffer=...) at store.cc:784 #11 0x000000000096cf76 in Client::storeReplyBody (this=0x857558118, data=0x859deeb80 "\221\311^\312&\t\032^\314ԃ\275\265̳\241\255|\236\321\3348dz\2357pi\352:\020\332\367\256\230\205}\357\2401V\306\352\2279=@", len=10136) at Client.cc:1057 #12 0x000000000096b1f5 in Client::addVirginReplyBody (this=0x857558118, data=0x859deeb80 "\221\311^\312&\t\032^\314ԃ\275\265̳\241\255|\236\321\3348dz\2357pi\352:\020\332\367\256\230\205}\357\2401V\306\352\2279=@", len=10136) at Client.cc:1049 #13 0x000000000083b93c in HttpStateData::writeReplyBody (this=0x857558118) at http.cc:1415 #14 0x000000000083c095 in HttpStateData::processReplyBody (this=0x857558118) at http.cc:1498 #15 0x0000000000837cd3 in HttpStateData::processReply (this=0x857558118) at http.cc:1295 #16 0x000000000083a153 in HttpStateData::readReply (this=0x857558118, io=...) at http.cc:1265 #17 0x000000000084742b in CommCbMemFunT<HttpStateData, CommIoCbParams>::doDial (this=0x859d9d0d8) at ../src/CommCalls.h:190 #18 0x0000000000845f48 in JobDialer<HttpStateData>::dial (this=0x859d9d0d8, call=...) at ../src/base/AsyncJobCalls.h:175 #19 0x00000000008476bc in AsyncCallT<CommCbMemFunT<HttpStateData, CommIoCbParams> >::fire (this=0x859d9d0a0) at ../src/base/AsyncCall.h:147 #20 0x00000000009d78de in AsyncCall::make (this=0x859d9d0a0) at AsyncCall.cc:44 #21 0x00000000009d8c34 in AsyncCallQueue::fire (this=0x832ee49a0) at AsyncCallQueue.cc:27 #22 0x00000000006d41a9 in EventLoop::dispatchCalls (this=0x821a0dc40) at EventLoop.cc:144 #23 0x00000000006d4081 in EventLoop::runOnce (this=0x821a0dc40) at EventLoop.cc:121 #24 0x00000000006d3ef0 in EventLoop::run (this=0x821a0dc40) at EventLoop.cc:83 #25 0x0000000000857fb4 in SquidMain (argc=5, argv=0x821a0dd50) at main.cc:1710 #26 0x00000000008570e5 in SquidMainSafe (argc=5, argv=0x821a0dd50) at main.cc:1353 #27 0x00000000008570b2 in main (argc=5, argv=0x821a0dd50) at main.cc:1341 (gdb) frame 4 #4 0x0000000000895dfa in mem_hdr::freeDataUpto (this=0x859d94628, target_offset=857) at stmem.cc:98 98 assert (lowestOffset () <= target_offset); (gdb) p target_offset $4 = 857 This doesn't really provide much more info, but I think the upstream fix is not sufficient. I'll report upstream.
(In reply to Dimitry Andric from comment #8) Oh, I see in https://bugs.squid-cache.org/show_bug.cgi?id=5309 that you're also supposed to apply https://github.com/squid-cache/squid/pull/1562, which simply deletes the assertion. :D Yes, that will certainly get rid of the crashes, but I am not entirely convinced that the root cause was found...
Looks like we should probably do a few more, https://gitlab.archlinux.org/archlinux/packaging/packages/squid/-/blob/2bb4fd902e7bf1782def10b1420e8da61ff09537/PKGBUILD#L25
Created attachment 246180 [details] port patch to update to 6.5 Can you try 6.5?
Same for me with 6.4. Just updated to 6.5 - will write result tomorrow.
Work fine for me. Time to commit update to 6.5, IMHO.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=c37e92a06589cf9e0de1c6df283134701d727538 commit c37e92a06589cf9e0de1c6df283134701d727538 Author: Pavel Timofeev <timp87@gmail.com> AuthorDate: 2023-11-10 11:22:30 +0000 Commit: Vladimir Druzenko <vvd@FreeBSD.org> CommitDate: 2023-11-10 11:27:30 +0000 www/squid: update to 6.5 - dumps core after updating from 6.3 to 6.4 Fixed bug in 6.4 after fix CVE. PR: 274825 Approved by: arrowd (mentor) MFH: 2023Q4 www/squid/Makefile | 2 +- www/squid/distinfo | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-)
A commit in branch 2023Q4 references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=9c5ab2ee134ac5f7006980bff790305d393023d3 commit 9c5ab2ee134ac5f7006980bff790305d393023d3 Author: Pavel Timofeev <timp87@gmail.com> AuthorDate: 2023-11-10 11:22:30 +0000 Commit: Vladimir Druzenko <vvd@FreeBSD.org> CommitDate: 2023-11-10 11:32:56 +0000 www/squid: update to 6.5 - dumps core after updating from 6.3 to 6.4 Fixed bug in 6.4 after fix CVE. PR: 274825 Approved by: arrowd (mentor) MFH: 2023Q4 (cherry picked from commit c37e92a06589cf9e0de1c6df283134701d727538) www/squid/Makefile | 2 +- www/squid/distinfo | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-)
Commited, thanks!