I can't remember when it started but recently mail delivery of mailing list significantly delays on weekend. What is happening to mailing list system?
Can you elaborate on the delay you're seeing ? Unfortunatly, our insight into mail delays is not as good as it should be 8-(
(In reply to Kurt Jaeger from comment #1) For example, https://lists.freebsd.org/pipermail/svn-ports-all/2019-February/206871.html This messages was delivered to my home mail server as following. https://www.utahime.org/FreeBSD/mailing-list/svn-ports-all.r492038.txt Accoding to headers it was delivered with following steps. 01. Sun, 3 Feb 2019 09:48:17 Posted to repo.freebsd.org. 02. Sun, 3 Feb 2019 09:48:17 Recieved from repo.freebsd.org by repo.freebsd.org 03. Sun, 3 Feb 2019 09:48:18 Recieved from repo.freebsd.org by by mxrelay.nyi.freebsd.org 04. Sun, 3 Feb 2019 09:48:18 Recieved from mxrelay.nyi.freebsd.org by mx1.freebsd.org 05. Sun, 3 Feb 2019 09:48:18 Recieved from mx1.freebsd.org by mailman.ysv.freebsd.org 06. Sun, 3 Feb 2019 ??:??:?? Delivered to svn-ports-all@mailman.ysv.freebsd.org 07. Sun, 3 Feb 2019 12:56:10 Recieved from mailman.ysv.freebsd.org by mx1.freebsd.org 08. Sun, 3 Feb 2019 12:56:13 Recieved from mx1.freebsd.org by mx2.freebsd.org 09. Sun, 3 Feb 2019 12:56:14 Recieved from mx2.freebsd.org by sv.hiroma.net 10. Sun, 3 Feb 2019 12:56:15 Recieved from sv.hiroma.net by localhost 11. Sun, 3 Feb 2019 12:56:18 Recieved from localhost by sv.hiroma.net 12. Sun, 3 Feb 2019 12:56:19 Recieved from sv.hiroma.net by gate.utahime.jp 13. Sun, 3 Feb 2019 12:56:20 Recieved from gate.utahime.jp by eastasia.home.utahime.org 14. Sun, 3 Feb 2019 12:56:24 Recieved from eastasia.home.utahime.org by localhost-backdoor.home.utahime.org 15. Sun, 3 Feb 2019 ??:??:?? Delivered to yasu@mail.home.utahime.org As you can see there is delay of about 3 hours from step 05 to step 07. It seems that it took about 3 hours for Mailman to deliver this mail to my registered mail address after receiving it from mx.freebsd.org.
~mailman/logs/smtp has this for 201902030948.x139mHPl054367 Feb 03 12:56:15 2019 (71317) <201902030948.x139mHPl054367@repo.freebsd.org> smtp to svn-ports-all for 177 recips, completed in 17.965 seconds Feb 03 13:35:36 2019 (71318) <201902030948.x139mHPl054367@repo.freebsd.org> smtp to svn-ports-head for 27 recips, completed in 3.541 seconds
(In reply to Kurt Jaeger from comment #3) Timestamp of first message almost matches that of step 07 in my comment #2. So it actually takes about 3 hours for Mailman to delver one mail to all registered mail addresses. Then you should investigate why so long time is required to deliver only one mail on weekend. Assuming that this machine is used only for mailing list server, my suggestions are, 1. See /var/log/cron and check if there are cron jobs that run on weekend. 2. See following periodic-related files and check what jobs runs on weekend. - /etc/periodic.conf - /etc/periodic/weekly/* - /usr/local/etc/periodic/weekly/*
mailman runs as a freebsd jail, so there might have been other things hogging the cpu at that time. We'll investigate.
A possible cause might be that the list freebsd-pkg-fallout archive mbox is very large (38 GB). We're discussing if we can remove that archive in total.
(In reply to Kurt Jaeger from comment #6) 38GB of mail archive is really surprising. But I wonder how it causes mail delivery delay on weekend. For example, is there huge traffic to freebsd-pkg-fallout ML on weekend ?
At that time it looked like several mailman qrunner processes stepped on each others toes trying to update the web archive. Approx. 1100 mails were in the queue, approx. 950 were for freebsd-pkg-fallout. My guess is that there are monthly package builder jobs which cause a wave of sent-out failure mails, which cause the box to IO-starve. It's still a guess and needs more analysis.
I note that the weekend (more specifically, Sunday afternoon in US time zones) is when the (older, pre-Mailman) WAIS-based archive does its weekly consolidation run.
Recently there is no mail delivery delay on weekend. So I close this bug report now.