Bug 297177 - mail/exim 4.99 has subprocess issues with user forward compared to 4.98
Summary: mail/exim 4.99 has subprocess issues with user forward compared to 4.98
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: Any Any
: --- Affects Only Me
Assignee: Dima Panov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2026-07-30 20:42 UTC by Pete French
Modified: 2026-08-05 08:38 UTC (History)
1 user (show)

See Also:
bugzilla: maintainer-feedback? (fluffy)


Attachments
My .forward file which is causing the problem (3.79 KB, text/plain)
2026-07-31 13:04 UTC, Pete French
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Pete French 2026-07-30 20:42:09 UTC
I am runnign exim, with a local delivery and a .forward file that forwards on some of those. Under heavy load with 4.99 I am seeing errors like this:

2026-07-30 09:37:48 1wpMuT-00000000KWB-0fCM internal problem in userforward router (recipient is liveerrors@reboxetine.ingresso.co.uk): failure to transfer data from subprocess: status=0100 readerror='No such file or directory'

This does not happen under 4.98 (which I have rolled back to).

Note that these are transient - they only happen during the flood of messages.The messages get placed into the queue correctly, and then are delivered later on.

At a guess, there is some problem with it trying to deliver too many at once, but processing them sequentially later works fine.
Comment 1 Oleksandr Kryvulia 2026-07-31 05:30:48 UTC
I think better to report this upstream.
Does exim process segfaults? Maybe you can provide a core dump?
Comment 2 Pete French 2026-07-31 09:50:00 UTC
Unfortunately, there is no segfault, just that error. It looks as if the subprocess creates a file, which then vanishes from underneath it possibly ? I am running on top of ZFS here, but I havent changed the operating system at all, only upgraded exim, and I can confirm that downgrading it makes the problem go away.

The change from 4.98 to 4.99 did modify some the local FreeBSD patched the 'files' directory, so I wondered if it could be something to do with them. But I cant see anything which looks like it might be related.

[ Its hard to test, as I need a situation. where I get a flood of warning emails form our production system - and I try and avoid floods of warnings in production! ]
Comment 3 Oleksandr Kryvulia 2026-07-31 12:13:54 UTC
(In reply to Pete French from comment #2)
There were some changes in redirect router between 4.98.2 and 4.99.* I do not have .forward files involved in my workload, but if you share your exim's router configuration I try to reproduce it.
Comment 4 Pete French 2026-07-31 13:03:52 UTC
Sure, here is the forward. We do a lot of filtering here to drop stuff into separate mailboxes, based on headers (this is where our euro emails go). But the problems only seem to happen with the rule at the end which sends a copy to the 'monitor; address
Comment 5 Pete French 2026-07-31 13:04:41 UTC
Created attachment 273352 [details]
My .forward file which is causing the problem
Comment 6 Pete French 2026-07-31 13:09:03 UTC
Thi bit appears to be the bit which has the temporary errors when we get a lot of them.

# Make anything with 'Cron' in the subject go to a separate mailbox
# We also copy it to the monitor copy alias
if (    
        $header_subject: contains "Cron "
) then  
        save $home/mail/.INBOX.Cron
        deliver monitor-copy@core.ingresso.co.uk
        finish
endif

I havent checked to see if the file save works but the deliver fails, but its a Burt of cron errors whcih causes the issue - and these are not huge volumes actually having calcukated - I have 24 machines, chekcing database lag once per minute, and sending an email if so. i.e we are looking at 24 emails a minute here when it shows up.
Comment 7 Pete French 2026-07-31 13:32:54 UTC
Oh! I just noticed! Along with each of these errors is one on the preceding line, saying it cannot set the user!

"2026-07-30 09:37:48 1wpMuT-00000000KWB-0fCM unable to set gid=1003 or uid=1003 (euid=26): userforward router (recipient is liveerrors@reboxetine.ingresso.co.uk)"

is the line before

"2026-07-30 09:37:48 1wpMuT-00000000KWB-0fCM internal problem in userforward router (recipient is liveerrors@reboxetine.ingresso.co.uk): failure to transfer data from subprocess: status=0100 readerror='No such file or directory'"

So actually, the inability to set the user ID may be the cause ?
Comment 8 Pete French 2026-07-31 13:53:42 UTC
I got Claude to compare the two exim versions and look at the error messages, and it found this (trimmed, as it can be very verbose)


  New 4.99 feature, ChangeLog JH/01 — "Use fewer forks & execs for sending many messages to a single host … we can loop there." It adds an in-process message chaining loop to deliver_message():

  - deliver.c:6739 — routing starts at a CONTINUED_ID: label ("Liable to be called as root").
  - deliver.c:8469 — after deliveries finish, it permanently drops root: exim_setugid(exim_uid, exim_gid, …, "post-delivery tidying"). This is irreversible — root can never be regained in that
  process.
  - deliver.c:8984–8997 — if the SMTP transport handed back a follow-on message id (continue_next_id, set back at deliver.c:3744 before the drop), it does goto CONTINUED_ID — re-routing the next
  message with euid=26.

  So: message A (a remote delivery to a smarthost) finishes → process drops to exim → loops to message B → routes B from scratch as exim → B has a local liveerrors@ recipient that needs its .forward run as uid 1003 → setuid fails.

  In 4.98.2 this loop does not exist (grep continue_next_id src/deliver.c → 0 hits; 4.99.5 → 10). Previously every message got its own fresh root delivery process. I also confirmed the entire process-spawn / re-exec / deliver_drop_privilege privilege model is byte-for-byte identical between the versions — this loop is the sole differentiator.

  Why only under heavy load

  continue_next_id is only populated when the SMTP transport finds more messages queued for the same host in the wait-<transport> hints DB and keeps the connection open (needs remote_max_parallel, default 4, to be reached). That backlog only exists when lots of mail is queued for one destination — a queue-run flood. No backlog → the goto at 8984 is skipped → the bug can't fire. That's your "only under heavy load."

This is a genuine Exim 4.99 privilege regression; worth filing at bugs.exim.org and tracking a fixed 4.99.x release rather than carrying a local patch long-term.
Comment 9 Pete French 2026-08-02 20:20:31 UTC
Files in upstream as https://code.exim.org/exim/exim/issues/3235
Comment 10 Oleksandr Kryvulia 2026-08-03 07:04:03 UTC
I was able to reproduce it with a simple test script:

#!/bin/sh

COUNT=100
MAILTO="o.kryvulia@exim-test.flex-it.com.ua)"

i=1
while [ ${i} -le ${COUNT} ]
do
        echo TEST | mail -s "Cron test #"${i} ${MAILTO}
        i=`expr ${i} + 1`
done

It generates 100 mails in a fraction of second:
# /usr/bin/time ./send-mail.sh
        0.66 real         0.21 user         0.52 sys

And there are many errors in exim log after attempt to deliver mails from queue (exiqgrep -f . -i | xargs exim -M):
2026-08-03 09:43:57 1wqmOW-000000009rv-3lkH unable to set gid=0 or uid=1001 (euid=26): userforward router (recipient is o.kryvulia@exim-test.flex-it.com.ua)
2026-08-03 09:43:57 1wqmOW-000000009rv-3lkH internal problem in userforward router (recipient is o.kryvulia@exim-test.flex-it.com.ua): failure to transfer data from subprocess: status=0100 readerror='No such file or directory'

However despite this errors all 100 mails were delivered to the remote host and mail/.INBOX.Cron.
Comment 11 Pete French 2026-08-03 08:22:47 UTC
"However despite this errors all 100 mails were delivered to the remote host and mail/.INBOX.Cron."

Yup - If you check one of them, you probably will find it got queued and then retried later on. Thats what I observed. Indeed my first symptoms of the bug were cron emails turning up much later than they were sent (the errors in them are timestamped).

Good to have a way to reproduce this though, thankyou!
Comment 12 Oleksandr Kryvulia 2026-08-03 08:42:20 UTC
(In reply to Pete French from comment #11)
Yes, I have 65 messages delivered immediately and 35 stuck in queue.
Comment 13 Pete French 2026-08-03 09:03:31 UTC
(In reply to Oleksandr Kryvulia from comment #12)
Over on the exim bug tracker, I got asked this:

"Can debug information be gathered, from a run demonstrating the problem?"

but any not sure what other debug info we can produce, aside from the log messages - do you have any idea what else we could get ? I should be able to reproduce the issue too using your script. [ will try in a moment ]
Comment 14 Pete French 2026-08-03 09:17:42 UTC
Yup, I can reproduce it on a local test machine using your little loop, excellent!
Comment 15 Pete French 2026-08-05 08:38:01 UTC
This is fixed in upstream, in version 4.100