Bug 235657 - /usr/libexec/atrun race causes missed jobs
Summary: /usr/libexec/atrun race causes missed jobs
Status: Closed Overcome By Events
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.2-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs (Nobody)
Keywords: patch
Depends on:
Reported: 2019-02-11 05:46 UTC by karl
Modified: 2022-09-18 13:15 UTC (History)
5 users (show)

See Also:

Diff against /usr/src/libexec/atrun directory (1.84 KB, patch)
2019-02-11 05:46 UTC, karl
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description karl 2019-02-11 05:46:11 UTC
Created attachment 201915 [details]
Diff against /usr/src/libexec/atrun directory

I have no idea why this hasn't bit people before, or isn't biting people now.... but it is biting me.

/usr/libexec/atrun is the "batch" job executor out of the cron and by default runs every 5 minutes.

The code has an unlink call in it that attempts to remove old jobs from the queue but unfortunately the queue code can select a job to run, call fork() to start it, post-fork() the child can give up the CPU before it opens the file containing the job and thus the queue code (which is in the parent) can execute the unlink before the child process gets the file open.  If this happens you get a "file not found" error in the cron log and the job doesn't run.

The attached patch fixes the potential race by moving the unlink into the child; it may not be the most-elegant, but it works.  Unfortunately due to the code's structure (it performs multiple tests on the file to be run for security reasons) there are multiple error exits and, in the event of any of those, you must unlink the file as well or it will try to run repeatedly -- yet you can't unlink it immediately after it is opened because some of the tests require it still be on the filesystem.
Comment 1 karl 2021-05-29 16:01:25 UTC
This is still present in 12.2-STABLE as of today... updated a multi-core system and had it repeatedly drop submitted "batch" jobs due to it.
Comment 2 karl 2021-05-30 14:31:40 UTC
This is still "active" as of 12.2 as a problem; updated the version impacted to reflect that fact.
Comment 3 Dave Baukus 2021-09-17 18:20:28 UTC
In my usage this is not an intermittent irritation on Stable-13, the "race" is hit every time. It would be nice to see this fix pulled into the current releases. It has been broken since Stable-12.
Comment 4 karl 2021-09-17 19:17:00 UTC
(In reply to Dave Baukus from comment #3)
It has been "nearly every time" for me since I went to 12.2 on one specific box, but is intermittent on others.  I suspect it has to do with how the scheduler interacts with the various cores in specific configurations.

Nonetheless I agree with you -- this needs fixed since when it happens the job disappears without a trace.
Comment 5 bitbucket63-it 2022-09-17 16:14:20 UTC
In case anyone is looking at this now...

I had a problem with at jobs disappearing and traced it down to a conflict with my old /etc/crontab and the new /etc/cron.d/at way of doing things. Both files had the schedule for atrun and I suspect cron was running it twice which was causing my problem.

When I removed the old atrun line from /etc/crontab the at service began returning expected results.

This problem began for me when I upgraded my system to 12.x-RELEASE from 10.x-RELEASE
Comment 6 karl 2022-09-18 13:15:43 UTC
At some point in more-recent builds this has disappeared; with 12.3 I cannot duplicate it without the patch, thus closed.

My system did not have the "extra" (old) entry as bitbucket noted.