Bug 295840 - Accumulation of zombie processes on FreeBSD 14.3 systems during sustained high-volume audit trail rotation on vendor machines
Summary: Accumulation of zombie processes on FreeBSD 14.3 systems during sustained hig...
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: 14.3-STABLE
Hardware: Any Any
: --- Affects Many People
Assignee: Dag-Erling Smørgrav
URL: https://reviews.freebsd.org/D57451
Keywords:
Depends on:
Blocks:
 
Reported: 2026-06-04 06:15 UTC by Anuj Telkhade
Modified: 2026-07-03 11:26 UTC (History)
2 users (show)

See Also:
des: mfc-stable15+
des: mfc-stable14+


Attachments
Added zombie collection script with output and data for analysis (23.47 KB, application/x-zip-compressed)
2026-06-04 06:15 UTC, Anuj Telkhade
no flags Details
Attached zombie tracker script and respective logs (21.60 KB, application/x-zip-compressed)
2026-06-09 06:15 UTC, Anuj Telkhade
no flags Details
zombie monitor logs (163 bytes, text/plain)
2026-06-23 16:10 UTC, Anuj Telkhade
no flags Details
Zombie detection script output (118.05 KB, application/x-zip-compressed)
2026-06-29 11:13 UTC, Anuj Telkhade
no flags Details
Zombie detection script for auditd - 30th June (106.05 KB, application/x-zip-compressed)
2026-06-30 14:54 UTC, Anuj Telkhade
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Anuj Telkhade 2026-06-04 06:15:32 UTC
Created attachment 271466 [details]
Added zombie collection script with output and data for analysis

We are observing an issue on FreeBSD 14.3 where auditd appears to accumulate zombie child processes during frequent audit trail rotation.
 
The issue is seen on systems with high audit event volume and relatively small audit trail file size configuration. Under this condition, audit trail files rotate frequently. Each rotation appears to invoke /etc/security/audit_warn with the closefile argument. Over time, multiple audit_warn processes remain in a defunct/zombie state.
 
Observed behavior:
- auditd is running normally.
- audit trail files under /var/audit rotate frequently.
- Increasing the audit trail file size reduces the rate at which zombies appear, but does not fully eliminate the issue.
- The zombie accumulation appears correlated with audit trail rotation frequency.
 
We are attaching logs and system outputs showing the observed zombie processes and audit trail behavior.

Let us know if any other details required from our end.
Comment 1 Kyle Evans freebsd_committer freebsd_triage 2026-06-05 13:30:49 UTC
des has a stack in review to address this, so sending it his way.
Comment 2 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-06-05 13:46:25 UTC
auditd appears to be failing to react to SIGCHLD, most likely because the variables used for communication between the signal handler and the main loop are not volatile and have been optimized out.  You can test this by manually sending a SIGCHLD to auditd and seeing if that clears up the zombies, or even running auditd in debug mode in a terminal (service auditd stop ; auditd -d) over a period of time and checking if it logs debug messages about SIGCHLD.

Exact behavior will depend on the compiler that auditd was compiled with.

The minimal fix is to mark these variables volatile: https://reviews.freebsd.org/D57451

The main loop should really use ppoll() in order to block signals when not sleeping, but the audit trigger device does not currently support poll() / select(), so that needs to be implemented first: https://reviews.freebsd.org/D57457 then https://reviews.freebsd.org/D57458

We can also reduce the amount of memory used by the trigger queue: https://reviews.freebsd.org/D57464
Comment 3 Kyle Evans freebsd_committer freebsd_triage 2026-06-05 16:40:10 UTC
(In reply to Anuj Telkhade from comment #0)

Can you confirm how the OS image is built? Is this just using release artifacts produced by re@, or are they hand rolled with custom flags or a different compiler?

If you could send along a copy of auditd and associated symbols (from /usr/lib/debug) if it's not produced by the project, that would be quite helpful.
Comment 4 Kyle Evans freebsd_committer freebsd_triage 2026-06-05 18:35:07 UTC
Thinking some more, read(2) at the beginning of the loop probably acts as enough of a compiler barrier that we'd need a clang bug for the missing volatiles to cause a problem in practice.

The other problem that des fixed- racing signal delivery and read(2)- seems like a more likely candidate.  It'd be useful if you could try the exercise he described, hit auditd with a SIGCHLD and see if it reaps all of the outstanding zombies.  If it doesn't, a ktrace(2) while you hit it with another SIGCHLD might be enlightening.
Comment 5 commit-hook freebsd_committer freebsd_triage 2026-06-08 22:47:49 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=5bd78cfc800339fd7f3945498052d67553af9e3c

commit 5bd78cfc800339fd7f3945498052d67553af9e3c
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2026-06-08 22:45:34 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2026-06-08 22:45:34 +0000

    auditd: Fix signal handling

    Rewrite the main loop to use ppoll() instead of just blocking on read,
    blocking the signals we care about when we aren't polling.

    I didn't bother replacing alarm() with setitimer(); the alarm code
    is dead anyway since there is no way for max_idletime to acquire a
    non-zero value.

    While here, avoid leaking the pid file and trigger descriptors to the
    log child.

    PR:             295840
    MFC after:      1 week
    Sponsored by:   Klara, Inc.
    Reviewed by:    kevans
    Differential Revision:  https://reviews.freebsd.org/D57451

 contrib/openbsm/bin/auditd/audit_warn.c  |  4 ++
 contrib/openbsm/bin/auditd/auditd.c      | 50 ++++++++++++++++----
 contrib/openbsm/bin/auditd/auditd.h      |  3 ++
 contrib/openbsm/bin/auditd/auditd_fbsd.c | 79 ++++++++++++++++----------------
 4 files changed, 86 insertions(+), 50 deletions(-)
Comment 6 Anuj Telkhade 2026-06-09 06:15:25 UTC
Created attachment 271644 [details]
Attached zombie tracker script and respective logs
Comment 7 Anuj Telkhade 2026-06-09 06:16:52 UTC
Hello Team,

Thank you for the patches provided (D57451, D57457, D57458, D57464). We have
attempted to apply them on a FreeBSD 14.3 host and are reporting our findings
along with observations of continued zombie behavior.

---

Environment:
  FreeBSD 14.3 (amd64)
  Host: 10.113.199.102
  auditd enabled, audit trails written under /var/audit

Current /etc/security/audit_control:
  dir:/var/audit
  dist:off
  minfree:5
  filesz:2M
  expire-after:10M
  flags:fw,fa,fm,fc,fd,pc,ap,ex
  naflags:fw,fa,fm,fc,fd,pc,ap,ex
  policy:cnt,argv

---

Patch Application Summary:

The host did not have a source tree. We cloned releng/14.3 from
https://git.freebsd.org/src.git into /usr/src.

Patches were applied in the following order:

1. D57451 (auditd_fbsd.c):
   All 4 hunks applied successfully.

2. D57457 (sys/security/audit/audit_trigger.c):
   All 6 hunks applied successfully (offset 1 line each).

3. D57458 (auditd_fbsd.c):
   *** The patch tool reported: "Reversed (or previously applied) patch
   detected! Assume -R? [y]"
   The operator answered 'y', causing D57458 to be applied in REVERSE.
   This means D57458 effectively UNDID the changes introduced by D57451
   in contrib/openbsm/bin/auditd/auditd_fbsd.c.

4. D57464 (sys/security/audit/audit_trigger.c):
   All 6 hunks applied successfully (offset 1 line each).

---

Build and Installation:

Only the userland auditd binary was rebuilt:
  cd /usr/src/usr.sbin/auditd
  make
  make install
  service auditd restart

The kernel sources (sys/security/audit/audit_trigger.c) were patched in the
source tree but NO kernel rebuild was performed (no 'make buildkernel' /
'make installkernel' / reboot). The kernel patches from D57457 and D57464
have therefore NOT taken effect in the running kernel.

---

Immediate Result After auditd Restart:

  root@:/usr/src/usr.sbin/auditd # ps auxw | grep Z
  root    83376   0.0  0.0    440   268  5  D+   15:34   0:00.00 grep Z

No zombie processes were observed immediately after the restart.

---

Overnight Monitoring Result:

After running an overnight monitoring script, zombie processes parented to
/usr/sbin/auditd were detected:

  Zombie PID : 29522
  Parent PID : 73560
  Parent Info: 73560  1  root  Ss  /usr/sbin/auditd

  PID   PPID  PGID   SID  TSID  THR  LOGIN    WCHAN   EMUL           COMM
  73560    1  73560  73560   0    1   zkckacp  auditd  FreeBSD ELF64  auditd

The zombie is directly parented to auditd (PID 73560), consistent with the
original behavior reported in this bug — auditd forks audit_warn on trail
rotation but does not reap the child in a timely manner.

---

Our Assessment:

We believe the continued zombie behavior is explained by the two issues noted
above:

1. D57458 conflicted with D57451 (both modify auditd_fbsd.c). When applied
   after D57451, the patch tool detected a reversal and the operator answered
   'y', causing D57451's changes to be undone. The net result on auditd_fbsd.c
   may be that neither fix is present.

   Could you clarify: are D57451 and D57458 intended to be applied together
   (with D57458 building on top of D57451), or are they mutually exclusive
   alternatives? If they conflict, which one represents the correct/final
   fix for auditd_fbsd.c?

2. The kernel patch (audit_trigger.c via D57457 / D57464) has not been
   compiled into the running kernel. Does this fix require a full kernel
   rebuild and reboot to take effect? If so, we will proceed with that step
   once the correct patch ordering for auditd_fbsd.c is confirmed.

We would like to confirm the intended application order for all four patches
before proceeding further. We are happy to re-apply and test once the correct
sequence is clarified.

Attached zombie tracker script and respective logs

Thank you!
Comment 8 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-06-09 08:03:00 UTC
The changes have been committed to main.  You should reset your tree and cherry-pick the committed changes:

$ git fetch
$ git reset --hard releng/14.3
$ git cherry-pick 0620c99d278b 77e894cb09af 5bd78cfc8003

Then rebuild and reinstall auditd and the kernel and reboot.

If you do not rebuild the kernel, auditd will sleep with signals blocked in read() instead of sleeping with signals unblocked in ppoll() and the problem will be worse than before.
Comment 9 commit-hook freebsd_committer freebsd_triage 2026-06-16 01:01:02 UTC
A commit in branch stable/15 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=03f078d90629f45ea19a48203cb5abeb7691dff6

commit 03f078d90629f45ea19a48203cb5abeb7691dff6
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2026-06-08 22:45:34 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2026-06-16 01:00:12 +0000

    auditd: Fix signal handling

    Rewrite the main loop to use ppoll() instead of just blocking on read,
    blocking the signals we care about when we aren't polling.

    I didn't bother replacing alarm() with setitimer(); the alarm code
    is dead anyway since there is no way for max_idletime to acquire a
    non-zero value.

    While here, avoid leaking the pid file and trigger descriptors to the
    log child.

    PR:             295840
    MFC after:      1 week
    Sponsored by:   Klara, Inc.
    Reviewed by:    kevans
    Differential Revision:  https://reviews.freebsd.org/D57451

    (cherry picked from commit 5bd78cfc800339fd7f3945498052d67553af9e3c)

 contrib/openbsm/bin/auditd/audit_warn.c  |  4 ++
 contrib/openbsm/bin/auditd/auditd.c      | 50 ++++++++++++++++----
 contrib/openbsm/bin/auditd/auditd.h      |  3 ++
 contrib/openbsm/bin/auditd/auditd_fbsd.c | 79 ++++++++++++++++----------------
 4 files changed, 86 insertions(+), 50 deletions(-)
Comment 10 commit-hook freebsd_committer freebsd_triage 2026-06-16 01:01:05 UTC
A commit in branch stable/14 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=872c046a4e97fb8356b1620186dff2d2649106bc

commit 872c046a4e97fb8356b1620186dff2d2649106bc
Author:     Dag-Erling Smørgrav <des@FreeBSD.org>
AuthorDate: 2026-06-08 22:45:34 +0000
Commit:     Dag-Erling Smørgrav <des@FreeBSD.org>
CommitDate: 2026-06-16 01:00:22 +0000

    auditd: Fix signal handling

    Rewrite the main loop to use ppoll() instead of just blocking on read,
    blocking the signals we care about when we aren't polling.

    I didn't bother replacing alarm() with setitimer(); the alarm code
    is dead anyway since there is no way for max_idletime to acquire a
    non-zero value.

    While here, avoid leaking the pid file and trigger descriptors to the
    log child.

    PR:             295840
    MFC after:      1 week
    Sponsored by:   Klara, Inc.
    Reviewed by:    kevans
    Differential Revision:  https://reviews.freebsd.org/D57451

    (cherry picked from commit 5bd78cfc800339fd7f3945498052d67553af9e3c)

 contrib/openbsm/bin/auditd/audit_warn.c  |  4 ++
 contrib/openbsm/bin/auditd/auditd.c      | 50 ++++++++++++++++----
 contrib/openbsm/bin/auditd/auditd.h      |  3 ++
 contrib/openbsm/bin/auditd/auditd_fbsd.c | 79 ++++++++++++++++----------------
 4 files changed, 86 insertions(+), 50 deletions(-)
Comment 11 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-06-16 01:03:30 UTC
Can you confirm that this cleared up your zombie infestation?
Comment 12 Anuj Telkhade 2026-06-17 09:40:07 UTC
Hello Team,

tried below patches after executing these steps - 

The changes have been committed to main.  You should reset your tree and cherry-pick the committed changes:

$ git fetch
$ git reset --hard releng/14.3
$ git cherry-pick 0620c99d278b 77e894cb09af 5bd78cfc8003

Then rebuild and reinstall auditd and the kernel and reboot.

If you do not rebuild the kernel, auditd will sleep with signals blocked in read() instead of sleeping with signals unblocked in ppoll() and the problem will be worse than before.


make buildworld o/p (truncated)

 

cons30r-mv cons30r-v cons43 cons43-m cons43-w cons43l1 cons43l1-m  cons43l1-w cons43l2 cons43l2-m cons43l7 cons43l7-m cons43r cons43r-m  cons43r-mv cons43r-v cons50 cons50-m cons50-w cons50l1 cons50l1-m  cons50l1-w cons50l2 cons50l2-m cons50l7 cons50l7-m cons50r cons50r-m  cons50r-mv cons50r-v cons60 cons60-m cons60-w cons60l1 cons60l1-m  cons60l1-w cons60l2 cons60l2-m cons60l7 cons60l7-m cons60r cons60r-m  cons60r-mv cons60r-v ecma[+]italics vt100 xterm xterm-256color xterm-basic  xterm-clear xterm-color xterm-new xterm-r6 xterm-r6-clear;  do  echo; awk "/${tcname}[:|]/{ f = 1; print; next } /[^\t]/{ f = 0 } f" /usr/src/share/termcap/termcap;  done) > termcap.small
===> etc/sendmail (all)
rm -f freebsd.cf
m4 -D_CF_DIR_=/usr/src/contrib/sendmail/cf/ -D_NO_MAKEINFO_    /usr/src/contrib/sendmail/cf/m4/cf.m4 /usr/src/etc/sendmail/freebsd.mc > freebsd.cf
chmod 444 freebsd.cf
rm -f freebsd.submit.cf
m4 -D_CF_DIR_=/usr/src/contrib/sendmail/cf/ -D_NO_MAKEINFO_    /usr/src/contrib/sendmail/cf/m4/cf.m4 /usr/src/etc/sendmail/freebsd.submit.mc > freebsd.submit.cf
chmod 444 freebsd.submit.cf
     6088.32 real      2436.02 user       259.10 sys

 

--------------------------------------------------------------
>>> World build completed on Fri Jun 12 03:00:31 IST 2026
>>> World built in 36977 seconds, ncpu: 2
--------------------------------------------------------------

 

make installworld o/p (truncated)

 

===> libheimipcs (install)
install  -o root -g wheel -m 444  -C libprivateheimipcs.a /usr/lib32/
install  -s -o root -g wheel -m 444  -C -S  libprivateheimipcs.so.11 /usr/lib32/
install  -o root -g wheel -m 444  -C  libprivateheimipcs.so.11.debug /usr/lib/debug/usr/lib32/
install -l rs -o root -g wheel -m 755 -S   libprivateheimipcs.so.11 /usr/lib32/libprivateheimipcs.so
===> libkafs5 (install)
install  -o root -g wheel -m 444  -C libkafs5.a /usr/lib32/
install  -s -o root -g wheel -m 444  -C -S  libkafs5.so.11 /usr/lib32/
install  -o root -g wheel -m 444  -C  libkafs5.so.11.debug /usr/lib/debug/usr/lib32/
install -l rs -o root -g wheel -m 755 -S   libkafs5.so.11 /usr/lib32/libkafs5.so
===> libgssapi_krb5 (install)
install  -o root -g wheel -m 444  -C libgssapi_krb5.a /usr/lib32/
install  -s -o root -g wheel -m 444  -C -S  libgssapi_krb5.so.10 /usr/lib32/
install  -o root -g wheel -m 444  -C  libgssapi_krb5.so.10.debug /usr/lib/debug/usr/lib32/
install -l rs -o root -g wheel -m 755 -S   libgssapi_krb5.so.10 /usr/lib32/libgssapi_krb5.so
===> libgssapi_ntlm (install)
install  -o root -g wheel -m 444  -C libgssapi_ntlm.a /usr/lib32/
install  -s -o root -g wheel -m 444  -C -S  libgssapi_ntlm.so.10 /usr/lib32/
install  -o root -g wheel -m 444  -C  libgssapi_ntlm.so.10.debug /usr/lib/debug/usr/lib32/
install -l rs -o root -g wheel -m 755 -S   libgssapi_ntlm.so.10 /usr/lib32/libgssapi_ntlm.so
===> libgssapi_spnego (install)
install  -o root -g wheel -m 444  -C libgssapi_spnego.a /usr/lib32/
install  -s -o root -g wheel -m 444  -C -S  libgssapi_spnego.so.10 /usr/lib32/
install  -o root -g wheel -m 444  -C  libgssapi_spnego.so.10.debug /usr/lib/debug/usr/lib32/
install -l rs -o root -g wheel -m 755 -S   libgssapi_spnego.so.10 /usr/lib32/libgssapi_spnego.so
       15.03 real         4.45 user         2.37 sys
--------------------------------------------------------------
>>> Installing everything completed on Tue Jun 16 15:47:16 IST 2026
--------------------------------------------------------------
      176.06 real        68.97 user        50.28 sys


Still we are facing the same issue, zombie is still reported sharing logs snippets - 

===== Wed Jun 17 14:47:06 IST 2026 =====
New zombie process(es) detected
 
Zombie PID : 35577
Parent PID : 660
Parent Info: 660    1 root   Ss /usr/sbin/auditd
 
Cause Analysis: Parent process is '/usr/sbin/auditd'
 
--- procstat parent (660) ---
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM        
  660     1   660   660     0   1 root     select    FreeBSD ELF64 auditd      
 
===== QUALYS PROCESS SNAPSHOT =====
root    48460   0.0  0.1 14020 2608  8  SC+  14:44      0:00.00 tail -f qualys_zombie_tracker.log
 
2:47PM  up 1 day, 32 mins, 9 users, load averages: 1.29, 1.37, 1.31
==========================================


Please help in checking this further.
Comment 13 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-06-17 10:36:47 UTC
We're going to need a trace.  Start by stopping the normal auditd service:

    service auditd stop

Then restart it in debug mode under ktrace:

    ktrace -t acnstu -f /tmp/auditd.ktr -p /usr/sbin/auditd -d >/tmp/auditd.log 2>&1

It will now be running in the foreground.  Once you are certain that additional zombies have been created, kill auditd (press Ctrl-C on the console) and restart normal service:

    service auditd start

then zip up auditd.ktr and auditd.log and send them to Kyle and me by email.  If possible, include the PIDs of the zombies that were created while auditd was being traced.
Comment 14 Kyle Evans freebsd_committer freebsd_triage 2026-06-17 14:39:55 UTC
(In reply to Anuj Telkhade from comment #12)

You only reported doing buildworld and installworld, can you confirm whether you also did a buildkernel + installkernel and just omitted the output, please?  world is only the userland bits, there.
Comment 15 Anuj Telkhade 2026-06-18 05:25:01 UTC
(In reply to Kyle Evans from comment #14)

kernel build is done prior to installworld.
Comment 16 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-06-18 06:05:38 UTC
Yes, but installworld does not install the kernel, you must run `make installkernel` as well.  Did you?
Comment 17 Anuj Telkhade 2026-06-18 06:10:03 UTC
(In reply to Dag-Erling Smørgrav from comment #13)

Once auditd is stopped zombies are not reported, isn't it obvious as zombies are coming from audit only then what's the point in stopping auditd and tracing for the zombie.

Can you elaborate this?
Comment 18 Anuj Telkhade 2026-06-18 06:10:46 UTC
(In reply to Dag-Erling Smørgrav from comment #16)

Yes we have also done it.
Comment 19 Anuj Telkhade 2026-06-22 04:37:04 UTC
Hi Team,
Can we schedule a call so that we can debug this issue in live session?
Comment 20 Anuj Telkhade 2026-06-22 11:10:21 UTC
(In reply to Anuj Telkhade from comment #17)
We have tried the below steps

    service auditd stop

Then restart it in debug mode under ktrace:

    ktrace -t acnstu -f /tmp/auditd.ktr -p /usr/sbin/auditd -d >/tmp/auditd.log 2>&1

It will now be running in the foreground.  Once you are certain that additional zombies have been created, kill auditd (press Ctrl-C on the console) and restart normal service:

    service auditd start

But after performing this, auditd not started and we weren't able to reproduce.
Comment 21 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-06-23 09:17:59 UTC
Sorry, there was a typo in my instructions, the correct command is:

    ktrace -t acnstu -f /tmp/auditd.ktr /usr/sbin/auditd -d >/tmp/auditd.log 2>&1
Comment 22 Anuj Telkhade 2026-06-23 16:09:05 UTC
Followed steps :
 
# service auditd stop
#
# ktrace -t acnstu -f /tmp/auditd.ktr /usr/sbin/auditd -d >/tmp/auditd.log 2>&1
#
# service audtid status
audtid does not exist in /etc/rc.d or the local startup
directories (/usr/local/etc/rc.d), or is not executable
#

No any zombie is logged 

As per the FreeBSD team's recommendation, once additional zombie processes are observed, we should terminate `auditd` (using Ctrl+C on the console) and restart the normal service.
 
However, we were unable to perform this step because no zombie processes were detected during our testing.
 
But we didn't able to do this step as no zombi detcted

Added zombie_monitor.log in description
Comment 23 Anuj Telkhade 2026-06-23 16:10:07 UTC
Created attachment 272079 [details]
zombie monitor logs
Comment 24 Anuj Telkhade 2026-06-24 10:54:41 UTC
Hi Team
 
We followed the requested steps to collect an auditd trace.
Steps executed:
# service auditd stop
# ktrace -t acnstu -f /tmp/auditd.ktr /usr/sbin/auditd -d >/tmp/auditd.log 2>&1

However, the command did not start auditd under ktrace. Instead, it failed immediately with the following error:
ktrace: unknown facility in acnstu
usage: ktrace [-aCcdi] [-f trfile] [-g pgrp | -p pid] [-t trstr]
       ktrace [-adi] [-f trfile] [-t trstr] command

As a result, we were unable to collect the requested trace.
 
Evidence:
No auditd process was running after executing the command.

# pgrep -fl auditd
<no output>
# ps -ax | grep auditd
94294  3  S+   0:00.00 grep auditd

No trace file was generated.

# ls -lh /tmp/auditd.ktr
ls: /tmp/auditd.ktr: No such file or directory
No active audit trail (current) was created.
# ls -lh /var/audit/current
ls: /var/audit/current: No such file or directory
Audit control operations were unsuccessful.
# audit -s
Error sending trigger: Operation not supported by device
# audit -n
Error sending trigger: Operation not supported by device

Since the ktrace command failed, auditd was never started under tracing, and therefore we could not continue with the requested trace collection or capture zombie creation during the traced execution.
 
Could you please confirm whether the ktrace command is correct for the FreeBSD version being used, or provide the appropriate tracing command if different?
 
Separately, during our previous testing, no additional zombie processes were observed while attempting to reproduce the issue.(Note : when audit is stop no zombie process observed , when audit is active zombie process observed)
Comment 25 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-06-24 11:12:18 UTC
I'm sorry, I assumed you were running FreeBSD 15.  Use this instead:

    ktrace -t cnstu -f /tmp/auditd.ktr /usr/sbin/auditd -d >/tmp/auditd.log 2>&1
Comment 26 Anuj Telkhade 2026-06-26 14:24:47 UTC
Hi Team,
We are not able to detect zombie by shared command as well

Steps we performed this time

service auditd stop

Then restart it in debug mode under ktrace:

ktrace -t cnstu -f /tmp/auditd.ktr /usr/sbin/auditd -d >/tmp/auditd.log 2>&1

After additional zombies detected, did CTRL+C and started the normal auditd service.

service auditd start

I have added trace logs and output from zombie detection script in attachments
Comment 27 Anuj Telkhade 2026-06-29 11:13:10 UTC
Created attachment 272257 [details]
Zombie detection script output
Comment 28 Anuj Telkhade 2026-06-29 11:17:49 UTC
Hi Team,
Logs were not uploaded properly last time, I have uploaded it again(we are able to generate issue this time)

Attachment name - Zombie detection script output
Comment 29 Dag-Erling Smørgrav freebsd_committer freebsd_triage 2026-06-29 11:46:45 UTC
The zombie in the log is not a child of auditd, it is a grandchild or great-grandchild.  We don't know for sure because your zombie detector did not include any information about the zombie except its PID and PPID.  It is either the logger process forked by audit_warn, or the Casper helper forked by logger.

We see auditd fork 41070 here:

 27811 auditd   2150.879612638 CALL  fork
 27811 auditd   2150.879729565 RET   fork 41070/0xa06e

then 41071:

 27811 auditd   2150.880494426 CALL  fork
 27811 auditd   2150.880585291 RET   fork 41071/0xa06f

Then we see the SIGCHLD for 41070 arrive:

 27811 auditd   2150.880678140 CALL  ppoll(0x51cf68e87c8,0x1,0,0x514d593ab1c)
 27811 auditd   2150.884087081 RET   ppoll -1 errno 4 Interrupted system call
 27811 auditd   2150.884100262 PSIG  SIGCHLD caught handler=0x514d5938240 mask=0x86001 code=CLD_EXITED
 27811 auditd   2150.884110492 CALL  sigreturn(0x51cf68e8200)
 27811 auditd   2150.884115487 RET   sigreturn JUSTRETURN

auditd immediately collects it:

 27811 auditd   2150.884122125 CALL  wait4(0xffffffff,0x51cf68e877c,0x1<WNOHANG>,0)
 27811 auditd   2150.884127117 RET   wait4 41070/0xa06e

it checks again but 41071 is still running, so wait4() returns 0:

 27811 auditd   2150.884130087 CALL  wait4(0xffffffff,0x51cf68e877c,0x1<WNOHANG>,0)
 27811 auditd   2150.884132708 RET   wait4 0

it returns to the main loop and almost immediately gets another SIGCHLD, this time for 41071:

 27811 auditd   2150.884135596 CALL  ppoll(0x51cf68e87c8,0x1,0,0x514d593ab1c)
 27811 auditd   2150.885304190 RET   ppoll -1 errno 4 Interrupted system call
 27811 auditd   2150.885314731 PSIG  SIGCHLD caught handler=0x514d5938240 mask=0
x86001 code=CLD_EXITED
 27811 auditd   2150.885320736 CALL  sigreturn(0x51cf68e8200)
 27811 auditd   2150.885324444 RET   sigreturn JUSTRETURN

so it collects it:

 27811 auditd   2150.885328568 CALL  wait4(0xffffffff,0x51cf68e877c,0x1<WNOHANG>,0)
 27811 auditd   2150.885332931 RET   wait4 41071/0xa06f

and checks again but now wait4() says auditd has no children at all:

 27811 auditd   2150.885336035 CALL  wait4(0xffffffff,0x51cf68e877c,0x1<WNOHANG>,0)
 27811 auditd   2150.885338542 RET   wait4 -1 errno 10 No child processes

which is odd because we know that 41071 forked 41074 and somehow failed to collect it, so it has now been reparented to the auditd process, as shown by your zombie detector.

There is a gap in the PID sequence.  We know that 41070 and 41071 are audit_warn processes.  Each of them will have forked a logger process, and each logger process will have forked a casper helper.  We know that 40174 is a descendant of 41071, but we don't know if it's the logger child or the casper helper, and we don't know what 41072 and 40173 were, though we can surmise that one of them was 41070's logger child.  41074 is more likely to be 41071's logger than 41071's logger's casper helper, but we can't know for sure.

So unfortunately, this looks like a kernel bug, which will take time to investigate and fix.

In the meantime, replacing the contents of /etc/security/audit_warn with the following may help:

#!/bin/sh
exec /usr/bin/logger -p security.warning "audit warning: $@"

If the zombie problem still occurs with this change, we know that it is logger's casper helper, not logger itself, which is failing to be collected.

We should also look into adding support for logging directly to syslog.
Comment 30 Anuj Telkhade 2026-06-29 13:33:01 UTC
Hi Dag-Erling Smørgrav

We will check on suggested points, meanwhile can you tell me the point of contact who can check help in checking this kernal bug?
Comment 31 Anuj Telkhade 2026-06-30 14:53:37 UTC
Hi Team,

As suggested by the FreeBSD team, applied the recommended changes. However, zombie processes are still being observed.
 
# cat /etc/security/audit_warn
#!/bin/sh
 
exec /usr/bin/logger -p security.warning "audit warning: $@"
 
#logger -p security.warning "audit warning: $@"
#

# tail -f /tmp/qualys_zombie_tracker.log
==========================================
Zombie monitoring started: Tue Jun 30 17:09:50 IST 2026
Hostname:
==========================================
Taking backup of existing zombies...
Existing zombie count:        0
Backup file: /tmp/zombie_existing_backup.txt
 
===== Tue Jun 30 17:19:11 IST 2026 =====
New zombie process(es) detected
 
Zombie PID : 94988
Parent PID : 46829
Parent Info: 46829    1 root   Ss /usr/sbin/auditd
 
Cause Analysis: Parent process is '/usr/sbin/auditd'
 
--- procstat parent (46829) ---
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
46829     1 46829 46829     0   1 zkckacp  select    FreeBSD ELF64 auditd
 
Zombie PID : 94996
Parent PID : 1
Parent Info:   1    0 root  SLs /sbin/init
 
Cause Analysis: Parent process is '/sbin/init'
 
--- procstat parent (1) ---
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
    1     0     1     1     0   1 root     wait      FreeBSD ELF64 init
 
===== QUALYS PROCESS SNAPSHOT =====
root    90900   0.0  0.1 14020 2580  4  TC   17:05       0:00.00 tail -f /tmp/qualys_zombie_tracker.log
root    91246   0.0  0.1 14020 2524  6  SC+  17:10       0:00.00 tail -f /tmp/qualys_zombie_tracker.log
 
5:19PM  up 14 days,  3:04, 7 users, load averages: 0.31, 0.29, 0.25
==========================================
 
===== Tue Jun 30 17:23:48 IST 2026 =====
New zombie process(es) detected
 
Zombie PID : 96954
Parent PID : 1
Parent Info:   1    0 root  SLs /sbin/init
 
Cause Analysis: Parent process is '/sbin/init'
 
--- procstat parent (1) ---
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
    1     0     1     1     0   1 root     wait      FreeBSD ELF64 init
 
===== QUALYS PROCESS SNAPSHOT =====
root    90900   0.0  0.1 14020 2580  4  TC   17:05       0:00.00 tail -f /tmp/qualys_zombie_tracker.log
root    91246   0.0  0.1 14020 2524  6  SC+  17:10       0:00.00 tail -f /tmp/qualys_zombie_tracker.log
 
5:23PM  up 14 days,  3:08, 7 users, load averages: 0.33, 0.27, 0.25
==========================================


Adding logs for more information
Comment 32 Anuj Telkhade 2026-06-30 14:54:25 UTC
Created attachment 272298 [details]
Zombie detection script for auditd - 30th June
Comment 33 Anuj Telkhade 2026-06-30 14:55:12 UTC
Please check attachment named "Zombie detection script for auditd - 30th June"
Comment 34 Anuj Telkhade 2026-07-03 11:26:39 UTC
Hi Team,
Please help in providing update on this ticket.