Bug 249179

Summary: Several audit framework/tool issues
Product: Base System Reporter: Eirik Oeverby <ltning-freebsd>
Component: miscAssignee: Christian S.J. Peron <csjp>
Status: New ---    
Severity: Affects Some People CC: ben, cem, csjp, emaste, kevans
Priority: ---    
Version: 12.1-STABLE   
Hardware: Any   
OS: Any   

Description Eirik Oeverby 2020-09-07 20:52:06 UTC
We use the audit framework extensively across our fleet. The configuration is relatively simple, so it has kept working reasonably well for >10 years, but it is now falling apart at the seams. Even the briefest attempts at working with it will reveal a number of problems; I'm listing some here but providing examples is difficult while reproducing should be very simple:

- all events (at least those in jails) seem to have 'naflags' as opposed to 'flags' applied, even if an audit identifier is available (user names/IDs are shown and correct in the audit log)
- all execve() calls are logged as failures (return,failure: Unknown error: 201,4294967295), regardless of their actual result
- auditreduce cannot select events by class (specifying -c XX returns nothing)
- auditreduce cannot invert its search (-v causes all events to be shown, including those that match)

See also issue 248025 - it appears as if detaching from or attaching to the audit pipe (bsmtrace) migth be causing this.

My audit_control, which is the only part of the configuration that has been modified:

dir:/var/audit
dist:off
flags:lo,aa,ad,ex
minfree:5
naflags:lo,aa,ad,ex
policy:cnt,argv
filesz:0
host:foo.tld
expire-after:90d
Comment 1 Mark Linimon freebsd_committer freebsd_triage 2020-09-07 21:22:50 UTC
^Triage: Cc: the most recent committers in this area for advice.
Comment 2 Kyle Evans freebsd_committer 2020-09-07 22:43:57 UTC
Tagging in csjp as well, as a domain expert-
Comment 3 Eirik Oeverby 2020-10-23 10:21:54 UTC
Is this still not fixed for 12.2? May I point out that the audit framework seems to be more-or-less non-functional at the moment? Successful execs aren't picked up by e.g. bsmtrace, presumably because of this.

Example output from praudit:

header_ex,132,11,execve(2),0,10.1.10.10,Fri Oct 23 10:17:36 2020, + 944 msec
exec arg,./t.sh
path,/tmp/./t.sh
attribute,755,root,0,968485463,32027,2160726950
subject,ltning,root,0,root,0,83815,83621,61751,192.168.127.2
return,failure: Unknown error: 201,4294967295
trailer,132


Content of /tmp/t.sh:
--
#!/bin/sh
echo "illegal stuff"
--
Comment 4 Kyle Evans freebsd_committer 2020-10-23 21:01:38 UTC
I've made an attempt at addressing this: https://reviews.freebsd.org/D26922
Comment 5 Kyle Evans freebsd_committer 2020-10-24 00:21:01 UTC
> - all events (at least those in jails) seem to have 'naflags' as opposed to 'flags' applied, even if an audit identifier is available (user names/IDs are shown and correct in the audit log)

You might try and reproduce what you're doing and take a look at a sample audit record for this; it seems to work as expected here:

header,147,11,execve(2),0,Sat Oct 24 00:14:50 2020, + 500 msec
exec arg,/root/grep/a.out
path,/root/grep/a.out
attribute,755,root,0,897360606,160136,3214960080
zone,foo0
subject,kevans,root,0,root,0,1937,1921,53793,10.0.2.2
return,success,0
trailer,147

Note that flags specifically gets applied if the second field ("kevans") in the subject line is not -1. e.g.:

header,124,11,execve(2),0,Sat Oct 24 00:19:41 2020, + 978 msec
exec arg,less
path,/usr/bin/less
attribute,555,root,0,897360606,1125807,2160857538
subject,-1,root,0,root,0,1995,0,0,0.0.0.0
return,success,0
trailer,124

This is considered not-attributable despite being directly executed by me as root.

root@viper:~/grep# id -A
auid=-1
mask.success=0x00000000
mask.failure=0x00000000
asid=0
termid.port=0x00000000
termid.machine=0x00000000
Comment 6 Christian S.J. Peron freebsd_committer 2020-10-24 00:22:43 UTC
Hi Eirik,

Are you same consistent results between the audit trail and the audit pipe for the flags issue?
Comment 7 Christian S.J. Peron freebsd_committer 2020-10-24 01:20:58 UTC
That last comment should have read: "Are you seeing consistent results between the audit trail and the audit pipe for the flags issue?"
Comment 8 commit-hook freebsd_committer 2020-10-24 14:40:15 UTC
A commit references this bug:

Author: kevans
Date: Sat Oct 24 14:39:18 UTC 2020
New revision: 367002
URL: https://svnweb.freebsd.org/changeset/base/367002

Log:
  audit: correct reporting of *execve(2) success

  r326145 corrected do_execve() to return EJUSTRETURN upon success so that
  important registers are not clobbered. This had the side effect of tapping
  out 'failures' for all *execve(2) audit records, which is less than useful
  for auditing purposes.

  Audit exec returns earlier, where we can know for sure that EJUSTRETURN
  translates to success. Note that this unsets TDP_AUDITREC as we commit the
  audit record, so the usual audit in the syscall return path will do nothing.

  PR:		249179
  Reported by:	Eirik Oeverby <ltning-freebsd anduin net>
  Reviewed by:	csjp, kib
  MFC after:	1 week
  Sponsored by:	Klara, Inc.
  Differential Revision:	https://reviews.freebsd.org/D26922

Changes:
  head/sys/compat/freebsd32/freebsd32_misc.c
  head/sys/kern/kern_exec.c
  head/sys/kern/subr_syscall.c
Comment 9 Eirik Oeverby 2020-10-24 22:13:29 UTC
(In reply to Christian S.J. Peron from comment #7)

If you mean
  auditreduce -c ex /dev/auditpipe | praudit
vs
  auditreduce -c ex /var/audit/<file> | praudit

then my previous tests were reading already-produced files, and I just now tried using /dev/auditpipe as input. Both show the same result - that is to say, no result at all. Am I doing it wrong?

/Eirik
Comment 10 Eirik Oeverby 2020-10-24 22:22:00 UTC
(In reply to Kyle Evans from comment #5)

Is this really the expected behaviour? At least from a user perspective (mine), and despite having tried to read the relevant manual pages, this is not immediately obvious, nor does it seem terribly useful.

I am also reasonably certain I have demonstrated auditreduce/praudit to auditors in the (distant*) past, and been able to show how the flags/naflags are applied differently, specifically by comparing the logged events for processes started during bootup (non-attributable or root) to those (re)started by one of us admins (attributed to an individual, since there is no direct root login enabled on the systems; only sudo).

*This obviously means I may be remembering what I was trying to show, not necessarily what was shown :)
Comment 11 Christian S.J. Peron freebsd_committer 2020-10-24 23:50:42 UTC
I have tracked down why auditreduce -c is broken. auditreduce now runs in capability mode, therefore the attempt to grab an fd for the event database fails. I am thinking about introducing a set_eventdb_fd() and set_classdb_fd() interface that would allow us to set the file descriptor that libbsm should be using or worse case we could introduce a privileged process, but that might be a bit overkill.
Comment 12 Christian S.J. Peron freebsd_committer 2020-10-25 01:01:45 UTC
I have a PR opened to fix auditreduce -c upstream: https://github.com/openbsm/openbsm/pull/68 We will likely refine this a bit and merge it into FreeBSD after
Comment 13 ben 2020-10-27 11:47:43 UTC
*** Bug 242938 has been marked as a duplicate of this bug. ***
Comment 14 commit-hook freebsd_committer 2020-10-27 13:13:40 UTC
A commit references this bug:

Author: kevans
Date: Tue Oct 27 13:13:06 UTC 2020
New revision: 367080
URL: https://svnweb.freebsd.org/changeset/base/367080

Log:
  MFC r367002, r367060

  r367002:
  audit: correct reporting of *execve(2) success

  r326145 corrected do_execve() to return EJUSTRETURN upon success so that
  important registers are not clobbered. This had the side effect of tapping
  out 'failures' for all *execve(2) audit records, which is less than useful
  for auditing purposes.

  Audit exec returns earlier, where we can know for sure that EJUSTRETURN
  translates to success. Note that this unsets TDP_AUDITREC as we commit the
  audit record, so the usual audit in the syscall return path will do nothing.

  r367060:
  audit: also correctly audit linux_execve()

  Linux execve() gets audited as AUE_EXECVE as well, we should also interpret
  the return from this correctly for the same reasoning as in r367002.

  PR:		249179, 242938

Changes:
_U  stable/12/
  stable/12/sys/amd64/linux/linux_machdep.c
  stable/12/sys/amd64/linux32/linux32_machdep.c
  stable/12/sys/arm64/linux/linux_machdep.c
  stable/12/sys/compat/freebsd32/freebsd32_misc.c
  stable/12/sys/i386/linux/linux_machdep.c
  stable/12/sys/kern/kern_exec.c
  stable/12/sys/kern/subr_syscall.c
Comment 15 Kyle Evans freebsd_committer 2020-10-27 15:30:16 UTC
MFC'd as 367080 and an EN request has been submitted for the *execve(2) part.
Comment 16 Ed Maste freebsd_committer 2020-10-27 16:23:45 UTC
execve issue staged as EN-20:19