Comment 1 Edwin Groothuis 2008-08-26 13:10:14 UTC

Responsible Changed
From-To: freebsd-ports-bugs->secteam

Over to maintainer (via the GNATS Auto Assign Tool)

The last line should read "And the below patch teaches portaudit..."

Good day.

A while ago I had created the new utility that serves as VuXML
filter for the installed packages:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/126853

My primary intention was to speed up the process of auditing the
vulnerable ports: I needed to run portaudit checks with Nagios and to
avoid large timeouts.

The new utility is called pkg_audit and it serves as a simple text
filter: on input it takes the full VuXML feed and on output it puts
VuXML entries that matches ports that are installed in the system with
port version specification substituted with the actual port versions.

No harm is done to the actual poartudit -- if pkg_audit is missing, old
code path is activated.

If someone is interested and will be able to test -- I am all ears.

Thanks!
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

Have you also posted this to ports@?

rik

Eygene Ryabinkin wrote:
> Good day.
>
> A while ago I had created the new utility that serves as VuXML
> filter for the installed packages:
>   http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/126853
>
> My primary intention was to speed up the process of auditing the
> vulnerable ports: I needed to run portaudit checks with Nagios and to
> avoid large timeouts.
>
> The new utility is called pkg_audit and it serves as a simple text
> filter: on input it takes the full VuXML feed and on output it puts
> VuXML entries that matches ports that are installed in the system with
> port version specification substituted with the actual port versions.
>
> No harm is done to the actual poartudit -- if pkg_audit is missing, old
> code path is activated.
>
> If someone is interested and will be able to test -- I am all ears.
>
> Thanks!
>

Roman, good day.

Sat, Sep 27, 2008 at 08:18:08PM +0400, Roman Kurakin wrote:
> Have you also posted this to ports@?


No, forgot to do it.  CC'ing ports@

Thanks!

The original posting to hackers@ goes below.  It will be double-posted
to the bug-followup@ -- sorry for this.

> Eygene Ryabinkin wrote:
> > Good day.
> >
> > A while ago I had created the new utility that serves as VuXML
> > filter for the installed packages:
> >   http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/126853
> >
> > My primary intention was to speed up the process of auditing the
> > vulnerable ports: I needed to run portaudit checks with Nagios and to
> > avoid large timeouts.
> >
> > The new utility is called pkg_audit and it serves as a simple text
> > filter: on input it takes the full VuXML feed and on output it puts
> > VuXML entries that matches ports that are installed in the system with
> > port version specification substituted with the actual port versions.
> >
> > No harm is done to the actual poartudit -- if pkg_audit is missing, old
> > code path is activated.
> >
> > If someone is interested and will be able to test -- I am all ears.


Additional clarifications inspired by the off-line talk with rik@:
I could take another route and add this functionality to the pkg_info.
I took another approach for the following reasons.

1. pkg_info's option list is already quite big -- around 32 options
   and switches.

2. It is easier to test for the presence of the new tool (pkg_audit)
   and use it, instead of checking the support for the new option in
   pkg_info.

3. I see no options in pkg_info that can be naturally extended to
   absorbe the new functionality.  The closest is '-E', but pkg_audit
   needs to read VuXML entries, choose ones that are present in the system
   and output the found VuXML entries with version templates substituted
   with the real entries, so pkg_audit is filter-like utility.  In my
   opinion, such extension of pkg_info's "-E" will be very unnatural.

4. I feel that it is Unix-way to do the things: create small utilities
   that do their (small) job in a proper fashion.  Moreover, since the
   majority of a code sits in the pkg_install's library, there is a very
   slight code duplication, if any.

Thanks for you time.
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

Eygene Ryabinkin wrote:
> Roman, good day.
> 
> Sat, Sep 27, 2008 at 08:18:08PM +0400, Roman Kurakin wrote:
> 
>>Have you also posted this to ports@?
> 
> 
> No, forgot to do it.  CC'ing ports@
> 
> Thanks!
> 
> The original posting to hackers@ goes below.  It will be double-posted
> to the bug-followup@ -- sorry for this.
> 
> 
>>Eygene Ryabinkin wrote:
>>
>>>Good day.
>>>
>>>A while ago I had created the new utility that serves as VuXML
>>>filter for the installed packages:
>>>  http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/126853
>>>
>>>My primary intention was to speed up the process of auditing the
>>>vulnerable ports: I needed to run portaudit checks with Nagios and to
>>>avoid large timeouts.
>>>
>>>The new utility is called pkg_audit and it serves as a simple text
>>>filter: on input it takes the full VuXML feed and on output it puts
>>>VuXML entries that matches ports that are installed in the system with
>>>port version specification substituted with the actual port versions.
>>>
>>>No harm is done to the actual poartudit -- if pkg_audit is missing, old
>>>code path is activated.
>>>
>>>If someone is interested and will be able to test -- I am all ears.
> 
> 
> Additional clarifications inspired by the off-line talk with rik@:
> I could take another route and add this functionality to the pkg_info.
> I took another approach for the following reasons.
> 
> 1. pkg_info's option list is already quite big -- around 32 options
>    and switches.
> 
> 2. It is easier to test for the presence of the new tool (pkg_audit)
>    and use it, instead of checking the support for the new option in
>    pkg_info.
> 
> 3. I see no options in pkg_info that can be naturally extended to
>    absorbe the new functionality.  The closest is '-E', but pkg_audit
>    needs to read VuXML entries, choose ones that are present in the system
>    and output the found VuXML entries with version templates substituted
>    with the real entries, so pkg_audit is filter-like utility.  In my
>    opinion, such extension of pkg_info's "-E" will be very unnatural.
> 
> 4. I feel that it is Unix-way to do the things: create small utilities
>    that do their (small) job in a proper fashion.  Moreover, since the
>    majority of a code sits in the pkg_install's library, there is a very
>    slight code duplication, if any.

Is there any possibility to cooperate portaudit / pkg_audit with 
pkg_version to show vulnerable package with information if newer (not 
vulnerable) package (or port) version is available for upgrade to?

If I read nightly security e-mail with for example 4 vulnerable 
packages, then I need to log in to server and manualy try, if newer 
(fixed) packages are available. It seems not so hard to check output of 
`pkg_version -vIL =` and compare both versions (installed and available) 
with portaudit in some shellscript, I didn't start to write it yet ;).

Miroslav Lachman

Miroslav, good day.

Sun, Sep 28, 2008 at 01:15:01PM +0200, Miroslav Lachman wrote:
> Is there any possibility to cooperate portaudit / pkg_audit with 
> pkg_version to show vulnerable package with information if newer (not 
> vulnerable) package (or port) version is available for upgrade to?
> 
> If I read nightly security e-mail with for example 4 vulnerable 
> packages, then I need to log in to server and manualy try, if newer 
> (fixed) packages are available. It seems not so hard to check output of 
> `pkg_version -vIL =` and compare both versions (installed and available) 
> with portaudit in some shellscript, I didn't start to write it yet ;).


I think it won't be very hard: I'll try to see how to extend portaudit
with such functionality -- it would be very handy, in my opinion.

Hadn't you have a chance to test my patch?

Thanks!
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

Sun, Oct 05, 2008 at 11:03:17PM +0400, Eygene Ryabinkin wrote:
> I had also changed the output format for pkg_audit, so I am attaching
> another version of the second patch for the pkg_install bundle.


One neat about new pkg_audit utility: if you already have the build
directory for pkg_install in the /usr/obj, you should create
subdirectory for the pkg_audit,
-----
mkdir /usr/obj/usr/src/usr.sbin/pkg_install/audit
-----
or completely remove /usr/obj/usr/src/usr.sbin/pkg_install

World build should do it automatically, at least it worked for me.
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

I had noticed that the attached patches are being transformed to one
large chunk by a Gnats interface.  Reattaching them properly (I hope ;))

I had also slightly rearranged statements in the second patch to make it
compile under FreeBSD 4.x with old GCC that expects the variable
definitions to be located strictly at the top of the code blocks.

Patches for pkg_install:
--- 0001-Add-functions-for-traversing-package-database-and-ma.patch begins here ---
From ce2fbab57ffb6f91d0a13a07cc82ec5a97cb06ff Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Tue, 26 Aug 2008 14:59:29 +0400
Subject: [PATCH] Add functions for traversing package database and matching package names

This functions will help one who wants to search package database for
the given package names or name globs and do something more than just
collecting the names of the matched packages.  They are skeleton-like
functions and are supposed to be used as package database iterator
and a handy match function:
-----
session = match_begin(MATCH_STYLE);

while (match_next_package(session)) {
	...
	pkg_name = match_get_pkgname(session);
	...
	match_matches(session, template);
	...
}

match_end(session);
-----

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 lib/lib.h   |    5 +++
 lib/match.c |  100 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 105 insertions(+), 0 deletions(-)

diff --git a/lib/lib.h b/lib/lib.h
index 773278a..2890df6 100644
--- a/lib/lib.h
+++ b/lib/lib.h
@@ -228,6 +228,11 @@ char		**matchbyorigin(const char *, int *);
 char		***matchallbyorigin(const char **, int *);
 int		isinstalledpkg(const char *name);
 int		pattern_match(match_t MatchType, char *pattern, const char *pkgname);
+struct match_session *match_begin(match_t);
+void            match_end(struct match_session *);
+Boolean         match_next_package(struct match_session *);
+int             match_matches(struct match_session *, char *);
+const char      *match_get_pkgname(struct match_session *);
 
 /* Dependencies */
 int		sortdeps(char **);
diff --git a/lib/match.c b/lib/match.c
index 07c136b..01f09ee 100644
--- a/lib/match.c
+++ b/lib/match.c
@@ -37,6 +37,16 @@ struct store {
     char **store;
 };
 
+/*
+ * Structure that holds information about package traversal
+ * and matching session.
+ */
+struct match_session {
+	FTS *ftsp;
+	FTSENT *f;
+	match_t match_type;;
+};
+
 static int rex_match(const char *, const char *, int);
 static int csh_match(const char *, const char *, int);
 struct store *storecreate(struct store *);
@@ -416,6 +426,96 @@ errout:
 }
 
 /*
+ * Prepares package database for the traversal.
+ */
+struct match_session *
+match_begin(match_t MatchType)
+{
+	struct match_session *sess;
+	const char *paths[2] = {LOG_DIR, NULL};
+
+	if (!isdir(paths[0]))
+		return NULL;
+
+	sess = (struct match_session *)malloc(sizeof(*sess));
+	if (sess == NULL)
+		return NULL;
+	bzero((void *)sess, sizeof(*sess));
+
+	sess->match_type = MatchType;
+	sess->ftsp = fts_open((char * const *)(uintptr_t)paths,
+	    FTS_LOGICAL | FTS_NOCHDIR | FTS_NOSTAT, fname_cmp);
+	if (sess->ftsp == NULL) {
+		free((void *)sess);
+		return NULL;
+	}
+
+	return sess;
+}
+
+/*
+ * Ends package traversal session.
+ */
+void
+match_end(struct match_session *sess)
+{
+	if (sess == NULL)
+		return;
+	if (sess->ftsp != NULL)
+		fts_close(sess->ftsp);
+	free((void *)sess);
+	return;
+}
+
+/*
+ * Proceeds to the next package in the session.
+ * Returns TRUE if package was selected, FALSE otherwise.
+ */
+Boolean
+match_next_package(struct match_session *sess)
+{
+	if (sess == NULL || sess->ftsp == NULL)
+		return FALSE;
+
+	while ((sess->f = fts_read(sess->ftsp)) != NULL) {
+		if (sess->f->fts_info == FTS_D &&
+		    sess->f->fts_level == 1) {
+			fts_set(sess->ftsp, sess->f, FTS_SKIP);
+			return TRUE;
+		}
+	}
+	return FALSE;
+}
+
+/*
+ * Matches the current package name against the given pattern.
+ * Returns 1 if pattern matches, 0 if not matches and -1 for
+ * the error condition.
+ */
+int
+match_matches(struct match_session *sess, char *pattern)
+{
+	if (sess == NULL || sess->ftsp == NULL || sess->f == NULL ||
+	    pattern == NULL)
+		return FALSE;
+
+	return pattern_match(sess->match_type, pattern,
+	    sess->f->fts_name);
+}
+
+/*
+ * Returns name of the current package.
+ */
+const char *
+match_get_pkgname(struct match_session *sess)
+{
+	if (sess == NULL || sess->ftsp == NULL || sess->f == NULL)
+		return NULL;
+	
+	return (const char *)(sess->f->fts_name);
+}
+
+/*
  * Returns 1 if specified pkgname matches RE pattern.
  * Otherwise returns 0 if doesn't match or -1 if RE
  * engine reported an error (usually invalid syntax).
-- 
1.6.0.2
--- 0001-Add-functions-for-traversing-package-database-and-ma.patch ends here ---
--- 0002-New-utility-pkg_audit.patch begins here ---
From bd13aaff2e2a61c0298383c6f61b9f03ff35d4ae Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Tue, 26 Aug 2008 15:08:46 +0400
Subject: [PATCH] New utility: pkg_audit

It is mainly a helper for portupgrade to avoid awk scripting and
numerous calls for pkg_info.  This utility speeds up portaudit by a
factor of 10 on a system with 521 installed ports and the auditfile that
contains 3213 entries:
-----
$ ls -d /var/db/pkg/*  | wc -l
     521

$ tar xOf /var/db/portaudit/auditfile.tbz auditfile | sed -e'/^#/d' | wc -l
    3213

$ time ./portaudit
Affected package: ruby-1.8.6.111_4,1
Type of problem: ruby -- DNS spoofing vulnerability.
Reference: <http://www.FreeBSD.org/ports/portaudit/959d384d-6b59-11dd-9d79-001fc61c2a55.html>

Affected package: ruby-1.8.6.111_4,1
Type of problem: ruby -- DoS vulnerability in WEBrick.
Reference: <http://www.FreeBSD.org/ports/portaudit/f7ba20aa-6b5a-11dd-9d79-001fc61c2a55.html>

Affected package: ruby-1.8.6.111_4,1
Type of problem: ruby -- multiple vulnerabilities in safe level.
Reference: <http://www.FreeBSD.org/ports/portaudit/c329712a-6b5b-11dd-9d79-001fc61c2a55.html>

3 problem(s) in your installed packages found.

You are advised to update or deinstall the affected package(s) immediately.

real    0m0.107s
user    0m0.116s
sys     0m0.012s

$ time portaudit
Affected package: ruby-1.8.6.111_4,1
Type of problem: ruby -- multiple vulnerabilities in safe level.
Reference: <http://www.FreeBSD.org/ports/portaudit/c329712a-6b5b-11dd-9d79-001fc61c2a55.html>

Affected package: ruby-1.8.6.111_4,1
Type of problem: ruby -- DoS vulnerability in WEBrick.
Reference: <http://www.FreeBSD.org/ports/portaudit/f7ba20aa-6b5a-11dd-9d79-001fc61c2a55.html>

Affected package: ruby-1.8.6.111_4,1
Type of problem: ruby -- DNS spoofing vulnerability.
Reference: <http://www.FreeBSD.org/ports/portaudit/959d384d-6b59-11dd-9d79-001fc61c2a55.html>

3 problem(s) in your installed packages found.

You are advised to update or deinstall the affected package(s) immediately.

real    0m1.583s
user    0m0.560s
sys     0m1.057s
-----

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 Makefile          |    2 +-
 audit/Makefile    |   14 +++
 audit/audit.h     |   43 +++++++++
 audit/main.c      |  166 ++++++++++++++++++++++++++++++++++
 audit/parse.c     |  259 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 audit/pkg_audit.1 |   63 +++++++++++++
 6 files changed, 546 insertions(+), 1 deletions(-)
 create mode 100644 audit/Makefile
 create mode 100644 audit/audit.h
 create mode 100644 audit/main.c
 create mode 100644 audit/parse.c
 create mode 100644 audit/pkg_audit.1

diff --git a/Makefile b/Makefile
index fefbd08..abc1e65 100644
--- a/Makefile
+++ b/Makefile
@@ -2,7 +2,7 @@
 
 .include <bsd.own.mk>
 
-SUBDIR=	lib add create delete info updating version
+SUBDIR=	lib add create delete info updating version audit
 
 .include <bsd.subdir.mk>
 
diff --git a/audit/Makefile b/audit/Makefile
new file mode 100644
index 0000000..2ece5f8
--- /dev/null
+++ b/audit/Makefile
@@ -0,0 +1,14 @@
+# $FreeBSD$
+
+PROG=	pkg_audit
+SRCS=	main.c parse.c
+
+CFLAGS+= -I${.CURDIR}/../lib
+
+WARNS?=	6
+WFORMAT?=	1
+
+DPADD=	${LIBINSTALL} ${LIBFETCH} ${LIBMD}
+LDADD=	${LIBINSTALL} -lfetch -lmd
+
+.include <bsd.prog.mk>
diff --git a/audit/audit.h b/audit/audit.h
new file mode 100644
index 0000000..1f0a369
--- /dev/null
+++ b/audit/audit.h
@@ -0,0 +1,43 @@
+/*
+ *
+ * FreeBSD install - a package for the installation and maintainance
+ * of non-core utilities.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * Eygene Ryabinkin
+ * 26 August 2008
+ *
+ * Parsing module for pkg_audit, header file.
+ *
+ */
+
+#ifndef __AUDIT_H__
+#define __AUDIT_H__
+
+#include <sys/queue.h>
+
+SLIST_HEAD(audit_contents, audit_entry);
+
+struct audit_entry {
+	char	*pkgglob;	/* Package name glob */
+	char	*url;		/* URL of advisory */
+	char	*descr;		/* Description of vulnerability */
+	size_t	pfx_size;	/* Metacharacter-less glob part size */
+	SLIST_ENTRY(audit_entry) entries;
+};
+
+
+/* Function prototypes */
+int
+parse_auditfile(FILE *_fp, struct audit_contents *_head);
+
+
+#endif /* defined(__AUDIT_H__) */
diff --git a/audit/main.c b/audit/main.c
new file mode 100644
index 0000000..2ef2b2e
--- /dev/null
+++ b/audit/main.c
@@ -0,0 +1,166 @@
+/*
+ *
+ * FreeBSD install - a package for the installation and maintainance
+ * of non-core utilities.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * Eygene Ryabinkin
+ * 26 August 2008
+ *
+ * This is the audit module -- fast helper for the portaudit script.
+ *
+ * It is filter-like utility: it reads the audit file from the
+ * standard input, intersects the vulnerable port list with the
+ * packages installed in the system and outputs the entries for
+ * the vulnerable ports that are present in the system to the
+ * standard output.
+ *
+ * The installed package can be listed multiple times, since it
+ * can be vulnerable to more than one bug at a time.  But the
+ * whole output entries will be unique -- package name and
+ * vulnerability details should produce unique entry.
+ *
+ * One more field is prepended to the list of the input fields --
+ * the name of the matched port.
+ *
+ */
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#ifdef PROFILING
+#include <sys/time.h>
+#endif
+
+#include <sys/types.h>
+#include <sys/sysctl.h>
+#include <getopt.h>
+#include <err.h>
+#include <stdio.h>
+
+#include "lib.h"
+#include "audit.h"
+
+static inline void
+audit_package(const char *_pkgname, struct audit_contents *_head,
+    struct match_session *_msess, FILE *_fp);
+
+int
+main(int argc, char *argv[])
+{
+	char freebsd[sizeof("FreeBSD-XXYYZZXXYYZZ")];
+	unsigned long reldate;
+	size_t reldate_size = sizeof(reldate);
+	int mib[2];
+
+	FILE *in = stdin, *out = stdout;
+	struct match_session *msess;
+	struct audit_entry *item;
+#ifdef PROFILING
+	struct timeval t1, t2;
+	double dt;
+#endif
+
+	struct audit_contents head =
+	    SLIST_HEAD_INITIALIZER(head);
+
+	/* Make compiler happy */
+	if (argv[argc] == NULL) {};
+
+	mib[0] = CTL_KERN;
+	mib[1] = KERN_OSRELDATE;
+	if (sysctl(mib, 2, (void *)&reldate, &reldate_size, NULL, 0) != 0)
+		errx(1, "Unable to get kern.osreldate");
+	snprintf(freebsd, sizeof(freebsd), "%s-%lu", "FreeBSD", reldate);
+	
+	SLIST_INIT(&head);
+#ifdef PROFILING
+	gettimeofday(&t1, NULL);
+#endif
+	if (parse_auditfile(in, &head) != 0) {
+		errx(1, "Failed to parse audit entries");
+	}
+#ifdef PROFILING
+	gettimeofday(&t2, NULL);
+	dt = t2.tv_sec - t1.tv_sec + 1e-6 * (t2.tv_usec - t1.tv_usec);
+	fprintf(stderr, "parse_auditfile(): %.6lf sec\n", dt);
+#endif
+
+	msess = match_begin(MATCH_GLOB);
+	if (msess == NULL)
+		return 1;
+	
+#ifdef PROFILING
+	gettimeofday(&t1, NULL);
+#endif
+
+	/* Special check: FreeBSD itself */
+	SLIST_FOREACH (item, &head, entries) {
+		if (strncmp(item->pkgglob,
+		    "FreeBSD", sizeof("FreeBSD") - 1) == 0 &&
+		    pattern_match(MATCH_GLOB, item->pkgglob, freebsd)) {
+			fprintf(out, "%s|%s|%s\n",
+			    freebsd, item->url, item->descr);
+		}
+	}
+
+	/* Installed packages */
+	while (match_next_package(msess))
+		audit_package(match_get_pkgname(msess), &head, msess, out);
+
+#ifdef PROFILING
+	gettimeofday(&t2, NULL);
+	dt = t2.tv_sec - t1.tv_sec + 1e-6 * (t2.tv_usec - t1.tv_usec);
+	fprintf(stderr, "match loop: %.6lf sec\n", dt);
+#endif
+
+	match_end(msess);
+
+	SLIST_FOREACH (item, &head, entries) {
+		free((void *)item->pkgglob);
+		free((void *)item);
+	}
+
+	return 0;
+}
+
+void
+cleanup(int sig)
+{
+	sig = 0;
+	return;
+}
+
+/*
+ * Loops over audit file contents and checks each entry in turn.
+ *
+ * The great speedup is to test the package prefix at the first
+ * place and only if it matches perform full match -- match_matches
+ * uses slow matching routines without precompilation and other
+ * tricks.  For hundreds of installed ports and a couple of thousands
+ * audit entries this slows things down very well.
+ */
+static inline void
+audit_package(const char *pkgname, struct audit_contents *head,
+    struct match_session *msess, FILE *fp)
+{
+	struct audit_entry *item;
+
+	SLIST_FOREACH (item, head, entries) {
+		if (strncmp(pkgname, item->pkgglob,
+		    item->pfx_size) == 0 &&
+		    match_matches(msess, item->pkgglob)) {
+			fprintf(fp, "%s|%s|%s|%s\n",
+			    pkgname,
+			    item->pkgglob, item->url, item->descr);
+		}
+	}
+}
diff --git a/audit/parse.c b/audit/parse.c
new file mode 100644
index 0000000..fb33f7c
--- /dev/null
+++ b/audit/parse.c
@@ -0,0 +1,259 @@
+/*
+ *
+ * FreeBSD install - a package for the installation and maintainance
+ * of non-core utilities.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * Eygene Ryabinkin
+ * 26 August 2008
+ *
+ * Parsing module for pkg_audit.
+ *
+ */
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <getopt.h>
+#include <err.h>
+#include <stdio.h>
+
+#include "lib.h"
+#include "audit.h"
+
+/* Simple exponentially-growing buffer. */
+
+struct dyn_buffer {
+	char *buf;
+	size_t size;
+};
+
+/* Prototypes */
+static int
+parse_audit_entry(struct dyn_buffer *_b, struct audit_entry *_e);
+
+static int
+read_line(FILE *_fp, struct dyn_buffer *_b);
+
+static struct dyn_buffer *
+buf_init(size_t _size);
+static void
+buf_destroy(struct dyn_buffer *_b);
+static int
+buf_grow(struct dyn_buffer *_b);
+
+/*
+ * Parses audit file to the linked list of single entries.
+ *
+ * Return values:
+ * 0 -- file was successfully parsed;
+ * 1 -- parsing or read error occured;
+ * -1 -- bad arguments, memory allocation problems, etc.
+ */
+int
+parse_auditfile(FILE *fp, struct audit_contents *head)
+{
+	struct audit_entry *e;
+	struct dyn_buffer *b;
+	int errcode;
+
+	b = buf_init(256);
+	if (b == NULL)
+		return 1;
+
+	while ((errcode = read_line(fp, b)) == 0) {
+		if (b->buf[0] == '#')
+			continue;
+		e = (struct audit_entry *)malloc(sizeof(*e));
+		if (e == NULL) {
+			buf_destroy(b);
+			return -1;
+		}
+		bzero((void *)e, sizeof(*e));
+		if ((errcode = parse_audit_entry(b, e)) != 0) {
+			buf_destroy(b);
+			return errcode;
+		}
+		SLIST_INSERT_HEAD(head, e, entries);
+	}
+
+	buf_destroy(b);
+
+	if (errcode != 1)
+		return 1;
+	else
+		return 0;
+}
+
+/*
+ * Helpers for audit file parsing routine.
+ */
+
+static struct dyn_buffer *
+buf_init(size_t size)
+{
+	struct dyn_buffer *b;
+
+	if (size <= 0)
+		return NULL;
+
+	b = (struct dyn_buffer *)malloc(sizeof(*b));
+	if (b == NULL)
+		return NULL;
+
+	bzero((void *)b, sizeof(*b));
+	b->size = size;
+	b->buf = (char *)malloc(b->size * sizeof(b->buf[0]));
+	if (b->buf == NULL) {
+		free((void *)b);
+		return NULL;
+	}
+
+	bzero((void *)b->buf, b->size);
+	return b;
+}
+
+static void
+buf_destroy(struct dyn_buffer *b)
+{
+	if (b == NULL)
+		return;
+	
+	if (b->buf != NULL)
+		free((void *)b->buf);
+	free((void *)b);
+	return;
+}
+
+static int
+buf_grow(struct dyn_buffer *b)
+{
+	char *newbuf;
+
+	if (b == NULL || b->buf == NULL || b->size <= 0)
+		return -1;
+
+	newbuf = (char *)malloc(2 * b->size * sizeof(newbuf));
+	if (newbuf == NULL)
+		return 1;
+
+	bzero(newbuf, 2 * b->size);
+	bcopy((void *)b->buf, (void *)newbuf, b->size);
+	b->buf = newbuf;
+	b->size *= 2;
+
+	return 0;
+}
+
+/*
+ * fgets()-like function that reads the whole input line
+ * Returns 0 on the successful read, 1 for the end-of-file
+ * condition, -1 for any error.
+ *
+ * Terminating '\n' is removed from the line.
+ */
+static int
+read_line(FILE *fp, struct dyn_buffer *b)
+{
+	size_t offset = 0, len = 0;
+
+	if (fp == NULL || b == NULL)
+		return -1;
+
+	if (feof(fp))
+		return 1;
+
+	/* We need at least two-character buffer */
+	if (b->size == 1 && buf_grow(b) != 0)
+		return -1;
+
+	b->buf[b->size - 1] = '\0';
+	offset = 0;
+	while (fgets(b->buf + offset, b->size - offset, fp) != NULL) {
+		len = strlen(b->buf);
+		/*
+		 * Read zero characters or buffer even shrinked?
+		 * Strange, let's indicate error.
+		 */
+		if (len <= offset)
+			return -1;
+		if (b->buf[len - 1] == '\n') {
+			b->buf[len - 1] = '\0';
+			return 0;
+		}
+
+		offset = len;
+		if (buf_grow(b) != 0)
+			return -1;
+
+		/* Should not happen, but who knows */
+		if (offset >= b->size)
+			return -1;
+	}
+
+	if (feof(fp)) {
+		/*
+		 * If we read no characters, if means that we were
+		 * at the EOF, but it was detected only by fgets(),
+		 * not the first feof().
+		 */
+		if (len == 0)
+			return 1;
+		else
+			return 0;
+	} else {
+		return -1;
+	}
+}
+
+/*
+ * Parses single audit line and places it to the structure.
+ * Calculates length of the package name suffix that is free
+ * from metacharacters -- it is used for the quick matches
+ * against port name.
+ */
+static int
+parse_audit_entry(struct dyn_buffer *b, struct audit_entry *e)
+{
+	size_t len;
+	char *string = NULL, *d1 = NULL, *d2 = NULL;
+	static const char globset[] = "{*?><=!";
+
+	/*
+	 * At least 5 characters:
+	 * two delimiters and three non-empty fields.
+	 */
+	len = strlen(b->buf);
+	if (len < 5)
+		return 1;
+
+	/* Locate delimiters. */
+	d1 = strchr(b->buf, '|');
+	if (d1 == NULL)
+		return 1;
+	d2 = strchr(d1 + 1, '|');
+	if (d2 == NULL)
+		return 1;
+
+	string = (char *)malloc((len + 1) * sizeof(string[0]));
+	if (string == NULL)
+		return -1;
+
+	bcopy((void *)b->buf, (void *)string, (len + 1) * sizeof(string[0]));
+	string[d1 - b->buf] = '\0';
+	string[d2 - b->buf] = '\0';
+	e->pkgglob = string;
+	e->url = string + (d1 - b->buf) + 1;
+	e->descr = string + (d2 - b->buf) + 1;
+	e->pfx_size = strcspn(e->pkgglob, globset);
+
+	return 0;
+}
diff --git a/audit/pkg_audit.1 b/audit/pkg_audit.1
new file mode 100644
index 0000000..cd4abbc
--- /dev/null
+++ b/audit/pkg_audit.1
@@ -0,0 +1,63 @@
+.\"
+.\" FreeBSD install - a package for the installation and maintenance
+.\" of non-core utilities.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\"
+.\" Eygene Ryabinkin
+.\"
+.\"
+.\"     @(#)pkg_audit.1
+.\" $FreeBSD$
+.\"
+.Dd Aug 26, 2008
+.Dt PKG_AUDIT 1
+.Os
+.Sh NAME
+.Nm pkg_audit
+.Nd lists vulnerable ports installed in the system
+.Sh SYNOPSIS
+.Nm
+.Sh DESCRIPTION
+The
+.Nm
+command is used to extract vulnerability information from the audit
+file and list vulnerable packages that are present in the system.
+It is main purpose to help
+.Xr portaudit 1
+utility to avoid time-consuming scripting.
+.Nm
+reads vulnerability information from the standard input and writes
+the list of vulnerable ports to the standard output.
+Format of the output lines is the same as for the audit file, but
+package matching globs are substituted with the actual package names.
+.Sh TECHNICAL DETAILS
+First the audit file is parsed to the internal representation
+(currently it is linked list).
+Then we are traversing installed packages database and trying to
+match the package name against each audit entry.
+The crucial step for the speeding up the process is to first
+match the package prefix that has no CSH-like metacharacters
+and perform full comparison only if match is found.
+One more name is tested prior to the installed packages: it is
+.Qo FreeBSD-`sysctl -n kern.osreldate` Qc ,
+the version of
+.Fx
+the current system is running.
+.Sh SEE ALSO
+.Xr portaudit 1 ,
+.Xr pkg_add 1 ,
+.Xr pkg_create 1 ,
+.Xr pkg_delete 1 ,
+.Xr pkg_version 1 .
+.Sh AUTHORS
+.An Eygene Ryabinkin Aq rea-fbsd@codelabs.ru
+.Sh BUGS
+Sure to be some.
-- 
1.6.0.2
--- 0002-New-utility-pkg_audit.patch ends here ---

Individual patches for the portaudit:
--- 0001-Avoid-usage-of-global-variables-N-in-the-print_affe.patch begins here ---
From b5fc2033e39aecd8b65f3bda45bf71572b72262a Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Sun, 5 Oct 2008 18:04:10 +0400
Subject: [PATCH 1/4] Avoid usage of global variables $N in the print_affected

I will need this functionality in the next commits: some invocations of
print_affected will be called with shifted variables $N.  But this is
good anyway: globals are always been bad and now functions are looking
more sanely -- one does not need to guess the meaning of $1, $2 and $3.

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 files/portaudit-cmd.sh |   68 ++++++++++++++++++++++++-----------------------
 1 files changed, 35 insertions(+), 33 deletions(-)

diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh
index c0eb67b..4583c9c 100755
--- a/files/portaudit-cmd.sh
+++ b/files/portaudit-cmd.sh
@@ -140,7 +140,8 @@ audit_installed()
 		$1 ~ /^FreeBSD[<=>!]/ {
 			if (fixedre && $2 ~ fixedre) next
 			if (!system("'"$pkg_version"' -T \"FreeBSD-'"$osversion"'\" \"" $1 "\"")) {
-				print_affected("FreeBSD-'"$osversion"'", \
+				print_affected("FreeBSD-'"$osversion"'", 
+					$1, $2, $3, \
 					"To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PREFIX%%/etc/portaudit.conf")
 			}
 			next
@@ -152,7 +153,7 @@ audit_installed()
 			cmd="'"$pkg_info"' -E \"" $1 "\""
 			while((cmd | getline pkg) > 0) {
 				vul++
-				print_affected(pkg, "")
+				print_affected(pkg, $1, $2, $3, "")
 			}
 			close(cmd)
 		}
@@ -207,7 +208,7 @@ audit_file()
 				if ($2 !~ /'"$opt_restrict"'/)
 					continue
 				vul++
-				print_affected(pkg, "")
+				print_affected(pkg, $1, $2, $3, "")
 			}
 			close(cmd)
 		}
@@ -244,7 +245,8 @@ audit_args()
 				' | $pkg_version -T "$1" -`; then
 				VULCNT=$(($VULCNT+1))
 				echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"'
-					{ print_affected("'"$1"'", "") }
+					{ print_affected("'"$1"'",
+					    $1, $2, $3, "") }
 				'
 			fi
 			;;
@@ -277,7 +279,7 @@ audit_cwd()
 			{ print }
 		' | $pkg_version -T "$PKGNAME" -`; then
 		echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"'
-			{ print_affected("'"$PKGNAME"'", "") }
+			{ print_affected("'"$PKGNAME"'", $1, $2, $3, "") }
 		'
 		return 1
 	fi
@@ -425,38 +427,38 @@ prerequisites_checked=false
 
 if $opt_quiet; then
 	PRINTAFFECTED_AWK='
-		function print_affected(apkg, note) {
-			print apkg
-		}
-		'
+function print_affected(apkg, glob, refs, descr, note) {
+	print apkg
+}
+'
 elif $opt_verbose; then
 	PRINTAFFECTED_AWK='
-		function print_affected(apkg, note) {
-			split(apkg, thepkg)
-			print "Affected package: " thepkg[1] " (matched by " $1 ")"
-			print "Type of problem: " $3 "."
-			split($2, ref, / /)
-			for (r in ref)
-				print "Reference: <" ref[r] ">"
-			if (note)
-				print "Note: " note
-			print ""
-		}
-		'
+function print_affected(apkg, glob, refs, descr, note) {
+	split(apkg, thepkg)
+	print "Affected package: " thepkg[1] " (matched by " glob ")"
+	print "Type of problem: " descr "."
+	split(refs, ref, / /)
+	for (r in ref)
+		print "Reference: <" ref[r] ">"
+	if (note)
+		print "Note: " note
+	print ""
+}
+'
 else
 	PRINTAFFECTED_AWK='
-		function print_affected(apkg, note) {
-			split(apkg, thepkg)
-			print "Affected package: " thepkg[1]
-			print "Type of problem: " $3 "."
-			split($2, ref, / /)
-			for (r in ref)
-				print "Reference: <" ref[r] ">"
-			if (note)
-				print "Note: " note
-			print ""
-		}
-		'
+function print_affected(apkg, glob, refs, descr, note) {
+	split(apkg, thepkg)
+	print "Affected package: " thepkg[1]
+	print "Type of problem: " descr "."
+	split(refs, ref, / /)
+	for (r in ref)
+		print "Reference: <" ref[r] ">"
+	if (note)
+		print "Note: " note
+	print ""
+}
+'
 fi
 
 if $opt_audit; then
-- 
1.6.0.2
--- 0001-Avoid-usage-of-global-variables-N-in-the-print_affe.patch ends here ---
--- 0002-Separate-vulnerable-ports-search-from-the-formatter.patch begins here ---
From b595cf7da5b81489b5e21d85df4dd8e79f3a0b1f Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Sun, 5 Oct 2008 18:56:13 +0400
Subject: [PATCH 2/4] Separate vulnerable ports search from the formatter routine

I am planning to insert some other filters between the code that outputs
the vulnerable port entries and the code that formats the messages about
vulnerabilities that had been found.  Such a split provides a big
flexibility: one can insert more filters that will transform entries and
one can substitute search routine for something else (I am planning to
substitute it with the binary utility written in a compiled language).

This was done only for the routine that checks the installed ports,
because it is my primary target for now.  May be later I will split
other auditing functions in the same way.

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 files/portaudit-cmd.sh |  101 ++++++++++++++++++++++++++++++------------------
 1 files changed, 63 insertions(+), 38 deletions(-)

diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh
index 4583c9c..5fc0d1d 100755
--- a/files/portaudit-cmd.sh
+++ b/files/portaudit-cmd.sh
@@ -125,50 +125,75 @@ portaudit_prerequisites()
 	return 0
 }
 
+#
+# Helper for audit_installed that actually finds vulnerable packages.
+#
+# It processes the auditfile entries (that are read from the stdin)
+# in the form "glob|refs|desc" and outputs entries in the form
+# "pkgname|glob|refs|desc", where "pkgname" is the matched package name.
+#
+findvuln_installed()
+{
+	local fixedre=`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' | tr -s ' \t\n' '|'`
+	local installedre=`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d '|' -`
+	local osversion=`sysctl -n kern.osreldate`
+
+	awk -F\| \
+	    -v fixedre="$fixedre" -v installedre="$installedre" \
+	    -v pkg_version="$pkg_version" -v pkg_info="$pkg_info" \
+	    -v osversion="$osversion" -v opt_restrict="$opt_restrict" \
+	    '
+/^(#|\$)/ { next }
+opt_restrict && $2 !~ opt_restrict { next }
+$1 ~ /^FreeBSD[<=>!]/ {
+	if (fixedre && $2 ~ fixedre) next
+	if (!system(pkg_version " -T \"FreeBSD-" osversion "\" \"" $1 "\"")) {
+		printf("FreeBSD-%s|%s\n", osversion, $0);
+	}
+	next
+}
+$1 ~ /^[^{}*?]*[<=>!]/ {
+	if ($1 !~ "^(" installedre ")[<=>!]") next;
+}
+{
+	cmd=pkg_info " -E \"" $1 "\""
+	while((cmd | getline pkg) > 0) {
+		printf("%s|%s\n", pkg, $0);
+	}
+	close(cmd)
+}
+'
+}
+
 audit_installed()
 {
 	local rc=0
 	local osversion=`sysctl -n kern.osreldate`
 
-	fixedre=`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' | tr -s ' \t\n' '|'`
-	installedre=`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d '|' -`
-
-	extract_auditfile | awk -F\| "$PRINTAFFECTED_AWK"'
-		BEGIN { vul=0; fixedre="'"$fixedre"'" }
-		/^(#|\$)/ { next }
-		$2 !~ /'"$opt_restrict"'/ { next }
-		$1 ~ /^FreeBSD[<=>!]/ {
-			if (fixedre && $2 ~ fixedre) next
-			if (!system("'"$pkg_version"' -T \"FreeBSD-'"$osversion"'\" \"" $1 "\"")) {
-				print_affected("FreeBSD-'"$osversion"'", 
-					$1, $2, $3, \
-					"To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PREFIX%%/etc/portaudit.conf")
-			}
-			next
-		}
-		$1 ~ /^[^{}*?]*[<=>!]/ {
-			if ($1 !~ "^('"$installedre"')[<=>!]") next;
-		}
-		{
-			cmd="'"$pkg_info"' -E \"" $1 "\""
-			while((cmd | getline pkg) > 0) {
-				vul++
-				print_affected(pkg, $1, $2, $3, "")
-			}
-			close(cmd)
-		}
-		END {
-			if ("'$opt_quiet'" == "false") {
-				print vul " problem(s) in your installed packages found."
-			}
-			if (vul > 0) {
-				if ("'$opt_quiet'" == "false") {
-					print "\nYou are advised to update or deinstall" \
-						" the affected package(s) immediately."
-				}
-				exit(1)
-			}
+	extract_auditfile | findvuln_installed | \
+	awk -F\| "$PRINTAFFECTED_AWK"'
+BEGIN { vul=0; }
+$1 ~ /^FreeBSD-/ {
+	print_affected($1, $2, $3, $4, \
+		"To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PREFIX%%/etc/portaudit.conf")
+	next
+}
+{
+	print_affected($1, $2, $3, $4, "");
+	vul++;
+}
+END {
+	if ("'$opt_quiet'" == "false") {
+		print vul " problem(s) in your installed packages found."
+	}
+	if (vul > 0) {
+		if ("'$opt_quiet'" == "false") {
+			print "\nYou are advised to update or deinstall" \
+				" the affected package(s) immediately."
 		}
+		exit(1)
+	}
+}
 	' || rc=$?
 
 	return $rc
-- 
1.6.0.2
--- 0002-Separate-vulnerable-ports-search-from-the-formatter.patch ends here ---
--- 0003-Use-pkg_audit-utility-if-it-is-available.patch begins here ---
From 100bf18057bb16f21f0058e31279b9c3730e097d Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Sun, 5 Oct 2008 19:20:34 +0400
Subject: [PATCH 3/4] Use pkg_audit utility if it is available

Pkg_audit provides a good speed-up to the search of vulnerable packages
within installed ones.  It can be unavailable, so its usage is
conditionalized.

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 files/portaudit-cmd.sh |   15 +++++++++++++++
 files/portaudit.conf   |    4 ++++
 2 files changed, 19 insertions(+), 0 deletions(-)

diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh
index 5fc0d1d..32c121d 100755
--- a/files/portaudit-cmd.sh
+++ b/files/portaudit-cmd.sh
@@ -43,6 +43,8 @@ portaudit_confs()
 
 	: ${portaudit_fixed=""}
 
+	: ${portaudit_pkg_audit="/usr/sbin/pkg_audit"}
+
 	if [ -r %%PREFIX%%/etc/portaudit.conf ]; then
 		. %%PREFIX%%/etc/portaudit.conf
 	fi
@@ -135,6 +137,19 @@ portaudit_prerequisites()
 findvuln_installed()
 {
 	local fixedre=`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' | tr -s ' \t\n' '|'`
+
+	if [ -x "${portaudit_pkg_audit}" ]; then
+		"${portaudit_pkg_audit}" | awk -F \| \
+		    -v fixedre="$fixedre" -v opt_restrict="$opt_restrict" \
+		    '
+opt_restrict && $3 !~ opt_restrict { next }
+$2 ~ /^FreeBSD[<=>!]/ && fixedre && $3 ~ fixedre { next }
+{ print }
+'
+
+		return
+	fi
+
 	local installedre=`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d '|' -`
 	local osversion=`sysctl -n kern.osreldate`
 
diff --git a/files/portaudit.conf b/files/portaudit.conf
index 4eb08d7..9ea2c4a 100644
--- a/files/portaudit.conf
+++ b/files/portaudit.conf
@@ -17,3 +17,7 @@
 
 # this vulnerability has been fixed in your FreeBSD version
 #portaudit_fixed="d2102505-f03d-11d8-81b0-000347a4fa7d"
+
+# this command will be used to find the vulnerable packages
+# instead of awk(1) script.
+# portaudit_pkg_audit="/usr/sbin/pkg_audit"
-- 
1.6.0.2
--- 0003-Use-pkg_audit-utility-if-it-is-available.patch ends here ---
--- 0004-Implement-checking-for-a-new-package-versions.patch begins here ---
From f4ef2c5c03ea9543ecdfc68a62b26449b8d492ed Mon Sep 17 00:00:00 2001
From: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
Date: Sun, 5 Oct 2008 22:27:53 +0400
Subject: [PATCH 4/4] Implement checking for a new package versions

Flag '-n' tells portaudit to check if updated packages are available.
It is currently done only for the 'all installed packages' mode, i.e.
when '-a' flag is given.

To do this, one additional utility, portaudit-checknew, is introduced.
It downloads the ports INDEX file for the current FreeBSD version and
checks if new versions of packages are available.  This utility is
spawned at the late stage when vulnerable packages are already
determined and thus only they are checked.  So the number of input items
shouldn't be very large and portaudit-checknew is a plain shell script.

Signed-off-by: Eygene Ryabinkin <rea-fbsd@codelabs.ru>
---
 Makefile                    |    4 +-
 files/portaudit-checknew.sh |  159 +++++++++++++++++++++++++++++++++++++++++++
 files/portaudit-cmd.sh      |   67 +++++++++++++++---
 files/portaudit.1           |    4 +-
 pkg-plist                   |    1 +
 5 files changed, 222 insertions(+), 13 deletions(-)
 create mode 100755 files/portaudit-checknew.sh

diff --git a/Makefile b/Makefile
index 885dc27..6083235 100644
--- a/Makefile
+++ b/Makefile
@@ -37,7 +37,8 @@ SED_SCRIPT=	-e 's|%%PREFIX%%|${PREFIX}|g' \
 		-e "s|%%BZIP2_CMD%%|${BZIP2_CMD}|g" \
 
 do-build:
-.for f in portaudit-cmd.sh portaudit.sh portaudit.1 portaudit.conf
+.for f in portaudit-cmd.sh portaudit-checknew.sh portaudit.sh \
+    portaudit.1 portaudit.conf
 	@${SED} ${SED_SCRIPT} ${FILESDIR}/${f} >${WRKDIR}/${f}
 .endfor
 
@@ -54,6 +55,7 @@ pre-install:
 
 do-install:
 	@${INSTALL_SCRIPT} ${WRKDIR}/portaudit-cmd.sh ${PREFIX}/sbin/portaudit
+	@${INSTALL_SCRIPT} ${WRKDIR}/portaudit-checknew.sh ${PREFIX}/sbin/portaudit-checknew
 	@${INSTALL_DATA} ${WRKDIR}/portaudit.conf ${PREFIX}/etc/portaudit.conf.sample
 	@${INSTALL_MAN} ${WRKDIR}/portaudit.1 ${MAN1PREFIX}/man/man1
 	@${MKDIR} ${PERIODICDIR}/security
diff --git a/files/portaudit-checknew.sh b/files/portaudit-checknew.sh
new file mode 100755
index 0000000..6bc7dfe
--- /dev/null
+++ b/files/portaudit-checknew.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+# $FreeBSD$
+#
+# Script to check if newer port versions then the ones installed
+# in the system are available.  Uses downloaded ports index from
+# the master site.
+#
+# Utility acts as a filter: reads input strings, processes them
+# and outputs the result to the standard output.  Each input string
+# is treated as the collection of '|'-separated fields.  First field
+# should be the package name with version, other fields can have
+# any contents -- they won't be touched.
+#
+# One more field can be added -- the newest port version, if it can
+# be deduced.  This field will be added to the end fieldset.
+#
+# For any error the utility will just output the unmodified input
+# contents.
+#
+# Eygene Ryabinkin, rea-fbsd@codelabs.ru.  September 2008.
+
+BASEURL=http://www.freebsd.org/ports/INDEX-
+PKGBASE=/var/db/pkg
+
+# Obtains FreeBSD major version number from uname output.
+freebsd_major () {
+	uname -r | cut -f1 -d- | cut -f1 -d.
+}
+
+# Returns the OS version for the INDEX file.  It is just a wrapper
+# for the freebsd_major routine that applies the fixups for the
+# known unsupported versions.
+index_version () {
+	local major
+
+	major=`freebsd_major`
+	case "$major" in
+	1|2|3|4)
+		major=5
+		;;
+	esac
+	echo "$major"
+}
+
+# Guesses (installed) port origin.
+# Arguments:
+# - port name with version specification.
+# - name of file wit
+guess_origin () {
+	local contents origin
+
+	if [ -z "$1" ]; then
+		echo "guess_origin(): called without arguments" >&2
+		exit 255
+	fi
+	if [ -z "$2" ]; then
+		echo "guess_origin(): called without second argument" >&2
+		exit 255
+	fi
+
+	contents="$PKGBASE"/"$1"/"+CONTENTS"
+	if [ -s "$contents" ]; then
+		origin=`grep '^@comment ORIGIN:' "$contents" | \
+		    sed -e's/^@comment ORIGIN://'`
+		if [ -n "$origin" ]; then
+			echo "$origin"
+			return
+		fi
+	fi
+	# Not yet implemented: loop over INDEX file and try to get
+	# the origin from the matched ports description.
+}
+
+# Fetches bzipped port index file and outputs it to the stdout
+# Arguments:
+# - URL to get file from.
+#
+# Uses 'opt_cachedir' (if it is not empty) as the name of the directory
+# to place the downloaded file to.  Download is made in the mirror mode,
+# so if file's timestamps and size are checked and no download takes
+# place if the remote file has the same characteristics.
+fetch_index () {
+	if [ -z "$1" ]; then
+		echo "fetch_index(): called without arguments" >&2
+		exit 255
+	fi
+
+	if [ -z "$opt_cachedir" -o ! -d "$opt_cachedir" ]; then
+		fetch -qpo - "$1"
+		return
+	fi
+
+	local outfile="$opt_cachedir"/`basename "$1"`
+	fetch -mpo "$outfile" "$1" && cat "$outfile"
+}
+
+TMPINDEX=`mktemp -q -t versionaudit-INDEX`
+if [ -z "$TMPINDEX" ]; then
+	echo "Unable to create temporary file for ports index." >&2
+	cat
+	exit 1
+fi
+
+trap "rm -f \"$TMPINDEX\"" 0 1 2 3 15
+
+opt_cachedir=
+
+while getopts c: opt; do
+	case "$opt" in
+	c)
+		opt_cachedir="$OPTARG"
+		;;
+	?)
+		echo "Usage: $0 [-c cachedir]"
+		exit 2
+		;;
+	esac
+done
+
+url="$BASEURL"`index_version`.bz2
+fetch_index "$url" | bunzip2 -c > "$TMPINDEX"
+if ! [ -s "$TMPINDEX" ]; then
+	echo "Unable to download port index from $url" >&2
+	cat
+	exit 1
+fi
+
+IFS='|'
+while read portspec rest; do
+	portname=`echo "$portspec" | sed -e's/-[^-]*$//'`
+	origin=`guess_origin "$portspec" "$TMPINDEX"`
+	if [ -z "$origin" ]; then
+		echo "Unable to get port origin for '$portspec'." >&2
+		continue
+	fi
+	# The while cycle is hackish: we make "$?" non-zero if no
+	# matches were found and it is set to zero if match was
+	# found.  All exit paths from the cycle should set "$?"
+	# properly.
+	#
+	# Another way to proceed is to make subroutine that will
+	# either print something or not, because we can't pass
+	# variables outside the while loop -- it is done in the
+	# separate process.
+	grep "^$portname-" "$TMPINDEX" | while read nportspec ndir nrest
+	do
+		nportname=`echo "$nportspec" | sed -e's/-[^-]*$//'`
+		norigin=`echo "$ndir" | sed -e's|^/[^/]*/[^/]*/||'`
+		if [ "$nportname" = "$portname" -a \
+		     "$norigin" = "$origin" ]; then
+			echo "$portspec|$rest|$nportspec"
+			true
+			break
+		fi
+		false
+	done || echo "$portspec|$rest"
+done
+
+exit 0
diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh
index 32c121d..0ddd3d3 100755
--- a/files/portaudit-cmd.sh
+++ b/files/portaudit-cmd.sh
@@ -127,6 +127,15 @@ portaudit_prerequisites()
 	return 0
 }
 
+checknew()
+{
+	if [ -n "$opt_checknew" ]; then
+		portaudit-checknew -c "${portaudit_dir}"
+	else
+		cat
+	fi
+}
+
 #
 # Helper for audit_installed that actually finds vulnerable packages.
 #
@@ -185,16 +194,16 @@ audit_installed()
 	local rc=0
 	local osversion=`sysctl -n kern.osreldate`
 
-	extract_auditfile | findvuln_installed | \
-	awk -F\| "$PRINTAFFECTED_AWK"'
+	extract_auditfile | findvuln_installed | checknew | \
+	awk -F\| -v pkg_version="$pkg_version" "$PRINTAFFECTED_AWK"'
 BEGIN { vul=0; }
 $1 ~ /^FreeBSD-/ {
-	print_affected($1, $2, $3, $4, \
+	print_affected($1, $2, $3, $4, "", \
 		"To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PREFIX%%/etc/portaudit.conf")
 	next
 }
 {
-	print_affected($1, $2, $3, $4, "");
+	print_affected($1, $2, $3, $4, $5, "");
 	vul++;
 }
 END {
@@ -248,7 +257,7 @@ audit_file()
 				if ($2 !~ /'"$opt_restrict"'/)
 					continue
 				vul++
-				print_affected(pkg, $1, $2, $3, "")
+				print_affected(pkg, $1, $2, $3, "", "")
 			}
 			close(cmd)
 		}
@@ -286,7 +295,7 @@ audit_args()
 				VULCNT=$(($VULCNT+1))
 				echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"'
 					{ print_affected("'"$1"'",
-					    $1, $2, $3, "") }
+					    $1, $2, $3, "", "") }
 				'
 			fi
 			;;
@@ -319,7 +328,7 @@ audit_cwd()
 			{ print }
 		' | $pkg_version -T "$PKGNAME" -`; then
 		echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"'
-			{ print_affected("'"$PKGNAME"'", $1, $2, $3, "") }
+			{ print_affected("'"$PKGNAME"'", $1, $2, $3, "", "") }
 		'
 		return 1
 	fi
@@ -389,12 +398,13 @@ opt_restrict=
 opt_verbose=false
 opt_version=false
 opt_expiry=
+opt_checknew=
 
 if [ $# -eq 0 ] ; then
 	opt_audit=true
 fi
 
-while getopts aCdf:Fqr:vVX: opt; do
+while getopts aCdf:Fnqr:vVX: opt; do
 	case "$opt" in
 	a)
 		opt_audit=true;;
@@ -406,6 +416,8 @@ while getopts aCdf:Fqr:vVX: opt; do
 		opt_file="$OPTARG";;
 	F)
 		opt_fetch=true;;
+	n)
+		opt_checknew=true;;
 	q)
 		opt_quiet=true;;
 	r)
@@ -465,15 +477,42 @@ fi
 
 prerequisites_checked=false
 
+# This awk code demands 'pkg_version' variable to be set
+# if 'newver' variable is non-empty.
+NEWVERS_AWK='
+
+nparts=split(thepkg[1], verarr, /-/)
+gtglob=verarr[1]
+for (i=2; i<nparts; i++)
+	gtglob=gtglob "-" verarr[i]
+gtglob=gtglob ">" verarr[i]
+
+updated=""
+avail=0
+if (system(pkg_version " -T \"" newver "\" '\''" gtglob "'\''") == 0) {
+	updated="available, " newver
+	avail=1
+} else {
+	updated="not available"
+}
+if (avail != 0) {
+	if (system(pkg_version " -T \"" newver "\" '\''" glob "'\''") == 0)
+		updated=updated ", still vulnerable"
+	else
+		updated=updated ", not vulnerable"
+}
+print "Updated package: " updated
+'
+
 if $opt_quiet; then
 	PRINTAFFECTED_AWK='
-function print_affected(apkg, glob, refs, descr, note) {
+function print_affected(apkg, glob, refs, descr, newver, note) {
 	print apkg
 }
 '
 elif $opt_verbose; then
 	PRINTAFFECTED_AWK='
-function print_affected(apkg, glob, refs, descr, note) {
+function print_affected(apkg, glob, refs, descr, newver, note) {
 	split(apkg, thepkg)
 	print "Affected package: " thepkg[1] " (matched by " glob ")"
 	print "Type of problem: " descr "."
@@ -482,12 +521,15 @@ function print_affected(apkg, glob, refs, descr, note) {
 		print "Reference: <" ref[r] ">"
 	if (note)
 		print "Note: " note
+	if (newver) {
+'"$NEWVERS_AWK"'
+	}
 	print ""
 }
 '
 else
 	PRINTAFFECTED_AWK='
-function print_affected(apkg, glob, refs, descr, note) {
+function print_affected(apkg, glob, refs, descr, newver, note) {
 	split(apkg, thepkg)
 	print "Affected package: " thepkg[1]
 	print "Type of problem: " descr "."
@@ -496,6 +538,9 @@ function print_affected(apkg, glob, refs, descr, note) {
 		print "Reference: <" ref[r] ">"
 	if (note)
 		print "Note: " note
+	if (newver) {
+'"$NEWVERS_AWK"'
+	}
 	print ""
 }
 '
diff --git a/files/portaudit.1 b/files/portaudit.1
index c982b41..5cb0ec2 100644
--- a/files/portaudit.1
+++ b/files/portaudit.1
@@ -42,7 +42,7 @@
 .Sh SYNOPSIS
 .
 .Nm
-.Op Fl aCdFqvV
+.Op Fl aCdFnqvV
 .Op Fl X Ar days
 .Op Fl f Ar file
 .Op Fl r Ar eregex
@@ -85,6 +85,8 @@ Print the creation date of the database.
 Fetch the current database from the
 .Fx
 servers.
+.It Fl n
+Check if new versions of vulnerable ports are present.
 .It Fl q
 Quiet mode.
 .It Fl V
diff --git a/pkg-plist b/pkg-plist
index 8edf7bb..3c9f775 100644
--- a/pkg-plist
+++ b/pkg-plist
@@ -1,4 +1,5 @@
 sbin/portaudit
+sbin/portaudit-checknew
 etc/portaudit.conf.sample
 %%PERIODICDIR%%/security/410.portaudit
 @dirrmtry %%PERIODICDIR%%/security
-- 
1.6.0.2

--- 0004-Implement-checking-for-a-new-package-versions.patch ends here ---

And a mega-patch for portaudit that squishes previous four patches
into a single diff:
--- portaudit-megapatch_pkg_audit-and-checknew.diff begins here ---
This is the megapatch that adds both pkg_audit support and the ability
to check if new versions of vulnerable ports are present.

diff --git a/Makefile b/Makefile
index 885dc27..6083235 100644
--- a/Makefile
+++ b/Makefile
@@ -37,7 +37,8 @@ SED_SCRIPT=	-e 's|%%PREFIX%%|${PREFIX}|g' \
 		-e "s|%%BZIP2_CMD%%|${BZIP2_CMD}|g" \
 
 do-build:
-.for f in portaudit-cmd.sh portaudit.sh portaudit.1 portaudit.conf
+.for f in portaudit-cmd.sh portaudit-checknew.sh portaudit.sh \
+    portaudit.1 portaudit.conf
 	@${SED} ${SED_SCRIPT} ${FILESDIR}/${f} >${WRKDIR}/${f}
 .endfor
 
@@ -54,6 +55,7 @@ pre-install:
 
 do-install:
 	@${INSTALL_SCRIPT} ${WRKDIR}/portaudit-cmd.sh ${PREFIX}/sbin/portaudit
+	@${INSTALL_SCRIPT} ${WRKDIR}/portaudit-checknew.sh ${PREFIX}/sbin/portaudit-checknew
 	@${INSTALL_DATA} ${WRKDIR}/portaudit.conf ${PREFIX}/etc/portaudit.conf.sample
 	@${INSTALL_MAN} ${WRKDIR}/portaudit.1 ${MAN1PREFIX}/man/man1
 	@${MKDIR} ${PERIODICDIR}/security
diff --git a/files/portaudit-checknew.sh b/files/portaudit-checknew.sh
new file mode 100755
index 0000000..6bc7dfe
--- /dev/null
+++ b/files/portaudit-checknew.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+# $FreeBSD$
+#
+# Script to check if newer port versions then the ones installed
+# in the system are available.  Uses downloaded ports index from
+# the master site.
+#
+# Utility acts as a filter: reads input strings, processes them
+# and outputs the result to the standard output.  Each input string
+# is treated as the collection of '|'-separated fields.  First field
+# should be the package name with version, other fields can have
+# any contents -- they won't be touched.
+#
+# One more field can be added -- the newest port version, if it can
+# be deduced.  This field will be added to the end fieldset.
+#
+# For any error the utility will just output the unmodified input
+# contents.
+#
+# Eygene Ryabinkin, rea-fbsd@codelabs.ru.  September 2008.
+
+BASEURL=http://www.freebsd.org/ports/INDEX-
+PKGBASE=/var/db/pkg
+
+# Obtains FreeBSD major version number from uname output.
+freebsd_major () {
+	uname -r | cut -f1 -d- | cut -f1 -d.
+}
+
+# Returns the OS version for the INDEX file.  It is just a wrapper
+# for the freebsd_major routine that applies the fixups for the
+# known unsupported versions.
+index_version () {
+	local major
+
+	major=`freebsd_major`
+	case "$major" in
+	1|2|3|4)
+		major=5
+		;;
+	esac
+	echo "$major"
+}
+
+# Guesses (installed) port origin.
+# Arguments:
+# - port name with version specification.
+# - name of file wit
+guess_origin () {
+	local contents origin
+
+	if [ -z "$1" ]; then
+		echo "guess_origin(): called without arguments" >&2
+		exit 255
+	fi
+	if [ -z "$2" ]; then
+		echo "guess_origin(): called without second argument" >&2
+		exit 255
+	fi
+
+	contents="$PKGBASE"/"$1"/"+CONTENTS"
+	if [ -s "$contents" ]; then
+		origin=`grep '^@comment ORIGIN:' "$contents" | \
+		    sed -e's/^@comment ORIGIN://'`
+		if [ -n "$origin" ]; then
+			echo "$origin"
+			return
+		fi
+	fi
+	# Not yet implemented: loop over INDEX file and try to get
+	# the origin from the matched ports description.
+}
+
+# Fetches bzipped port index file and outputs it to the stdout
+# Arguments:
+# - URL to get file from.
+#
+# Uses 'opt_cachedir' (if it is not empty) as the name of the directory
+# to place the downloaded file to.  Download is made in the mirror mode,
+# so if file's timestamps and size are checked and no download takes
+# place if the remote file has the same characteristics.
+fetch_index () {
+	if [ -z "$1" ]; then
+		echo "fetch_index(): called without arguments" >&2
+		exit 255
+	fi
+
+	if [ -z "$opt_cachedir" -o ! -d "$opt_cachedir" ]; then
+		fetch -qpo - "$1"
+		return
+	fi
+
+	local outfile="$opt_cachedir"/`basename "$1"`
+	fetch -mpo "$outfile" "$1" && cat "$outfile"
+}
+
+TMPINDEX=`mktemp -q -t versionaudit-INDEX`
+if [ -z "$TMPINDEX" ]; then
+	echo "Unable to create temporary file for ports index." >&2
+	cat
+	exit 1
+fi
+
+trap "rm -f \"$TMPINDEX\"" 0 1 2 3 15
+
+opt_cachedir=
+
+while getopts c: opt; do
+	case "$opt" in
+	c)
+		opt_cachedir="$OPTARG"
+		;;
+	?)
+		echo "Usage: $0 [-c cachedir]"
+		exit 2
+		;;
+	esac
+done
+
+url="$BASEURL"`index_version`.bz2
+fetch_index "$url" | bunzip2 -c > "$TMPINDEX"
+if ! [ -s "$TMPINDEX" ]; then
+	echo "Unable to download port index from $url" >&2
+	cat
+	exit 1
+fi
+
+IFS='|'
+while read portspec rest; do
+	portname=`echo "$portspec" | sed -e's/-[^-]*$//'`
+	origin=`guess_origin "$portspec" "$TMPINDEX"`
+	if [ -z "$origin" ]; then
+		echo "Unable to get port origin for '$portspec'." >&2
+		continue
+	fi
+	# The while cycle is hackish: we make "$?" non-zero if no
+	# matches were found and it is set to zero if match was
+	# found.  All exit paths from the cycle should set "$?"
+	# properly.
+	#
+	# Another way to proceed is to make subroutine that will
+	# either print something or not, because we can't pass
+	# variables outside the while loop -- it is done in the
+	# separate process.
+	grep "^$portname-" "$TMPINDEX" | while read nportspec ndir nrest
+	do
+		nportname=`echo "$nportspec" | sed -e's/-[^-]*$//'`
+		norigin=`echo "$ndir" | sed -e's|^/[^/]*/[^/]*/||'`
+		if [ "$nportname" = "$portname" -a \
+		     "$norigin" = "$origin" ]; then
+			echo "$portspec|$rest|$nportspec"
+			true
+			break
+		fi
+		false
+	done || echo "$portspec|$rest"
+done
+
+exit 0
diff --git a/files/portaudit-cmd.sh b/files/portaudit-cmd.sh
index c0eb67b..0ddd3d3 100755
--- a/files/portaudit-cmd.sh
+++ b/files/portaudit-cmd.sh
@@ -43,6 +43,8 @@ portaudit_confs()
 
 	: ${portaudit_fixed=""}
 
+	: ${portaudit_pkg_audit="/usr/sbin/pkg_audit"}
+
 	if [ -r %%PREFIX%%/etc/portaudit.conf ]; then
 		. %%PREFIX%%/etc/portaudit.conf
 	fi
@@ -125,49 +127,97 @@ portaudit_prerequisites()
 	return 0
 }
 
+checknew()
+{
+	if [ -n "$opt_checknew" ]; then
+		portaudit-checknew -c "${portaudit_dir}"
+	else
+		cat
+	fi
+}
+
+#
+# Helper for audit_installed that actually finds vulnerable packages.
+#
+# It processes the auditfile entries (that are read from the stdin)
+# in the form "glob|refs|desc" and outputs entries in the form
+# "pkgname|glob|refs|desc", where "pkgname" is the matched package name.
+#
+findvuln_installed()
+{
+	local fixedre=`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' | tr -s ' \t\n' '|'`
+
+	if [ -x "${portaudit_pkg_audit}" ]; then
+		"${portaudit_pkg_audit}" | awk -F \| \
+		    -v fixedre="$fixedre" -v opt_restrict="$opt_restrict" \
+		    '
+opt_restrict && $3 !~ opt_restrict { next }
+$2 ~ /^FreeBSD[<=>!]/ && fixedre && $3 ~ fixedre { next }
+{ print }
+'
+
+		return
+	fi
+
+	local installedre=`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d '|' -`
+	local osversion=`sysctl -n kern.osreldate`
+
+	awk -F\| \
+	    -v fixedre="$fixedre" -v installedre="$installedre" \
+	    -v pkg_version="$pkg_version" -v pkg_info="$pkg_info" \
+	    -v osversion="$osversion" -v opt_restrict="$opt_restrict" \
+	    '
+/^(#|\$)/ { next }
+opt_restrict && $2 !~ opt_restrict { next }
+$1 ~ /^FreeBSD[<=>!]/ {
+	if (fixedre && $2 ~ fixedre) next
+	if (!system(pkg_version " -T \"FreeBSD-" osversion "\" \"" $1 "\"")) {
+		printf("FreeBSD-%s|%s\n", osversion, $0);
+	}
+	next
+}
+$1 ~ /^[^{}*?]*[<=>!]/ {
+	if ($1 !~ "^(" installedre ")[<=>!]") next;
+}
+{
+	cmd=pkg_info " -E \"" $1 "\""
+	while((cmd | getline pkg) > 0) {
+		printf("%s|%s\n", pkg, $0);
+	}
+	close(cmd)
+}
+'
+}
+
 audit_installed()
 {
 	local rc=0
 	local osversion=`sysctl -n kern.osreldate`
 
-	fixedre=`echo -n $portaudit_fixed | tr -c '[:alnum:]- \t\n' 'x' | tr -s ' \t\n' '|'`
-	installedre=`$pkg_info -aE | sed -e 's/-[^-]*$//g' | paste -s -d '|' -`
-
-	extract_auditfile | awk -F\| "$PRINTAFFECTED_AWK"'
-		BEGIN { vul=0; fixedre="'"$fixedre"'" }
-		/^(#|\$)/ { next }
-		$2 !~ /'"$opt_restrict"'/ { next }
-		$1 ~ /^FreeBSD[<=>!]/ {
-			if (fixedre && $2 ~ fixedre) next
-			if (!system("'"$pkg_version"' -T \"FreeBSD-'"$osversion"'\" \"" $1 "\"")) {
-				print_affected("FreeBSD-'"$osversion"'", \
-					"To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PREFIX%%/etc/portaudit.conf")
-			}
-			next
-		}
-		$1 ~ /^[^{}*?]*[<=>!]/ {
-			if ($1 !~ "^('"$installedre"')[<=>!]") next;
-		}
-		{
-			cmd="'"$pkg_info"' -E \"" $1 "\""
-			while((cmd | getline pkg) > 0) {
-				vul++
-				print_affected(pkg, "")
-			}
-			close(cmd)
-		}
-		END {
-			if ("'$opt_quiet'" == "false") {
-				print vul " problem(s) in your installed packages found."
-			}
-			if (vul > 0) {
-				if ("'$opt_quiet'" == "false") {
-					print "\nYou are advised to update or deinstall" \
-						" the affected package(s) immediately."
-				}
-				exit(1)
-			}
+	extract_auditfile | findvuln_installed | checknew | \
+	awk -F\| -v pkg_version="$pkg_version" "$PRINTAFFECTED_AWK"'
+BEGIN { vul=0; }
+$1 ~ /^FreeBSD-/ {
+	print_affected($1, $2, $3, $4, "", \
+		"To disable this check add the uuid to \`portaudit_fixed'"'"' in %%PREFIX%%/etc/portaudit.conf")
+	next
+}
+{
+	print_affected($1, $2, $3, $4, $5, "");
+	vul++;
+}
+END {
+	if ("'$opt_quiet'" == "false") {
+		print vul " problem(s) in your installed packages found."
+	}
+	if (vul > 0) {
+		if ("'$opt_quiet'" == "false") {
+			print "\nYou are advised to update or deinstall" \
+				" the affected package(s) immediately."
 		}
+		exit(1)
+	}
+}
 	' || rc=$?
 
 	return $rc
@@ -207,7 +257,7 @@ audit_file()
 				if ($2 !~ /'"$opt_restrict"'/)
 					continue
 				vul++
-				print_affected(pkg, "")
+				print_affected(pkg, $1, $2, $3, "", "")
 			}
 			close(cmd)
 		}
@@ -244,7 +294,8 @@ audit_args()
 				' | $pkg_version -T "$1" -`; then
 				VULCNT=$(($VULCNT+1))
 				echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"'
-					{ print_affected("'"$1"'", "") }
+					{ print_affected("'"$1"'",
+					    $1, $2, $3, "", "") }
 				'
 			fi
 			;;
@@ -277,7 +328,7 @@ audit_cwd()
 			{ print }
 		' | $pkg_version -T "$PKGNAME" -`; then
 		echo "$VLIST" | awk -F\| "$PRINTAFFECTED_AWK"'
-			{ print_affected("'"$PKGNAME"'", "") }
+			{ print_affected("'"$PKGNAME"'", $1, $2, $3, "", "") }
 		'
 		return 1
 	fi
@@ -347,12 +398,13 @@ opt_restrict=
 opt_verbose=false
 opt_version=false
 opt_expiry=
+opt_checknew=
 
 if [ $# -eq 0 ] ; then
 	opt_audit=true
 fi
 
-while getopts aCdf:Fqr:vVX: opt; do
+while getopts aCdf:Fnqr:vVX: opt; do
 	case "$opt" in
 	a)
 		opt_audit=true;;
@@ -364,6 +416,8 @@ while getopts aCdf:Fqr:vVX: opt; do
 		opt_file="$OPTARG";;
 	F)
 		opt_fetch=true;;
+	n)
+		opt_checknew=true;;
 	q)
 		opt_quiet=true;;
 	r)
@@ -423,40 +477,73 @@ fi
 
 prerequisites_checked=false
 
+# This awk code demands 'pkg_version' variable to be set
+# if 'newver' variable is non-empty.
+NEWVERS_AWK='
+
+nparts=split(thepkg[1], verarr, /-/)
+gtglob=verarr[1]
+for (i=2; i<nparts; i++)
+	gtglob=gtglob "-" verarr[i]
+gtglob=gtglob ">" verarr[i]
+
+updated=""
+avail=0
+if (system(pkg_version " -T \"" newver "\" '\''" gtglob "'\''") == 0) {
+	updated="available, " newver
+	avail=1
+} else {
+	updated="not available"
+}
+if (avail != 0) {
+	if (system(pkg_version " -T \"" newver "\" '\''" glob "'\''") == 0)
+		updated=updated ", still vulnerable"
+	else
+		updated=updated ", not vulnerable"
+}
+print "Updated package: " updated
+'
+
 if $opt_quiet; then
 	PRINTAFFECTED_AWK='
-		function print_affected(apkg, note) {
-			print apkg
-		}
-		'
+function print_affected(apkg, glob, refs, descr, newver, note) {
+	print apkg
+}
+'
 elif $opt_verbose; then
 	PRINTAFFECTED_AWK='
-		function print_affected(apkg, note) {
-			split(apkg, thepkg)
-			print "Affected package: " thepkg[1] " (matched by " $1 ")"
-			print "Type of problem: " $3 "."
-			split($2, ref, / /)
-			for (r in ref)
-				print "Reference: <" ref[r] ">"
-			if (note)
-				print "Note: " note
-			print ""
-		}
-		'
+function print_affected(apkg, glob, refs, descr, newver, note) {
+	split(apkg, thepkg)
+	print "Affected package: " thepkg[1] " (matched by " glob ")"
+	print "Type of problem: " descr "."
+	split(refs, ref, / /)
+	for (r in ref)
+		print "Reference: <" ref[r] ">"
+	if (note)
+		print "Note: " note
+	if (newver) {
+'"$NEWVERS_AWK"'
+	}
+	print ""
+}
+'
 else
 	PRINTAFFECTED_AWK='
-		function print_affected(apkg, note) {
-			split(apkg, thepkg)
-			print "Affected package: " thepkg[1]
-			print "Type of problem: " $3 "."
-			split($2, ref, / /)
-			for (r in ref)
-				print "Reference: <" ref[r] ">"
-			if (note)
-				print "Note: " note
-			print ""
-		}
-		'
+function print_affected(apkg, glob, refs, descr, newver, note) {
+	split(apkg, thepkg)
+	print "Affected package: " thepkg[1]
+	print "Type of problem: " descr "."
+	split(refs, ref, / /)
+	for (r in ref)
+		print "Reference: <" ref[r] ">"
+	if (note)
+		print "Note: " note
+	if (newver) {
+'"$NEWVERS_AWK"'
+	}
+	print ""
+}
+'
 fi
 
 if $opt_audit; then
diff --git a/files/portaudit.1 b/files/portaudit.1
index c982b41..5cb0ec2 100644
--- a/files/portaudit.1
+++ b/files/portaudit.1
@@ -42,7 +42,7 @@
 .Sh SYNOPSIS
 .
 .Nm
-.Op Fl aCdFqvV
+.Op Fl aCdFnqvV
 .Op Fl X Ar days
 .Op Fl f Ar file
 .Op Fl r Ar eregex
@@ -85,6 +85,8 @@ Print the creation date of the database.
 Fetch the current database from the
 .Fx
 servers.
+.It Fl n
+Check if new versions of vulnerable ports are present.
 .It Fl q
 Quiet mode.
 .It Fl V
diff --git a/files/portaudit.conf b/files/portaudit.conf
index 4eb08d7..9ea2c4a 100644
--- a/files/portaudit.conf
+++ b/files/portaudit.conf
@@ -17,3 +17,7 @@
 
 # this vulnerability has been fixed in your FreeBSD version
 #portaudit_fixed="d2102505-f03d-11d8-81b0-000347a4fa7d"
+
+# this command will be used to find the vulnerable packages
+# instead of awk(1) script.
+# portaudit_pkg_audit="/usr/sbin/pkg_audit"
diff --git a/pkg-plist b/pkg-plist
index 8edf7bb..3c9f775 100644
--- a/pkg-plist
+++ b/pkg-plist
@@ -1,4 +1,5 @@
 sbin/portaudit
+sbin/portaudit-checknew
 etc/portaudit.conf.sample
 %%PERIODICDIR%%/security/410.portaudit
 @dirrmtry %%PERIODICDIR%%/security
--- portaudit-megapatch_pkg_audit-and-checknew.diff ends here ---
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

Comment 10 Simon L. B. Nielsen 2008-10-05 21:55:40 UTC

On 2008.10.05 20:00:14 +0000, Eygene Ryabinkin wrote:

>  I had noticed that the attached patches are being transformed to one
>  large chunk by a Gnats interface.  Reattaching them properly (I hope ;))

Generally base64/uuencoding is the only way to avoid patch mangling in
GNATS.  Or just put the patch on a webserver.

I haven't had chance to look at the code yet.  I plan to, but I just
don't know when.

-- 
Simon L. Nielsen
FreeBSD Security Team

Miroslav, good day.

Mon, Oct 06, 2008 at 12:41:05AM +0200, Miroslav Lachman wrote:
> I am busy these days, but it is nice to read about your progress. I hope 
> I will get some time to test all of these large patches in a few days 
> and I will report back my experiences!


Fine, thank you!  I am re-CC'ing bug-followup@ to track this letter,
since it contains some useful information that should go into GNATS.

> One note before tests... do -n flag always download new INDEX file, or 
> is it possible to use one already existing in /usr/ports?


Currently, it is downloads bzipped INDEX file to /var/db/portaudit every
time, but it uses mirror mode, so if remote file hadn't changed at all,
all network expences are just the HTTP's HEAD request and reply.

I can add another variable to the portaudit to force the usage of the
existing INDEX file, if it is needed.  By the way, how are you keeping
your INDEX file up to date (your proposed usage of 'pkg_version -I'
implies that you're always rely on it)?  I am just curious -- my INDEX
files are almost always stay unupdated, even if I am using portupgrade.

And there can be another way if one keeps ports tree updated: utility
can use 'make' to determine the version that is currently available on
the examined host.

But downloading the INDEX file from the central server seemed to be the
best way, since it almost always gives one the latest port versions, so
I had implemented this in a first place.

Don't know, however, how the badly the load to the central HTTP server
will be raised.  I am using just two first fields from the INDEX file,
so I can use such a stripped file.  For me, the reduction was about
6x: SIZE(INDEX-7.bz2) = 1126189, SIZE(INDEX-7.stripped.bz2) = 184345.

I am CC'ing the portmgr team.  Guys, could you quickly glance over these
patches and determine if they are useful to the project in large?  If
yes, then may be such a stripped INDEX can be created on the FreeBSD
servers (via cut -f1-2 -d'|' INDEX-N)?

Thanks!
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

As per note of Simon Nielsen, I am putting the patches to the Web and
giving the URLs for them.  Simon, thanks for stepping into this problem!

Patches for pkg_install:
  http://codelabs.ru/fbsd/patches/portaudit/0001-Add-functions-for-traversing-package-database-and-ma.patch
  http://codelabs.ru/fbsd/patches/portaudit/0002-New-utility-pkg_audit.patch

Individual patches for the portaudit:
  http://codelabs.ru/fbsd/patches/portaudit/0001-Avoid-usage-of-global-variables-N-in-the-print_affe.patch
  http://codelabs.ru/fbsd/patches/portaudit/0002-Separate-vulnerable-ports-search-from-the-formatter.patch
  http://codelabs.ru/fbsd/patches/portaudit/0003-Use-pkg_audit-utility-if-it-is-available.patch
  http://codelabs.ru/fbsd/patches/portaudit/0004-Implement-checking-for-a-new-package-versions.patch

And a mega-patch for portaudit that squishes previous four patches
into a single diff:
  http://codelabs.ru/fbsd/patches/portaudit/portaudit-megapatch_pkg_audit-and-checknew.diff
-- 
Eygene

Eygene Ryabinkin wrote:
> Miroslav, good day.
> 
> Mon, Oct 06, 2008 at 12:41:05AM +0200, Miroslav Lachman wrote:
> 
>>I am busy these days, but it is nice to read about your progress. I hope 
>>I will get some time to test all of these large patches in a few days 
>>and I will report back my experiences!
> 
> 
> Fine, thank you!  I am re-CC'ing bug-followup@ to track this letter,
> since it contains some useful information that should go into GNATS.
> 
> 
>>One note before tests... do -n flag always download new INDEX file, or 
>>is it possible to use one already existing in /usr/ports?
> 
> 
> Currently, it is downloads bzipped INDEX file to /var/db/portaudit every
> time, but it uses mirror mode, so if remote file hadn't changed at all,
> all network expences are just the HTTP's HEAD request and reply.
> 
> I can add another variable to the portaudit to force the usage of the
> existing INDEX file, if it is needed.  By the way, how are you keeping
> your INDEX file up to date (your proposed usage of 'pkg_version -I'
> implies that you're always rely on it)?  I am just curious -- my INDEX
> files are almost always stay unupdated, even if I am using portupgrade.

I have '/usr/sbin/portsnap cron' and '/usr/sbin/portsnap -I update' in 
my crontab, so I get INDEX updated every night before nightly security 
e-mail is generated.

> And there can be another way if one keeps ports tree updated: utility
> can use 'make' to determine the version that is currently available on
> the examined host.
> 
> But downloading the INDEX file from the central server seemed to be the
> best way, since it almost always gives one the latest port versions, so
> I had implemented this in a first place.

My previous question was not against your solution, it seems useful to 
have really actual data from the fresh INDEX. It was just a question 
"how it is done". Maybe someone will be happier to use the existing 
INDEX because of traffic on some GPRS internet connection or because of 
the own INDEX creation. (it is not my case, I have all machines as the 
servers with enough connectivity) ;)

> Don't know, however, how the badly the load to the central HTTP server
> will be raised.  I am using just two first fields from the INDEX file,
> so I can use such a stripped file.  For me, the reduction was about
> 6x: SIZE(INDEX-7.bz2) = 1126189, SIZE(INDEX-7.stripped.bz2) = 184345.
> 
> I am CC'ing the portmgr team.  Guys, could you quickly glance over these
> patches and determine if they are useful to the project in large?  If
> yes, then may be such a stripped INDEX can be created on the FreeBSD
> servers (via cut -f1-2 -d'|' INDEX-N)?
> 
> Thanks!

Hello,

On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote:

> But downloading the INDEX file from the central server seemed to be the
> best way, since it almost always gives one the latest port versions, so
> I had implemented this in a first place.

I've been following this, but I don't agree that (port|pkg_)audit should do 
this, from the very perspective you're writing this program from:

On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote:
> 4. I feel that it is Unix-way to do the things: create small utilities
>    that do their (small) job in a proper fashion.

Instead, it can provide installed-pkgname<seperator>pkgorigin output. Then, 
any utility can check whether a new version is available, using what ever 
source it finds relevant.
For example, it is completely irrelevant if a new version is available on the 
FreeBSD servers, when your machine uses a buildserver in a local network. For 
those machines it's relevant whether their build server has a new version and 
one can automatically upgrade if one so desires.
Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're 
again reporting false information.

It's also quite trivial to provide this availibility information in a daily 
security script, for the "majority of cases" and it's better to have tunables 
like _use_remote_portindex, _use_portsdir=/bigdisk/usr/ports in a script.
-- 
Mel

Mel, good day.

Mon, Oct 06, 2008 at 11:24:54AM +0200, Mel wrote:
> On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote:
> > But downloading the INDEX file from the central server seemed to be the
> > best way, since it almost always gives one the latest port versions, so
> > I had implemented this in a first place.
> 
> I've been following this, but I don't agree that (port|pkg_)audit should do 
> this, from the very perspective you're writing this program from:


The download is done not by the portaudit itself, but by the helper
script, portaudit-checknew.

> On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote:
> > 4. I feel that it is Unix-way to do the things: create small utilities
> >    that do their (small) job in a proper fashion.
> 
> Instead, it can provide installed-pkgname<seperator>pkgorigin output. Then, 
> any utility can check whether a new version is available, using what ever 
> source it finds relevant.
>
> For example, it is completely irrelevant if a new version is available on the 
> FreeBSD servers, when your machine uses a buildserver in a local network. For 
> those machines it's relevant whether their build server has a new version and 
> one can automatically upgrade if one so desires.
> Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're 
> again reporting false information.


I hear you, but it seems to me that I should just equip
portaudit-checknew with the other sources of a new ports information and
provide tunables for their location (on-disk path, URL, etc).  I am
planning to do this, but first I want to know if these patches will be
viable for the project: feeding these into the /dev/null or just using
them locally, but equipping with a lot of functionality, is not what I
really want ;))

> It's also quite trivial to provide this availibility information in a daily 
> security script, for the "majority of cases"


Didn't get it, sorry.  Could you, please, elaborate a bit?

> and it's better to have tunables 
> like _use_remote_portindex, _use_portsdir=/bigdisk/usr/ports in a script.


Yes, it was what I had talked about above in this mail.

Thanks for the input!
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

Miroslav,

Mon, Oct 06, 2008 at 10:59:46AM +0200, Miroslav Lachman wrote:
> I have '/usr/sbin/portsnap cron' and '/usr/sbin/portsnap -I update' in 
> my crontab, so I get INDEX updated every night before nightly security 
> e-mail is generated.


Ah, I see.  Thanks!

> > But downloading the INDEX file from the central server seemed to be the
> > best way, since it almost always gives one the latest port versions, so
> > I had implemented this in a first place.
> 
> My previous question was not against your solution, it seems useful to 
> have really actual data from the fresh INDEX. It was just a question 
> "how it is done". Maybe someone will be happier to use the existing 
> INDEX because of traffic on some GPRS internet connection or because of 
> the own INDEX creation. (it is not my case, I have all machines as the 
> servers with enough connectivity) ;)


OK, fine.  I will implement the usage of the local INDEX file in some
days.
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

On Monday 06 October 2008 12:28:48 Eygene Ryabinkin wrote:
> Mel, good day.
>
> Mon, Oct 06, 2008 at 11:24:54AM +0200, Mel wrote:
> > On Monday 06 October 2008 07:23:37 Eygene Ryabinkin wrote:
> > > But downloading the INDEX file from the central server seemed to be the
> > > best way, since it almost always gives one the latest port versions, so
> > > I had implemented this in a first place.
> >
> > I've been following this, but I don't agree that (port|pkg_)audit should
> > do this, from the very perspective you're writing this program from:
>
> The download is done not by the portaudit itself, but by the helper
> script, portaudit-checknew.
>
> > On Sunday 28 September 2008 11:49:18 Eygene Ryabinkin wrote:
> > > 4. I feel that it is Unix-way to do the things: create small utilities
> > >    that do their (small) job in a proper fashion.
> >
> > Instead, it can provide installed-pkgname<seperator>pkgorigin output.
> > Then, any utility can check whether a new version is available, using
> > what ever source it finds relevant.
> >
> > For example, it is completely irrelevant if a new version is available on
> > the FreeBSD servers, when your machine uses a buildserver in a local
> > network. For those machines it's relevant whether their build server has
> > a new version and one can automatically upgrade if one so desires.
> > Similarly, if your /usr/ports is ahead of the FreeBSD's INDEX.bz2, you're
> > again reporting false information.
>
> I hear you, but it seems to me that I should just equip
> portaudit-checknew with the other sources of a new ports information and
> provide tunables for their location (on-disk path, URL, etc).  I am
> planning to do this, but first I want to know if these patches will be
> viable for the project: feeding these into the /dev/null or just using
> them locally, but equipping with a lot of functionality, is not what I
> really want ;))
>
> > It's also quite trivial to provide this availibility information in a
> > daily security script, for the "majority of cases"
>
> Didn't get it, sorry.  Could you, please, elaborate a bit?

Once you have the origin of the port, you can:
- make -C $PORTSDIR/$origin -V PKGNAME
- get the matching origin(s) out of ${INDEXDIR}/${INDEXFILE}
- get the matching origin(s) out of a downloaded INDEX.bz2

This covers the majority of cases.

What portaudit lacks, is providing the origin along with the installed package 
name in easily parseable format. So, a central server wanting to query all 
the machines for vulnerable packages, now has to do an extra step of going 
into $PKG_DBDIR/$pkgname/+CONTENTS and getting the @comment ORIGIN: line, 
while (port|pkg_)audit has just been there.

This would be something I'd expect:
ssh clientmachine "/usr/sbin/pkg_audit -l"
foo-1.2,3:misc/foo
bar-4.5_6:devel/bar
...

-- 
Mel

Mel,

Mon, Oct 06, 2008 at 01:07:51PM +0200, Mel wrote:
> On Monday 06 October 2008 12:28:48 Eygene Ryabinkin wrote:
> Once you have the origin of the port, you can:
> - make -C $PORTSDIR/$origin -V PKGNAME
> - get the matching origin(s) out of ${INDEXDIR}/${INDEXFILE}
> - get the matching origin(s) out of a downloaded INDEX.bz2
> 
> This covers the majority of cases.
> 
> What portaudit lacks, is providing the origin along with the installed package
> name in easily parseable format. So, a central server wanting to query all 
> the machines for vulnerable packages, now has to do an extra step of going 
> into $PKG_DBDIR/$pkgname/+CONTENTS and getting the @comment ORIGIN: line, 
> while (port|pkg_)audit has just been there.
> 
> This would be something I'd expect:
> ssh clientmachine "/usr/sbin/pkg_audit -l"
> foo-1.2,3:misc/foo
> bar-4.5_6:devel/bar
> ...


OK, got it.  There is one neat: pkg_audit should be feeded with the
contents of the auditfile and the latter is located in the tar archive.
So, if you wouldn't mind about the following sequence
-----
tar xf /var/db/portaudit/auditfile.tbz
pkg_audit < auditfile | portaudit-checknew -o | cut -d '|' -f1,4,5
-----
then I can add the flag '-o' to the portaudit-checknew: it will
additionally output the port origin along with the new version.

Is that what you meant?
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

On Monday 06 October 2008 14:22:13 Eygene Ryabinkin wrote:
> Mel,
>
> Mon, Oct 06, 2008 at 01:07:51PM +0200, Mel wrote:
> > On Monday 06 October 2008 12:28:48 Eygene Ryabinkin wrote:
> > Once you have the origin of the port, you can:
> > - make -C $PORTSDIR/$origin -V PKGNAME
> > - get the matching origin(s) out of ${INDEXDIR}/${INDEXFILE}
> > - get the matching origin(s) out of a downloaded INDEX.bz2
> >
> > This covers the majority of cases.
> >
> > What portaudit lacks, is providing the origin along with the installed
> > package name in easily parseable format. So, a central server wanting to
> > query all the machines for vulnerable packages, now has to do an extra
> > step of going into $PKG_DBDIR/$pkgname/+CONTENTS and getting the @comment
> > ORIGIN: line, while (port|pkg_)audit has just been there.
> >
> > This would be something I'd expect:
> > ssh clientmachine "/usr/sbin/pkg_audit -l"
> > foo-1.2,3:misc/foo
> > bar-4.5_6:devel/bar
> > ...
>
> OK, got it.  There is one neat: pkg_audit should be feeded with the
> contents of the auditfile and the latter is located in the tar archive.
> So, if you wouldn't mind about the following sequence
> -----
> tar xf /var/db/portaudit/auditfile.tbz
> pkg_audit < auditfile | portaudit-checknew -o | cut -d '|' -f1,4,5
> -----
> then I can add the flag '-o' to the portaudit-checknew: it will
> additionally output the port origin along with the new version.
>
> Is that what you meant?

What I meant is the '-o' flag in pkg_audit, so I can figure out myself whether 
it's new or not and my buildserver can prioritize it's builds based on 
vulnerable packages it's clients have installed. The origin is the unique key 
that identifies any port, so that's vital information in a pipeline.

-- 
Mel

Mel,

Mon, Oct 06, 2008 at 02:40:48PM +0200, Mel wrote:
> What I meant is the '-o' flag in pkg_audit, so I can figure out myself whether
> it's new or not and my buildserver can prioritize it's builds based on 
> vulnerable packages it's clients have installed. The origin is the unique key 
> that identifies any port, so that's vital information in a pipeline.


Ah, OK: no problems, will do it.
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual   
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook 
    {_.-``-'         {_/            #

Mon, Oct 06, 2008 at 02:30:29PM +0400, Eygene Ryabinkin wrote:
> OK, fine.  I will implement the usage of the local INDEX file in some
> days.

OK, I had implemented both '-o' option to pkg_audit and the usage of the
local INDEX file.  I had reworked pkg_audit and portaudit a bit further,
mostly fixing some issues (both mine and existing).  Here we go.

Patches for pkg_install that adds pkg_audit:
  http://codelabs.ru/fbsd/patches/portaudit/0001-Add-functions-for-traversing-package-database-and-ma.patch
  http://codelabs.ru/fbsd/patches/portaudit/0002-Add-function-match_get_pkgorigin.patch
  http://codelabs.ru/fbsd/patches/portaudit/0003-New-utility-pkg_audit.patch
  http://codelabs.ru/fbsd/patches/portaudit/0004-pkg_audit-add-option-to-print-origins.patch

Mega-patch for pkg_install:
  http://codelabs.ru/fbsd/patches/portaudit/pkg_install-megapatch-pkg_audit.diff

Patches for portaudit:
  http://codelabs.ru/fbsd/patches/portaudit/0001-Avoid-usage-of-global-variables-N-in-the-print_affe.patch
  http://codelabs.ru/fbsd/patches/portaudit/0002-Separate-vulnerable-ports-search-from-the-formatter.patch
  http://codelabs.ru/fbsd/patches/portaudit/0003-Use-pkg_audit-utility-if-it-is-available.patch
  http://codelabs.ru/fbsd/patches/portaudit/0004-Implement-checking-for-a-new-package-versions.patch

Mega-patch for portaudit:
  http://codelabs.ru/fbsd/patches/portaudit/portaudit-megapatch_pkg_audit-and-checknew.diff

Opinions are welcome!
-- 
Eygene

Forgot to say:

Tue, Oct 07, 2008 at 12:30:58AM +0400, Eygene Ryabinkin wrote:
> OK, I had implemented both '-o' option to pkg_audit and the usage of the
> local INDEX file.

The latter can be activated by writing something like
-----
portaudit_pkg_index=3D"file:///usr/ports/INDEX-%d"
-----
to the /usr/local/etc/portaudit.conf.
--=20
Eygene

Gentlemen, good day.

Just a reminder about this PR -- it is already a bit old.  But it is
still viable and kicking on many machines of mine.  I am seeing speedups
=66rom 10x to 26x comparing to the plain portaudit.  Since VuXML database
will only grow, this will be good to consider these patches and (likely)
integrate them into main trees.

Could someone, please, look at the patches?  I had uploaded slightly
modified patches to the old locations.  Most of changes were cosmetic:
whitespace and so on.  No real code was changed.

In the case of suggestions, bugs, etc -- I am all ears.  Thanks!
--=20
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook
    {_.-``-'         {_/            #

Fri, May 01, 2009 at 10:42:21PM +0400, Eygene Ryabinkin wrote:
> Gentlemen, good day.
> 
> Just a reminder about this PR -- it is already a bit old.  But it is
> still viable and kicking on many machines of mine.  I am seeing speedups
> from 10x to 26x comparing to the plain portaudit.  Since VuXML database
> will only grow, this will be good to consider these patches and (likely)
> integrate them into main trees.
> 
> Could someone, please, look at the patches?  I had uploaded slightly
> modified patches to the old locations.  Most of changes were cosmetic:
> whitespace and so on.  No real code was changed.

Hmm, I am going to be a bit nasty this time -- the PR lies for 1.5 years
and no one really looked at it.  Though, Simon and Martin promised to
do so.  If you really don't want this patch to go in -- just say, I'll
try to rework it to suit the project's needs.  But for me it is rediculous
that no one is really interested in speeding up the stuff: number of
installed ports and number of VuXML entries will only grow and the patch
provides great opportunity to keep things very fast for the vast amount
of time.

Sorry for a slightly harsh tone, but I am really disappointed with
the handling of this PR.
-- 
Eygene
 _                ___       _.--.   #
 \`.|\..----...-'`   `-._.-'_.-'`   #  Remember that it is hard
 /  ' `         ,       __.--'      #  to read the on-line manual
 )/' _/     \   `-_,   /            #  while single-stepping the kernel.
 `-'" `"\_  ,_.-;_.-\_ ',  fsc/as   #
     _.-'_./   {_.'   ; /           #    -- FreeBSD Developers handbook
    {_.-``-'         {_/            #

Comment 25 Stanislav Sedov 2009-10-25 15:02:49 UTC

Responsible Changed
From-To: secteam->stas

I'll take it.

Hi!

> > The modified portaudit took the same time. What am I doing wrong ?
> 
> What will the command
> {{{
> time tar -xf /var/db/portaudit/auditfile.tbz -O | pkg_audit > /dev/null
> }}}
> give you?

I now tested with both patches.

fa8# time tar -xf /var/db/portaudit/auditfile.tbz -O | pkg_audit > /dev/null

real    0m0.131s
user    0m0.120s
sys     0m0.014s
fa8# 

Nice. So, what's missing to commit this ?

-- 
pi@opsec.eu            +49 171 3101372                        10 years to go !

Comment 27 Eygene Ryabinkin 2011-01-02 07:50:38 UTC

Kurt, good day.

Sat, Jan 01, 2011 at 09:02:38PM +0100, Kurt Jaeger wrote:
> I seem to need the symlink "b" to get the patch to work on my side -
> is it possible to describe how I might apply the patch without
> the symlink ?


'patch -p1' will help here.

Sat, Jan 01, 2011 at 10:25:31PM +0100, Kurt Jaeger wrote:
> > cc -O2 -pipe  -I/usr/src/usr.sbin/pkg_install/audit/../lib -std=gnu99 -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wcast-align -Wunused-parameter -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wno-pointer-sign -Wformat=2 -Wno-format-extra-args -Werror -c main.c
> > cc1: warnings being treated as errors
> > main.c: In function 'audit_package':
> > main.c:197: warning: cast discards qualifiers from pointer target type
> > *** Error code 1
> > 
> > Stop in /usr/src/usr.sbin/pkg_install/audit.
> > *** Error code 1
> 
> Hmm, either lib/match.c:match_get_pkgorigin should return a
> char * instead of a const char *.
> 
> Fixed that, works now.


Will fix it in my patch too, thanks!

Sat, Jan 01, 2011 at 10:29:06PM +0100, Kurt Jaeger wrote:
> I now tested with both patches.
> 
> fa8# time tar -xf /var/db/portaudit/auditfile.tbz -O | pkg_audit > /dev/null
> 
> real    0m0.131s
> user    0m0.120s
> sys     0m0.014s
> fa8# 
> 
> Nice.


Thanks!

> So, what's missing to commit this ?


I am waiting for Florent Thoumie to review the second version of the
patch for -CURRENT.  He just got a new $JOB, so things are moving slowly.
But I think that the pkg_audit will be committed to the tree.  And after
that I'll be asking secteam@freebsd.org to commit the modifications
to portaudit itself.

I think that pkg_audit will make its way to the 8.x only after release
of 8.2: new utility should be tested on -CURRENT first and then it
will be merged to the -STABLE.

Thanks for testing!
-- 
Eygene Ryabinkin                                        ,,,^..^,,,
[ Life's unfair - but root password helps!           | codelabs.ru ]
[ 82FE 06BC D497 C0DE 49EC  4FF0 16AF 9EAE 8152 ECFB | freebsd.org ]

Comment 28 Thomas Abthorpe 2011-11-05 04:11:26 UTC

Responsible Changed
From-To: stas->freebsd-ports-bugs

- back to the heap

Comment 29 Eitan Adler 2011-11-12 02:34:57 UTC

Responsible Changed
From-To: freebsd-ports-bugs->freebsd-secteam

this one is owned by secteam

Comment 30 Eitan Adler 2011-11-12 02:35:35 UTC

Responsible Changed
From-To: freebsd-secteam->secteam

woops

Comment 31 Eitan Adler 2013-02-04 05:45:05 UTC

State Changed
From-To: open->closed

OBE, we are now at the area of "pkg audit"