Bug 214783 - [patch] awk(1) POSIX character classes never match backslash
Summary: [patch] awk(1) POSIX character classes never match backslash
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 10.1-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: Warner Losh
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2016-11-23 14:00 UTC by martin
Modified: 2021-10-11 16:29 UTC (History)
3 users (show)

See Also:


Attachments
Patch to escape chars in POSIX character classes (673 bytes, patch)
2016-11-23 14:00 UTC, martin
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description martin 2016-11-23 14:00:33 UTC
Created attachment 177332 [details]
Patch to escape chars in POSIX character classes

The commands below should print "yes" but on FreeBSD they print nothing:

echo '\' | awk '/[[:print:]]/ { print "yes" }'

The attached patch fixes this (and also a theoretical problem with matching hyphen).
Comment 1 Pedro F. Giffuni freebsd_committer 2017-10-08 17:41:35 UTC
Warner is collecting awk patches.
Comment 2 Stefan Hagen 2018-08-28 20:07:18 UTC
Hi,

Is there an Update to this? We've just spend a good day with 3 people to figure out why FreeBSD awk is behaving different than all other BSD awks, just to find an unapplied patch from years ago that would solve the problem.

Can we have this one applied, please?

Best Regards,
Stefan
Comment 3 Warner Losh freebsd_committer 2018-08-28 20:15:11 UTC
I didn't apply this patch to the tree because it was one of several patches I staged up, only to find that one of them was causing problems... Not sure it was this one or not, and got pulled away on other things.
Comment 4 Warner Losh freebsd_committer 2021-07-08 01:52:09 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=f39dd6a9784467f0db5886012b3f4b13899be6b8

commit f39dd6a9784467f0db5886012b3f4b13899be6b8
Merge: 7cd22ac43418 746b7396bb3e
Author:     Warner Losh <imp@FreeBSD.org>
AuthorDate: 2021-07-07 23:30:35 +0000
Commit:     Warner Losh <imp@FreeBSD.org>
CommitDate: 2021-07-08 01:25:43 +0000

    one-true-awk: import 20210221 (1e4bc42c53a1) which fixes a number of bugs

    Import the latest bsd-features branch of the one-true-awk upstream:

    o Move to bison for $YACC
    o Set close-on-exec flag for file and pipe redirects that aren't std*
    o lots of little fixes to modernize ocde base
    o free sval member before setting it
    o fix a bug where a{0,3} could match aaaa
    o pull in systime and strftime from NetBSD awk
    o pull in fixes from {Net,Free,Open}BSD (normalized our code with them)
    o add BSD extensions and, or, xor, compl, lsheift, rshift (mostly a nop)

    Also revert a few of the trivial FreeBSD changes that were done slightly
    differently in the upstreaming process. Also, our PR database may have
    been mined by upstream for these fixes, and Mikolaj Golub may deserve
    credit for some of the fixes in this update.

    Suggested by:           Mikolaj Golub <to.my.trociny@gmail.com>
    PR:                     143363,143365,143368,143369,143373,143375,214782
    Sponsored by:           Netflix

 contrib/one-true-awk/ChangeLog                     | 108 +++
 contrib/one-true-awk/FIXES                         | 261 +++++-
 contrib/one-true-awk/REGRESS                       |   4 +
 contrib/one-true-awk/awk.1                         | 159 +++-
 contrib/one-true-awk/awk.h                         |  67 +-
 contrib/one-true-awk/awkgram.y                     |  48 +-
 contrib/one-true-awk/b.c                           | 438 +++++++---
 .../one-true-awk/bugs-fixed/missing-precision.ok   |   2 +-
 contrib/one-true-awk/bugs-fixed/negative-nf.ok     |   2 +-
 contrib/one-true-awk/lex.c                         |  77 +-
 contrib/one-true-awk/lib.c                         | 327 +++++---
 contrib/one-true-awk/main.c                        | 163 ++--
 contrib/one-true-awk/makefile                      |  74 +-
 contrib/one-true-awk/maketab.c                     |  66 +-
 contrib/one-true-awk/parse.c                       |  29 +-
 contrib/one-true-awk/proctab.c                     | 202 ++---
 contrib/one-true-awk/proto.h                       |  41 +-
 contrib/one-true-awk/run.c                         | 918 ++++++++++++++-------
 contrib/one-true-awk/tran.c                        | 164 ++--
 usr.bin/awk/Makefile                               |  10 +-
 20 files changed, 2192 insertions(+), 968 deletions(-)
Comment 5 Warner Losh freebsd_committer 2021-07-08 01:52:45 UTC
and yes, the commit had the wrong bug number in it :(
Comment 6 martin 2021-07-09 17:46:11 UTC
(In reply to Warner Losh from comment #4)

I've reopened this because the update has missed an important part of my patch:

-						if (!adjbuf((char **) &buf, &bufsz, bp-buf+1, 100, (char **) &bp, "relex2"))
+						if (!adjbuf((char **) &buf, &bufsz, bp-buf+2, 100, (char **) &bp, "relex2"))

This is needed to prevent heap buffer overflow, e.g. for 

awk '/[[:cntrl:]01234567[:graph:]]/'

See also https://github.com/onetrueawk/awk/issues/121

I'll make a new PR if you prefer.
Comment 7 Warner Losh freebsd_committer 2021-07-09 18:08:37 UTC
(In reply to martin from comment #6)

Please make a pull request upstream, and link the issue here. While I generally prefer new bugs for this situation, since you've already reopened this one, let's use it to track the issue upstream. If they merge it soon, I'll reimport. If they say they'll merge it but take a while, then I'll make a local change and drop it when it actually gets merged.

Thanks for spotting this!
Comment 8 commit-hook freebsd_committer 2021-07-10 17:09:32 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=666abb0888d277e82c6468851e015798e9a7629f

commit 666abb0888d277e82c6468851e015798e9a7629f
Author:     Warner Losh <imp@FreeBSD.org>
AuthorDate: 2021-07-07 23:30:35 +0000
Commit:     Warner Losh <imp@FreeBSD.org>
CommitDate: 2021-07-10 17:07:26 +0000

    one-true-awk: import 20210221 (1e4bc42c53a1) which fixes a number of bugs

    Import the latest bsd-features branch of the one-true-awk upstream:

    o Move to bison for $YACC
    o Set close-on-exec flag for file and pipe redirects that aren't std*
    o lots of little fixes to modernize ocde base
    o free sval member before setting it
    o fix a bug where a{0,3} could match aaaa
    o pull in systime and strftime from NetBSD awk
    o pull in fixes from {Net,Free,Open}BSD (normalized our code with them)
    o add BSD extensions and, or, xor, compl, lsheift, rshift (mostly a nop)

    Also revert a few of the trivial FreeBSD changes that were done slightly
    differently in the upstreaming process. Also, our PR database may have
    been mined by upstream for these fixes, and Mikolaj Golub may deserve
    credit for some of the fixes in this update.

    Suggested by:           Mikolaj Golub <to.my.trociny@gmail.com>
    PR:                     143363, 143365, 143368, 143369, 143373, 143375, 214783
    Sponsored by:           Netflix

    (cherry picked from commit f39dd6a9784467f0db5886012b3f4b13899be6b8)

 contrib/one-true-awk/ChangeLog                     | 108 +++
 contrib/one-true-awk/FIXES                         | 261 +++++-
 contrib/one-true-awk/REGRESS                       |   4 +
 contrib/one-true-awk/awk.1                         | 159 +++-
 contrib/one-true-awk/awk.h                         |  67 +-
 contrib/one-true-awk/awkgram.y                     |  48 +-
 contrib/one-true-awk/b.c                           | 438 +++++++---
 .../one-true-awk/bugs-fixed/missing-precision.ok   |   2 +-
 contrib/one-true-awk/bugs-fixed/negative-nf.ok     |   2 +-
 contrib/one-true-awk/lex.c                         |  77 +-
 contrib/one-true-awk/lib.c                         | 327 +++++---
 contrib/one-true-awk/main.c                        | 163 ++--
 contrib/one-true-awk/makefile                      |  74 +-
 contrib/one-true-awk/maketab.c                     |  66 +-
 contrib/one-true-awk/parse.c                       |  29 +-
 contrib/one-true-awk/proctab.c                     | 202 ++---
 contrib/one-true-awk/proto.h                       |  41 +-
 contrib/one-true-awk/run.c                         | 918 ++++++++++++++-------
 contrib/one-true-awk/tran.c                        | 164 ++--
 usr.bin/awk/Makefile                               |  10 +-
 20 files changed, 2192 insertions(+), 968 deletions(-)
Comment 9 Piotr Pawel Stefaniak freebsd_committer 2021-10-03 09:21:51 UTC
Warner, do you think we can close this now?
Comment 10 martin 2021-10-11 14:37:26 UTC
Upstream now has a pull request to fix the heap buffer overflow:

https://github.com/onetrueawk/awk/pull/132
Comment 11 Warner Losh freebsd_committer 2021-10-11 16:29:54 UTC
We should pull in the change from upstream once it lands. Given 12.x timelines, I may try to pull it in soon if it isn't resolved in the next couple of days. It's still a problem.