Bug 206295 - sh(1)/test(1) bug (precedence)
Summary: sh(1)/test(1) bug (precedence)
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 9.3-RELEASE
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-01-15 18:08 UTC by nibbana
Modified: 2016-09-19 05:56 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description nibbana 2016-01-15 18:08:25 UTC
The following demonstrates the bug and the fix.
It appears to be a "hack" in the test(1) code
for some very peculiar exception; the cause
for this exception is unknown ...

This should be fixed regardless of what originally
caused this exception since it is obviously a stupid
and illogical hack to base system utilities (sh(1)/test(1))
that breaks their standard and normally intended usage.

> > If you have authoritative knowledge on the subject,
> > please state if this functionality is correct:
> >
> > $ [ ! "" -a "" ] && echo pass || echo fail
> > pass
> > $ [ ! 11 -a "" ] && echo pass || echo fail
> > pass
> >
> > The "-a" operator binds stronger than the "!" operator.
> > Intuition based on functionality in awk/C would suppose
> > that the "!" operator would bind stronger than the "-a"
> > operator, especially since "-a" does in fact have higher
> > precedence than the "-o" operator, as in awk/C.
> >
> > In order to make it work as "expected", it gets ugly:
> >
> > $ [ ! "" -a "" ] && echo pass || echo fail
> > pass
> > $ [ \( ! "" \) -a "" ] && echo pass || echo fail
> > fail
> >
> > $ [ ! 11 -a "" ] && echo pass || echo fail
> > pass
> > $ [ \( ! 11 \) -a "" ] && echo pass || echo fail
> > fail
> >
> > I never noticed this in 20 years, so I don't know if it always
> > worked this way, or if something changed in my upgrade from
> > 4.11 to 9.3.
>
> --
> REPLY#1
> #######
> I think that while unexpected in comparison to other languages which
> have inherited C-like operator precedence rules, this is according to
> the POSIX standard for test(1).  As the man page says:
>
>      The test grammar is inherently ambiguous.  In order to assure a degree
>      of consistency, the cases described in the IEEE Std 1003.2
>      (``POSIX.2''), section D11.2/4.62.4, standard are evaluated
>      consistently according to the rules specified in the standards
>      document.  All other cases are subject to the ambiguity in the
>      command semantics.
>
> and it notes that the ambiguous cases are those involving (), -a and -o.
>
> Your test might be more clearly expressed as:
>
> $ [ ! "" ] && [ "" ] && echo pass || echo fail
> fail
>
> (although I'd recommend the -z and -n operators for testing the
> emptiness / undefinedness or not of strings.)
>
> --
> REPLY#2
> #######
> I would never claim to be an authority, but a quick look at the test(1)
> code shows that although the comments starting line 53 suggest the
> grammar one would expect, there's a special case in main() for an
> expression of exactly four parts starting with "!". The comment on line
> 218 is relevant to your tests.
>
> 217          if (nargc == 4 && strcmp(*t_wp, "!") == 0) {
> 218       /* Things like ! "" -o x do not fit in the normal grammar. */
> 219                  --nargc;
> 220                  ++t_wp;
> 221                  res = oexpr(t_lex(*t_wp));
> 222          } else
> 223                  res = !oexpr(t_lex(*t_wp));
>
> I presume the shells use a version of this code as well. My guess would
> be that this is some sort of kludge going back to the original bourne
> shell, but someone else more knowledgeable than me will have to deal
> with that.

In regards to Reply#1, yes, I did read that previously, but that leaves
the sensibility of the cited functionality open to whatever is most
sensible.  And the current functionality does not seem sensible due
to its apparent senseless incongruity with standard functionality
in most/all other languages, in addition to making simple expressions
more lengthy/tedious/ugly than necessary.  Of course, these were just
examples to demonstrate the functionality.

In regards to Reply#2 - that sure is odd!

$ [ ! "" -a "" ] && echo pass || echo fail
pass
$ [ ! "" -a "" -a "" ] && echo pass || echo fail
fail

You hit the nail on the head!  This is terrible.
Thank you for making that insightful effort to find the real
reason for this functionality - this sure is a strange exception.
Comment 1 Jilles Tjoelker freebsd_committer 2016-01-22 14:11:30 UTC
(In reply to nibbana from comment #0)
This may not make much sense, but per a literal reading of the 'test' page in POSIX.1-2008 XCU, the -a or -o indeed binds more strongly than the !. This is because -a and -o are explicitly said to be "binary primaries" and therefore '' -a '' matches the first case of 3 arguments and ! '' -a '' matches the first case of 4 arguments.

In bash, dash, zsh and yash, [ ! "" -a "" ] likewise returns true.

I recommend writing [ ! "" ] && [ "" ] or [ -z "" ] && [ -n "" ] instead.
Comment 2 nibbana 2016-09-19 04:35:16 UTC
From the test(1) manpage:

     ! expression  True if expression is false.

NOTE: it is stating that (based on the use of the term 'expression',
that ! should have higher precedence than -a/-o ... there is no
indication that one should beware of any -a/-o operators that follow.
Also, this bizzare behavior only applies to the -a operator, not -o,
so that makes the current implementation doubly strange, and it only
applies to a 1st and not in further instances, eg:

"! expression -a expression -o ! expression -a expression"

$ [ ! "" -a "" ] && echo pass || echo fail
pass
$ [ ! "" -a "" -o ! "" -a "" ] && echo pass || echo fail
fail

     expression1 -a expression2
                   True if both expression1 and expression2 are true.

     expression1 -o expression2
                   True if either expression1 or expression2 are true.

     ( expression )
                   True if expression is true.

     The -a operator has higher precedence than the -o operator.

GRAMMAR AMBIGUITY
     The test grammar is inherently ambiguous.  In order to assure a degree of
     consistency, the cases described in the IEEE Std 1003.2 (``POSIX.2''),
     section D11.2/4.62.4, standard are evaluated consistently according to
     the rules specified in the standards document.  All other cases are sub-
     ject to the ambiguity in the command semantics.

NOTE: unfortunately, these documents are not easily accessible, and users
are, in a practical sense, forced to rely on the manpages.
Comment 3 nibbana 2016-09-19 05:56:06 UTC
I apologize for this somewhat incorrect statement (see below):

"this bizzare behavior only applies to the -a operator, not -o"

--

    the -a or -o indeed binds more strongly than the !.
    This is because -a and -o are explicitly said to be "binary
    primaries" and therefore '' -a ' ' matches the first case of
    3 arguments and !  '' -a '' matches the first case of
    4 arguments.

    In bash, dash, zsh and yash, [ !  "" -a "" ] likewise
    returns true.

    I recommend writing [ !  "" ] && [ "" ] or
    [ -z "" ] && [ -n "" ] instead.

--

Thanks for that reply: the fact that other shells do this is
important, and probably everything is working as it should,
but the manpage doesn't explain any of the logic.

The other justification is vague:

    This is because -a and -o are explicitly said to be "binary
    primaries" and therefore '' -a ' ' matches the first case of
    3 arguments and !  '' -a '' matches the first case of
    4 arguments.

'binary' only signifies that they are operators that require
two expressions and doesn't signify precedence over 'unary'
operators/primaries.  Whatever the case, it's a mess:

$ [ ! "" -a "" -o ! "" -a "" ] && echo pass || echo fail
fail

Accordingly, since the manpage explicitly states that -a has higher
precedence than -o, we should have "pass -o pass" above, but we get
"fail" instead.  At the very least, the manpage should state the
logic behind this ... is it:

(pass -o pass) -a fail = fail?
pass -o (pass -a fail) = fail?

--

$ [ ! "" -a "" -o 11 -a "" ] && echo pass || echo fail
fail

pass -o (pass -a fail) = .....  no, this is not correct.
(pass -o pass) -a fail = fail?  -o has higher precedence than
                                   the 2nd -a ... which violates
                                   the specified manpage rule.

$ [ ! "" -a "" -o "" -o "" ] && echo pass || echo fail
fail

pass -o (fail -o fail) = .....  no, this is not correct.
(pass -o fail) -o fail = .....  no, this is not correct.

wtf???

$ [ ! "" -a "" -o "" -a 11 ] && echo pass || echo fail
fail

pass -o (fail -a pass) = ..... no, this is not correct.
(pass -o fail) -a pass = ..... no, this is not correct.

wtf???

Is there any good reason for this matter to be
such a mess other than "family tradition?"
Did it make code more efficient in 1975?
It appears to be intentionally broken
without any rational basis whatsoever.