Bug 239058 - lex: input() returns EOF instead of 0 on end-of-file
Summary: lex: input() returns EOF instead of 0 on end-of-file
Status: Closed Not A Bug
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.0-STABLE
Hardware: Any Any
: --- Affects Some People
Assignee: Jung-uk Kim
URL:
Keywords:
: 256066 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-07-09 10:21 UTC by Robert Clausecker
Modified: 2021-06-21 20:32 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Clausecker 2019-07-09 10:21:31 UTC
The flex version shipped with FreeBSD as /usr/bin/lex generates an input() function that returns EOF on EOF.  This is contrary to POSIX which mandates that 0 be returned on EOF.
Comment 1 Robert Clausecker 2019-07-09 11:26:46 UTC
This issue was fixed upstream in flex 2.6.1.  Perhaps it's a good idea to update the flex version bundled with FreeBSD.
Comment 2 Mark Johnston freebsd_committer 2020-06-09 13:52:19 UTC
Jung-uk, you did the previous update of flex to 2.5.37 - would you be willing to import the new version?  2.6.4 seems to be the latest.
Comment 3 Jung-uk Kim freebsd_committer 2020-06-10 18:12:44 UTC
Okay, I'll update flex.
Comment 4 Jung-uk Kim freebsd_committer 2020-06-18 18:19:30 UTC
head is updated to 2.6.4.
Comment 5 Jung-uk Kim freebsd_committer 2021-04-07 15:50:55 UTC
Note the change introduced regressions.  Therefore, I had to make the following commit.

https://cgit.freebsd.org/src/commit/?id=6b7e592c215fb76ea027f25030ddc9a697184fbe

If you need POSIX compatibility, please use '-l' option.
Comment 6 Robert Clausecker 2021-04-07 17:15:04 UTC
This does not fully solve the issue.  Option -X should also change the behaviour to be compatible to POSIX as it is documented as “maximal compatibility with POSIX lex.”  Right now, only -l does while -X does not fix the incorrect behaviour.  Please amend the patch so behaviour is correct with -X, too.
Comment 7 Jung-uk Kim freebsd_committer 2021-04-07 17:23:55 UTC
(In reply to Robert Clausecker from comment #6)
Actually, '-X' option does nothing in flex.

https://cgit.freebsd.org/src/tree/contrib/flex/src/main.c#n1492

If you really need this "feature" for the base for some reason, please let me know.  Otherwise, there are many alternatives in the ports tree, e.g., textproc/flex, textproc/reflex.
Comment 8 Robert Clausecker 2021-04-07 17:46:45 UTC
I need this feature for my software to compile and work on base FreeBSD.  I can add workarounds, but I rather didn't.  And FreeBSD should strive to have it for POSIX compliance.  Why have an OS standard if people don't try to implement it?

Yes, -X does nothing right now, but it's the documented place to patch this in.  Note also that the same switch triggered by -X is also triggered by POSIXLY_CORRECT, so it's definitely what should trigger compliant behaviour.

An alternative to rolling back the POSIX compliance change would be to fix the dtrace lexer.  I suppose it might not be too hard.
Comment 9 Robert Clausecker 2021-04-07 17:52:57 UTC
If the FreeBSD team decides that it is not interested in achieving POSIX compliance, this issue should be closed as “Not Accepted” because the problem persists and has most definitely not been fixed.
Comment 10 Jung-uk Kim freebsd_committer 2021-04-07 18:16:36 UTC
(In reply to Robert Clausecker from comment #8)
> I need this feature for my software to compile and work on base FreeBSD.

Please install textproc/flex if you need the POSIX behavior unconditionally.

> Why have an OS standard if people don't try to implement it?

It is a traditional *BSD* behavior and we should not cause regression just to make it compatible with POSIX.  Actually, flex was written to improve AT&T lex and these incompatibilities are well documented in flex(1).  Please read "INCOMPATIBILITIES WITH LEX AND POSIX" section.

> An alternative to rolling back the POSIX compliance change would be to fix the
> dtrace lexer.  I suppose it might not be too hard.

Yes, it is not too hard.  However, I noticed there are many projects relying on the traditional behavior.
Comment 11 Jung-uk Kim freebsd_committer 2021-04-07 18:23:56 UTC
(In reply to Robert Clausecker from comment #9)
> If the FreeBSD team decides that it is not interested in achieving POSIX
> compliance, this issue should be closed as “Not Accepted” because the problem
> persists and has most definitely not been fixed.

We do care about POSIX compliance.  However, it is difficult to pursue both flex improvement and POSIX compliance at the same time.  In fact, that's the reason why the upstream made '-X' option no-op in the first place.  I will mark it as "Not A Bug" for now because it was actually intended behavior.
Comment 12 Jung-uk Kim freebsd_committer 2021-06-21 20:32:23 UTC
*** Bug 256066 has been marked as a duplicate of this bug. ***