The flex version shipped with FreeBSD as /usr/bin/lex generates an input() function that returns EOF on EOF. This is contrary to POSIX which mandates that 0 be returned on EOF.
This issue was fixed upstream in flex 2.6.1. Perhaps it's a good idea to update the flex version bundled with FreeBSD.
Jung-uk, you did the previous update of flex to 2.5.37 - would you be willing to import the new version? 2.6.4 seems to be the latest.
Okay, I'll update flex.
head is updated to 2.6.4.
Note the change introduced regressions. Therefore, I had to make the following commit.
If you need POSIX compatibility, please use '-l' option.
This does not fully solve the issue. Option -X should also change the behaviour to be compatible to POSIX as it is documented as “maximal compatibility with POSIX lex.” Right now, only -l does while -X does not fix the incorrect behaviour. Please amend the patch so behaviour is correct with -X, too.
(In reply to Robert Clausecker from comment #6)
Actually, '-X' option does nothing in flex.
If you really need this "feature" for the base for some reason, please let me know. Otherwise, there are many alternatives in the ports tree, e.g., textproc/flex, textproc/reflex.
I need this feature for my software to compile and work on base FreeBSD. I can add workarounds, but I rather didn't. And FreeBSD should strive to have it for POSIX compliance. Why have an OS standard if people don't try to implement it?
Yes, -X does nothing right now, but it's the documented place to patch this in. Note also that the same switch triggered by -X is also triggered by POSIXLY_CORRECT, so it's definitely what should trigger compliant behaviour.
An alternative to rolling back the POSIX compliance change would be to fix the dtrace lexer. I suppose it might not be too hard.
If the FreeBSD team decides that it is not interested in achieving POSIX compliance, this issue should be closed as “Not Accepted” because the problem persists and has most definitely not been fixed.
(In reply to Robert Clausecker from comment #8)
> I need this feature for my software to compile and work on base FreeBSD.
Please install textproc/flex if you need the POSIX behavior unconditionally.
> Why have an OS standard if people don't try to implement it?
It is a traditional *BSD* behavior and we should not cause regression just to make it compatible with POSIX. Actually, flex was written to improve AT&T lex and these incompatibilities are well documented in flex(1). Please read "INCOMPATIBILITIES WITH LEX AND POSIX" section.
> An alternative to rolling back the POSIX compliance change would be to fix the
> dtrace lexer. I suppose it might not be too hard.
Yes, it is not too hard. However, I noticed there are many projects relying on the traditional behavior.
(In reply to Robert Clausecker from comment #9)
> If the FreeBSD team decides that it is not interested in achieving POSIX
> compliance, this issue should be closed as “Not Accepted” because the problem
> persists and has most definitely not been fixed.
We do care about POSIX compliance. However, it is difficult to pursue both flex improvement and POSIX compliance at the same time. In fact, that's the reason why the upstream made '-X' option no-op in the first place. I will mark it as "Not A Bug" for now because it was actually intended behavior.
*** Bug 256066 has been marked as a duplicate of this bug. ***