Bug 273196 - xargs(1): remove the limitation on the length of strings with replacements
Summary: xargs(1): remove the limitation on the length of strings with replacements
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Only Me
Assignee: freebsd-bugs (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-08-18 05:17 UTC by Yuri Victorovich
Modified: 2023-08-29 14:58 UTC (History)
4 users (show)

See Also:


Attachments
testcase.sh (222 bytes, text/plain)
2023-08-18 05:18 UTC, Yuri Victorovich
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Yuri Victorovich freebsd_committer freebsd_triage 2023-08-18 05:17:53 UTC
This script:
> echo aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | \
> xargs -I '{}' echo "'{}' xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx '{}'"

fails:
> xargs: command line cannot be assembled, too long

The resulting command line has only 3 arguments: 80, 100, and 80 characters.
This is quite a short command line, but xargs(1) fails for some reason.

xargs(1) should be able to assemble commands as long as they can be executed, which is much longer than the above example.


FreeBSD 13.2
Comment 1 Yuri Victorovich freebsd_committer freebsd_triage 2023-08-18 05:18:58 UTC
Created attachment 244183 [details]
testcase.sh
Comment 2 Andrew "RhodiumToad" Gierth 2023-08-18 05:33:25 UTC
The command being assembled has _one_ arg, not three, and that arg is 266 characters long.

"The resulting arguments, after replacement is done, will not be allowed to grow beyond replsize (or 255 if no -S flag is specified) bytes..."

266 is greater than 255 so it fails.
Comment 3 Yuri Victorovich freebsd_committer freebsd_triage 2023-08-18 06:32:42 UTC
Yes, it has one argument.

I think it would be a good improvement to get rid of the -S limit.
Currently xargs's manpage says:
>     -S replsize
>             Specify the amount of space (in bytes) that -I can use for
>             replacements.  The default for replsize is 255.

It isn't clear why should this limit even be applied. The limitation should be the same as the limitation that the system imposes on the command line length that can be run.

It isn't beneficial for xargs(1) to have another, lower limit in case when replacements are done.

I found this problem when xargs failed during tests of devel/util-linux. xargs is used there and the tests run fine on Linux, but they fail on FreeBSD due to this problem.
Comment 4 Andrew "RhodiumToad" Gierth 2023-08-18 07:06:09 UTC
(In reply to Yuri Victorovich from comment #3)

I concur that the limit is silly and should be removed.

What the spec currently requires is that _if_ there is a limit, it must be at least 255 bytes. However, a former version of the spec specified the 255 character limit:

v5 (1997) and v6 (2004): "Constructed arguments cannot grow larger than 255 bytes."

v7 (2008): "Each of these constructed arguments cannot grow larger than an implementation-defined limit greater than or equal to 255 bytes."
Comment 5 Fernando Apesteguía freebsd_committer freebsd_triage 2023-08-18 07:09:32 UTC
^Triage: changing Product.
Comment 6 Fernando Apesteguía freebsd_committer freebsd_triage 2023-08-25 15:29:07 UTC
^Triage: shorten summary.
Comment 7 Fernando Apesteguía freebsd_committer freebsd_triage 2023-08-25 15:53:53 UTC
I can't reproduce this. When I execute the testcase, this is what I get:

~/test$ sh run.sh 
'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx '{}'

The second replstr is not replaced.

If I shorten the match string, then both are replaced.

~/test$ sh run.sh 
'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
Comment 8 Andrew "RhodiumToad" Gierth 2023-08-25 21:02:22 UTC
(In reply to Fernando Apesteguía from comment #7)

see commit f058359ba5 for the change in behavior
Comment 9 Fernando Apesteguía freebsd_committer freebsd_triage 2023-08-29 14:58:36 UTC
(In reply to Andrew "RhodiumToad" Gierth from comment #8)
I see.

So the problem is not that there wasn't a limit before f058359ba5 and after f058359ba5 there is one.

The problem f058359ba5 fixed was that the output of xargs was broken for very long substitution strings and after f058359ba5 an error is reported.