Bug 241710 - please increase ARG_MAX
Summary: please increase ARG_MAX
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: Any Any
: --- Affects Some People
Assignee: freebsd-bugs mailing list
URL:
Keywords: patch
Depends on:
Blocks: 242655
  Show dependency treegraph
 
Reported: 2019-11-04 20:09 UTC by Thierry Thomas
Modified: 2019-12-21 03:08 UTC (History)
4 users (show)

See Also:


Attachments
Bump arg_max (511 bytes, patch)
2019-11-05 22:46 UTC, Pedro F. Giffuni
no flags Details | Diff
Double ARG_MAX (512 bytes, patch)
2019-11-12 23:27 UTC, Pedro F. Giffuni
no flags Details | Diff
Attempt to make ARG_MAX platform dependent (618 bytes, patch)
2019-11-17 04:37 UTC, Pedro F. Giffuni
no flags Details | Diff
Option 1: double ARG_MAX for all platforms (519 bytes, patch)
2019-11-21 00:10 UTC, Pedro F. Giffuni
no flags Details | Diff
Option 2: leave ARG_MAX untouched for IPL32 but increment for LP64 (625 bytes, patch)
2019-11-21 00:16 UTC, Pedro F. Giffuni
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Thierry Thomas freebsd_committer 2019-11-04 20:09:35 UTC
[See also the closed PR 208154]

I'm trying to upgrade the port french/aster to the latest stable
release. This is a complex port: it has no Makefile (in the upstream
tarball, but of course there is a Makefile for the port), and it uses a
combination of setup.py (Python) and a bundled waf.
     
At this point, it builds, but the latest step (linkage) fails with the
following message:
     
gfortran9: fatal error: cannot execute
'/usr/local/libexec/gcc9/gcc/x86_64-portbld-freebsd12.0/9.2.0/collect2':
execv: Argument list too long

Yes, linkage is done by gfortran, in a classical way:
gfortran9 (some -Wl parameters) (a very long list of object files .o) (a
list of several libraies with their paths)

but the problem is not caused neither by gfortran nor by the final
linker: if I execute the same command line manually from my shell, it
succeeds and the aster program is built.

So I guess that the problem is caused partly by the arguments list, but
also by the environment variables brought by the build system.

On my machine, `getconf ARG_MAX' returns 262144.

There's a sysctl kern.argmax, but it's read-only and initialized to ARG_MAX.

On PR 208154, for the same problem with Libreoffice, a work-around was to disable-mergelibs. I'll try to apply the same with Code_Aster, but it's not that easy with their build system (no configure).

Anyway, it would be a good thing to increase this limit, as other OSes do.
Comment 1 Pedro F. Giffuni freebsd_committer 2019-11-05 22:46:58 UTC
Created attachment 208895 [details]
Bump arg_max

I think what Solaris/illumos calls _ARG_MAX32 would be a reasonable default.
Is this enough?

I don't know any alternative solution. :(
Comment 2 Pedro F. Giffuni freebsd_committer 2019-11-12 23:27:09 UTC
Created attachment 209120 [details]
Double ARG_MAX

On further thought: adopting the Solaris/illumos ARG_MAX32 limit is a huge bump: it involves practically multiplying the existing value by four!

It is better to be more conservative: perhaps just double it. I also like the NetBSD notation better. Please confirm this solves the issue.

FWIW, macOS also has the same limit we currently have.
Comment 3 Thierry Thomas freebsd_committer 2019-11-16 18:05:07 UTC
Great! With this patch, `getconf ARG_MAX' returns 524288, and the previous failure in french/aster (linkage) is solved!

Note: at this point I'm not yet ready to upgrade french/aster, and anyway it would fail as long as your patch is not committed and MFC!
Comment 4 Pedro F. Giffuni freebsd_committer 2019-11-16 18:38:46 UTC
I think kib is already CC'd (through PR 208154), but I would like to check with him if the change is OK, since I know he has been working hard to lower the KVA memory use.

Just a heads up: if there is silence in a week time I will commit it.
Comment 5 Konstantin Belousov freebsd_committer 2019-11-16 18:54:44 UTC
(In reply to Pedro F. Giffuni from comment #4)
Did you ever looked at the kernel memory map after your doubling, on KVA-starved architectures ?
Comment 6 Pedro F. Giffuni freebsd_committer 2019-11-16 19:18:52 UTC
(In reply to Konstantin Belousov from comment #5)
No ... I don't have any KVA-starved platform. i386 counts? I can test that.

INHO the default is already high though, so there would be two options:

- Any clean way to change it only for non KVA-starved platforms?
- If you or someone more knowledgeable than me could kindly suggest an acceptable scale factor.
Comment 7 Konstantin Belousov freebsd_committer 2019-11-16 20:02:40 UTC
(In reply to Pedro F. Giffuni from comment #6)
I did not asked for tests (I assumed that this is something not requiring a mention).  I want to see how this change affects KVA allocation for other things, if any.

i386 with 4/4, in 13-CURRENT, is not too starved because it has around 3.5G of KVA total.  I think armv7 platforms are much more at risk there, because they cannot have more that 1G of KVA, but practically provide around 0.5G, I believe.
Comment 8 Pedro F. Giffuni freebsd_committer 2019-11-16 20:28:56 UTC
(In reply to Konstantin Belousov from comment #7)

According to r142039, which was the last time ARG_MAX was bumped (by obrien):

"Note that this results in more kernel virtual memory being reserved for
temporary storage of the args.  The args temporary space is allocated out
of exec_map (a submap of kernel_map).  This will use roughly 4MB of KVM."

So we are talking of about another 4MB.
Comment 9 Konstantin Belousov freebsd_committer 2019-11-16 21:40:45 UTC
(In reply to Pedro F. Giffuni from comment #8)
Could you, please, read some code instead of commit messages, before coming with conclusions ?

The sizing of exec_map is scaled with number of cpus.  It is already pared down on ILP32 due to KVA exhaustion issues.  Even if we assume that the bump is from 4M to 8M (which is not), it is already quite sensitive for low-KVA machines.

On my low-end (8 threads but amd64) desktop machine, I have ~17M reserved for the submap.
Comment 10 Pedro F. Giffuni freebsd_committer 2019-11-17 04:37:44 UTC
Created attachment 209195 [details]
Attempt to make ARG_MAX platform dependent

It seems like we are in the same situation as Illumos/Solaris which added _ARG_MAX32 and _ARG_MAX64 to handle this. Looking at the illumos limits.h header, I came up with the attached patch.

Still untested and probably ugly: is there a better way?
Comment 11 Konstantin Belousov freebsd_committer 2019-11-18 13:28:07 UTC
(In reply to Pedro F. Giffuni from comment #10)
2048 is a strange multiplicator.

But you still ignore the crucial question: does increase cause issues for KVA starved arches.  If it is not, then introducing such gratuitous difference is pointless.  If it is, might be we should bump the size for LP64 much more aggressive.
Comment 12 Pedro F. Giffuni freebsd_committer 2019-11-18 14:43:47 UTC
(In reply to Konstantin Belousov from comment #11)
> (In reply to Pedro F. Giffuni from comment #10)
> 2048 is a strange multiplicator.

I can write it 2 * 1024. I looked around and I noticed the value was a multiple of 1024 on most platforms. It is admitedly an arbitrarily a number between what we have and what Illumos uses for 32 bit archs. Having uncertain multipliers is better than uncertain numbers.

> But you still ignore the crucial question: does increase cause issues for KVA
> starved arches.  If it is not, then introducing such gratuitous difference is 
> pointless.  If it is, might be we should bump the size for LP64 much more 
> aggressive.

I am indeed ignoring the question :(. I admitedly don't know what I am doing here (note that I haven't grabbed the PR), I am just doing an educated guess in the hopes that someone else comes with a real solution.

I understand it would be better to have a unique value for all platforms, I just don't have a KVA-starved platform to test it or sufficient understanding on the kernel to determine it (I am looking at exec_alloc_args_kva() and I see a linked list, beyond that the numbers escape me).  OTOH, I see historic evidence that we don't want to jump such values arbitrarily.

If we are severely KVA limited on non _LP64 platforms, then it makes perfect sense to avoid the bump on those platforms (I doubt we want to run Code Aster on a Raspberry Pi anyways), and Illumos discriminates archs already although with much higher values. I personally don't see a reason to bump ARG_MAX more than absolutely necessary: I just want software to compile and wasting more precious KVA memory doesn't serve any purpose. If we have to revise the value every ten years, so be it: people can always check the ARG_MAX value with getconf and report it.
Comment 13 Pedro F. Giffuni freebsd_committer 2019-11-18 19:27:05 UTC
For the record ...

I am currently testing on an i386 VM with 2G RAM, and after the kernel rebuilds (it's slow), I'll start starving the main memory in the VM.
Comment 14 Konstantin Belousov freebsd_committer 2019-11-18 19:30:43 UTC
(In reply to Pedro F. Giffuni from comment #13)
It is rather useless.  The issue is not a memory, but KVA which is completely different resource.

Also, on i386 we have 4/4 split, which gives the kernel around 3.5G of KVA.  Note above, where I said that i386 is unlikely to note this change.  On the other hand, arm platforms typically have 0.5G KVA or less, and even tens of MB of KVA means a lot for them, often.
Comment 15 Pedro F. Giffuni freebsd_committer 2019-11-18 20:57:34 UTC
(In reply to Konstantin Belousov from comment #14)

Ugh.. yes, illustrating my ignorance on the subject, I confirm memory is unaffected.
Comment 16 Pedro F. Giffuni freebsd_committer 2019-11-21 00:10:39 UTC
Created attachment 209299 [details]
Option 1: double ARG_MAX for all platforms

Attempt to summarise the options:

Option 1: Double ARG_MAX and keep the same value consistently in all archs.

(May break some platforms like ARM7 that are KVA constrained - needs approval by ARM expert).
Comment 17 Pedro F. Giffuni freebsd_committer 2019-11-21 00:16:02 UTC
Created attachment 209300 [details]
Option 2: leave ARG_MAX untouched for IPL32 but increment for LP64

Option 2: discriminate ARG_MAX for different args but greatly increment the value for others.

2X is enough to build Code Aster, but perhaps 3X is a better value(?). Illumos uses ~4X for ILP32 (they don't support ARM) and ~8X for _LP64.
Comment 18 John Baldwin freebsd_committer freebsd_triage 2019-11-21 00:59:41 UTC
Perhaps we could just let the ARM folks figure out if there is a problem and if so they can add an #ifdef?  On a quad-core ARMv7 (e.g. my RPi), doubling ARG_MAX would use an additional 2MB of KVA (256K * (2 * 4 CPUs)).  I don't know if we have any non-terrible ways to see the kernel_map.  I have some gdb scripts that I've only tested on amd64 but do handle submaps.  I thought we had a hack to export the kernel map via pid 0 for procstat -v, but that doesn't seem to work for me.
Comment 19 Konstantin Belousov freebsd_committer 2019-11-21 12:00:21 UTC
(In reply to John Baldwin from comment #18)
We do not have export for kernel map, but in this case it is not needed.  It is enough to manually print the submaps from kgdb, and perhaps to look at the kmem sizing.  Surplus would be to run a stress test to see how much threads can be created on unpatched vs. patched kernel.

I do not want to become a target for accusation of breaking [*] smaller arches.

* Breaking there has very subtle meaning since I do not expect that the boot to singlemode would be broken, but I would be not surprised and even expect that some loads that worked before the change, start breaking after it, sometimes in non-obvious ways.
Comment 20 Thierry Thomas freebsd_committer 2019-11-21 18:17:48 UTC
(In reply to Konstantin Belousov from comment #19)
Please note that with the current limitation, we already have at least a case of breakage caused by this limitation.
Comment 21 Pedro F. Giffuni freebsd_committer 2019-11-24 03:15:53 UTC
(In reply to Thierry Thomas from comment #20)
Hmm ...

While we need a fix for this issue for Code Aster (rather important package in my line of engineering, so yes .. I am biased), we need a solution that we can MFC with confidence that it will not cause stability issues.

So far the least invasive change is Option 2 (but only doubling ARG_MAX), and then let the ARM developers decide when/if they want to match the behaviour and keep consistency in all platforms. Sounds like a plan?
Comment 22 Thierry Thomas freebsd_committer 2019-11-24 10:09:13 UTC
Just a naive question: would it be difficult to make it tuneable via sysctl kern.argmax?
Comment 23 Konstantin Belousov freebsd_committer 2019-11-24 11:39:09 UTC
(In reply to Thierry Thomas from comment #22)
Strings are copied to the KVA allocated from submap which is sized at boot.  If the size of individual allocation can be changed at runtime it could result both in over-provisioning of the submap and in fragmentation.  Both of the problems might cause execve(2) hang waiting for free KVA to proceed.

We might allow a tunable to size the maxargs and correspondingly size submap at boot.  But this still makes the changes to the kernel map, which are now on user.  I just doubt that user can do any reasonable choice there, it requires understanding too much of the kernel layout even to guess what is safe.
Comment 24 Thierry Thomas freebsd_committer 2019-12-15 21:10:12 UTC
Note that PR 242655 is blocked by this one.
Comment 25 Pedro F. Giffuni freebsd_committer 2019-12-15 22:50:21 UTC
(In reply to Thierry Thomas from comment #24)

OK, this has been stuck for a while without a clear way out :-(.

Unless there is an objection I will go for the conservative approach and double the value for LP64 only.
Comment 26 commit-hook freebsd_committer 2019-12-16 23:56:14 UTC
A commit references this bug:

Author: pfg
Date: Mon Dec 16 23:55:31 UTC 2019
New revision: 355828
URL: https://svnweb.freebsd.org/changeset/base/355828

Log:
  Double the size of ARG_MAX on LP64 platforms.

  As modern software keeps growing in size, we get requests to update the
  value of ARG_MAX in order to link the resulting object files. Other OSs
  have much higher values but Increasiong ARG_MAX has a multiplied effect on
  KVA, so just bumping this value is dangerous in some archs like ARM32 that
  can exhaust KVA rather easily.

  While it would be better to have a unique value for all archs, other OSs
  (Illumos in partidular) can have different ARG_MAX limits depending on the
  platform,  For now we want to be really conservative so we are avoidng
  the change on ILP32 and in the alternative case we only double it since that
  seems to work well enough for recent Code Aster.

  I was planning to bump the _FreeBSD_version but it was bumped recently
  (r355798) so we can reuse the 1300068 value for this change.

  PR:		241710
  MFC after:	5 days

Changes:
  head/sys/sys/syslimits.h
Comment 27 commit-hook freebsd_committer 2019-12-21 02:41:07 UTC
A commit references this bug:

Author: pfg
Date: Sat Dec 21 02:40:41 UTC 2019
New revision: 355968
URL: https://svnweb.freebsd.org/changeset/base/355968

Log:
  MFC r355828:
  Double the size of ARG_MAX on LP64 platforms.

  As modern software keeps growing in size, we get requests to update the
  value of ARG_MAX in order to link the resulting object files. Other OSs
  have much higher values but increasiong ARG_MAX has a multiplied effect on
  KVA, so just bumping this value is dangerous in some archs like ARM32 that
  can exhaust KVA rather easily.

  While it would be better to have a unique value for all archs, other OSs
  (Illumos in particular) can have different ARG_MAX limits depending on the
  platform,  For now we want to be really conservative so we are avoidng
  the change on ILP32 and in the alternative case we only double it since that
  seems to work well enough for recent Code Aster.

  Bump the _FreeBSD_version for this change.

  PR:		241710

Changes:
_U  stable/12/
  stable/12/sys/sys/param.h
  stable/12/sys/sys/syslimits.h
Comment 28 commit-hook freebsd_committer 2019-12-21 03:01:10 UTC
A commit references this bug:

Author: pfg
Date: Sat Dec 21 03:01:06 UTC 2019
New revision: 53702
URL: https://svnweb.freebsd.org/changeset/doc/53702

Log:
  Double the size of ARG_MAX on LP64 platforms.

  Document the _FreeBSD_version updates corresponding to this change.

  PR:		241710

Changes:
  head/en_US.ISO8859-1/books/porters-handbook/versions/chapter.xml
Comment 29 Pedro F. Giffuni freebsd_committer 2019-12-21 03:08:06 UTC
OK, the new limit should hold ...

- i386 is being unfairly penalized,
- Still pending for someone with ARM32 to determine if we can re-unify the ARG_MAX value, but that should probably be handled in another PR.