[See also the closed PR 208154] I'm trying to upgrade the port french/aster to the latest stable release. This is a complex port: it has no Makefile (in the upstream tarball, but of course there is a Makefile for the port), and it uses a combination of setup.py (Python) and a bundled waf. At this point, it builds, but the latest step (linkage) fails with the following message: gfortran9: fatal error: cannot execute '/usr/local/libexec/gcc9/gcc/x86_64-portbld-freebsd12.0/9.2.0/collect2': execv: Argument list too long Yes, linkage is done by gfortran, in a classical way: gfortran9 (some -Wl parameters) (a very long list of object files .o) (a list of several libraies with their paths) but the problem is not caused neither by gfortran nor by the final linker: if I execute the same command line manually from my shell, it succeeds and the aster program is built. So I guess that the problem is caused partly by the arguments list, but also by the environment variables brought by the build system. On my machine, `getconf ARG_MAX' returns 262144. There's a sysctl kern.argmax, but it's read-only and initialized to ARG_MAX. On PR 208154, for the same problem with Libreoffice, a work-around was to disable-mergelibs. I'll try to apply the same with Code_Aster, but it's not that easy with their build system (no configure). Anyway, it would be a good thing to increase this limit, as other OSes do.
Created attachment 208895 [details] Bump arg_max I think what Solaris/illumos calls _ARG_MAX32 would be a reasonable default. Is this enough? I don't know any alternative solution. :(
Created attachment 209120 [details] Double ARG_MAX On further thought: adopting the Solaris/illumos ARG_MAX32 limit is a huge bump: it involves practically multiplying the existing value by four! It is better to be more conservative: perhaps just double it. I also like the NetBSD notation better. Please confirm this solves the issue. FWIW, macOS also has the same limit we currently have.
Great! With this patch, `getconf ARG_MAX' returns 524288, and the previous failure in french/aster (linkage) is solved! Note: at this point I'm not yet ready to upgrade french/aster, and anyway it would fail as long as your patch is not committed and MFC!
I think kib is already CC'd (through PR 208154), but I would like to check with him if the change is OK, since I know he has been working hard to lower the KVA memory use. Just a heads up: if there is silence in a week time I will commit it.
(In reply to Pedro F. Giffuni from comment #4) Did you ever looked at the kernel memory map after your doubling, on KVA-starved architectures ?
(In reply to Konstantin Belousov from comment #5) No ... I don't have any KVA-starved platform. i386 counts? I can test that. INHO the default is already high though, so there would be two options: - Any clean way to change it only for non KVA-starved platforms? - If you or someone more knowledgeable than me could kindly suggest an acceptable scale factor.
(In reply to Pedro F. Giffuni from comment #6) I did not asked for tests (I assumed that this is something not requiring a mention). I want to see how this change affects KVA allocation for other things, if any. i386 with 4/4, in 13-CURRENT, is not too starved because it has around 3.5G of KVA total. I think armv7 platforms are much more at risk there, because they cannot have more that 1G of KVA, but practically provide around 0.5G, I believe.
(In reply to Konstantin Belousov from comment #7) According to r142039, which was the last time ARG_MAX was bumped (by obrien): "Note that this results in more kernel virtual memory being reserved for temporary storage of the args. The args temporary space is allocated out of exec_map (a submap of kernel_map). This will use roughly 4MB of KVM." So we are talking of about another 4MB.
(In reply to Pedro F. Giffuni from comment #8) Could you, please, read some code instead of commit messages, before coming with conclusions ? The sizing of exec_map is scaled with number of cpus. It is already pared down on ILP32 due to KVA exhaustion issues. Even if we assume that the bump is from 4M to 8M (which is not), it is already quite sensitive for low-KVA machines. On my low-end (8 threads but amd64) desktop machine, I have ~17M reserved for the submap.
Created attachment 209195 [details] Attempt to make ARG_MAX platform dependent It seems like we are in the same situation as Illumos/Solaris which added _ARG_MAX32 and _ARG_MAX64 to handle this. Looking at the illumos limits.h header, I came up with the attached patch. Still untested and probably ugly: is there a better way?
(In reply to Pedro F. Giffuni from comment #10) 2048 is a strange multiplicator. But you still ignore the crucial question: does increase cause issues for KVA starved arches. If it is not, then introducing such gratuitous difference is pointless. If it is, might be we should bump the size for LP64 much more aggressive.
(In reply to Konstantin Belousov from comment #11) > (In reply to Pedro F. Giffuni from comment #10) > 2048 is a strange multiplicator. I can write it 2 * 1024. I looked around and I noticed the value was a multiple of 1024 on most platforms. It is admitedly an arbitrarily a number between what we have and what Illumos uses for 32 bit archs. Having uncertain multipliers is better than uncertain numbers. > But you still ignore the crucial question: does increase cause issues for KVA > starved arches. If it is not, then introducing such gratuitous difference is > pointless. If it is, might be we should bump the size for LP64 much more > aggressive. I am indeed ignoring the question :(. I admitedly don't know what I am doing here (note that I haven't grabbed the PR), I am just doing an educated guess in the hopes that someone else comes with a real solution. I understand it would be better to have a unique value for all platforms, I just don't have a KVA-starved platform to test it or sufficient understanding on the kernel to determine it (I am looking at exec_alloc_args_kva() and I see a linked list, beyond that the numbers escape me). OTOH, I see historic evidence that we don't want to jump such values arbitrarily. If we are severely KVA limited on non _LP64 platforms, then it makes perfect sense to avoid the bump on those platforms (I doubt we want to run Code Aster on a Raspberry Pi anyways), and Illumos discriminates archs already although with much higher values. I personally don't see a reason to bump ARG_MAX more than absolutely necessary: I just want software to compile and wasting more precious KVA memory doesn't serve any purpose. If we have to revise the value every ten years, so be it: people can always check the ARG_MAX value with getconf and report it.
For the record ... I am currently testing on an i386 VM with 2G RAM, and after the kernel rebuilds (it's slow), I'll start starving the main memory in the VM.
(In reply to Pedro F. Giffuni from comment #13) It is rather useless. The issue is not a memory, but KVA which is completely different resource. Also, on i386 we have 4/4 split, which gives the kernel around 3.5G of KVA. Note above, where I said that i386 is unlikely to note this change. On the other hand, arm platforms typically have 0.5G KVA or less, and even tens of MB of KVA means a lot for them, often.
(In reply to Konstantin Belousov from comment #14) Ugh.. yes, illustrating my ignorance on the subject, I confirm memory is unaffected.
Created attachment 209299 [details] Option 1: double ARG_MAX for all platforms Attempt to summarise the options: Option 1: Double ARG_MAX and keep the same value consistently in all archs. (May break some platforms like ARM7 that are KVA constrained - needs approval by ARM expert).
Created attachment 209300 [details] Option 2: leave ARG_MAX untouched for IPL32 but increment for LP64 Option 2: discriminate ARG_MAX for different args but greatly increment the value for others. 2X is enough to build Code Aster, but perhaps 3X is a better value(?). Illumos uses ~4X for ILP32 (they don't support ARM) and ~8X for _LP64.
Perhaps we could just let the ARM folks figure out if there is a problem and if so they can add an #ifdef? On a quad-core ARMv7 (e.g. my RPi), doubling ARG_MAX would use an additional 2MB of KVA (256K * (2 * 4 CPUs)). I don't know if we have any non-terrible ways to see the kernel_map. I have some gdb scripts that I've only tested on amd64 but do handle submaps. I thought we had a hack to export the kernel map via pid 0 for procstat -v, but that doesn't seem to work for me.
(In reply to John Baldwin from comment #18) We do not have export for kernel map, but in this case it is not needed. It is enough to manually print the submaps from kgdb, and perhaps to look at the kmem sizing. Surplus would be to run a stress test to see how much threads can be created on unpatched vs. patched kernel. I do not want to become a target for accusation of breaking [*] smaller arches. * Breaking there has very subtle meaning since I do not expect that the boot to singlemode would be broken, but I would be not surprised and even expect that some loads that worked before the change, start breaking after it, sometimes in non-obvious ways.
(In reply to Konstantin Belousov from comment #19) Please note that with the current limitation, we already have at least a case of breakage caused by this limitation.
(In reply to Thierry Thomas from comment #20) Hmm ... While we need a fix for this issue for Code Aster (rather important package in my line of engineering, so yes .. I am biased), we need a solution that we can MFC with confidence that it will not cause stability issues. So far the least invasive change is Option 2 (but only doubling ARG_MAX), and then let the ARM developers decide when/if they want to match the behaviour and keep consistency in all platforms. Sounds like a plan?
Just a naive question: would it be difficult to make it tuneable via sysctl kern.argmax?
(In reply to Thierry Thomas from comment #22) Strings are copied to the KVA allocated from submap which is sized at boot. If the size of individual allocation can be changed at runtime it could result both in over-provisioning of the submap and in fragmentation. Both of the problems might cause execve(2) hang waiting for free KVA to proceed. We might allow a tunable to size the maxargs and correspondingly size submap at boot. But this still makes the changes to the kernel map, which are now on user. I just doubt that user can do any reasonable choice there, it requires understanding too much of the kernel layout even to guess what is safe.
Note that PR 242655 is blocked by this one.
(In reply to Thierry Thomas from comment #24) OK, this has been stuck for a while without a clear way out :-(. Unless there is an objection I will go for the conservative approach and double the value for LP64 only.
A commit references this bug: Author: pfg Date: Mon Dec 16 23:55:31 UTC 2019 New revision: 355828 URL: https://svnweb.freebsd.org/changeset/base/355828 Log: Double the size of ARG_MAX on LP64 platforms. As modern software keeps growing in size, we get requests to update the value of ARG_MAX in order to link the resulting object files. Other OSs have much higher values but Increasiong ARG_MAX has a multiplied effect on KVA, so just bumping this value is dangerous in some archs like ARM32 that can exhaust KVA rather easily. While it would be better to have a unique value for all archs, other OSs (Illumos in partidular) can have different ARG_MAX limits depending on the platform, For now we want to be really conservative so we are avoidng the change on ILP32 and in the alternative case we only double it since that seems to work well enough for recent Code Aster. I was planning to bump the _FreeBSD_version but it was bumped recently (r355798) so we can reuse the 1300068 value for this change. PR: 241710 MFC after: 5 days Changes: head/sys/sys/syslimits.h
A commit references this bug: Author: pfg Date: Sat Dec 21 02:40:41 UTC 2019 New revision: 355968 URL: https://svnweb.freebsd.org/changeset/base/355968 Log: MFC r355828: Double the size of ARG_MAX on LP64 platforms. As modern software keeps growing in size, we get requests to update the value of ARG_MAX in order to link the resulting object files. Other OSs have much higher values but increasiong ARG_MAX has a multiplied effect on KVA, so just bumping this value is dangerous in some archs like ARM32 that can exhaust KVA rather easily. While it would be better to have a unique value for all archs, other OSs (Illumos in particular) can have different ARG_MAX limits depending on the platform, For now we want to be really conservative so we are avoidng the change on ILP32 and in the alternative case we only double it since that seems to work well enough for recent Code Aster. Bump the _FreeBSD_version for this change. PR: 241710 Changes: _U stable/12/ stable/12/sys/sys/param.h stable/12/sys/sys/syslimits.h
A commit references this bug: Author: pfg Date: Sat Dec 21 03:01:06 UTC 2019 New revision: 53702 URL: https://svnweb.freebsd.org/changeset/doc/53702 Log: Double the size of ARG_MAX on LP64 platforms. Document the _FreeBSD_version updates corresponding to this change. PR: 241710 Changes: head/en_US.ISO8859-1/books/porters-handbook/versions/chapter.xml
OK, the new limit should hold ... - i386 is being unfairly penalized, - Still pending for someone with ARM32 to determine if we can re-unify the ARG_MAX value, but that should probably be handled in another PR.