|Summary:||Compiling world fails on /usr/src/contrib/llvm/tools/clang/lib/CodeGen/CGCXXABI.cpp|
|Component:||kern||Assignee:||Dimitry Andric <dim>|
|Status:||In Progress ---|
|Severity:||Affects Only Me||CC:||dim, imp, marklmi26-fbsd, rgrimes, toolchain, tschweikle|
Description tschweikle 2018-08-08 11:54:26 UTC
c++: error: unable to execute command: Killed c++: error: clang frontend command failed due to signal (use -v to see invocation) FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) Target: x86_64-unknown-freebsd12.0 Thread model: posix InstalledDir: /usr/bin c++: note: diagnostic msg: PLEASE submit a bug report to https://bugs.freebsd.org/submit/ and include the crash backtrace, preprocessed source, and associated run script. c++: note: diagnostic msg:
Comment 2 tschweikle 2018-08-08 12:12:43 UTC
Second attachment was not accepted because of size (has 14MB, max. accepted 1MB).
Comment 3 Dimitry Andric 2018-08-11 06:30:39 UTC
Most likely, you are running out of memory. Try lowering your make -j level, or adding some more swap (or memory :).
Comment 4 Rodney W. Grimes 2018-08-11 14:00:18 UTC
(In reply to Dimitry Andric from comment #3) From other issues we have seen on aarch64 and small memory amd64 VM's adding swap is rarely the solution to this, you either have to add memory or reduce -j level.
Comment 5 Warner Losh 2018-08-11 14:15:12 UTC
(In reply to Rodney W. Grimes from comment #4) You need to add non-flash swap typically to make any headway, although reducing -j is more effective. Spinning disks suffer much less from these issues, but are slower. Also, on rpi3 there appears to be a hanging bug in the USB stack which can sometimes be triggered. The arm64 issues stem from crappy nand and bad drivers causing undue (multi-second) delays. But the original poster is running on amd64 where that's not an issue because most people aren't putting swap on crappy SD cards, crappy USB thumb drives and our SATA drivers there are much, much more mature and resilient.
Comment 6 Mark Millard 2018-08-11 16:43:46 UTC
(In reply to Warner Losh from comment #5) While Bob P.'s reports were with large gstat reporting large ms/w and/or ms/r, Trev had made reports of getting the problem without gstat indicating multi-second delays (averages for some time intervals) and having lots of swap still available, starting at: https://lists.freebsd.org/pipermail/freebsd-arm/2018-June/018151.html (But Trev only provided short segments of the records in messages instead of posting the whole logs somewhere. This limits confirmation.) Note to avoid misinterpretation: I am in general agreement with the following as appropriate experiments . . . "You need to add non-flash swap" and "Spinning disks suffer much less from these issues" although I would explicitly add that there are SSD-based USB drives and at least some of these might be about as good as using a direct SATA SSD environment with good SSDs. (Correct me if I'm wrong for some reason.) Notes on testing: Having records from, say, periodic gstat runs, is a good cross check in all cases. Having the information-reporting patches from Mark Johnston are appropriate for testing as well. http://www.zefox.net/~fbsd/rpi3/swaptests/tools/ has various things that have been used in testing. But batchqueue.patch did not pan out and likely should be avoided. Similarly for config_options. The readme is what talks about: sysctl vm.pageout_oom_seq=120 The rest are patches for reporting of additional information during operation. These are likely more important than the gstat ms/w and ms/r records. The context for the patches is recent head, not 11.x .
Comment 7 Mark Millard 2018-08-12 17:50:32 UTC
(In reply to Mark Millard from comment #6) [This is extracted from another context that involved the Pine64+ 2GB.] As of updating to -r337400 the Pine64+ 2GB no longer will boot from the e.MMC on the microsd adapter card. (I switched to tracking fully modern dts use, u-boot, etc.) So I tried a build via a USB SSD as the root file system and swap partition. As reported in: https://lists.freebsd.org/pipermail/freebsd-arm/2018-August/018605.html it failed with an Out Of Memory kill. (More detail there. I did not adjust vm.pageout_oom_seq .) This should have avoided I/O latency problems being involved. (That message is part of a long on-going thread tied to OOM kills, most of the reports involving large I/O latencies being involved.) Very little swap was observed to be in use. This was a context with Mark Johnston's reporting patches in place, but they only reported the kill itself, not waiting for I/O or such. I can not change the Affects Only Me status. Dimitry Andric is possibly the wrong assignee: not the compiler's problem if the system did an Out Of Memory kill of a process. Other bugzilla's are: 227609 230402, not just this 230454.
Comment 8 Mark Millard 2020-03-11 04:45:45 UTC
I'll note that folks have been reporting problems with 1 GiByte machines for -j1 buildworld's based on the compilation of gmock-matchers_test.cc . An example is: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241848 It might be better to make such the base so the information is not as old as here: declare older submittals as duplicates of the modern information example? (llvm materials have grown as well.) Head has also added an paging/swapping I/O related OOM criteria (that was temporarily broken a littel while back). I've not used a 1 GiBYTe context but I use two settings, one of which is for avoiding I/O related kills happening as easilly: # # Delay when persistent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=120 # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: vm.pfault_oom_attempts=-1 I do this on 2 GiByte+ contexts to better avoid OOM kills. vm.pfault_oom_attempts is head-only last I knew but vm.pageout_oom_seq is much older. Some folks have used much larger vm.pageout_oom_seq values.
Comment 9 Mark Millard 2020-03-31 18:48:24 UTC
(In reply to Mark Millard from comment #8) I've tried a 1 GiByte aarch64 context for a -j4 buildworld buildkernel . . . I had a RPi3 that was based on head -r358966 do a build world buildkernel of the same version, from scratch, -j4 style. The RPi3 is a 1 GiByte RAM context. I had 3072 GiBytes for the swap partition. That ,and the ufs file system, were on a USB SSD, not the microsd card. The build completed without any /var/log/message or console output during the build. My modified version of top reported (details copied from a ssh window) . . . For Mem: 738512Ki MaxObsActive, 190608Ki MaxObsWired, 906372Ki MaxObs(Act+Wir) For Swap: 1927Mi MaxObsUsed (top was started before the build. "MaxObs" is short for "Maximum Observed".) The build took a few minutes under 31 hrs. The build used (my PINE64 media are also set up to boot the RPi3, explaining some naming): vm.pageout_oom_seq=120 vm.pfault_oom_attempts=-1 vfs.root.mountfrom="ufs:/dev/gpt/PINE642Groot" dumpdev="/dev/gpt/PINE642Gswp2" /dev/gpt/PINE642Groot / ufs rw,noatime 1 1 /dev/gpt/PINE642Gswap none swap sw 0 0 (So this avoided the microsd card for ufs and swap/page space.) Overall, it looks like having more than 2 GiBytes of swap partition is appropriate for -j4 : 1927 MiByte is not much less than 2048 MiByte. But, with appropriate configuration anyway, the RPi3 can do buildworld buildkernel for head 13, even -j4 style. This was aarch64. armv7 style with 1 GiByte RAM does not allow as much swap/page space without complaining at boot. It does not appear that such a -j4 build would be appropriate for armv7. But I've not investigated what would fit.