gcc11-11.3.0 build gets stuck in poudriere, seemingly doing nothing. It creates 126 lto1 processes and all of them wait for something. | | | \-+- 09463 nobody gmake -f Makefile -j8 MAKEINFOFLAGS=--no-split bootstrap-lean | | | \-+- 09473 nobody (gmake) | | | \-+- 81022 nobody (gmake) | | | \-+- 10107 nobody (gmake) | | | |-+- 51886 nobody (xg++) | | | | \-+- 51897 nobody (collect2) | | | | \-+- 51900 nobody /usr/local/bin/ld -plugin /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/liblto_plugin.so -plugin-opt=/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper -plugin-opt=-fresolution=/tmp/ccsaUS6c.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh --eh-frame-hdr -m elf_x86_64_fbsd -export-dynamic -dynamic-linker /libexec/ld-elf.so.1 -o f951 /usr/lib/crt1.o /usr/lib/crti.o /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtbegin.o -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs -L/usr/local/lib -L/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc -L/usr/local/x86_64-portbld-freebsd13.1/bin -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs fortran/arith.o fortran/array.o fortran/bbt.o fortran/check.o fortran/class.o fortran/constructor.o fortran/cpp.o fortran/data.o fortran/decl.o fortran/dump-parse-tree.o fortran/error.o fortran/expr.o fortran/interface.o fortran/intrinsic.o fortran/io.o fortran/iresolve.o fortran/match.o fortran/matchexp.o fortran/misc.o fortran/module.o fortran/openmp.o fortran/options.o fortran/parse.o fortran/primary.o fortran/resolve.o fortran/scanner.o fortran/simplify.o fortran/st.o fortran/symbol.o fortran/target-memory.o fortran/convert.o fortran/dependency.o fortran/f95-lang.o fortran/trans.o fortran/trans-array.o fortran/trans-common.o fortran/trans-const.o fortran/trans-decl.o fortran/trans-expr.o fortran/trans-intrinsic.o fortran/trans-io.o fortran/trans-openmp.o fortran/trans-stmt.o fortran/trans-types.o fortran/frontend-passes.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a --library=z li | | | | \-+- 51923 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper @/tmp/cc2byeZq.lto_wrapper_args | | | | \-+- 51935 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/xg++ @/tmp/ccMZdRyD | | | | \-+- 51937 nobody lto1: lto1-wpa (lto1) | | | | |--- 55581 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55583 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55586 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55588 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55592 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55597 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55599 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55600 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55601 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55602 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55603 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55609 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55610 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55611 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55612 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55613 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55614 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55615 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55618 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55619 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55620 nobody lto1: lto1-wpa-streaming (lto1) | | | | \--- 55622 nobody lto1: lto1-wpa-streaming (lto1) | | | |-+- 51888 nobody (xg++) | | | | \-+- 51896 nobody (collect2) | | | | \-+- 51903 nobody /usr/local/bin/ld -plugin /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/liblto_plugin.so -plugin-opt=/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper -plugin-opt=-fresolution=/tmp/ccuGYwnP.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh --eh-frame-hdr -m elf_x86_64_fbsd -Bshareable -o libgccjit.so.0.0.1 /usr/lib/crti.o /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtbeginS.o -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs -L/usr/local/lib -L/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc -L/usr/local/x86_64-portbld-freebsd13.1/bin -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs attribs.o jit/dummy-frontend.o jit/libgccjit.o jit/jit-logging.o jit/jit-recording.o jit/jit-playback.o jit/jit-result.o jit/jit-tempdir.o jit/jit-builtins.o jit/jit-spec.o gcc.o libbackend.a libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a libcommon.a ../libcpp/libcpp.a ../libbacktrace/.libs/libbacktrace.a ../libiberty/pic/libiberty.a ../libdecnumber/libdecnumber.a --library=mpc --library=mpfr --library=gmp --library=z driver-i386.o --version-script /wrkdirs/usr/ports/lang/gcc11/work/gcc-11.3.0/gcc/jit/libgccjit.map -soname libgccjit.so.0 -Bstatic --library=stdc++ -Bdynamic --library=m --library=gcc --library=gcc_eh --library=c --library=gcc --library=gcc_eh /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtendS.o /usr/lib/crtn.o | | | | \-+- 51938 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper @/tmp/ccjfOa88.lto_wrapper_args | | | | \-+- 51940 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/xg++ @/tmp/ccWDpIYQ | | | | \-+- 51941 nobody lto1: lto1-wpa (lto1) | | | | |--- 55561 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55562 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55563 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55564 nobody <defunct> | | | | |--- 55565 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55566 nobody <defunct> | | | | |--- 55567 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55568 nobody <defunct> | | | | |--- 55569 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55570 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55571 nobody <defunct> | | | | |--- 55572 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55573 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55574 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55575 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55576 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55577 nobody <defunct> | | | | |--- 55578 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55579 nobody <defunct> | | | | |--- 55580 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55582 nobody <defunct> | | | | |--- 55584 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55585 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55587 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55589 nobody <defunct> | | | | |--- 55590 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55591 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55593 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55594 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55596 nobody lto1: lto1-wpa-streaming (lto1) | | | | \--- 55598 nobody lto1: lto1-wpa-streaming (lto1) | | | |-+- 51890 nobody (xg++) | | | | \-+- 51898 nobody (collect2) | | | | \-+- 51901 nobody /usr/local/bin/ld -plugin /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/liblto_plugin.so -plugin-opt=/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper -plugin-opt=-fresolution=/tmp/ccFHucKA.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh --eh-frame-hdr -m elf_x86_64_fbsd -export-dynamic -dynamic-linker /libexec/ld-elf.so.1 -o lto1 /usr/lib/crt1.o /usr/lib/crti.o /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtbegin.o -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs -L/usr/local/lib -L/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc -L/usr/local/x86_64-portbld-freebsd13.1/bin -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs lto/lto-lang.o lto/lto.o lto/lto-object.o attribs.o lto/lto-partition.o lto/lto-symtab.o lto/lto-common.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a --library=mpc --library=mpfr --library=gmp --library=z libcommon.a ../libcpp/libcpp.a ../libbacktrace/.libs/libbacktrace.a ../libiberty/pic/libiberty.a ../libdecnumber/libdecnumber.a -Bstatic --library=stdc++ -Bdynamic --library=m --library=gcc --library=gcc_eh --library=c --library=gcc --library=gcc_eh /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtend.o /usr/lib/crtn.o | | | | \-+- 51916 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper @/tmp/cc2ZSUWL.lto_wrapper_args | | | | \--- 51918 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/xg++ @/tmp/ccUq3Dkz | | | |-+- 51892 nobody (xg++) | | | | \-+- 51899 nobody (collect2) | | | | \-+- 51902 nobody /usr/local/bin/ld -plugin /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/liblto_plugin.so -plugin-opt=/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper -plugin-opt=-fresolution=/tmp/ccESamwQ.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh --eh-frame-hdr -m elf_x86_64_fbsd -export-dynamic -dynamic-linker /libexec/ld-elf.so.1 -o lto-dump /usr/lib/crt1.o /usr/lib/crti.o /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtbegin.o -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs -L/usr/local/lib -L/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc -L/usr/local/x86_64-portbld-freebsd13.1/bin -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs lto/lto-lang.o lto/lto-object.o attribs.o lto/lto-partition.o lto/lto-symtab.o lto/lto-dump.o lto/lto-common.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a --library=mpc --library=mpfr --library=gmp --library=z libcommon.a ../libcpp/libcpp.a ../libbacktrace/.libs/libbacktrace.a ../libiberty/pic/libiberty.a ../libdecnumber/libdecnumber.a -Bstatic --library=stdc++ -Bdynamic --library=m --library=gcc --library=gcc_eh --library=c --library=gcc --library=gcc_eh /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtend.o /usr/lib/crtn.o | | | | \-+- 51920 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper @/tmp/cc06rbGO.lto_wrapper_args | | | | \-+- 51922 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/xg++ @/tmp/ccyTHGjo | | | | \-+- 51924 nobody lto1: lto1-wpa (lto1) | | | | |--- 55462 nobody <defunct> | | | | |--- 55463 nobody <defunct> | | | | |--- 55464 nobody <defunct> | | | | |--- 55465 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55466 nobody <defunct> | | | | |--- 55467 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55468 nobody <defunct> | | | | |--- 55469 nobody <defunct> | | | | |--- 55470 nobody <defunct> | | | | |--- 55471 nobody <defunct> | | | | |--- 55472 nobody <defunct> | | | | |--- 55473 nobody <defunct> | | | | |--- 55474 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55475 nobody <defunct> | | | | |--- 55476 nobody <defunct> | | | | |--- 55477 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55478 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55479 nobody <defunct> | | | | |--- 55480 nobody <defunct> | | | | |--- 55481 nobody <defunct> | | | | |--- 55482 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55483 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55484 nobody <defunct> | | | | |--- 55485 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55486 nobody <defunct> | | | | |--- 55487 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55488 nobody <defunct> | | | | |--- 55489 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55490 nobody <defunct> | | | | |--- 55491 nobody <defunct> | | | | \--- 55492 nobody <defunct> | | | |-+- 51927 nobody (xg++) | | | | \-+- 51929 nobody (collect2) | | | | \-+- 51930 nobody /usr/local/bin/ld -plugin /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/liblto_plugin.so -plugin-opt=/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper -plugin-opt=-fresolution=/tmp/cclmHA1C.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh --eh-frame-hdr -m elf_x86_64_fbsd -export-dynamic -dynamic-linker /libexec/ld-elf.so.1 -o cc1 /usr/lib/crt1.o /usr/lib/crti.o /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtbegin.o -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs -L/usr/local/lib -L/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc -L/usr/local/x86_64-portbld-freebsd13.1/bin -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs c/c-lang.o c-family/stub-objc.o attribs.o c/c-errors.o c/c-decl.o c/c-typeck.o c/c-convert.o c/c-aux-info.o c/c-objc-common.o c/c-parser.o c/c-fold.o c/gimple-parser.o c-family/c-common.o c-family/c-cppbuiltin.o c-family/c-dump.o c-family/c-format.o c-family/c-gimplify.o c-family/c-indentation.o c-family/c-lex.o c-family/c-omp.o c-family/c-opts.o c-family/c-pch.o c-family/c-ppoutput.o c-family/c-pragma.o c-family/c-pretty-print.o c-family/c-semantics.o c-family/c-ada-spec.o c-family/c-ubsan.o c-family/known-headers.o c-family/c-attribs.o c-family/c-warn.o c-family/c-spellcheck.o i386-c.o default-c.o cc1-checksum.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a libcommon.a ../libcpp/libcpp.a ../libbacktrace/.libs/libbacktrace.a ../libiberty/pic/libiberty.a ../libdecnumber/libdecnumber.a --library=mpc --library=mpfr --library=gmp --library=z -Bstatic --library=stdc++ -Bdynamic --library=m | | | | \-+- 51982 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper @/tmp/ccZ0X3Y7.lto_wrapper_args | | | | \-+- 51987 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/xg++ @/tmp/ccnZlPjF | | | | \-+- 51988 nobody lto1: lto1-wpa (lto1) | | | | |--- 55495 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55497 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55498 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55499 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55500 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55502 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55504 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55505 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55506 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55507 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55508 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55510 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55512 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55514 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55515 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55519 nobody lto1: lto1-wpa-streaming (lto1) | | | | \--- 55520 nobody lto1: lto1-wpa-streaming (lto1) | | | |-+- 51932 nobody (xg++) | | | | \-+- 51933 nobody (collect2) | | | | \-+- 51936 nobody /usr/local/bin/ld -plugin /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/liblto_plugin.so -plugin-opt=/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper -plugin-opt=-fresolution=/tmp/ccHPswfI.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh --eh-frame-hdr -m elf_x86_64_fbsd -export-dynamic -dynamic-linker /libexec/ld-elf.so.1 -o cc1obj /usr/lib/crt1.o /usr/lib/crti.o /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtbegin.o -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs -L/usr/local/lib -L/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc -L/usr/local/x86_64-portbld-freebsd13.1/bin -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs objc/objc-lang.o objc/objc-act.o hash-table.o objc/objc-runtime-shared-support.o objc/objc-gnu-runtime-abi-01.o objc/objc-next-runtime-abi-01.o objc/objc-next-runtime-abi-02.o objc/objc-encoding.o objc/objc-map.o attribs.o c/c-errors.o c/c-decl.o c/c-typeck.o c/c-convert.o c/c-aux-info.o c/c-objc-common.o c/c-parser.o c/c-fold.o c/gimple-parser.o c-family/c-common.o c-family/c-cppbuiltin.o c-family/c-dump.o c-family/c-format.o c-family/c-gimplify.o c-family/c-indentation.o c-family/c-lex.o c-family/c-omp.o c-family/c-opts.o c-family/c-pch.o c-family/c-ppoutput.o c-family/c-pragma.o c-family/c-pretty-print.o c-family/c-semantics.o c-family/c-ada-spec.o c-family/c-ubsan.o c-family/known-headers.o c-family/c-attribs.o c-family/c-warn.o c-family/c-spellcheck.o i386-c.o default-c.o cc1obj-checksum.o libbackend.a main.o libcommon-target.a libcommon.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a libcommon.a ../libcpp/libcpp.a ../libbacktra | | | | \-+- 51981 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper @/tmp/ccccyoAO.lto_wrapper_args | | | | \-+- 51984 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/xg++ @/tmp/cczm4Pl7 | | | | \-+- 51985 nobody lto1: lto1-wpa (lto1) | | | | |--- 55530 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55531 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55532 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55533 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55534 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55535 nobody <defunct> | | | | |--- 55536 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55537 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55538 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55539 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55540 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55541 nobody <defunct> | | | | |--- 55542 nobody <defunct> | | | | |--- 55543 nobody <defunct> | | | | |--- 55544 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55545 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55546 nobody <defunct> | | | | |--- 55547 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55548 nobody <defunct> | | | | |--- 55549 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55550 nobody <defunct> | | | | |--- 55551 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55552 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55553 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55554 nobody <defunct> | | | | |--- 55555 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55556 nobody lto1: lto1-wpa-streaming (lto1) | | | | |--- 55557 nobody <defunct> | | | | |--- 55558 nobody <defunct> | | | | |--- 55559 nobody lto1: lto1-wpa-streaming (lto1) | | | | \--- 55560 nobody lto1: lto1-wpa-streaming (lto1) | | | \-+- 51944 nobody (xg++) | | | \-+- 51945 nobody (collect2) | | | \-+- 51946 nobody /usr/local/bin/ld -plugin /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/liblto_plugin.so -plugin-opt=/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper -plugin-opt=-fresolution=/tmp/cc0x7OLe.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_eh --eh-frame-hdr -m elf_x86_64_fbsd -export-dynamic -dynamic-linker /libexec/ld-elf.so.1 -o cc1plus /usr/lib/crt1.o /usr/lib/crti.o /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/crtbegin.o -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs -L/usr/local/lib -L/wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc -L/usr/local/x86_64-portbld-freebsd13.1/bin -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/src/.libs -L/wrkdirs/usr/ports/lang/gcc11/work/.build/prev-x86_64-portbld-freebsd13.1/libstdc++-v3/libsupc++/.libs cp/cp-lang.o c-family/stub-objc.o cp/call.o cp/class.o cp/constexpr.o cp/constraint.o cp/coroutines.o cp/cp-gimplify.o cp/cp-objcp-common.o cp/cp-ubsan.o cp/cvt.o cp/cxx-pretty-print.o cp/decl.o cp/decl2.o cp/dump.o cp/error.o cp/except.o cp/expr.o cp/friend.o cp/init.o cp/lambda.o cp/lex.o cp/logic.o cp/mangle.o cp/mapper-client.o cp/mapper-resolver.o cp/method.o cp/module.o cp/name-lookup.o cp/optimize.o cp/parser.o cp/pt.o cp/ptree.o cp/rtti.o cp/search.o cp/semantics.o cp/tree.o cp/typeck.o cp/typeck2.o cp/vtable-class-hierarchy.o attribs.o incpath.o c-family/c-common.o c-family/c-cppbuiltin.o c-family/c-dump.o c-family/c-format.o c-family/c-gimplify.o c-family/c-indentation.o c-family/c-lex.o c-family/c-omp.o c-family/c-opts.o c-family/c-pch.o c-family/c-ppoutput.o c-family/c-pragma.o c-family/c-pretty-print.o c-family/c-semantics.o c-family/c-ada-spec.o c-family/c-ubsan.o c-family/known-headers.o c-family/c-attribs.o c-family/c-wa | | | \-+- 51989 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper @/tmp/ccieVE2K.lto_wrapper_args | | | \-+- 51991 nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/xg++ @/tmp/cczeSj6Z | | | \-+- 51992 nobody lto1: lto1-wpa (lto1) | | | |--- 55651 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55652 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55653 nobody <defunct> | | | |--- 55654 nobody <defunct> | | | |--- 55655 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55656 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55657 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55658 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55659 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55660 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55661 nobody <defunct> | | | |--- 55662 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55663 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55664 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55665 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55666 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55667 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55668 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55669 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55670 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55671 nobody <defunct> | | | |--- 55672 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55673 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55674 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55675 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55676 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55677 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55678 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55679 nobody lto1: lto1-wpa-streaming (lto1) | | | |--- 55680 nobody <defunct> | | | \--- 55681 nobody lto1: lto1-wpa-streaming (lto1)
My guess is that is due to the high amount of resources needed to build GCC with LTO_BOOTSTRAP enabled, see also pkg-help and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264949 . I suggest you to disable LTO_BOOTSTRAP. I CC pkubaj who is the author of commit https://cgit.freebsd.org/ports/commit/?id=2f6be9fb8281fbffc66e18608bcfc4df0043c613 and probably has a better understanding of the topic.
(In reply to Lorenzo Salvadore from comment #1) It creates too many processes which each have size ~1GB. So total memory for all of them would be ~130GB and they keep competing with each other for memory. Build should only create up to NCPU parallel processes. I have 8 CPUs but it creates 130 processes - this kills the machine.
I think the LTO_BOOTSTRAP should be first disabled, then fixed, and then re-enabled again. Otherwise it would hurt a lot of people.
It was suggested "I suggest you to disable LTO_BOOTSTRAP" and I'd 2nd that. But this note goes in a different direction, not so limited to lang/gcc* despite it being a driving example. I see 7 separate /usr/local/bin/ld commands running at once, each being an example of LTO processing. But I generalize below to principles beyond just lang/gcc* . lang/gcc* are far from being the only resource intensive port builds. The FreeBSD build servers avoid this for port building by using not only ALLOW_MAKE_JOBS=yes in /usr/local/etc/poudriere.conf but also in the *make.conf that it uses in/for poudriere: # Build ALLOW_MAKE_JOBS_PACKAGES with 2 jobs MAKE_JOBS_NUMBER=2 That makes the maximum make jobs 2*(number of builders). Without the MAKE_JOBS_NUMBER (or analogous) but using ALLOW_MAKE_JOBS=yes the number of make jobs is: (number of freebsd cpus)*(number of builders) And if the number of builders is set to the number freebsd cpus, the number of make jobs can the square of the freebsd cpu count. FreeBSD build servers are configured to avoid such. In fact they commonly configure for having: 2*(number of builders) < (number of freebsd cpus) You can configure your build context in such ways as well. It would make these sorts of issues less likely. FreeBSD is unlikely to make things such that you do not need to deal with such configuration of your environment. I'll note that . . . However, various ports have steps that create more processes and/or use multi-threaded processes beyond this, outside of the poudriere's control and even outside make's control. There is no global control on the load average that will/can occur during port builds, even if one sets up only one builder and avoids ALLOW_MAKE_JOBS=yes . This is part of why the FreeBSD build servers use MAKE_JOBS_NUMBER=2 and: 2*(number of builders) < (number of freebsd cpus) There are also questions relative to memory use, such as if you are using USE_TMPFS=data or USE_TMPFS=no vs. some other value. All the other values lead to various examples of huge amounts of memory-space use by tmpfs for some ports. (There is a poudriere mechanism for making only specific ports avoid tmpfs use but I do not use it.) Personally I use USE_TMPFS=data and ALLOW_MAKE_JOBS=yes and as many builders as freebsd cpus (for example, 32 or 16), no use of the likes of MAKE_JOBS_NUMBER. But I configured everything for high load average activity, including having a mean of 4 GiBytes of RAM per freebsd cpu on the machines used for such builds. I also have SwapSpace == 3.6*RAM or so and fairly fast storage space media (including for swapping/paging to/from swap space). No spinning rust. My builds have not produced examples of "this kills the machine" for how I've configured things. On very rare occasions I test doing "bulk -a -c". I also have some system tailoring: in /boot/loader.conf : vm.pageout_oom_seq=120 in /etc/sysctl.conf : vm.swap_enabled=0 in /etc/sysctl.conf : vm.swap_idle_enabled=0 I'll not get into why here. I do some experimentation with an environment with a mean of only 2GiBytes of RAM per FreeBSD cpu, but only 4 such cpus. This has not had problems yet for my poudriere use.
(In reply to Yuri Victorovich from comment #3) While I also would like to disable LTO by default, others consider that LTO should be enabled by default so that official packages are built as optimized as possible, while users building their own ports should just disable the option if they are unhappy with it. At the moment the approach I have taken is leaving LTO on by default but warning about it as much as possible: I have created a pkg-help file in all relevant ports, I am informing on IRC and I am adding a note to the latest gcc quarterly report ( https://github.com/freebsd/freebsd-quarterly/pull/489 , I am going to update it with your case). And of course I am tracking bugzilla, as you have seen. Now, if I have understood correctly, your issue is not a bug, but it is instead the consequence of an optimization which requires a lot of resources (too much) in your particular configuration (not shared by the package builders, which are able to compile gcc11 successfully), is that right? If it is a bug, we need to fix it. If it is not, I am sorry, but you need to tweak your configuration: if you happen to do anything more special than disabling LTO_BOOTSTRAP to do so, please let me know so that I can inform other users about it.
(In reply to Lorenzo Salvadore from comment #5) > not shared by the package builders, which are able to compile gcc11 successfully Comment #4, in part, has notes related to those builder's use of MAKE_JOBS_NUMBER and a limited number of builders that are allowed at once on the build machines. The structure of the techniques that the build machines use are available for others to do similarly if they want.
(In reply to Lorenzo Salvadore from comment #5) > While I also would like to disable LTO by default, others consider that LTO should be enabled by default so that official packages are built as optimized as possible, while users building their own ports should just disable the option if they are unhappy with it. The problem is not that LTO is "very resource consuming". The problem is that GCC mal-parallelized the build and it is spawning exceedingly many processes. Such mal-parallelization is a pitfall of GNU Make. All one has to do is to use -jN at multiple levels of the directory hierarchy and similar effect would be achieved. They should fix the bug first before others use this feature. I am not sure why this buggy feature was enabled in the FreeBSD port in the first place.
LTO should be enabled only after it is fixed.
(In reply to Yuri Victorovich from comment #8) I've built the lang/gcc* ports many times with LTO_BOOTSTRAP, including lang/gcc11 . The FreeBSD build servers do as well. It is not broken, just very resource intensive that requires explicit management to limit the resource use.
(In reply to Mark Millard from comment #9) > It is not broken, just very resource intensive that > requires explicit management to limit the resource > use. It is broken. It creates 130+ processes when only 8 were requested.
(In reply to Mark Millard from comment #9) FYI from a past LTO-BOOTSTRAP build of lang/gcc12-devel (no other builders active) on a 8 GiByte RPi4B, so 4 cores: ---Begin OPTIONS List--- ===> The following configuration options are available for gcc12-devel-12.0.1.s20220306_2: GRAPHITE=off: Support for Graphite loop optimizations ====> Options available for the radio BOOTSTRAP: you can only select none or one of them LTO_BOOTSTRAP=on: Build using a full LTO bootstrap STANDARD_BOOTSTRAP=off: Build using a full bootstrap without LTO ===> Use 'make config' to modify these settings ---End OPTIONS List--- [21:06:37] [01] [21:06:11] Finished lang/gcc12-devel | gcc12-devel-12.0.1.s20220306_2: Success It is a UFS context, not ZFS. (ZFS gets into more competing for memory, something I forgot to mention as a tradeoffs in setting things up.) I had 30720Mi swap set up but, around 2 GiBytes of it was used at one point, not more. I've also built lang/gcc11 in the past but not have old notes for the details.
(In reply to Yuri Victorovich from comment #10) There is no way to request a global limit on the load average in/for poudriere or other forms of building ports. You got 7 of 8 make jobs in what you show: 7instances of xg++ started by make in what you show. You requested allowing 8 make jobs, not a load average of 8, because there is no way to directly request the load average constraint.
(In reply to Mark Millard from comment #12) Since the subject came up: that lang/gcc12-devel build had maximum observed load averages of: 18.00, 9.87, 6.81 (1min) (5min) (15min) but there were 4 FreeBSD cpus (4 cores). One thing I have never gotten that you got was the large number of <defunct> processes. But I've no clue of anything potentially useful to say about that part of what you got. [I have a patched top that records and reports various "MaxObs" (maximum observed) figures.]
(In reply to Mark Millard from comment #12) > There is no way to request a global limit on the load average [...] No. But there's always a way to fix makefiles such that they would respect the -jN setting.
(In reply to Mark Millard from comment #13) I forgot one of the numbers, but not a load average: 39 MaxObsRunning on a 3 FreeBSD cpu (4 core) system.
(In reply to Yuri Victorovich from comment #14) The Makefile did respect the -j8: it only started the: nobody (xg++) nobody processes, and there were only 7 of those in the evidence that you gave. Each nobody (xg++) [not make!] in turn started: nobody (collect2) This is outside make's control. This is gcc11 materials that are doing such, not make. Each nobody (collect2) [not make!] in turn started a: nobody /usr/local/bin/ld This is outside make's control. Each /usr/local/bin/ld [not make!] in turn started a: nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/lto-wrapper This is outside make's control. Each of those [not make!] in turn started a: nobody /wrkdirs/usr/ports/lang/gcc11/work/.build/./prev-gcc/xg++ @/tmp/* This is outside make's control. Each of those [not make!] in turn started: nobody lto1: lto1-wpa (lto1) This is outside make's control. Each of those [not make!] in turn started a bunch of: nobody lto1: lto1-wpa-streaming (lto1) This is outside make's control. There is a lot not under make's control: things internal to how gcc11 materials operate.
(In reply to Mark Millard from comment #16) > The Makefile did respect the -j8 No. It started 130+ processes instead.
Makefiles should be written in such way that they respect -jN. If this is too difficult - there's always an option to switch to cmake or meson. Both tools nicely respect NCPU by themselves. There's no need to even ever think about it.
(In reply to Yuri Victorovich from comment #17 and #18) There are no instances gmake below any "nobody (xg++)" in your description text. Below xg++, gmake was not involved in any of the subprocesses trees in what you provided. There are only 7 examples of "nobody (xg++)". There are only 4 instances of gmake in that description text and they are all above the "nobody (xg++)" in the process parent/child tree that you present. The vast majority of the processes shown were not created by gmake but instead by otehr processes in teh parent/child tree that you provided. I do not think I can make it any clearer than that about which processes created which other processes --and so which were not created by gmake. I do not plan any more notes on this subject.
So there's some discussion where people are seemingly talking at different levels. Yuri is observing an exploding number of jobs, and if some parent is swapped out under memory pressure - and possibly with ZFS filling up memory and/or disks getting very slow - then the parent process can't collect the children that have exited, hence many <defunct> zombies. Grim process reaper caught up in a traffic jam if you will. Then some of the GCC drivers seem to be trying to interact with the GNU make jobservers to avoid that, but either this is not configured (port but, either upstream or FreeBSD's port) or not working in a bootstrap => someone could investigate that and I am not sure off-hand if it pertains to GCC11. I recall that GCC has a hefty discussion when it was introducing LTO that it going slow because you have, say, 16 processes generate code and the LTO link just run one because at that time (again, not sure which linker exactly) the assumption still was that linking is serial, disregarding the "LTO global optimizer and code generation" phases. Then there are fat and thin LTO variants... So it is a complex matter and GCC11 is not the only offender, apparently mongodb50 was recently told not to use LTO in FreeBSD's port factory setting, and there is more. I frequently see my builds killed because my many-GB 8-core 2-threads-per-core Ryzen 1700 fills up the disk during compiler builds, and I FREQUENTLY see multiple compilers competing in poudriere. It's one or two LLVMs, one GCC, and Rust at the same time, and then I usually fill up the disk. Especially if anything generates debugging information intermediately without my setting WITHOUT_DEBUG because someone thought it wise to have -flto -g or something. Python 3.8 did the latter on my mailserver and could not build with 1 GB RAM + 1 GB swap, but the Python port has since been fixed. So yes, arguably my own FreeBSD builer VM should have more disk space ;-)
sorry for the unusual typo quote, I accidentally clicked save changes before proofreading, so here is my same comment written in an intelligible way: ----- So there's some discussion where people are seemingly talking at different levels. Yuri is observing an exploding number of jobs, and if some parent is swapped out under memory pressure - and possibly with ZFS filling up memory and/or disks getting very slow - then the parent process can't collect the children that have exited, hence many <defunct> zombies. Grim process reaper caught up in a traffic jam if you will. Then I have seen some of the GCC drivers (not sure which versions) that attempt to interface with the GNU make jobservers to avoid that "nested GNU make process explosion", but either this is not configured (in the port, either upstream GCC or FreeBSD's port), or it is not working in a bootstrap => someone could investigate that and I am not sure off-hand if it pertains to GCC11, or if GCC11 did not have that interface. I recall that GCC used to have a hefty discussion when it was introducing LTO that it was going slow because there were 16 paperweight compilers that generate intermediate code and then linker ran single-threadedly because at that time (again, not sure which linker exactly) the assumption still was that linking is serial, disregarding the "LTO global optimizer and code generation" phases, when LLVM/clang were parallelizing the LTO link, too. I think this has been fixed in GCC since. Then there are fat and thin LTO variants... So it is a complex matter and GCC11 is not the only offender, apparently mongodb50 was recently told not to use LTO in FreeBSD's port factory setting, and there is more. I frequently see my builds killed because my many-GB 8-core 2-threads-per-core Ryzen 1700 fills up the disk during compiler builds, and I FREQUENTLY see multiple compilers competing in poudriere. It's one or two LLVMs, one GCC, and Rust at the same time, and then I usually fill up the disk. Especially if anything generates debugging information intermediately without my setting WITHOUT_DEBUG because someone thought it wise to have -flto -g or something. Python 3.8 did the latter on my mailserver and could not build with 1 GB RAM + 1 GB swap, but the Python port has since been fixed. So yes, arguably my own FreeBSD builer VM should have more disk space ;-) But still the enormous LTO resource hunger needs to be addressed in the ports tree as a whole.
(In reply to Matthias Andree from comment #21) Filling up disks: just rust uses 17+ GiBytes of temporary disk space when tmpfs is largely avoided. Otherwise RAM+SWAP ends up covering that 17+ GiBytes. I set up my environments for building such that having 10 or 20 or more examples of such builders going at once would not wipe out the free space intended for use. I avoid configurations that try to cover such via tmpfs use. And, yes, it is not uncommon for me to end up with 4+ compiler toolchains building at once. Note: For UFS, I consider most of the free space as available for use. But, for ZFS, I try to always keep 25%+ free at all times, per the notes in the book The Design and Implementation of the FeeBSD Operating Stystem 2nd Ed. about ZFS performance characteristics (based on its design).
As it can be read in the bug Yuri has reported upstream, the bug is confirmed. Now we should agree on a strategy to deal with it while we wait for an upstream resolution. Yuri suggested disabling LTO for lang/gcc11. It makes sense since there are cases that stucks the buil. However, I don't see those cases so common. Until now, Yuri is the only one who has noticed the issue (while LTO is quite new on default gcc version on FreeBSD, it has already been around for a long time in other systems, but it seems Yuri has reported the first bug upstream on the topic), so I think the issue does not affect most of our users. Moreover, we know it does not affect the official packages builders either. I think we should choose between these three strategies (but more ideas are welcome of course): 1 - Leave LTO enabled by default on all gcc ports having it. 2 - Disable LTO on all gcc ports having it. 3 - Leave LTO enabled on some ports and disable it on others. In that case, I would suggest to leave it enabled on -devel ports and disable it on gcc11 and gcc12. Moreover I believe we should update pkg-help to inform about this bug for all gcc ports where the LTO_BOOTSTRAP option exists: an additional warning is always better. Personally, I prefer strategy 3. I see the following advantages: - gcc11 (default version) builds flawlessly; - disabling LTO requires a PORTREVISION bump, but the gcc11 and gcc12 ports with standard bootstrap (which I would enable by default instead of lto bootstrap) are quick enough to compile; - leaving LTO enabled on -devel ports allows us to monitor lto state, notice bugs, fix them and then better understand when LTO is ready to be enabled on the production ports. Please let me know your thoughts about it.
(In reply to Lorenzo Salvadore from comment #23) No one has to wait for any changes --if they are willing to use the OPTIONS to choose to avoid LTO_BOOTSTRAP or do other things like the FreeBSD build servers do. I'll note that my own reports here indicated a 1 min. load average of 18 for a 4 FreeBSD-cpu environment, somewhat more than 4*4. Not as extreme but still notable. But I knew about the behavior up front and my configurations are set up to deal with such high load average contexts. Note that GCC may be configurable: The confirmations in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328 comments 2..4 by Richard Biener report: QUOTE GCCs WPA stage fork()s to write out LTRANS IL object files in parallel - those processes are not controlled by make END QUOTE But it also reports a "--param lto-max-streaming-parallelism" that could be controlled now, without waiting for gcc changes, although I've never experimented with such.
(In reply to Mark Millard from comment #24) Hmm. Looking around, apparently it is important to use BOOT_CFLAGS in order to avoid the option being stripped. QUOTE (from https://forums.gentoo.org/viewtopic-t-1104752-start-0.html ) Try adding `--param lto-max-streaming-parallelism=3` to BOOT_CFLAGS END QUOTE The later response was: QUOTE I am compiling again right now and BOOT_CFLAGS was not stripped as the others, so my options are passing through END QUOTE
*** Bug 268148 has been marked as a duplicate of this bug. ***
I assign the bug to me since in the meantime I have adopted all GCC ports of supported versions. I also rename the bug with a more precise summary, taken from the bug report submitted upstream by Yuri. Please note that the upstream commit referenced in https://gcc.gnu.org/bugzilla/show_bug.cgi?id=106328#c6 should have fixed the bug in upcoming GCC 13. I am still unsure about what to do with the LTO_BOOTSTRAP option while we wait for GCC 13. Since the build seems to work fine on the official package builders and people compiling their own ports have the option to disable it, I think I am going to leave things as they are and work on other higher priority issues. In the meantime the bug stays open, as a reference for anyone getting into the same issue (so I also understand how many they are and thus maybe reconsider my priorities) and as a place for discussion in case some new elements suggest that it is necessary to disable LTO_BOOTSTRAP option by default.
(In reply to Lorenzo Salvadore from comment #27) The later notes indicate for BSD: QUOTE And IIRC BSD 'make' is not GNU make but I think gmake is available from the ports repo. The documentation about -flto=jobserver mentions that already. END QUOTE I do not expect that they are making any claims about arbitrary variations of make. Will ports use GNU make for building lang/gcc13 ?
(In reply to Mark Millard from comment #28) > Will ports use GNU make for building lang/gcc13 ? Yes, indeed we are already using GNU make to build the GCC ports of all the supported versions (and I guess also the older ones that we still have in the ports tree, but I have not checked): all of them have USES=gmake.
(In reply to Lorenzo Salvadore from comment #29) Cool. It will be interesting to see what Yuri's context gets with the changes for gcc13. The environments that I've access to cleared the I/O fast enough to avoid ending up with an accumulating backlog of processes/threads for the I/O when/where I did LTO-based-gcc-build experiments, at least at the time. My testing would likely not show much as it never showed the problem in the first place.
(In reply to Mark Millard from comment #30) Test can already be done using gcc13-devel, if anyone wants to try it. The snapshots already include the resolving commit.
(In reply to Lorenzo Salvadore from comment #31) Using poudriere and with the LTO_BOOTSTRAP option enabled, I succeeded in building gcc13-devel (version 13.0.1 20230226). In /usr/local/etc/poudriere.conf, I had ALLOW_MAKE_JOBS=yes and in /usr/local/etc/poudriere.d/make.conf, I had MAKE_JOBS_NUMBER=2. The build time was a bit more than 6 hours. I finally had this success in building gcc with the LTO_BOOTSTRAP option enabled after attempting to build it a few times and failing to do it. For example, during one of my previous attempts to build it, I had #ALLOW_MAKE_JOBS=yes in poudriere.conf (meaning there was this # thing present) and didn't create the make.conf file. So, my computer crashed during this build attempt. I find it a bit puzzling because unless I am mistaken, if you disable ALLOW_MAKE_JOBS, it means that poudriere will make use of only one job. I tried using portmaster to build gcc(combined with LTO_BOOTSTRAP), but the computer crashed when I was compiling the port. But, as I said earlier, I finally had success and built the port.
I also want to mention that I always had PARALLEL_JOBS=1 setting in poudriere.conf. As I said earlier, I find it puzzling that my computer kept crashing after I commented out ALLOW_MAKE_JOBS setting. Was I wrong in my belief that when this setting is disabled, poudriere can only make use of one job? As mentioned earlier, re-enabling ALLOW_MAKE_JOBS and setting MAKE_JOBS_NUMBER to 2 in make.conf allowed me to avoid crashing the computer and build the port successfully.
(In reply to Oleg from comment #33) A hypothesis: "enabling ALLOW_MAKE_JOBS and setting MAKE_JOBS_NUMBER to 2" probably explicitly used: -j2 "commented out ALLOW_MAKE_JOBS" probably did not use: -j (explicitly at all). Lack of -j vs. an explicit -j1 are not equivalent in all respects as I understand. (Not limited to the gcc* issue for this bugzilla submittal.) To get an explicit -j1 might require something analogous to: enabling ALLOW_MAKE_JOBS and setting MAKE_JOBS_NUMBER to 1 .
People interested in this bug report might be intetested in bug #272508 too, that submits a patch adding an option to serialize linking.
I do observe the same issue with gcc12. It does NOT happen always. Most of the time the build performs correctly, only under certain (yet unclear) conditions does the number of processes explode. I reported this as PR268760, which was closed as "works as designed". The behaviour is NOT "works as designed", as gets obvious from my statistics (but that PR is a duplicate anyway): total compute cores runtime smp eff. date 29:16:42 4 7:34:27 1.03 230922124639 30:09:54 4 7:54:28 1.05 231128133946 25:17:04 10 2:57:21 1.17 231009122722 38:41:53 18 2:32:53 1.19 240107181715 153:49:18 20 8:12:02 1.07 240108181223 Normally the build uses rather reproducible 30 hours of compute (+- 25% due to hyperthreading, E5-2660v3 10/20c). But in the case of the many processes appearing, it takes five times as long, and the system gets very much unresponsive. I counted 215 processes that want to run simultaneously. It is NOT a memory exhaustion issue (doesn't pageout), just too many processes: last pid: 54768; load averages: 110.14, 111.74, 117.35 up 0+06:01:42 00:14:55 296 processes: 123 running, 102 sleeping, 12 zombie, 1 waiting, 58 lock CPU: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle Mem: 2423M Active, 11G Inact, 777M Laundry, 10G Wired, 1529M Buf, 2643M Free ARC: 4258M Total, 1577M MFU, 2438M MRU, 128K Anon, 43M Header, 198M Other 3779M Compressed, 7337M Uncompressed, 1.94:1 Ratio Swap: 15G Total, 15G Free These processes are different than the usual LTO processes. They say "lto1-wpa-streaming". It is probably not a "make -j" problem either, but rather some occasional runaway behaviour in the LTO processing.