Comment 1 Raphael Kubo da Costa 2019-11-03 11:31:01 UTC

> The bug above says devel/llvm60 is fine (I've tried 12.0-RELEASE's clang 6 and it also works), but llvm70 and llvm80 are also broken.

Actually, the patch in bug 241096 says clang 7 and 8 are broken on amd64.

Comment 2 Dimitry Andric 2019-11-03 12:26:29 UTC

I also tried with clang 9, but it's simply running out of RAM:

pid 1144 (clang), jid 0, uid 1000, was killed: out of swap space

This is on a VM with 6G RAM and 8G swap.  I can attempt to increase RAM to see if I can get it to build, to see what the maximum memory usage is.

Comment 3 Raphael Kubo da Costa 2019-11-03 12:30:08 UTC

For the record, I've been using a VM with a lot more RAM than that, and even when it works (on amd64, for example) it still takes a long time to build that file.

Comment 4 Dimitry Andric 2019-11-03 12:46:13 UTC

See also upstream bug https://bugs.llvm.org/show_bug.cgi?id=42515, and a Qt bug report: https://bugreports.qt.io/browse/QTBUG-57672

Comment 5 Dimitry Andric 2019-11-03 12:59:34 UTC

Ok, it takes about 150 seconds and 10 GiB of RAM:

$ /usr/bin/time -l clang -cc1 -triple i386-portbld-freebsd12.0 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name qgscoordinatereferencesystem.cpp -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -relaxed-aliasing -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu i586 -dwarf-column-info -debugger-tuning=gdb -coverage-notes-file /wrkdirs/usr/ports/graphics/qgis/work/.build/src/core/CMakeFiles/qgis_core.dir/qgscoordinatereferencesystem.cpp.gcno -D PROJ_VERSION_MAJOR=6 -D QT_CONCURRENT_LIB -D QT_CORE_LIB -D QT_DISABLE_DEPRECATED_BEFORE=0 -D QT_GUI_LIB -D QT_NETWORK_LIB -D QT_NO_CAST_TO_ASCII -D QT_NO_DEBUG -D QT_NO_FOREACH -D QT_POSITIONING_LIB -D QT_PRINTSUPPORT_LIB -D QT_SERIALPORT_LIB -D QT_SQL_LIB -D QT_SVG_LIB -D QT_WEBKITWIDGETS_LIB -D QT_WEBKIT_LIB -D QT_WIDGETS_LIB -D QT_XML_LIB -D SIP_VERSION=0x041313 -D TEST_DATA_DIR=\"/wrkdirs/usr/ports/graphics/qgis/work/QGIS-final-3_8_2/tests/testdata\" -D WITH_BINDINGS -D WITH_QTWEBKIT -D _HAVE_PTHREAD_ -D qgis_core_EXPORTS -O2 -Wall -Wextra -Wno-long-long -Wformat-security -Wno-strict-aliasing -Wno-return-type-c-linkage -Wno-overloaded-virtual -Wimplicit-fallthrough -std=gnu++11 -fdeprecated-macro -ferror-limit 19 -fmessage-length 190 -fvisibility hidden -stack-protector 2 -fobjc-runtime=gnustep -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -vectorize-loops -vectorize-slp -faddrsig -x c++ qgscoordinatereferencesystem-8a0c03.cpp
/wrkdirs/usr/ports/graphics/qgis/work/QGIS-final-3_8_2/src/core/qgscoordinatereferencesystem.cpp:1451:15: warning: unused variable 'errNo' [-Wunused-variable]
    const int errNo = proj_context_errno( ctx );
              ^
/wrkdirs/usr/ports/graphics/qgis/work/QGIS-final-3_8_2/src/core/qgscoordinatereferencesystem.cpp:2410:81: warning: unused parameter 'message' [-Wunused-parameter]
static void sync_db_proj_logger( void * /* user_data */, int level, const char *message )
                                                                                ^
2 warnings generated.
      153.92 real       146.75 user         7.09 sys
  10587924  maximum resident set size
     51645  average shared memory size
       299  average unshared data size
       858  average unshared stack size
   3153690  page reclaims
         2  page faults
         0  swaps
         2  block input operations
         9  block output operations
         0  messages sent
         0  messages received
         0  signals received
         6  voluntary context switches
      3495  involuntary context switches

That is indeed quite a lot.  It looks like lowering the optimization level to -O1 saves quite a bit, it then compiles in ~46 seconds, and using ~3.4 GiB RAM instead:

$ /usr/bin/time -l clang -cc1 -triple i386-portbld-freebsd12.0 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name qgscoordinatereferencesystem.cpp -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -relaxed-aliasing -masm-verbose -mconstructor-aliases -fuse-init-array -target-cpu i586 -dwarf-column-info -debugger-tuning=gdb -coverage-notes-file /wrkdirs/usr/ports/graphics/qgis/work/.build/src/core/CMakeFiles/qgis_core.dir/qgscoordinatereferencesystem.cpp.gcno -D PROJ_VERSION_MAJOR=6 -D QT_CONCURRENT_LIB -D QT_CORE_LIB -D QT_DISABLE_DEPRECATED_BEFORE=0 -D QT_GUI_LIB -D QT_NETWORK_LIB -D QT_NO_CAST_TO_ASCII -D QT_NO_DEBUG -D QT_NO_FOREACH -D QT_POSITIONING_LIB -D QT_PRINTSUPPORT_LIB -D QT_SERIALPORT_LIB -D QT_SQL_LIB -D QT_SVG_LIB -D QT_WEBKITWIDGETS_LIB -D QT_WEBKIT_LIB -D QT_WIDGETS_LIB -D QT_XML_LIB -D SIP_VERSION=0x041313 -D TEST_DATA_DIR=\"/wrkdirs/usr/ports/graphics/qgis/work/QGIS-final-3_8_2/tests/testdata\" -D WITH_BINDINGS -D WITH_QTWEBKIT -D _HAVE_PTHREAD_ -D qgis_core_EXPORTS -O1 -Wall -Wextra -Wno-long-long -Wformat-security -Wno-strict-aliasing -Wno-return-type-c-linkage -Wno-overloaded-virtual -Wimplicit-fallthrough -std=gnu++11 -fdeprecated-macro -ferror-limit 19 -fmessage-length 190 -fvisibility hidden -stack-protector 2 -fobjc-runtime=gnustep -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -vectorize-loops -vectorize-slp -faddrsig -x c++ qgscoordinatereferencesystem-8a0c03.cpp
/wrkdirs/usr/ports/graphics/qgis/work/QGIS-final-3_8_2/src/core/qgscoordinatereferencesystem.cpp:1451:15: warning: unused variable 'errNo' [-Wunused-variable]
    const int errNo = proj_context_errno( ctx );
              ^
/wrkdirs/usr/ports/graphics/qgis/work/QGIS-final-3_8_2/src/core/qgscoordinatereferencesystem.cpp:2410:81: warning: unused parameter 'message' [-Wunused-parameter]
static void sync_db_proj_logger( void * /* user_data */, int level, const char *message )
                                                                                ^
2 warnings generated.
       46.02 real        43.95 user         2.07 sys
   3582316  maximum resident set size
     51602  average shared memory size
       299  average unshared data size
       660  average unshared stack size
    960286  page reclaims
         0  page faults
         0  swaps
         0  block input operations
        13  block output operations
         0  messages sent
         0  messages received
         0  signals received
         4  voluntary context switches
       830  involuntary context switches

So maybe this can be used as a workaround for now.

Comment 6 Loïc Bartoletti 2019-11-03 14:36:27 UTC

For reference, we have also a discussion on the QGIS Developper ML:
https://www.mail-archive.com/qgis-developer@lists.osgeo.org/msg47062.html

Thank you.

Comment 7 Loïc Bartoletti 2019-11-06 07:19:19 UTC

Rainer, Raphael, Dimitry,

Can you try to build it with this option for qgscoordinatereferencesystem "-mllvm -inline-threshold=128"

The patch on OpenBSD: http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/geo/qgis/patches/patch-src_core_CMakeLists_txt?rev=1.22&content-type=text/x-cvsweb-markup&hideattic=1

Comment 8 Raphael Kubo da Costa 2019-11-06 10:48:30 UTC

The OpenBSD patch did not solve the crash here. However, replacing "-mllvm -inline-threshold=128" with "-O1" is essentially dim's suggestion from comment #5 and Rainer's question from https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241096#c6. Doing so has allowed me to build the port on my 12-i386 jail with no issues.

Comment 9 Loïc Bartoletti 2019-11-06 17:26:51 UTC

(In reply to Raphael Kubo da Costa from comment #8)

OK. Tested on my local machine with:
SET_SOURCE_FILES_PROPERTIES(qgscoordinatereferencesystem.cpp PROPERTIES COMPILE_FLAGS "-01")

added in src/core/CMakeLists.txt and it works for me too.

If it's ok also for you. I can upstream the patch.

Comment 10 Rainer Hurling 2019-11-06 18:20:57 UTC

Sorry for the delayed response. I first had to build a new jail 12.1 ie86 on Poudriere.

There I just tried the original method from the OpenBSD port and it also builds and installs with clang90! This is the patch I used for it:


# cat files/patch-src_core_CMakeLists.txt 
--- src/core/CMakeLists.txt.orig	2019-10-28 04:34:30 UTC
+++ src/core/CMakeLists.txt
@@ -889,6 +889,7 @@ ELSE(MSVC)
   SET_SOURCE_FILES_PROPERTIES(${QGIS_CORE_MOC_SRCS} PROPERTIES COMPILE_FLAGS "-Wno-deprecated-declarations" )
   IF ("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang" AND (CMAKE_BUILD_TYPE MATCHES Release OR CMAKE_BUILD_TYPE MATCHES RelWithDebInfo))
     SET_SOURCE_FILES_PROPERTIES(${CMAKE_CURRENT_BINARY_DIR}/qgsexpression_texts.cpp PROPERTIES COMPILE_FLAGS "-O1")
+    SET_SOURCE_FILES_PROPERTIES(qgscoordinatereferencesystem.cpp PROPERTIES COMPILE_FLAGS "-mllvm -inline-threshold=128")
   ENDIF ()
 ENDIF(MSVC)

Comment 11 Rainer Hurling 2019-11-06 19:22:41 UTC

(In reply to Rainer Hurling from comment #10)

The patch in comment #10 also works with clang8 (default compiler on 12.1).

Perhaps there is a chance to also remove LLVM_VER, CC, CPP and CXX vars now. I will investigate in the next days ...

Comment 12 Raphael Kubo da Costa 2019-11-24 18:44:06 UTC

The patch from comment #10 doesn't work for me on Poudriere as reported in comment #8, but adding "-O1" does. Are you OK with committing that patch instead?

Comment 13 Rainer Hurling 2019-11-25 06:10:05 UTC

(In reply to Raphael Kubo da Costa from comment #12)

I tested again your patch version (with '-O1') on Poudriere, especially for i368, and it seems to work also.

From my point of view, it would be ok to change the already patched QGIS sources[1] towards '-O1'. Loïc, what do you think about it?

Just out of curiosity, what exactly is the problem/error with the other patch?


[1] https://github.com/qgis/QGIS/commit/59a766e22a12cc274cf1b852d466f34848297658

Comment 14 Raphael Kubo da Costa 2019-11-25 09:57:22 UTC

(In reply to Rainer Hurling from comment #13)
> Just out of curiosity, what exactly is the problem/error with the other patch?

1.      <eof> parser at end of file
2.      Per-module optimization passes
3.      Running pass 'CallGraph Pass Manager' on module '/wrkdirs/usr/ports/graphics/qgis/work/QGIS-final-3_8_2/src/core/qgscoordinatereferencesystem.cpp'.
4.      Running pass 'Jump Threading' on function '@__cxx_global_var_init'
libunwind: EHHeaderParser::decodeTableEntry: bad fde: CIE ID is not zero
clang-9: error: unable to execute command: Abort trap (core dumped)
clang-9: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 9.0.0 (tags/RELEASE_900/final)
Target: i386-portbld-freebsd12.0
Thread model: posix
InstalledDir: /usr/local/llvm90/bin
clang-9: note: diagnostic msg: PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.

Comment 15 Rainer Hurling 2019-11-25 10:21:16 UTC

(In reply to Raphael Kubo da Costa from comment #14)

Perhaps a problem with to less memory? But this does not explain, why it works with '-O1'.

At least, this does not happen on my Poudriere box (for HEADi/a, 12.1i/a and 11.3i/a, 16GB RAM) ...

Thanks for the info.

Comment 16 Raphael Kubo da Costa 2019-11-25 10:33:19 UTC

FTR, the VM where I run Poudriere has >32GB RAM so I don't think that's the issue. I also don't really understand why the -mllvm change does not work for me but does work for you and others. Did you try using a 12.0-i386 jail by any chance?

Comment 17 Rainer Hurling 2019-11-27 07:03:03 UTC

(In reply to Raphael Kubo da Costa from comment #16)

> Did you try using a 12.0-i386 jail by any chance?

If you mean, that I try 12.0 instead of 12.1, no. I only tried with 12.1.

Comment 18 Loïc Bartoletti 2019-11-27 12:16:31 UTC

(In reply to Rainer Hurling from comment #13)

> From my point of view, it would be ok to change the already patched QGIS sources[1] towards '-O1'. Loïc, what do you think about it?

From a QGIS dev I prefer "-mllvm -inline-threshold=128" since it's 2x faster than "-O1" :) and works for macOS, FreeBSD (out of poudriere), OpenBSD and Linux (especially our CI with Travis).

But maybe, if I understood the problem, you can patch the src/core/CMakeLists.txt with "-O1" for i386 for poudriere.

Comment 19 Rainer Hurling 2019-11-28 13:41:49 UTC

(In reply to Loïc Bartoletti from comment #18)

Loïc,

Following your suggestion, for QGIS versions, which already includes [1], the new patch (of of your existing patch) could be

post-patch:
.if ${OPSYS} == FreeBSD && ${ARCH} == "i386"
	@${REINPLACE_CMD} -e 's|"-mllvm -inline-threshold=128"|"-O1"|g' \
		${WRKSRC}/src/core/CMakeLists.txt
.endif

For all other QGIS versions > 3.8.2 and < 3.12, all non i386 arches should patch src/core/CMakeLists.txt like [1], but i386 should use '-O1' for the patch, right?

This way, all non i386 arches should be able to build faster the QMap part ...


[1] https://github.com/qgis/QGIS/commit/59a766e22a12cc274cf1b852d466f34848297658

Comment 20 Raphael Kubo da Costa 2019-11-28 23:02:02 UTC

(In reply to Rainer Hurling from comment #17)
> If you mean, that I try 12.0 instead of 12.1, no. I only tried with 12.1.

I've set up a 12.1-i386 jail, and it works with devel/llvm90 and -mllvm -inline-threshold=128. It's 12.0 that's busted (I haven't checked 11.3, though you did mention you'd tested that one).

Comment 21 Loïc Bartoletti 2019-12-04 14:20:38 UTC

(In reply to Raphael Kubo da Costa from comment #20)

So, based on Rainer's snip, something like:

post-patch:
.if ${OPSYS} == FreeBSD && ${ARCH} == "i386" && ${OSVERSION} < 1201000
	@${REINPLACE_CMD} -e 's|"-mllvm -inline-threshold=128"|"-O1"|g' \
		${WRKSRC}/src/core/CMakeLists.txt
.endif

Comment 22 Rainer Hurling 2019-12-04 16:26:04 UTC

(In reply to Loïc Bartoletti from comment #21)

I am only waiting on the release of QGIS 3.10.1, because of some real trouble on QGIS > 3.4.x wih gdal3/proj6 [1].

After the arrival of QGIS 3.10.1, I will update the port(s) and put in the i386 patch.

Hope, it is ok to wait some more days?


[1] http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-QGIS-amp-PROJ-6-amp-SAGA-td5423735.html

Comment 23 Rainer Hurling 2019-12-10 21:05:24 UTC

I just filed a patch to update graphics/qgis from 3.8.2 to 3.10.1 [1]. After that patch is committed, this bug #241687 should be obsolet.


[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242557

Comment 24 commit-hook 2019-12-20 16:41:07 UTC

A commit references this bug:

Author: rakuco
Date: Fri Dec 20 16:40:11 UTC 2019
New revision: 520519
URL: https://svnweb.freebsd.org/changeset/ports/520519

Log:
  Update to 3.10.1.

  For some months now, the build process of graphics/qgis 3.8.2 had problems
  while using the QT5 QMap() function with large list like for proj6[1].
  Compiling such long QMap() list with clang takes several hours instead of
  some seconds or minutes[2], often it breaks the build.

  It turns out that building that QMap() code with clang takes exhaustive
  memory[3], for which a nice patch already exists on OpenBSD[4]. While this
  OpenBSD patch builds fine on amd64, it needs another patch on i386 (usage of
  '-O1' instead the clang specific patch '-mllvm -inline-threshold=128').

  After we found a solution for the QMap problem, it was necessary to wait for
  the release of QGIS 3.10.1 because of some tricky problems with proj6 / gdal
  combinations[5] that should be avoided.

  [1] https://github.com/qgis/QGIS/blob/master/src/core/qgscoordinatereferencesystem_legacy.h#L23
  [2] http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-QGIS-3-8-0-on-FreeBSD-build-takes-several-hours-td5408676.html
  [3] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241687
  [4] https://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/geo/qgis/patches/patch-src_core_CMakeLists_txt?rev=1.22&content-type=text/x-cvsweb-markup&hideattic=1
  [5] https://www.mail-archive.com/qgis-developer@lists.osgeo.org/msg48670.html

  Changes in the port:
  - Update from 3.8.2 to 3.10.1
  - Allow using default clang again
  - Update several Python dependencies
  - Add textproc/py-nltk, math/py-pandas, sysutils/py-psutil,
    graphics/py-pyrsgis, x11-toolkits/py-qt5-quick for Python extensions
  - Add USES=gl and USE_GL=gl
  - Add USE_PYQT=quickwidgets
  - Add several CMAKE ARGS
  - Add files/patch-src_core_CMakeLists.txt for QMap problem
  - Change above patch into '-O1' for i386
  - Update pkg-plist

  PR:		242557
  PR:		241687
  Submitted by:	Rainer Hurling <rhurlin@gwdg.de> (maintainer)

Changes:
  head/graphics/qgis/Makefile
  head/graphics/qgis/distinfo
  head/graphics/qgis/files/patch-src_core_CMakeLists.txt
  head/graphics/qgis/pkg-plist

Comment 25 Mark Linimon 2020-08-17 07:10:46 UTC

^Triage: assign to committer that resolved.