Bug 255753

Created attachment 224813 [details]
gzipped poudriere log file

I'm experiencing the same problem with the same 'Illegal instruction' error during build using synth on 11-STABLE.

This happens when I try to build the port manually.  The specific instructions leading to the problem are:

/usr/local/bin/cmake -E env LD_LIBRARY_PATH=/usr/ports/devel/pyside2/work-py38/.build/libpyside:/usr/local/lib /usr/local/bin/python3.8 -vvd /usr/ports/devel/pyside2/work-py38/pyside-setup-opensource-src-5.15.2/sources/pyside2/PySide2/Qt3DRender/../support/generate_pyi.py Qt3DRender --sys-path /usr/ports/devel/pyside2/work-py38/.build

The -vv option in the line above produces the following output before failing:

import 'PySide2.support.signature.lib.tool' # <_frozen_importlib_external.SourceFileLoader object at 0x8009b8af0>
# trying /usr/ports/devel/pyside2/work-py38/.build/PySide2/Qt3DRender.cpython-38.so
Illegal instruction

The problem is that files/patch-src_core_configure.json in graphics/qt5-3d turns on the avx2 instructions on cpus that do not have these instructions.

Removing the patch file allows pyside2 to be built on these machines, such as my penryn.

I can confirm that the the patch proposed by crahman works for me also. My machine holds two INTEL X5650 XEON processors that do not support  this avx2 functionality.

So the root of this problem is not pyside2, but graphics/qt5-3d.

In case it is helpful, here is an exerpt of the system startup log, that shows the processor capabilities:

May 25 08:25:12 arwen kernel: FreeBSD 12.2-RELEASE-p6 GENERIC amd64
May 25 08:25:12 arwen kernel: FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
May 25 08:25:12 arwen kernel: VT(vga): resolution 640x480
May 25 08:25:12 arwen kernel: CPU: Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz (2660.06-MHz K8-class CPU)
May 25 08:25:12 arwen kernel:   Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
May 25 08:25:12 arwen kernel:   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,F
XSR,SSE,SSE2,SS,HTT,TM,PBE>
May 25 08:25:12 arwen kernel:   Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,P
OPCNT,AESNI>
May 25 08:25:12 arwen kernel:   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
May 25 08:25:12 arwen kernel:   AMD Features2=0x1<LAHF>
May 25 08:25:12 arwen kernel:   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
May 25 08:25:12 arwen kernel:   TSC: P-state invariant, performance statistics
May 25 08:25:12 arwen kernel: real memory  = 94491377664 (90114 MB)

Refer to bug#256152, "graphics/qt5-3d: fails to work on processors with CPU < haswell (anything without AVX2)".

Comment 7 Steve Wills 2021-05-25 21:21:26 UTC

Could you please share a copy of the content of /usr/local/lib/qt5/mkspecs/qmodule.pri from qt5-qmake (devel/qt5-qmake)? If it mentions avx2, that might be the source of the problem.

Hello Steve,

here it is, indeed it seems to mention avx2:

QT_CPU_FEATURES.x86_64 = mmx sse sse2
QT.global_private.enabled_features = sse2 alloca avx2 dlopen network posix_fallocate reduce_ex
ports reduce_relocations relocatable sql system-zlib testlib xml
QT.global_private.disabled_features = alloca_h alloca_malloc_h android-style-assets private_te
sts dbus dbus-linked gc_binaries gui intelcet libudev release_tools stack-protector-strong wid
gets zstd
PKG_CONFIG_EXECUTABLE = pkgconf
QMAKE_LIBS_LIBDL =
QT_COORD_TYPE = double
QMAKE_LIBS_ZLIB = -lz
CONFIG -= precompile_header
CONFIG += sse2 aesni sse3 ssse3 sse4_1 sse4_2 avx avx2 avx512f avx512bw avx512cd avx512dq avx5
12er avx512ifma avx512pf avx512vbmi avx512vl compile_examples f16c largefile rdrnd rdseed shan
i x86SimdAlways
QT_BUILD_PARTS += libs tools
QT_HOST_CFLAGS_DBUS +=
~

Comment 9 Steve Wills 2021-05-25 21:45:45 UTC

(In reply to Frans van der Veer from comment #8)
Ok, thanks for confirming. Was your qmake built on your host that does not support AVX2 or on another host that does support AVX2? If it was built on a host that does not support AVX2 then I think it shouldn't mention avx2 and that might be the source of the issue. In the mean time, you may be able to simply edit it not to mention avx2 and other things that CPU does not support and avoid the issue.

(In reply to Steve Wills from comment #9)
In this case is was built on the same host, but within a poudriere jail. That is what you will regard as being the same host also I presume.

Come to think of that, the contents of the file is from the host filing system, not from the jail filing system.

Comment 11 Steve Wills 2021-05-26 11:04:51 UTC

(In reply to Frans van der Veer from comment #10)
Yes, that's all the same host really. Can you please share a complete copy of the poudriere log for devel/qt5-qmake?

Created attachment 225275 [details]
Poudriere log file for qt5-qmake-5.15.2

Steve,

here it is.

Comment 13 Steve Wills 2021-05-26 11:45:47 UTC

(In reply to Frans van der Veer from comment #12)
Thanks. Pretty strange to me that log contains:

Checking for AVX2 instructions... yes

I wonder why?

(In reply to Steve Wills from comment #13)
Hi Steve,

I also wonder why it reports 

UNAME_v=FreeBSD 12.2-RELEASE-p3
UNAME_r=12.2-RELEASE-p3

while I did a jail updates upto current p6. Just checked that once again:


# poudriere jail -u -j amd64
[00:00:00] Upgrading using ftp
/etc/resolv.conf -> /usr/local/poudriere/jails/amd64/etc/resolv.conf
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 12.2-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 12.2-RELEASE-p6.
12.2-RELEASE-p6
[00:01:45] Recording filesystem state for clean...cannot create snapshot 'zroot/poudriere/jails/amd64@clean': dataset already exists
 done

Even the latest builds of today still report p3 in the log. But OK, I presume that is quite another issue.


What I also find strange about this situation is that a poudriere build should to my opinion produce CPU architecture independent packages so you can distribute then anywhere. So making these decisions at package compile time seems odd to me. 

Regards,

Frans

Comment 15 Steve Wills 2021-05-26 13:02:25 UTC

(In reply to Frans van der Veer from comment #14)

> Even the latest builds of today still report p3 in the log. But OK, I presume that is quite another issue.

I think this may be due to the updates, where some cause kernel changes and others don't, and is expected.

> What I also find strange about this situation is that a poudriere build should to my opinion produce CPU architecture independent packages so you can distribute then anywhere.

I don't really understand what you mean here. Compiled binaries always have a target CPU architecture they run on. Even so-called "fat-binaries" or multi-architecture binaries have a list of architectures they support and don't run "anywhere".

Regardless, as far as this issue goes, it's not clear to me what qmake means here when it says "Checking for AVX2 instructions". I almost think it's just checking if the compiler supports them, not if the host CPU supports them. And then there's the question of if they should be used on the target CPU, which could actually be completely different. I will have to have someone who knows qmake better chime in on these questions I'm afraid.

Comment 16 Steve Wills 2021-05-26 15:27:03 UTC

(In reply to Frans van der Veer from comment #14)
If possible, could you share a copy of the config.log from the WRKSRC directory after running make build on devel/qt5-qmake on your host that doesn't support avx2?

Created attachment 225284 [details]
qt5-qmake build config.log file

Hi Steve,

I don't know how to retrieve this from the poudriere jail, since I seem only to be able to make poudriere  save a wrkdir on build failure.  

So this is taken outside poudriere build from the (updated) hosts ports tree itself as it was filed under:

/usr/ports/devel/qt5-qmake/work/qtbase-everywhere-src-5.15.2

I hope this helps.

Comment 18 Steve Wills 2021-05-26 16:31:27 UTC

(In reply to Frans van der Veer from comment #17)
That's fine, thanks. It's running:

> clang++ -c -pipe -mavx2 -O2 -w -fPIC -DNO_ATTRIBUTE -DQT_COMPILER_SUPPORTS_AVX2 -I. -I/usr/local/include -I/usr/ports/devel/qt5-qmake/work/qtbase-everywhere-src-5.15.2/mkspecs/freebsd-clang -o main.o main.cpp

without issue. Maybe qmodule.pri is meant to indicate what the compiler supports, not what the host or target support? It's quite confusing.

Comment 19 Steve Wills 2021-05-26 21:52:28 UTC

Created attachment 225295 [details]
patch for qmake

(In reply to Frans van der Veer from comment #17)
Ok, I did some more digging and I think the found the real source of the issue. Can you test rebuilding your qmake with this and then rebuilding graphics/qt5-3d and then also devel/pyside2 and share the contents of the /usr/local/lib/qt5/mkspecs/qmodule.pri from the new qmake?

Comment 20 Steve Wills 2021-05-26 21:55:31 UTC

(In reply to Steve Wills from comment #19)
Oh, I should have mentioned, also remove graphics/qt5-3d/files/patch-src_core_configure.json before rebuilding graphics/qt5-3d.

I've done the tests you requested - I added the patch file in comment#19 to qt5-qmake/files and recompiled it, and then built qt5-3d and py38-pyside2.

This is all on a machine without avx2 instructions.

qmodule.pri was unchanged by the qt5-qmake patch:

QT_CPU_FEATURES.x86_64 = mmx sse sse2
QT.global_private.enabled_features = sse2 alloca avx2 dlopen network posix_fallocate reduce_exports reduce_relocations relocatable sql system-zlib testlib xml
QT.global_private.disabled_features = alloca_h alloca_malloc_h android-style-assets private_tests dbus dbus-linked gc_binaries gui intelcet libudev release_tools stack-protector-strong widgets zstd
PKG_CONFIG_EXECUTABLE = pkgconf
QMAKE_LIBS_LIBDL = 
QT_COORD_TYPE = double
QMAKE_LIBS_ZLIB = -lz
CONFIG -= precompile_header
CONFIG += sse2 aesni sse3 ssse3 sse4_1 sse4_2 avx avx2 avx512f avx512bw avx512cd avx512dq avx512er avx512ifma avx512pf avx512vbmi avx512vl compile_examples f16c largefile rdrnd rdseed shani x86SimdAlways
QT_BUILD_PARTS += libs tools
QT_HOST_CFLAGS_DBUS += 

The patch from comment#19 did not change the avx2 issue in qt5-3d.  With or without the patch, if qt5-3d/files/patch-src_core_configure.json is present, qt5-3d is compiled with -mavx2, and matrix4x4_avx2.cpp is compiled, and the configure summary shows:

  Use SSE2 instructions .................. yes
  Use AVX2 instructions .................. yes

and pyside2 fails to build.

With or without the patch, if qt5-3d/files/patch-src_core_configure.json is not present, qt5-ed is compiled without -mavx2, matrix4x4.avx2.cpp is compiled, and the configure summary shows:

  Use SSE2 instructions .................. yes
  Use AVX2 instructions .................. no

and pyside2 builds successfully.

The matrix4x4_avx2_cpp contents are conditionalized with an #ifdef QT_COMPILER_SUPPORTS_AVX2, if that's of any use.

Comment 22 Steve Wills 2021-05-27 12:43:03 UTC

(In reply to crahman from comment #21)

> qmodule.pri was unchanged by the qt5-qmake patch:

Good! It should be.

> With or without the patch, if qt5-3d/files/patch-src_core_configure.json is not present, qt5-ed is compiled without -mavx2, matrix4x4.avx2.cpp is compiled, and the configure summary shows:
> 
>   Use SSE2 instructions .................. yes
>   Use AVX2 instructions .................. no
> 
> and pyside2 builds successfully.

Good!

The patch to qt5-3d was actually not quite right, but it allowed it to build qt5-3d with avx2 enabled.

Here we've removed the patch to qt5-3d and instead added this patch to qmake which fixes the qt5-3d issue in the avx2 case and also doesn't break things in the non-avx2 case.

I've tested the avx2 case and everything is fine, but I couldn't test the without avx2 case fully. You just tested the without avx2 case and confirmed that it all works.

So, I believe removing the patch to qt5-3d and adding this patch to qmake is the correct solution. It allows building everything properly both without avx2 and with avx2.

Just to be on the safe side, I'd like some input from KDE Team, but this seems right to me.

Created attachment 227351 [details]
patch-qmake_revert_248155_qt5-3d

Steve, this may be due to me still learning git but I had issues with your patch applying unless I included --directory=devel and applying from the devel directory. This is probably user error on my end. But to make things easier for folks like me: git apply with this patch from just being in /usr/ports without additional switches to git. I created this patch including reverting 248155 by removing the offending file. I applied these patches from a ports tree pulled about an hour prior, with everything in the tree that can build allowed to updated including qgis in the blocking notice. All this does is include your patch to qmake and delete the qt5-3d patch file for AVX. qt5-qmake and qt5-3d rebuilt, qt5-3d no longer shows using AVX in configure stage.

py38-pyside2 builds fine following patch
qgis runs without recompilation (though it was freshly updated, and failed prior to the patch with the newer version) without illegal instructions, and so is fixed PR 257402. It the qmake fix helped folks with AVX on qt5-3d then this should work for them too, but I can only test the non-AVX case.

patch produced by git diff --staged, and confirmed to apply to new test branch in my local tree. Maybe this will be useful to folks, but this gets me a little closer to trying to migrate to Releng 13.0

if there is something wrong with the format of my patch let me know, I want to learn how to do this properly.

It appears that this problem resurfaced again.

My "weekly" portsupdate and poudriere build tells the build of py38-pyside2-5.15.2_2 fails.

This is the last part of the build log file:

Qt3DRender --sys-path /wrkdirs/usr/ports/devel/pyside2/work-py38/.build /usr/local/lib/python3.8/site-packages/shiboken2
FAILED: PySide2/Qt3DRender/CMakeFiles/Qt3DRender_pyi /wrkdirs/usr/ports/devel/pyside2/work-py38/.build/PySide2/Qt3DRender/CMakeFiles/Qt3DRender_pyi 
cd /wrkdirs/usr/ports/devel/pyside2/work-py38/.build/PySide2/Qt3DRender && /usr/local/bin/cmake -E env LD_LIBRARY_PATH=/wrkdirs/usr/ports/devel/pyside2/work-py38/.build/libpyside:/usr/local/lib /usr/local/bin/python3.8 /wrkdirs/usr/ports/devel/pyside2/work-py38/pyside-setup-opensource-src-5.15.2/sources/pyside2/PySide2/Qt3DRender/../support/generate_pyi.py Qt3DRender --sys-path /wrkdirs/usr/ports/devel/pyside2/work-py38/.build /usr/local/lib/python3.8/site-packages/shiboken2
Illegal instruction
ninja: build stopped: subcommand failed.
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/pyside2
=>> Cleaning up wrkdir
===>  Cleaning for py38-pyside2-5.15.2_2
build of devel/pyside2 | py38-pyside2-5.15.2_2 ended at Thu Oct 28 07:11:33 CEST 2021
build time: 00:11:16
!!! build failure encountered !!!

(In reply to Frans van der Veer from comment #24)

not surprising, its never been patched in ports. This is one of a few I watch like a hawk. But if you apply my patch, or Steve's and revert the qt3d patch he references instead things will work. I just re-confirmed py38-pyside2 still builds without the illegal instruction error linking to the forced AVX instructions that end up in qt5-3d without patching qmake and reverting the change in qt5-3d that forced it on even on <Haswell machines without AVX.

Hi, 
any updates here ?

Comment 27 commit-hook 2021-12-19 14:12:37 UTC

A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/ports/commit/?id=183d079fd681df544b67cf2f03c7d1d5aa7853fc

commit 183d079fd681df544b67cf2f03c7d1d5aa7853fc
Author:     Dima Panov <fluffy@FreeBSD.org>
AuthorDate: 2021-12-19 14:04:58 +0000
Commit:     Dima Panov <fluffy@FreeBSD.org>
CommitDate: 2021-12-19 14:04:58 +0000

    graphics/qt5-q3d: convert AVX2 detection hack to OPTIONS (+)

    Force qt5-qmake to pass CXXFLAGS to cpu features test.
    Define AVX OPTION only amd64/i386 arches due to cpu used.

    Off by default to keep out-of-box behavior and allow to build
    qt5-q3d and its consumes on any old processors.

    Necromages, go forward! :)

    PR:             255753
    With hat:       desktop
    Sponsored by:   Netzkommune GmbH

 devel/qt5-qmake/Makefile                                          | 2 +-
 devel/qt5-qmake/files/patch-config.tests_arch_arch.pro (new)      | 6 ++++++
 graphics/qt5-3d/Makefile                                          | 8 +++++++-
 ...src_core_configure.json => extrapatch-src_core_configure.json} | 0
 4 files changed, 14 insertions(+), 2 deletions(-)

Comment 28 Adriaan de Groot 2023-05-19 20:34:13 UTC

Presumed fixed by comment #27