Bug 227659

Summary: [PATCH] src.opts.mk: fix to build WITH_GCC=yes when using an external compiler that supports c++11
Product: Base System Reporter: Kenneth Salerno <kennethsalerno>
Component: miscAssignee: freebsd-toolchain (Nobody) <toolchain>
Status: New ---    
Severity: Affects Only Me CC: bdrewery, marklmi26-fbsd
Priority: --- Keywords: patch
Version: 11.1-RELEASE   
Hardware: Any   
OS: Any   
Attachments:
Description Flags
patch to allow user to select WITH_GCC=yes when building with an external compiler that supports c++11 none

Description Kenneth Salerno 2018-04-20 19:07:01 UTC
Created attachment 192687 [details]
patch to allow user to select WITH_GCC=yes when building with an external compiler that supports c++11

We should not explicitly force MK_GCC to always be set to "no" when using a modern gcc XCC (that supports c++11). Seems this should be left up to the user (WITH_, WITHOUT_).

I agree with always setting MK_GNUCXX to "no" because it doesn't even compile (missing bits/c++config.h), but there are cases where you want a system compiler installed when building with an XCC, but don't want that system compiler to be clang.

One such example is the system /usr/bin/clang (LLVM 4.0.0) segfaults in powerpc64 linking ports-mgmt/pkg. So while the first and last task of /usr/bin/gcc is to just build lang/gcc6, you can't get very far if /usr/bin/cc is clang.

The system gcc 4.2.1 on powerpc64 can successfully build ports-mgmt/pkg:
/usr/local/sbin/pkg: ELF 64-bit MSB executable, 64-bit PowerPC or cisco 7500, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, FreeBSD-style, for FreeBSD 11.1, stripped
Comment 1 Kenneth Salerno 2018-04-28 22:05:42 UTC
On second thought, it is a real hassle to get g++ to use libc++... And I just could not get out of include hell building libstdc++ with devel/powerpc64-gcc (6.3.0) as my external compiler - the same exact parameters (isystem and sysroot etc.) would compile with no complaints with clang++ (4.0.0) but g++ (6.3.0) on the other hand would choke on __is_empty traits. I even checked with -E (-o *.i) and diffed the results between g++ and clang++ and they were almost identical!

I reverted this patch and now trying to use devel/llvm60 (yes, I'm caving in) as my external compiler and see if I can get past the ports-mgmt/pkg segfault with the system compiler under ppc64.
Comment 2 Mark Millard 2018-04-29 01:24:18 UTC
[In recent times my FreeBSD activity has been greatly limited
compared to before. My doing the powerpc64 and powerpc (32-bit)
clang/powerpc64-gcc experiments goes back to 10.x and has been
ongoing through 11 as head and now 12 as head. I've not tried
under an 11.x in some time.]

clang historically has had lots of powerpc64 and powerpc code
generation problems. See

https://bugs.llvm.org//show_bug.cgi?id=25780

and its depends-on list. (I'll ignore lld issues here.) There
has been progress in some areas, but not in others. Most of
the fixed items were fixed in 2017 or very late 2016 as
I remember. (FreeBSD picks up fixes from llvm generally.)

Even now, if you are trying to buildworld based on any llvm* 
then the built system libraries' support for handling C++
exceptions will be broken: For example, clang does not implement
__builtin_eh_return for powerpc64 or for powerpc and so there is
missing material in what it builds in the system libraries.

[llvm 26844 is a report of this. But the __builtin_eh_return
part of the discovery is far more recent than the original
submittal of lower level details about the bad exception code
generation for the ABI.]

I build for and run old powermacs via system-clang based on
as part of giving evidence of what is needed for FreeBSD to be
able to use clang as the system compiler. kyua can not be used
for this context because of its dependence on C++ exceptions,
for example.

[For powerpc (32-bit) clang6-being-involved can not build the
kernel currently. Some older vintages of system-clang could.
So currently I tend to build a gcc 4.2.1 based kernel and mix
it with a system-clang6 based world for 32-bit experiments.]

I also build via powerpc64-gcc and for it C++ exceptions support
built fine in the system. Where I've had problems is bad code
generation for support of lib32: crtbeginS.o ended up with code
that presumes R30 has content that it does not actually have in
the ABI and so ends up using it for a garbage-valued pointer
dereference.

[So unless I'm experimenting with the crtbeginS.o issue, I build
without lib32 support when I build via powerpc64-gcc.]

I use powerpc64-binutils for both clang-based and powerpc64
based buildworld and buildkernel targeting.

[Historically I've mostly done self-hosted builds but in recent
times I've done more amd64 -> powerpc64 (or powerpc) cross
builds. They take far less time for buildworld and buildkernel.]

My powerpc64 build/install experiments normally do not have
gcc 4.2.1 built, used, or installed at all. When I build
with powerpc64-gcc I also have the system-clang compiler
built and installed in the target powerpc64 environment.

[I have not completed building world and/or kernel using the
ports /head/base/{binutils,gcc} and testing the results. But
I've done enough to know that it has progressed since when I
first tried and submitted reports for what I found back then.]

In all cases that I could build and test, I did build various
ports, including pkg. But I'd need to do another update and
try again to know the current status for sure. The most recent
powerpc64 context that I've installed and used is:

# uname -apKU
FreeBSD FBSDG5L 12.0-CURRENT FreeBSD 12.0-CURRENT  r327364M  powerpc powerpc64 1200054 1200054

# dmesg -a | more
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT  r327364M powerpc
gcc version 6.3.0 (FreeBSD Ports Collection for powerpc64) 
. . .

(So, apparently, I wanted C++ exception support in the system
to be working at the time: powerpc64-gcc based, not based on
some clang variant.)

# cc --version
FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1)
Target: powerpc64-unknown-freebsd12.0
Thread model: posix
InstalledDir: /usr/bin

# pkg info pkg
pkg-1.10.3_1
Name           : pkg
Version        : 1.10.3_1
Installed on   : Sun Dec 24 12:47:25 2017 PST
Origin         : ports-mgmt/pkg
Architecture   : FreeBSD:12:powerpc64
. . .

So system-clang5 built pkg back then (in a head context).
Comment 3 Mark Millard 2018-04-29 19:40:18 UTC
(In reply to Mark Millard from comment #2)

I had forgotten, but back around the beginning of the year
when I last updated the old powermac's freebsd, the clang
build did not dynamically load kernel modules: attempting
such crashed. The powerpc64-gcc based build worked fine.

The clang build put out R_PPC64_JMP_SLOT (.rela.plt)
instead of R_PPC64_RELATIVE/R_PPC64_ADDR64 (.rela.dyn) in
places and the loading of .ko files does not handle
R_PPC64_JMP_SLOT : So the two sides are mismatched.

I ended up building-in a couple of .ko's that I use
into the kernel:

device          filemon
device          geom_label

filemon is involved in doing self hosted builds. This
avoided requiring cross builds.

Still, as of when I stopped back then, I was running
the powerpc64-gcc based build.
Comment 4 Kenneth Salerno 2018-04-30 12:32:10 UTC
(In reply to Mark Millard from comment #2)
Thank you, Mark, for sharing all of your experiences. I have been using an external comliler since 11.0-RELEASE, and have found 10.x to be impossible to hack for gcc6 on i386 (I can get it to compile, but it will segfault somewhere). I wasn't able to use 10.x on my ppc64 QEMU guest in any form so wasn't attempting external compilers for that version back then.

In general using an external compiler exposes just how fragile the whole system is because of idiocintricies or bugs of specific compiler versions, but it is getting easier and easier since 11.0. And I would prefer to not have to resort to running HEAD, so I retro-patch a currently supported releng version instead.

11.0 was the only version I was able to get a working kernel compiled with gcc5 and gcc, for both i386 and ppc64. Starting with 11.1 I needed to mix a kernel built with the system compiler, even for i386 where I am using devel/llvm60 that breaks the iwi device driver (won't scan for networks) and for some reason multimedia/mplayer segfaults when built with any compiler besides the system compiler.
Comment 5 Kenneth Salerno 2018-05-03 21:06:45 UTC
(In reply to Mark Millard from comment #2)
Hi, Mark. 

I had a segfault compiling libstand32 (https://bugs.llvm.org/show_bug.cgi?id=37330) that was caused by -mcpu=power9, so I tried building everything again without -mcpu=power9 and got the same result with ports-mgmt/pkg segfaulting when the system clang is linking pkg-static.

Here's my config:
MYLLVMSUFFIX = 60
_OSRELEASE != uname -r
OSREL = ${_OSRELEASE:C/-.*//}
XCC = /usr/local/bin/clang${MYLLVMSUFFIX}
XCXX = /usr/local/bin/clang++${MYLLVMSUFFIX}
XCPP = /usr/local/bin/clang-cpp${MYLLVMSUFFIX}
CROSS_BINUTILS_PREFIX = /usr/local/bin/powerpc64-unknown-freebsd${OSREL}-
XCFLAGS += -B/usr/local/bin/powerpc64-unknown-freebsd${OSREL}-
XCXXFLAGS += -B/usr/local/bin/powerpc64-unknown-freebsd${OSREL}-
XLDFLAGS += -B/usr/local/bin/powerpc64-unknown-freebsd${OSREL}-
NO_WERROR =
WERROR =
WITHOUT_GCC = yes
WITHOUT_GCC_BOOTSTRAP = yes
WITHOUT_GNUCXX = yes
WITH_CLANG = yes
WITH_CLANG_BOOTSTRAP = yes
WITH_CLANG_FULL = yes
WITH_CLANG_IS_CC = yes
WITH_LLD = yes

FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
Target: powerpc64-unknown-freebsd11.1
Thread model: posix
InstalledDir: /usr/bin

I'm about ready to give up on this one.
Comment 6 Bryan Drewery freebsd_committer freebsd_triage 2018-05-03 22:46:19 UTC
Looking over the commit that brought this in, r273406:
    Always use libc++ as the default c++ stack when building with an external gcc 4.8+
    While here disable building gcc from base when using gcc 4.8+


I do think it's fine to remove the MK_GCC=no line; there's no reason to force not building GCC when using an external compiler, assuming it all builds fine.