Bug 228007 - math/lapack: switch to flang
Summary: math/lapack: switch to flang
Status: New
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-fortran mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-05-06 05:33 UTC by robert.ayrapetyan
Modified: 2018-05-10 03:26 UTC (History)
5 users (show)

See Also:


Attachments
neccessary changes for swithing to flang (1.70 KB, patch)
2018-05-06 05:33 UTC, robert.ayrapetyan
no flags Details | Diff
poudriere log for a patched port (250.00 KB, text/plain)
2018-05-06 21:33 UTC, robert.ayrapetyan
no flags Details
necessary changes for switching to flang (2.59 KB, patch)
2018-05-07 05:38 UTC, robert.ayrapetyan
no flags Details | Diff
poudriere log for a patched port (255.38 KB, text/plain)
2018-05-07 05:39 UTC, robert.ayrapetyan
no flags Details
necessary changes for switching to flang (amd64) (2.61 KB, patch)
2018-05-07 07:00 UTC, robert.ayrapetyan
no flags Details | Diff
patch with a for loop for non-flang archs (2.71 KB, patch)
2018-05-07 22:00 UTC, robert.ayrapetyan
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description robert.ayrapetyan 2018-05-06 05:33:20 UTC
Created attachment 193072 [details]
neccessary changes for swithing to flang

There are lot of known problems with gfortran when clang is involved, e.g.:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196862

This patch switches port to flang compiler (on amd64 only) and resolves gcc linker issues.

Note:
- Added new file: files/flang-patch-make.inc.example
- patched "Makefile" contains a "ditry" arch check, I believe it's the best available option now until "flang" will be set by default for all amd64 ports in fortran.mk.
Comment 1 Bugzilla Automation freebsd_committer 2018-05-06 05:33:20 UTC
Maintainer informed via mail
Comment 2 Tijl Coosemans freebsd_committer 2018-05-06 10:16:33 UTC
This is probably going to be rejected because there are plenty of people that prefer gfortran.  Note that flang doesn't even build at the moment because devel/flang-clang has been marked broken.  If you wish to use flang you can add the following line to /etc/make.conf:

DEFAULT_VERSIONS+=	fortran=flang
Comment 3 robert.ayrapetyan 2018-05-06 21:33:28 UTC
Created attachment 193117 [details]
poudriere log for a patched port

Hi, thanks for a prompt reply!

Globally defined: DEFAULT_VERSIONS+=	fortran=flang
will break all ports using gfortran-specific FFLAGS (e.g.-ffixed-line-length-none) and not using flang's replacements (e.g. -Mextend).
In case of a math/lapack there are also incompatibilities with e.g. ETIME function: 
gfortran requires it to be INTERNAL (default make.inc), flang - EXTERNAL (requires make.inc patch) etc.
A suggested patch makes math/lapack to build using flang on amd64 only and passes all tests after that (see attached poudriere log).
Other platforms should be unaffected and will continue to use gfortran.
> devel/flang-clang has been marked broken
hm, thanks, I've missed that somehow... I've just commited a patch for it (see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222254#c12).
gfortran's FreeBSD issue is really annoying and affects virtually all scientific packages (e.g. Python's Keras, or recently added math/mlpack).
With FreeBSD's movement from gcc to clang, flang is the only right choice in a long-term perspective.
To avoid even more issues and incompatibilities in the future I'm encouraging FreeBSD's ports maintainers to make this switch ASAP.

Thanks!
Comment 4 sgk 2018-05-06 22:33:06 UTC
(In reply to robert.ayrapetyan from comment #3)

> hm, thanks, I've missed that somehow... I've just commited a patch
> for it (see >https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222254#c12).
> gfortran's FreeBSD issue is really annoying and affects virtually all
> scientific packages (e.g. Python's Keras, or recently added math/mlpack).
> With FreeBSD's movement from gcc to clang, flang is the only right choice
> in a long-term perspective. To avoid even more issues and incompatibilities
> in the future I'm encouraging FreeBSD's ports maintainers to make this
> switch ASAP.

As a long time user of FreeBSD (since it was called 386BSD+patchkit)
and a long time contributor to both FreeBSD and gfortran, can you
elaborate on gfortran's FreeBSD issue.

Switching to Flang without actually testing its performance, conformance
to standards, and quality is somewhat a questionable move.  Last time
I tried compiling gfortran's testsuite with Flang, it found somewhere
around 100 internal compiler errors.
Comment 5 robert.ayrapetyan 2018-05-06 23:22:00 UTC
The gfortran's issue I'm talking about is related to:

ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by /usr/local/lib/gcc48/libgfortran.so.3 not found from gfortran.so library

-like errors, you can find recent examples here:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196862
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=217459
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=209985
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211889
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207750
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196491

Also, I've personally got into this issue trying to build Python's Keras and mlpack recently.

Note also gcc46 is not provided by FreeBSD ports\packages anymore, so the problem become even more actual.

So far I've tested\recompiled 4 ports, including this one, with no issues:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228008
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228009
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228011

I wouldn't like users of the recently added port (math/mlpack) deal with:

"-L/usr/local/lib/gcc6 -Wl,-rpath,/usr/local/lib:/usr/local/lib/gcc6"

tricks to compile their own code with clang in FreeBSD just because mentioned libs above depend on a broken libgcc_s.so\gfortran.

If FreeBSD goes with clang in a long-term, flang is the only option there. As per https://github.com/flang-compiler/flang/wiki:

Flang+LLVM is a production-quality Fortran solution designed to be co-installed and is fully interoperable with Clang C++.
Flang single-core and OpenMP performance is now on par with GNU Fortran. Flang has implemented Fortran 2003 and has a near full implementation of OpenMP through version 4.5 targeting multicore CPUs.

Not sure how close is that to the reality (not close enough as per your feedback), but we need to start this movement and sooner is better (will catch\fix more issues\bugs in a shorten interval if people begin to move).

Thanks!
Comment 6 sgk 2018-05-07 00:26:11 UTC
(In reply to robert.ayrapetyan from comment #5)

> The gfortran's issue I'm talking about is related to:
>
> ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by
> /usr/local> \\/lib/gcc48/libgfortran.so.3 not found from gfortran.so library

That isn't a gfortran issue.  This is an issue with FreeBSD's
ldd, ldconfig, and the highjacking of a filename.

-% gfortran7 -o z h.f90
% ldd z
z:
        libgfortran.so.4 => /usr/local/lib/gcc7/libgfortran.so.4 (0x200645000)
        libm.so.5 => /lib/libm.so.5 (0x200a17000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x200a4b000)
        libquadmath.so.0 => /usr/local/lib/gcc7/libquadmath.so.0 (0x200a63000)
        libc.so.7 => /lib/libc.so.7 (0x200ca3000)


% ldconfig -r | grep libgcc
        6:-lgcc_s.1 => /lib/libgcc_s.so.1
        218:-lgccpp-threaded.1 => /usr/local/lib/libgccpp-threaded.so.1
        679:-lgccpp.1 => /usr/local/lib/libgccpp.so.1
        715:-lgcc_s.1 => /usr/local/lib/gcc7/libgcc_s.so.1

The problem is trivially solved by a number of means. (1)
If clang is the system compiler, then rename libgcc_s.so.1
to libclang_s.so.1.  It's rather odd that clang uses gcc
in the name of an important runtime library.  (2) Bump the
the major number on FreeBSD's libgcc_s.so.1 from 1 to 2 to
indicate that FreeBSD's libgcc_s has nothing to do with GCC.
You then have libgcc_s.so.2 and ldd should be able to 
differentiate between 1 and 2.  (3) Use -rpath.  (4)
Use -Wl to explicitly find the right libgcc_s.

This is not a problem with gfortran and naively changing the
default to flang is questionable. 1) There is no release version
of flang.  2) flang developers do not participate in discussions
with users (see crickets on flang-dev list).  3) AFAICT, flang
developers do not accept patches from outside of Nvidia.
Comment 7 Johannes M Dieterich freebsd_committer 2018-05-07 02:14:56 UTC
As the maintainer of devel/flang I agree with Steve here. flang is a) not yet ready to replace gfortran (for the reasons Steve discussed) and b) only available for amd64. Personally, I hope this will change eventually and I hope we (as in FreeBSD) can help upstream by filing bug reports and vetting the compiler.

I assume that you, Steve, filed these 100 internal compiler errors with flang upstream so that we can track progress?

Alternatively, could we get the Fortran test suite into the ports tree? I wouldn't mind having that myself to check flang updates.

I see as a more viable alternative to get BLAS and LAPACK to Fortran-free implementations and am working on making blis and flame viable. Reference source-code implementations for BLAS/LAPACK are anyways a bad idea to use in production. That'll still leave a bunch of Fortran ports but at least it would be a start.
Comment 8 sgk 2018-05-07 04:07:17 UTC
(In reply to Johannes M Dieterich from comment #7)

> I assume that you, Steve, filed these 100 internal compiler
> errors with flang upstream so that we can track progress?

The flang developers have never responded to an initial bug
report nor a post in flang-dev concerning a roadmap for an
official release.  Others have reported a few of the bugs
that can be easily reproduced by trying to compile the gfortran
testsuite.  Preparing 100+ bug reports is beyond what I'm
prepared to do.

> Alternatively, could we get the Fortran test suite into
> the ports tree? I wouldn't mind having that myself to
> check flang updates.

I am unaware of any stand alone testsuite.  gfortran's testsuite
tests for conformance where this includes programs that should
compile and execute, and programs that should fail and the
Fortran compiler must issue an error message.  The former
should compile and execute with flang.  The latter will fail,
but the testsuire isn't set up to test for flang error
messages.

> I see as a more viable alternative to get BLAS and LAPACK
> to Fortran-free implementations and am working on making
> blis and flame viable. Reference source-code implementations
> for BLAS/LAPACK are anyways a bad idea to use in production.

Never heard of blis and flame. I'll check them out. One thing
about Netlib BLAS and LAPACK is that the algorithms are designed
by well-recognized numerical analysist.  Yes, these routines
are considered reference implementations, but the people behind
them are well-respected.

Robert pointed out that flang is almost F2003 complaint, which
is commendable.  Unfortunately, the current Fortran standard
is Fortran 2008, and Fortran 2018 should be ratified within
the next few months.  gfortran, other than a bugs, is Fortran
2003 compliant, implements a significant portion of F2008, and
has implemented some of the upcoming Fortran 2018 standard.  I
know because I either implemented the features or probably
reviewed the patches that do.
Comment 9 Johannes M Dieterich freebsd_committer 2018-05-07 04:25:57 UTC
(In reply to sgk from comment #8)
Alright, so we do not know what the current status is w.r.t. the test suite. I want to track this.

I was referring to the gfortran test suite you referenced. Could you provide a link to the source?

I believe you will find blis and flame to have more than the required pedigree (van de Geijn).
Comment 10 robert.ayrapetyan 2018-05-07 04:29:47 UTC
Meanwhile would you accept a less "destructive" OPTION FLANG for this port (like it was done for math/R)? Thanks.
Comment 11 Johannes M Dieterich freebsd_committer 2018-05-07 04:38:37 UTC
(In reply to robert.ayrapetyan from comment #10)
Not speaking for fortran@: I would support such an option. If so, please provide similar option for math/blas.
Comment 12 sgk 2018-05-07 04:49:46 UTC
(In reply to Johannes M Dieterich from comment #9)


> I was referring to the gfortran test suite you referenced.
> Could you provide a link to the source?

It's where all GCC testsuite code is 

https://gcc.gnu.org/viewcvs/gcc/trunk/gcc/testsuite/

you want the gfortran.dg and gfortran.fortran-torture
directories.

The code is written to use the dejagnu framework.  Codes
that should compile and execute have an initial line
of '! { dg-do run }'.  There may be options and other
dejagnu directives in additional lines.
Comment 13 sgk 2018-05-07 05:01:22 UTC
(In reply to robert.ayrapetyan from comment #10)

> Meanwhile would you accept a less "destructive" OPTION FLANG
> for this port (like it was done for math/R)? Thanks.

IMHO, it is the only logical choice.  It would be nice to 
have a hook in ports/Mk/Uses/fortran.mk that can be set
in /etc/make.conf to set the desired compiler.  Then ports 
could have an 'OPTION FLANG' and/or 'OPTION GFORTRAN' knob.

Note, I know OpenCoarray requires a compiler that supports
coarray Fortran,  I'm not sure if flang supports coarrays.
Comment 14 robert.ayrapetyan 2018-05-07 05:38:57 UTC
Created attachment 193128 [details]
necessary changes for switching to flang

A less-destructive FLANG switch (OPTION).
Comment 15 robert.ayrapetyan 2018-05-07 05:39:31 UTC
Created attachment 193129 [details]
poudriere log for a patched port
Comment 16 robert.ayrapetyan 2018-05-07 06:39:09 UTC
(In reply to Johannes M Dieterich from comment #11)
Doesn't math/blas simply referencing math/lapack? I've checked make config inside math/blas - it successfully derives all lapack's options (having no own options).
Thanks.
Comment 17 robert.ayrapetyan 2018-05-07 07:00:41 UTC
Created attachment 193140 [details]
necessary changes for switching to flang (amd64)
Comment 18 sgk 2018-05-07 16:26:14 UTC
(In reply to Johannes M Dieterich from comment #7)

> As the maintainer of devel/flang I agree with Steve here.
> flang is a) not yet ready to replace gfortran (for the
> reasons Steve discussed) and b) only available for amd64.
> Personally, I hope this will change eventually and I hope
> we (as in FreeBSD) can help upstream by filing bug reports
> and vetting the compiler.

OK, guys, I'm really confused here.  Robert wants to make
flang the default Fortran compiler and Johannes is the
flang maintainer.  So, 

% su
password: *************
% cd /usr/ports
% svn update
% portmaster -Byd devel/flang
...
===>>> devel/flang >> devel/flang-clang (1/1)

===>>> Port directory: /usr/ports/devel/flang-clang

        ===>>> This port is marked BROKEN
        ===>>> fails to package

        ===>>> If you are sure you can build it, remove the
               BROKEN line in the Makefile and try again.


Is flang really ready to be used as a build/use dependency
in the Ports Collection?
Comment 19 robert.ayrapetyan 2018-05-07 17:39:20 UTC
(In reply to sgk from comment #18)
Well, after discussing here and other related threads it doesn't look like a good idea to make flang a default compiler right now, community and ports are not ready yet. 

From the other side, making flang a default compiler on certain ports should not break anything, e.g. here is armadillo built against mix of flang\gfortran:

ldd /usr/local/lib/libarmadillo.so
/usr/local/lib/libarmadillo.so:
	libopenblasp.so => /usr/local/lib/libopenblasp.so (0x801400000)
	liblapack.so.4 => /usr/local/lib/liblapack.so.4 (0x802400000)
	libhdf5.so.101 => /usr/local/lib/libhdf5.so.101 (0x802e00000)
	libsz.so.2 => /usr/local/lib/libsz.so.2 (0x80337d000)
	libz.so.6 => /lib/libz.so.6 (0x803596000)
	libm.so.5 => /lib/libm.so.5 (0x8037af000)
	libarpack.so.1 => /usr/local/lib/libarpack.so.1 (0x8039da000)
	libsuperlu.so.5 => /usr/local/lib/libsuperlu.so.5 (0x803c83000)
	libc++.so.1 => /usr/lib/libc++.so.1 (0x803f04000)
	libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x8041cb000)
	libc.so.7 => /lib/libc.so.7 (0x800824000)
	libgfortran.so.3 => /usr/local/lib/gcc6/libgfortran.so.3 (0x8043e9000)
	libquadmath.so.0 => /usr/local/lib/gcc6/libquadmath.so.0 (0x804710000)
	libthr.so.3 => /lib/libthr.so.3 (0x80494f000)
	libflang.so => /usr/local/flang/lib/libflang.so (0x804b77000)
	libflangrti.so => /usr/local/flang/lib/libflangrti.so (0x804f3b000)
	libompstub.so => /usr/local/flang/lib/libompstub.so (0x80514e000)
	libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x805353000)
	libopenblas.so => /usr/local/lib/libopenblas.so (0x805600000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x8063d7000)
	librt.so.1 => /usr/lib/librt.so.1 (0x8065e5000)
	libomp.so => /usr/local/llvm50/lib/libomp.so (0x8067ea000)
	libelf.so.2 => /lib/libelf.so.2 (0x806a74000)

There are 174 ports with USE=fortran in ports collection. I would rather add an optional FLANG compile param for all of them (on amd64). Of course flang should become a replacement in a long-term perspective (if FreeBSD will continue a clang movement).

Regarding broken flang-clang - it's a basic pkg-plist issue, I've committed a fix here - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222254#c12.
Comment 20 sgk 2018-05-07 18:21:34 UTC
(In reply to robert.ayrapetyan from comment #19)
>
> From the other side, making flang a default compiler on
> certain ports should not break anything, e.g. here is
> armadillo built against mix of flang\gfortran:

Are you sure?  After hacking devel/flang to install.
Simple testing by trying to compile and compile and
run programs from gfortran's testsuite I have found

% cd gfortran.fortran-torture/compile
implicit_1.f90
F90-S-0000-Internal compiler error. create_ref:IM_BASE op#2 not based
  sym     302 (implicit_1.f90: 22)
F90-S-0000-Internal compiler error. size_of: attempt to size assumed
  size character       0  (implicit_1.f90: 31)
  0 inform,   0 warnings,   2 severes, 0 fatal for foo

% cd ../execute
Wrong code generated for data_3.f90, dep_fails.f90, der_io.f90,
  entry_6.f90, forall_3.f90, forall_5.f90, forall_6.f90,
  forall_7.f90, getarg_1.f90, intrinsic_nearest.f90,
  intrinsic_pack.f90, intrinsic_sign.f90, where18.f90,
  where_7.f90, and where_8.f90

This might be a gfortran extension, but I suspect a bug in flang.
entry_2.f90
F90-S-0045-Data type of entry point e4 disagrees with function
f3 (entry_2.f90: 22)
F90-S-0045-Data type of entry point e3 disagrees with function
f3 (entry_2.f90: 21)
  0 inform,   0 warnings,   2 severes, 0 fatal for f3

Code generation bugs
entry_4.f90
/tmp/entry_4-b1e98a.ll:167:14: error: expected '=' after instruction id
        ret void %18, !dbg !35

entry_7.f90
/tmp/entry_7-7901bf.ll:83:52: error: pointers to void are invalid -
  use i8* instead
        %35 = bitcast i64* %__master_entry_rslt346 to void*

entry_8.f90
/tmp/entry_8-25b70f.ll:87:14: error: expected '=' after instruction id
        ret void %24, !dbg !32
                    ^
forall_1.f90
F90-F-0000-Internal compiler error. make_same_idx: illegal forall       2  (forall_1.f90: 59)

There isn't an internal procedure in this program.  This is a 
bogus error message.
intrinsic_associated.f90
F90-W-0155-END statement for internal procedure should be END SUBROUTINE (intrinsic_associated.f90: 125)
  0 inform,   1 warnings,   0 severes, 0 fatal for associated_2

Flang does not support the TRAILZ intrinsic
intrinsic_trailz.f90
F90-S-0038-Symbol, trailz, has not been explicitly declared (intrinsic_trailz.f90
)
  0 inform,   0 warnings,   1 severes, 0 fatal for test_trailz


BTW, FreeBSD's bugzilla is probably a poor place to have a discussion
concerning potential problems with flang.

PS: for those interested more thorough testing see

https://groups.google.com/forum/#!searchin/comp.lang.fortran/flang%7Csort:date/comp.lang.fortran/AIHRQ2kJv3c/0BhngsA-BAAJ
Comment 21 robert.ayrapetyan 2018-05-07 22:00:58 UTC
Created attachment 193165 [details]
patch with a for loop for non-flang archs

Because of the weird way of Makefile's processing logic, we have to EXCLUDE unsupported architectures rather than include only working one. Now it builds fine in e.g. i386 env.
Comment 22 robert.ayrapetyan 2018-05-07 22:03:20 UTC
(In reply to sgk from comment #20)
You're right, it doesn't seem production grade. So proposing just an optional switch for now. Thanks!
Comment 23 Johannes M Dieterich freebsd_committer 2018-05-08 02:41:16 UTC
(In reply to sgk from comment #18)
Steve, I'd really appreciate if you'd be less snarky about this. First, I am not released from mentorship yet, hence cannot commit patches without mentor approval and my main mentor has been and is currently only in very limited capacity able to review patches. We are working on assigning a new mentor.

Lastly, flang-clang pkg-list breaking happens b/c of devel/llvm50 updates that are not being coordinated with me. To insinuate that would be the fault of my maintainership or the flang compiler is not OK.