Bug 256897 - lang/go: Fails to build on armv7: runtime: this CPU has no floating point hardware, so it cannot run this GOARM=7 binary
Summary: lang/go: Fails to build on armv7: runtime: this CPU has no floating point har...
Status: Open
Alias: None
Product: Ports & Packages
Classification: Unclassified
Component: Individual Port(s) (show other bugs)
Version: Latest
Hardware: arm Any
: --- Affects Some People
Assignee: Dmitri Goutnik
URL:
Keywords: needs-patch, needs-qa
: 256903 (view as bug list)
Depends on:
Blocks:
 
Reported: 2021-06-30 02:22 UTC by Brad Davis
Modified: 2021-08-22 08:56 UTC (History)
7 users (show)

See Also:
bugzilla: maintainer-feedback? (dmgk)
koobs: merge-quarterly?


Attachments
patch to expose AT_HWCAP* auxvec (805 bytes, patch)
2021-07-01 04:22 UTC, Brad Davis
no flags Details | Diff
lang/go/files/patch-src_runtime_os__freebsd__arm.go (322 bytes, text/plain)
2021-07-01 04:24 UTC, Brad Davis
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Brad Davis freebsd_committer 2021-06-30 02:22:57 UTC
I have been trying to figure out why go won't build in an armv7 jail on a arm64 builder (AWS Graviton2 instance w/ HW FP).

This is the error:

runtime: this CPU has no floating point hardware, so it cannot run
this GOARM=7 binary. Recompile using GOARM=5.

So I added some debugging:

diff --git a/src/runtime/os_freebsd.go b/src/runtime/os_freebsd.go
index 730973a202..2ff154ce32 100644
--- a/src/runtime/os_freebsd.go
+++ b/src/runtime/os_freebsd.go
@@ -405,6 +405,7 @@ func sysauxv(auxv []uintptr) {
                        timekeepSharedPage = (*vdsoTimekeep)(unsafe.Pointer(val))
                }
 
+               print("calling archauxv with tag = ", tag, ", val = ", val, "\n")
                archauxv(tag, val)
        }
}
diff --git a/src/runtime/os_freebsd_arm.go b/src/runtime/os_freebsd_arm.go
index 3feaa5e225..a894e7de5e 100644
--- a/src/runtime/os_freebsd_arm.go
+++ b/src/runtime/os_freebsd_arm.go
@@ -15,6 +15,7 @@ func checkgoarm() {
        if goarm > 5 && cpu.HWCap&_HWCAP_VFP == 0 {
                print("runtime: this CPU has no floating point hardware, so it cannot run\n")
                print("this GOARM=", goarm, " binary. Recompile using GOARM=5.\n")
+               print("cpu.HWCap=", cpu.HWCap, "; vfp=", _HWCAP_VFP, ".\n")
                exit(1)
        }
        if goarm > 6 && cpu.HWCap&_HWCAP_VFPv3 == 0 {

and built a new bootstrap using github.com/dmgk/go-bootstrap repo.  Which showed me that cpu.HWCap is 0:

Building Go cmd/dist using /wrkdirs/usr/ports/lang/go/work/go-freebsd-arm7-bootstrap. ()
cmd/dist
calling archauxv with tag = 3, val = 65588
calling archauxv with tag = 4, val = 32
calling archauxv with tag = 5, val = 5
calling archauxv with tag = 6, val = 4096
calling archauxv with tag = 8, val = 0
calling archauxv with tag = 9, val = 471532
calling archauxv with tag = 7, val = 0
calling archauxv with tag = 24, val = 83886082
calling archauxv with tag = 15, val = 4294959023
calling archauxv with tag = 18, val = 1300139
calling archauxv with tag = 16, val = 4294958959
calling archauxv with tag = 17, val = 64
calling archauxv with tag = 19, val = 4
calling archauxv with tag = 20, val = 4294958944
calling archauxv with tag = 21, val = 12
calling archauxv with tag = 22, val = 4294959152
calling archauxv with tag = 23, val = 3
calling archauxv with tag = 27, val = 1
calling archauxv with tag = 28, val = 5
calling archauxv with tag = 29, val = 4294956876
calling archauxv with tag = 30, val = 55
calling archauxv with tag = 31, val = 4294956900
calling archauxv with tag = 32, val = 4294959088
runtime: this CPU has no floating point hardware, so it cannot run
this GOARM=7 binary. Recompile using GOARM=5.
cpu.HWCap=0; vfp=64.

So far I have tracked down these parts:

We never set cpu.HWCap here:
https://github.com/golang/go/blob/master/src/runtime/os_freebsd_arm.go#L34-L40

Because on FreeBSD we have_AT_HWCAP = 25
https://github.com/golang/go/blob/master/src/runtime/os_freebsd.go#L402-L419

I am not sure where to go from here.
Comment 1 Mikael Urankar freebsd_committer 2021-06-30 13:38:44 UTC
(In reply to Brad Davis from comment #0)
go retrieves the auxiliary vector with 'func sysauxv' in src/runtime/os_freebsd.go

If I understand this correctly the compat32 code doesn't fill the auxiliary info vector (I suppose it should be handled by elf32_freebsd_copyout_auxargs in sys/arm64/arm64/elf32_machdep.c but I can't find the definition of this function)

This code returns 'hwcap 2078934' on my nvidia armv7 board and 'hwcap 0' on my aarch64 board (with an armv7 chroot)

#include <stdlib.h>
#include <stdio.h>
#include <sys/auxv.h>

int
main(int argc, char **argv)
{
        unsigned long hwcap;
        elf_aux_info(AT_HWCAP, &hwcap, sizeof(hwcap));
        printf("hwcap %lu\n", hwcap);

        return(0);
}
Comment 2 Robert Clausecker 2021-06-30 14:00:54 UTC
(In reply to Mikael Urankar from comment #1)

Note that the elf_aux_info call in your example fails with ENOENT, so the value printed is undefined.  The example from bug #256903 has error handling to make this case visible.
Comment 3 Brad Davis freebsd_committer 2021-07-01 04:22:02 UTC
Created attachment 226146 [details]
patch to expose AT_HWCAP* auxvec
Comment 4 Brad Davis freebsd_committer 2021-07-01 04:24:27 UTC
Created attachment 226147 [details]
lang/go/files/patch-src_runtime_os__freebsd__arm.go

This is more complicated because this patch needs to be in the bootstrap and in the ports tree as: lang/go/files/patch-src_runtime_os__freebsd__arm.go
Comment 5 Mikael Urankar freebsd_committer 2021-07-01 06:07:03 UTC
(In reply to Brad Davis from comment #4)
Why do you want to change the value of HWCAP_VFP* on arm? They are correct per https://github.com/freebsd/freebsd-src/blob/main/sys/arm/include/elf.h#L90
Comment 6 Scott Long freebsd_committer 2021-07-01 08:43:58 UTC
I didn't see these 32bit ABI definitions before.  Ugh, this means that we can't share the elf_hwcap and elf_hwcap2 words between 32bit and 64bit ARM; elf32_machdep.c will need to populate this on its own.
Comment 7 Robert Clausecker 2021-07-01 09:31:17 UTC
Filed an upstream issue to have this worked around in the Go toolchain.

https://github.com/golang/go/issues/47006

This is something that should really be addressed upstream as otherwise only armv7 FreeBSD binaries built by the ports-patched toolchain will be able to run on arm64 FreeBSD 13.0-RELEASE.  However, as a temporary measure I can write a patch for the toolchain implementing a workaround as described in the Go issue listed above if desired.
Comment 8 Robert Clausecker 2021-07-01 09:34:50 UTC
(In reply to Scott Long from comment #6)

IIRC feature support for AArch32 is indicated by different special registers than feature support for AArch64, so even if the layout was the same, the checks would have to be reimplemented as it is conceivable that a CPU supports a feature for AArch64 but not AArch32.

If desired, I can try and write a patch that determines which feature bits to set.
Comment 9 Scott Long freebsd_committer 2021-07-01 09:37:42 UTC
Yeah, I'm looking at creating new feature checks too.  I only know a little about ARM, so I'm not sure which CPU registers to check to find 32bit features.  Are they the same registers as on a native 32bit ARM CPU?
Comment 10 Robert Clausecker 2021-07-01 09:52:01 UTC
(In reply to Scott Long from comment #9)

Hi Scott,

As far as I am concerned, yes, they should be, however the register might have a different number.  Otherwise refer to the ARMv8-A Architecture Reference Manual:

    https://developer.arm.com/documentation/ddi0487/latest/

The relevant registers should be those described in section D13.2.69 and following (AArch32 instruction set attributes, processor features).  Many of the features are architectural, so you can just hard-code them to "supported".
Comment 11 John Baldwin freebsd_committer freebsd_triage 2021-07-02 17:46:26 UTC
It seems we will need new 'elf32_hwcap' and 'elf32_hwcap2' globals that are set to the same values used in sys/arm for 32-bit arm.  Note that the port patch is wrong as it would break on a true 32-bit arm host.  We cannot change the value of HWCAP_* flags as that is an ABI breakage.  Likely we will need HWCAP32_* constants and HWCAP32_2_* constants or some such in sys/arm64/include/elf.h.
Comment 12 Peter Grehan freebsd_committer 2021-07-14 08:59:04 UTC
First cut at a fix for this at https://reviews.freebsd.org/D31175
Comment 13 Robert Clausecker 2021-07-14 10:40:31 UTC
*** Bug 256903 has been marked as a duplicate of this bug. ***
Comment 14 Robert Clausecker 2021-07-20 07:17:41 UTC
Upstream has proposed a workaround in the Go toolchain:

https://go-review.googlesource.com/c/go/+/335791/1/src/runtime/os_freebsd_arm.go

Could somebody test this and see if it successfully works around the issue?
Comment 15 commit-hook freebsd_committer 2021-07-25 08:32:37 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=bbe80bff7c3549128bd19862eea7899b3def1d7f

commit bbe80bff7c3549128bd19862eea7899b3def1d7f
Author:     Peter Grehan <grehan@FreeBSD.org>
AuthorDate: 2021-07-25 09:34:14 +0000
Commit:     Peter Grehan <grehan@FreeBSD.org>
CommitDate: 2021-07-25 09:39:32 +0000

    arm64: HWCAP/HWCAP2 aux args support for 32-bit ARM binaries.

    This fixes build/run of golang under COMPAT32 emulation.

    PR:     256897
    Reviewed by:    andrew, mmel, manu, jhb, cognet, Robert Clausecker
    Tested by:      brd, andrew, Robert Clausecker
    MFC after:      3 weeks
    Relnotes:       yes
    Sponsored by:   Rubicon Communications, LLC ("Netgate")
    Differential Revision:  https://reviews.freebsd.org/D31175

 sys/arm64/arm64/elf32_machdep.c |   5 +
 sys/arm64/arm64/identcpu.c      | 302 ++++++++++++++++++++++++++++++++++++++++
 sys/arm64/include/armreg.h      | 131 +++++++++++++++++
 sys/arm64/include/elf.h         |  29 ++++
 sys/arm64/include/md_var.h      |   4 +
 5 files changed, 471 insertions(+)
Comment 16 commit-hook freebsd_committer 2021-08-22 02:32:34 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=28e22482279f43dda9c78f3fec2189630e9b84cd

commit 28e22482279f43dda9c78f3fec2189630e9b84cd
Author:     Peter Grehan <grehan@FreeBSD.org>
AuthorDate: 2021-07-25 09:34:14 +0000
Commit:     Peter Grehan <grehan@FreeBSD.org>
CommitDate: 2021-08-22 04:17:46 +0000

    arm64: HWCAP/HWCAP2 aux args support for 32-bit ARM binaries.

    This fixes build/run of golang under COMPAT32 emulation.

    PR:     256897
    Reviewed by:    andrew, mmel, manu, jhb, cognet, Robert Clausecker
    Tested by:      brd, andrew, Robert Clausecker
    Relnotes:       yes
    Sponsored by:   Rubicon Communications, LLC ("Netgate")
    Differential Revision:  https://reviews.freebsd.org/D31175

    (cherry picked from commit bbe80bff7c3549128bd19862eea7899b3def1d7f)

 sys/arm64/arm64/elf32_machdep.c |   5 +
 sys/arm64/arm64/identcpu.c      | 302 ++++++++++++++++++++++++++++++++++++++++
 sys/arm64/include/armreg.h      | 131 +++++++++++++++++
 sys/arm64/include/elf.h         |  29 ++++
 sys/arm64/include/md_var.h      |   4 +
 5 files changed, 471 insertions(+)
Comment 17 Robert Clausecker 2021-08-22 08:56:29 UTC
As for fixing the Go port, there's a CL floating around the Go project with a fix for the problem.  Perhaps we could add it as a patch to the port?  Of course, dmgk would also have to regenerate the bootstrap toolchains with this patch so the port has a chance at building.

https://go-review.googlesource.com/c/go/+/335791/