I have been trying to figure out why go won't build in an armv7 jail on a arm64 builder (AWS Graviton2 instance w/ HW FP). This is the error: runtime: this CPU has no floating point hardware, so it cannot run this GOARM=7 binary. Recompile using GOARM=5. So I added some debugging: diff --git a/src/runtime/os_freebsd.go b/src/runtime/os_freebsd.go index 730973a202..2ff154ce32 100644 --- a/src/runtime/os_freebsd.go +++ b/src/runtime/os_freebsd.go @@ -405,6 +405,7 @@ func sysauxv(auxv []uintptr) { timekeepSharedPage = (*vdsoTimekeep)(unsafe.Pointer(val)) } + print("calling archauxv with tag = ", tag, ", val = ", val, "\n") archauxv(tag, val) } } diff --git a/src/runtime/os_freebsd_arm.go b/src/runtime/os_freebsd_arm.go index 3feaa5e225..a894e7de5e 100644 --- a/src/runtime/os_freebsd_arm.go +++ b/src/runtime/os_freebsd_arm.go @@ -15,6 +15,7 @@ func checkgoarm() { if goarm > 5 && cpu.HWCap&_HWCAP_VFP == 0 { print("runtime: this CPU has no floating point hardware, so it cannot run\n") print("this GOARM=", goarm, " binary. Recompile using GOARM=5.\n") + print("cpu.HWCap=", cpu.HWCap, "; vfp=", _HWCAP_VFP, ".\n") exit(1) } if goarm > 6 && cpu.HWCap&_HWCAP_VFPv3 == 0 { and built a new bootstrap using github.com/dmgk/go-bootstrap repo. Which showed me that cpu.HWCap is 0: Building Go cmd/dist using /wrkdirs/usr/ports/lang/go/work/go-freebsd-arm7-bootstrap. () cmd/dist calling archauxv with tag = 3, val = 65588 calling archauxv with tag = 4, val = 32 calling archauxv with tag = 5, val = 5 calling archauxv with tag = 6, val = 4096 calling archauxv with tag = 8, val = 0 calling archauxv with tag = 9, val = 471532 calling archauxv with tag = 7, val = 0 calling archauxv with tag = 24, val = 83886082 calling archauxv with tag = 15, val = 4294959023 calling archauxv with tag = 18, val = 1300139 calling archauxv with tag = 16, val = 4294958959 calling archauxv with tag = 17, val = 64 calling archauxv with tag = 19, val = 4 calling archauxv with tag = 20, val = 4294958944 calling archauxv with tag = 21, val = 12 calling archauxv with tag = 22, val = 4294959152 calling archauxv with tag = 23, val = 3 calling archauxv with tag = 27, val = 1 calling archauxv with tag = 28, val = 5 calling archauxv with tag = 29, val = 4294956876 calling archauxv with tag = 30, val = 55 calling archauxv with tag = 31, val = 4294956900 calling archauxv with tag = 32, val = 4294959088 runtime: this CPU has no floating point hardware, so it cannot run this GOARM=7 binary. Recompile using GOARM=5. cpu.HWCap=0; vfp=64. So far I have tracked down these parts: We never set cpu.HWCap here: https://github.com/golang/go/blob/master/src/runtime/os_freebsd_arm.go#L34-L40 Because on FreeBSD we have_AT_HWCAP = 25 https://github.com/golang/go/blob/master/src/runtime/os_freebsd.go#L402-L419 I am not sure where to go from here.
(In reply to Brad Davis from comment #0) go retrieves the auxiliary vector with 'func sysauxv' in src/runtime/os_freebsd.go If I understand this correctly the compat32 code doesn't fill the auxiliary info vector (I suppose it should be handled by elf32_freebsd_copyout_auxargs in sys/arm64/arm64/elf32_machdep.c but I can't find the definition of this function) This code returns 'hwcap 2078934' on my nvidia armv7 board and 'hwcap 0' on my aarch64 board (with an armv7 chroot) #include <stdlib.h> #include <stdio.h> #include <sys/auxv.h> int main(int argc, char **argv) { unsigned long hwcap; elf_aux_info(AT_HWCAP, &hwcap, sizeof(hwcap)); printf("hwcap %lu\n", hwcap); return(0); }
(In reply to Mikael Urankar from comment #1) Note that the elf_aux_info call in your example fails with ENOENT, so the value printed is undefined. The example from bug #256903 has error handling to make this case visible.
Created attachment 226146 [details] patch to expose AT_HWCAP* auxvec
Created attachment 226147 [details] lang/go/files/patch-src_runtime_os__freebsd__arm.go This is more complicated because this patch needs to be in the bootstrap and in the ports tree as: lang/go/files/patch-src_runtime_os__freebsd__arm.go
(In reply to Brad Davis from comment #4) Why do you want to change the value of HWCAP_VFP* on arm? They are correct per https://github.com/freebsd/freebsd-src/blob/main/sys/arm/include/elf.h#L90
I didn't see these 32bit ABI definitions before. Ugh, this means that we can't share the elf_hwcap and elf_hwcap2 words between 32bit and 64bit ARM; elf32_machdep.c will need to populate this on its own.
Filed an upstream issue to have this worked around in the Go toolchain. https://github.com/golang/go/issues/47006 This is something that should really be addressed upstream as otherwise only armv7 FreeBSD binaries built by the ports-patched toolchain will be able to run on arm64 FreeBSD 13.0-RELEASE. However, as a temporary measure I can write a patch for the toolchain implementing a workaround as described in the Go issue listed above if desired.
(In reply to Scott Long from comment #6) IIRC feature support for AArch32 is indicated by different special registers than feature support for AArch64, so even if the layout was the same, the checks would have to be reimplemented as it is conceivable that a CPU supports a feature for AArch64 but not AArch32. If desired, I can try and write a patch that determines which feature bits to set.
Yeah, I'm looking at creating new feature checks too. I only know a little about ARM, so I'm not sure which CPU registers to check to find 32bit features. Are they the same registers as on a native 32bit ARM CPU?
(In reply to Scott Long from comment #9) Hi Scott, As far as I am concerned, yes, they should be, however the register might have a different number. Otherwise refer to the ARMv8-A Architecture Reference Manual: https://developer.arm.com/documentation/ddi0487/latest/ The relevant registers should be those described in section D13.2.69 and following (AArch32 instruction set attributes, processor features). Many of the features are architectural, so you can just hard-code them to "supported".
It seems we will need new 'elf32_hwcap' and 'elf32_hwcap2' globals that are set to the same values used in sys/arm for 32-bit arm. Note that the port patch is wrong as it would break on a true 32-bit arm host. We cannot change the value of HWCAP_* flags as that is an ABI breakage. Likely we will need HWCAP32_* constants and HWCAP32_2_* constants or some such in sys/arm64/include/elf.h.
First cut at a fix for this at https://reviews.freebsd.org/D31175
*** Bug 256903 has been marked as a duplicate of this bug. ***
Upstream has proposed a workaround in the Go toolchain: https://go-review.googlesource.com/c/go/+/335791/1/src/runtime/os_freebsd_arm.go Could somebody test this and see if it successfully works around the issue?
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=bbe80bff7c3549128bd19862eea7899b3def1d7f commit bbe80bff7c3549128bd19862eea7899b3def1d7f Author: Peter Grehan <grehan@FreeBSD.org> AuthorDate: 2021-07-25 09:34:14 +0000 Commit: Peter Grehan <grehan@FreeBSD.org> CommitDate: 2021-07-25 09:39:32 +0000 arm64: HWCAP/HWCAP2 aux args support for 32-bit ARM binaries. This fixes build/run of golang under COMPAT32 emulation. PR: 256897 Reviewed by: andrew, mmel, manu, jhb, cognet, Robert Clausecker Tested by: brd, andrew, Robert Clausecker MFC after: 3 weeks Relnotes: yes Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D31175 sys/arm64/arm64/elf32_machdep.c | 5 + sys/arm64/arm64/identcpu.c | 302 ++++++++++++++++++++++++++++++++++++++++ sys/arm64/include/armreg.h | 131 +++++++++++++++++ sys/arm64/include/elf.h | 29 ++++ sys/arm64/include/md_var.h | 4 + 5 files changed, 471 insertions(+)
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=28e22482279f43dda9c78f3fec2189630e9b84cd commit 28e22482279f43dda9c78f3fec2189630e9b84cd Author: Peter Grehan <grehan@FreeBSD.org> AuthorDate: 2021-07-25 09:34:14 +0000 Commit: Peter Grehan <grehan@FreeBSD.org> CommitDate: 2021-08-22 04:17:46 +0000 arm64: HWCAP/HWCAP2 aux args support for 32-bit ARM binaries. This fixes build/run of golang under COMPAT32 emulation. PR: 256897 Reviewed by: andrew, mmel, manu, jhb, cognet, Robert Clausecker Tested by: brd, andrew, Robert Clausecker Relnotes: yes Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D31175 (cherry picked from commit bbe80bff7c3549128bd19862eea7899b3def1d7f) sys/arm64/arm64/elf32_machdep.c | 5 + sys/arm64/arm64/identcpu.c | 302 ++++++++++++++++++++++++++++++++++++++++ sys/arm64/include/armreg.h | 131 +++++++++++++++++ sys/arm64/include/elf.h | 29 ++++ sys/arm64/include/md_var.h | 4 + 5 files changed, 471 insertions(+)
As for fixing the Go port, there's a CL floating around the Go project with a fix for the problem. Perhaps we could add it as a patch to the port? Of course, dmgk would also have to regenerate the bootstrap toolchains with this patch so the port has a chance at building. https://go-review.googlesource.com/c/go/+/335791/