If I read the value of the sysctl kern.supported_archs on an arm64 FreeBSD 13.0-RELEASE system, I get the output kern.supported_archs: aarch64 armv7 However, this list is incomplete. Clearly, the kernel is also capable of executing armv6 binaries. This unfortunately means that Poudriere refuses to install an armv6 jail on the system, making it difficult for me to test armv6 ports. It seems like this is the consequence of unfortunate programming. sys/kern/kern_mib.c defines the sysctl like such: --- #ifdef COMPAT_FREEBSD32 #define MACHINE_ARCHES MACHINE_ARCH " " MACHINE_ARCH32 #else #define MACHINE_ARCHES MACHINE_ARCH #endif #endif SYSCTL_STRING(_kern, OID_AUTO, supported_archs, CTLFLAG_RD | CTLFLAG_MPSAFE, MACHINE_ARCHES, 0, "Supported architectures for binaries"); --- so if COMPAT_FREEBSD32 is enabled, the kernel claims support for exactly one additional 32 bit architecture, which is clearly insufficient: (a) there can be multiple supported 32 bit architectures and (b) support for 32 bit programs may depend on processor features. For example, on arm64 not all cores support executing 32 bit binaries, but the way the sysctl is set up, the kernel just wrongly claims it can, probably failing only when execution is tried (I have no such system to test this, but e.g. the Apple M1 chip is like this). This seems quite unexpected. Please fix the way kern.supported_archs is set up such that the list reflects both armv6 and armv7 for arm64 cores and that only if the AArch32 execution state is supported at all.
(In reply to Robert Clausecker from comment #0) On armv7 hardware, such as cortex-a7, armv6 likely is not listed as well. (I'm not sure if the armv7 kernel would support an armv6 chroot vs. not.) (A recent U-Boot update invalidated my root-on-USB context for cortex-a7 so I've not checked kern.supported_archs . I've yet to deal with the change on the 2 cortex-a7 contexts that I've access to.) (I've no clue of AArch32's implications for completeness of coverage of armv6 for chroot activity. FreeBSD has a separate kernel build for armv6, only targeting the older, pre-armv7, RPi*'s, if I understand right. But that does not determine the chroot handling.)
(In reply to Mark Millard from comment #1) Yes, on armv7, armv6 is not listed, either. This seems like an oversight. > (I've no clue of AArch32's implications for completeness of coverage of armv6 for chroot activity. FreeBSD has a separate kernel build for armv6, only targeting the older, pre-armv7, RPi*'s, if I understand right. But that does not determine the chroot handling.) As far as usermode is concerned, it should be fully backwards compatible. However, it does support unaligned memory access which armv6 does not. So perhaps the kernel would have to configure the CPU to trap on unaligned access while armv6 or armv7 binaries are runninig? Not sure.
(In reply to Robert Clausecker from comment #2) I put back older U-Boot materials and have the OPi+2e booting root-on-USB again. # sysctl kern.supported_archs kern.supported_archs: armv7 QUOTE However, it does support unaligned memory access which armv6 does not. END QUOTE As I remember, FreeBSD set a register field that disabled unaligned access for armv7, something like SCTLR bit[1]==1 was used. For a time things did not work until presumptions of unaligned access were removed, as I remember. clang itself had problems but llvm had dealt with a similar alignment requirement on SPARC and updated for a FreeBSD context. (Late 2015/early 2016.) So, I do not expect any support for unaligned access in FreeBSD for armv7.
(In reply to Robert Clausecker from comment #2) Hmm. Looking on the web ( https://en.wikipedia.org/wiki/ARM_architecture ): QUOTE ARMv6 and later, except some microcontroller versions, support unaligned accesses for half-word and single-word load/store instructions with some limitations, such as no guaranteed atomicity. END QUOTE So if FreeBSD for armv6 supports such unaligned accesses, then its armv7 kernel would have more to do than it now does before armv6 could be supported in full generality.
(In reply to Mark Millard from comment #4) I got it *backwards*: all the alignment issues and fixes lead to *disabling* the alignment requirement for armv7. I correctly remembered the presence of the activity but not the end result that stopped that type of activity. I finally found the old material in the E-mail archive: QUOTE (mid 2016): I think FreeBSD is the only major OS left that is enforcing strict alignment on armv6/v7 and it causes a lot of trouble for ports and other 3rd party software, and prevents us from enabling certain compiler options and optimizations. I'm very close to a commit to stop enforcing strict alignment (clear rather than CPU_CONTROL_AFLT_ENABLE). I've been testing it yesterday and today, and haven't run into any trouble at all. -- Ian Good to know. I had submitted at least one port bug report that will likely need to be canceled if this goes through. Effectively its an ABI change allowing a wider variety of code to be compliant. It was partly all that testing you did a few months ago, and the PRs and discussions coming out of that, which are driving these changes. If I could get away with procrastinating a bit more, I probably would (always too busy), but with the big hardfloat abi change and with a code freeze coming up later this week, this seems like the last chance to do some disruptive changes that are long overdue. Is the kernel involved in emulating access/instructions via some technique for misaligned access for armv6/armv7 for some types of instructions? Are there performance issues/tradeoffs that might contribute to sometimes choosing to be careful about alignment? Nope, no emulation, the hardware is able to do this, we've just always run with alignment faults enabled, partly because base freebsd already has to work on other strict-alignment hardware anyway. The driver of this change is ports more than anything -- increasingly you run into code that assumes #ifdef __arm__ is sufficient to mean "unaligned access will work". There are a few arm instructions that still require alignment, but (at least in theory) the compiler knows about that and only emits those instructions when it knows they're safe (such as it knowing that the stack stays aligned to 8-byte boundaries in non-leaf functions). We'll see; everything seems okay in testing I've done so far. Performance-wise, there is a cost for unaligned access. The hardware has to do more work so unaligned accesses take extra cycles. On the other hand, if the data is unaligned, you also have to use extra cycles, potentially a lot of them, to copy-align the data or access it a byte at a time and reassmble it in a register. In theory this should be faster than doing copy-align stuff. END QUOTE And in another place: QUOTE 4. ARM isn't as happy about unaligned accesses as x86. That's mostly not the case anymore. Using load/store-doubleword or load/store-multiple instructions requires 4-byte-aligned values (not a typo: doubleword access requires word alignment). Everything smaller than doubleword access in userland these days can be unaligned. The optimizer can combine adjacent 32-bit accesses into doubleword -instruction accesses, leading to alignment faults with unaligned data, but that shouldn't be the case here because malloc'd memory is aligned. -- Ian END QUOTE
Any progress on this issue?
(In reply to Robert Clausecker from comment #0) Looks to me like /usr/main-src/sys/arm64/include/param.h having something like: #define MACHINE_ARCH32 armv7 " " armv6 would deal with the issue. The only use of MACHINE_ARCH32 seems to be: #define MACHINE_ARCHES MACHINE_ARCH " " MACHINE_ARCH32
(In reply to Mark Millard from comment #7) I should have listed the MACHINE_ARCHES uses as well: # grep -r MACHINE_ARCHES /usr/main-src/ | more /usr/main-src/sys/riscv/include/param.h:#define MACHINE_ARCHES "riscv64 riscv64sf" /usr/main-src/sys/kern/kern_mib.c:#ifndef MACHINE_ARCHES /usr/main-src/sys/kern/kern_mib.c:#define MACHINE_ARCHES MACHINE_ARCH " " MACHINE_ARCH32 /usr/main-src/sys/kern/kern_mib.c:#define MACHINE_ARCHES MACHINE_ARCH /usr/main-src/sys/kern/kern_mib.c: MACHINE_ARCHES, 0, "Supported architectures for binaries"); Looks like: #define MACHINE_ARCH32 "armv7 armv6" would be the idiom, not what I listed in comment #6 : strings are needed so I had quotes missing.
(In reply to Mark Millard from comment #8) Ingore my comment #6 and comment #7 pair: sysctl hw.machine_arch would return: armv7 armv6 not just the appropriate one. (I missed a MACHINE_ARCH32 reference.)
(In reply to Mark Millard from comment #9) I should have said: Ignore my comment #7 and comment #8 pair: . . .
(In reply to Robert Clausecker from comment #0) One can have multiple /boot/ker*/ ( /boot/kernel/ being the default one used ). (Naming is just for illustration.) So a separate kernel built with with sys/arm64/include/param.h modified to indicate: #define MACHINE_ARCH32 "armv6" could be built and put in place, say in /boot/kerarmv6/ . Then a very modal way of selecting between armv7 and armv6 support is to reboot and select the kernel to use via the UEFI loader. No dynamic selection would be provided by this but is would allow for some types of activity.
(In reply to Mark Millard from comment #11) Better naming convention for reading my notes would have been: /boot/ker_aarch64_with_armv6_32bit/ (Not that one would be likely to use a name that was that long for one's aternate kernel directory. I've no clue if there is a limit on the name length.)
Downstream in CheriBSD we use a more complex scheme for alternate ABIs, and riscv64 used to make use of it to handle soft-float vs hard-float. For this case, I would say that you want a custom sv_machine_arch in sys/arm64/arm64/elf32_freebsd.c that returns either "armv6" or "armv7" based on the currently executing process. That will handle hw.machine_arch. For hw.supported_arches, I would just add a new cpu_supported_arches() hook that is used if it is non-null and define it to a valid function that returns "aarch64 armv6 armv7" if compat_32bit is supported on aarch64.