Bug 256132 - arm64 kernels with aarch32 support should claim support for armv6 in addition to armv7
Summary: arm64 kernels with aarch32 support should claim support for armv6 in addition...
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: arm (show other bugs)
Version: 13.0-RELEASE
Hardware: arm64 Any
: --- Affects Only Me
Assignee: freebsd-arm (Nobody)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-05-24 19:29 UTC by Robert Clausecker
Modified: 2021-11-02 22:17 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Clausecker 2021-05-24 19:29:45 UTC
If I read the value of the sysctl kern.supported_archs on an arm64 FreeBSD 13.0-RELEASE system, I get the output

    kern.supported_archs: aarch64 armv7

However, this list is incomplete.  Clearly, the kernel is also capable of executing armv6 binaries.  This unfortunately means that Poudriere refuses to install an armv6 jail on the system, making it difficult for me to test armv6 ports.

It seems like this is the consequence of unfortunate programming.  sys/kern/kern_mib.c defines the sysctl like such:

---
#ifdef COMPAT_FREEBSD32
#define MACHINE_ARCHES  MACHINE_ARCH " " MACHINE_ARCH32
#else
#define MACHINE_ARCHES  MACHINE_ARCH
#endif
#endif

SYSCTL_STRING(_kern, OID_AUTO, supported_archs, CTLFLAG_RD | CTLFLAG_MPSAFE,
    MACHINE_ARCHES, 0, "Supported architectures for binaries");
---

so if COMPAT_FREEBSD32 is enabled, the kernel claims support for exactly one additional 32 bit architecture, which is clearly insufficient: (a) there can be multiple supported 32 bit architectures and (b) support for 32 bit programs may depend on processor features.  For example, on arm64 not all cores support executing 32 bit binaries, but the way the sysctl is set up, the kernel just wrongly claims it can, probably failing only when execution is tried (I have no such system to test this, but e.g. the Apple M1 chip is like this).  This seems quite unexpected.

Please fix the way kern.supported_archs is set up such that the list reflects both armv6 and armv7 for arm64 cores and that only if the AArch32 execution state is supported at all.
Comment 1 Mark Millard 2021-05-24 21:30:38 UTC
(In reply to Robert Clausecker from comment #0)

On armv7 hardware, such as cortex-a7, armv6 likely is
not listed as well. (I'm not sure if the armv7 kernel
would support an armv6 chroot vs. not.)

(A recent U-Boot update invalidated my root-on-USB context
for cortex-a7 so I've not checked kern.supported_archs .
I've yet to deal with the change on the 2 cortex-a7
contexts that I've access to.)

(I've no clue of AArch32's implications for completeness
of coverage of armv6 for chroot activity. FreeBSD has a
separate kernel build for armv6, only targeting the older,
pre-armv7, RPi*'s, if I understand right. But that does
not determine the chroot handling.)
Comment 2 Robert Clausecker 2021-05-24 22:28:50 UTC
(In reply to Mark Millard from comment #1)

Yes, on armv7, armv6 is not listed, either.  This seems like an oversight.

> (I've no clue of AArch32's implications for completeness
of coverage of armv6 for chroot activity. FreeBSD has a
separate kernel build for armv6, only targeting the older,
pre-armv7, RPi*'s, if I understand right. But that does
not determine the chroot handling.)

As far as usermode is concerned, it should be fully backwards compatible.  However, it does support unaligned memory access which armv6 does not.  So perhaps the kernel would have to configure the CPU to trap on unaligned access while armv6 or armv7 binaries are runninig?  Not sure.
Comment 3 Mark Millard 2021-05-24 23:21:37 UTC
(In reply to Robert Clausecker from comment #2)

I put back older U-Boot materials and have the OPi+2e
booting root-on-USB again.

# sysctl kern.supported_archs
kern.supported_archs: armv7

QUOTE
However, it does support unaligned memory access which armv6 does not.
END QUOTE

As I remember, FreeBSD set a register field that disabled unaligned
access for armv7, something like SCTLR bit[1]==1 was used. For a
time things did not work until presumptions of unaligned access were
removed, as I remember. clang itself had problems but llvm had dealt
with a similar alignment requirement on SPARC and updated for a
FreeBSD context. (Late 2015/early 2016.)

So, I do not expect any support for unaligned access in FreeBSD
for armv7.
Comment 4 Mark Millard 2021-05-24 23:27:13 UTC
(In reply to Robert Clausecker from comment #2)

Hmm. Looking on the web ( https://en.wikipedia.org/wiki/ARM_architecture ):

QUOTE
ARMv6 and later, except some microcontroller versions, support unaligned accesses for half-word and single-word load/store instructions with some limitations, such as no guaranteed atomicity.
END QUOTE

So if FreeBSD for armv6 supports such unaligned accesses, then its armv7
kernel would have more to do than it now does before armv6 could be
supported in full generality.
Comment 5 Mark Millard 2021-05-25 00:40:51 UTC
(In reply to Mark Millard from comment #4)


I got it *backwards*: all the alignment issues and fixes
lead to *disabling* the alignment requirement for armv7.
I correctly remembered the presence of the activity
but not the end result that stopped that type of activity.

I finally found the old material in the E-mail archive:

QUOTE (mid 2016):
I think FreeBSD is the only major OS left that is enforcing strict
alignment on armv6/v7 and it causes a lot of trouble for ports and
other 3rd party software, and prevents us from enabling certain
compiler options and optimizations.  I'm very close to a commit to
stop
enforcing strict alignment (clear rather than
CPU_CONTROL_AFLT_ENABLE).
I've been testing it yesterday and today, and haven't run into any
trouble at all.

-- Ian


Good to know. I had submitted at least one port bug report that will
likely need to be canceled if this goes through. Effectively its an
ABI change allowing a wider variety of code to be compliant.



It was partly all that testing you did a few months ago, and the PRs
and discussions coming out of that, which are driving these changes. 
If I could get away with procrastinating a bit more, I probably would
(always too busy), but with the big hardfloat abi change and with a
code freeze coming up later this week, this seems like the last chance
to do some disruptive changes that are long overdue.



Is the kernel involved in emulating access/instructions via some
technique for misaligned access for armv6/armv7 for some types of
instructions? Are there performance issues/tradeoffs that might
contribute to sometimes choosing to be careful about alignment?


Nope, no emulation, the hardware is able to do this, we've just always
run with alignment faults enabled, partly because base freebsd already
has to work on other strict-alignment hardware anyway.  The driver of
this change is ports more than anything -- increasingly you run into
code that assumes #ifdef __arm__ is sufficient to mean "unaligned
access will work".

There are a few arm instructions that still require alignment, but (at
least in theory) the compiler knows about that and only emits those
instructions when it knows they're safe (such as it knowing that the
stack stays aligned to 8-byte boundaries in non-leaf functions).  We'll
see; everything seems okay in testing I've done so far.

Performance-wise, there is a cost for unaligned access.  The hardware
has to do more work so unaligned accesses take extra cycles.  On the
other hand, if the data is unaligned, you also have to use extra
cycles, potentially a lot of them, to copy-align the data or access it
a byte at a time and reassmble it in a register.  In theory this should
be faster than doing copy-align stuff.
END QUOTE

And in another place:

QUOTE
4. ARM isn't as happy about unaligned accesses as x86.


That's mostly not the case anymore.  Using load/store-doubleword or
load/store-multiple instructions requires 4-byte-aligned values (not a
typo: doubleword access requires word alignment).  Everything smaller
than doubleword access in userland these days can be unaligned.  The
optimizer can combine adjacent 32-bit accesses into doubleword
-instruction accesses, leading to alignment faults with unaligned data,
but that shouldn't be the case here because malloc'd memory is aligned.

-- Ian
END QUOTE
Comment 6 Robert Clausecker 2021-09-04 11:13:07 UTC
Any progress on this issue?
Comment 7 Mark Millard 2021-11-02 19:19:42 UTC
(In reply to Robert Clausecker from comment #0)

Looks to me like /usr/main-src/sys/arm64/include/param.h having something like:

#define MACHINE_ARCH32  armv7 " " armv6

would deal with the issue. The only use of MACHINE_ARCH32 seems to be:

#define       MACHINE_ARCHES  MACHINE_ARCH " " MACHINE_ARCH32
Comment 8 Mark Millard 2021-11-02 19:27:19 UTC
(In reply to Mark Millard from comment #7)

I should have listed the MACHINE_ARCHES uses as well:

# grep -r MACHINE_ARCHES /usr/main-src/ | more
/usr/main-src/sys/riscv/include/param.h:#define MACHINE_ARCHES  "riscv64 riscv64sf"
/usr/main-src/sys/kern/kern_mib.c:#ifndef MACHINE_ARCHES
/usr/main-src/sys/kern/kern_mib.c:#define       MACHINE_ARCHES  MACHINE_ARCH " " MACHINE_ARCH32
/usr/main-src/sys/kern/kern_mib.c:#define       MACHINE_ARCHES  MACHINE_ARCH
/usr/main-src/sys/kern/kern_mib.c:    MACHINE_ARCHES, 0, "Supported architectures for binaries");


Looks like:

#define MACHINE_ARCH32  "armv7 armv6"

would be the idiom, not what I listed in comment #6 : strings are
needed so I had quotes missing.
Comment 9 Mark Millard 2021-11-02 19:44:55 UTC
(In reply to Mark Millard from comment #8)

Ingore my comment #6 and comment #7 pair:

sysctl hw.machine_arch

would return:

armv7 armv6

not just the appropriate one. (I missed a MACHINE_ARCH32 reference.)
Comment 10 Mark Millard 2021-11-02 20:01:39 UTC
(In reply to Mark Millard from comment #9)

I should have said:

Ignore my comment #7 and comment #8 pair:

. . .
Comment 11 Mark Millard 2021-11-02 21:58:37 UTC
(In reply to Robert Clausecker from comment #0)

One can have multiple /boot/ker*/ ( /boot/kernel/ being the
default one used ). (Naming is just for illustration.)

So a separate kernel built with with sys/arm64/include/param.h
modified to indicate:

#define MACHINE_ARCH32  "armv6"

could be built and put in place, say in /boot/kerarmv6/ .

Then a very modal way of selecting between armv7 and
armv6 support is to reboot and select the kernel to use
via the UEFI loader.

No dynamic selection would be provided by this but is
would allow for some types of activity.
Comment 12 Mark Millard 2021-11-02 22:17:51 UTC
(In reply to Mark Millard from comment #11)

Better naming convention for reading my notes would
have been:

/boot/ker_aarch64_with_armv6_32bit/

(Not that one would be likely to use a name that
was that long for one's aternate kernel directory.
I've no clue if there is a limit on the name
length.)