Bug 254693

Summary: random crashes with armv7 thumb application on arm64
Product: Base System Reporter: Robert Clausecker <fuz>
Component: armAssignee: freebsd-arm (Nobody) <freebsd-arm>
Status: Closed Not A Bug    
Severity: Affects Some People CC: marklmi26-fbsd
Priority: ---    
Version: 13.0-STABLE   
Hardware: arm   
OS: Any   

Description Robert Clausecker freebsd_committer freebsd_triage 2021-04-01 10:17:38 UTC 
I'm trying to run Mecrisp Stellaris 2.5.8 on arm64 FreeBSD.  This is a Forth system written entirely in armv7 thumb2 assembly.  A FreeBSD port has existed for a while and has recently been made available as lang/mecrisp-stellaris in the ports tree.

It appears that with FreeBSD 13, rudimentary support for loading and running armv7 binaries from arm64 FreeBSD exists.  And indeed, mecrisp-stellaris loads and executes fine.  However, problems crop up once more definitions are loaded.  Specifically, the system crashes in random places with a wildly inconsistent program state.

To reproduce the issue, download Mecrisp Stellaris 2.5.8 from Sourceforge [1] onto an arm64 FreeBSD 13 system.  Navigate into the freebsd-ra directory and execute "buildcore-freebsd".  You should see a few pages of output followed by a crash.  The crash is a segmentation fault or an illegal instruction fault.

The same issue does not seem to appear on armv7 FreeBSD, so I'm a bit at a loss as to what may have caused this.

[1]: https://sourceforge.net/projects/mecrisp/files/mecrisp-stellaris-2.5.8.tar.gz
Comment 1 Robert Clausecker freebsd_committer freebsd_triage 2021-04-01 10:27:07 UTC 
One clue:  the code uses a sequence of barried instructions (DMB; DSB; ISB) instructions to synchronise caches before jumping to recently compiled code.  If these instructions are replaced with an appropriate sysarch(ARM_SYNC_ICACHE, ...) call, the problem goes away.  It is possible that I am misusing the barrier instructions, though if so, I don't understand how.
Comment 2 Robert Clausecker freebsd_committer freebsd_triage 2021-04-01 10:31:13 UTC 
My bad.  It appears that the synchronisation code is indeed incorrect.  Citing from the ARMv8 ARM, section E2.5.3:

> If software requires coherency between instruction execution and
> memory, it must manage this coherency using Context synchronization
> events and cache maintenance instructions. These can only be
> accessed from an Exception level that is higher than EL0, and
> therefore require a system call [...].

Closing this one as invalid.
Comment 3 Mark Millard 2021-04-01 11:16:47 UTC 
(In reply to Robert Clausecker from comment #2)

Using armv7 documentation instead of armv8:

https://developer.arm.com/documentation/ddi0406/c/Application-Level-Architecture/Application-Level-Memory-Model/Caches-and-memory-hierarchy/Implication-of-caches-for-the-application-programmer?lang=en

reports:

QUOTE
If software requires coherency between instruction execution and memory, it must manage this coherency using the ISB and DSB memory barriers and cache maintenance operations, see Ordering of cache and branch predictor maintenance operations. Many of these operations are only available to system software.
END QUOTE