Bug 262836

Summary: [arm64] cacheline flush instruction (dc cvau) causes SIGSEGV from userland
Product: Base System Reporter: Dave Cottlehuber <dch>
Component: armAssignee: Andrew Turner <Andrew>
Status: Closed FIXED    
Severity: Affects Some People CC: Andrew
Priority: ---    
Version: CURRENT   
Hardware: arm64   
OS: Any   
URL: https://github.com/erlang/otp/issues/5817
Attachments:
Description Flags
https://git.sr.ht/~dch/src/commit/ae9221b85d18b6816276b8401734548f03b89013.patch none

Description Dave Cottlehuber freebsd_committer freebsd_triage 2022-03-26 16:12:27 UTC
Created attachment 232744 [details]
https://git.sr.ht/~dch/src/commit/ae9221b85d18b6816276b8401734548f03b89013.patch

building OTP-25-RC2 from source yields a segfault during build. After
some build finagling, we can narrow this down to `dc cvau` failing:


```
sudo lldb ./bin/aarch64-unknown-freebsd14.0/beam.debug.jit
(lldb) target create "./bin/aarch64-unknown-freebsd14.0/beam.debug.jit"
Current executable set to '/tmp/otp/bin/aarch64-unknown-freebsd14.0/beam.debug.jit' (aarch64).

(lldb) run
Process 40356 launched: '/tmp/otp/bin/aarch64-unknown-freebsd14.0/beam.debug.jit' (aarch64)
Process 40356 stopped
* thread #1, name = 'beam.debug.jit', stop reason = signal SIGSEGV: address access protected (fault address: 0xdee00000)
    frame #0: 0x0000000000769f54 beam.debug.jit`__clear_cache(start=0x00000000dee00000, end=0x00000000dee0294c) at clear_cache.c:119:7
   116      const size_t dcache_line_size = 4 << ((ctr_el0 >> 16) & 15);
   117      for (addr = xstart & ~(dcache_line_size - 1); addr < xend;
   118           addr += dcache_line_size)
-> 119        __asm __volatile("dc cvau, %0" ::"r"(addr));
   120    }
   121    __asm __volatile("dsb ish");
   122

(lldb) bt
* thread #1, name = 'beam.debug.jit', stop reason = signal SIGSEGV: address access protected (fault address: 0xdee00000)
  * frame #0: 0x0000000000769f54 beam.debug.jit`__clear_cache(start=0x00000000dee00000, end=0x00000000dee0294c) at clear_cache.c:119:7
    frame #1: 0x00000000003feb60 beam.debug.jit`BeamAssembler::_codegen(this=0x00000000d4b0e000, allocator=0x000000008b16c018, executable_ptr=<unavailable>, writable_ptr=<unavailable>) at beam_jit_common.cpp:128:5
    frame #2: 0x00000000003ea4e8 beam.debug.jit`BeamGlobalAssembler::BeamGlobalAssembler(this=0x00000000d4b0e000, allocator=0x000000008b16c018) at beam_asm_global.cpp:71:9
    frame #3: 0x00000000004034dc beam.debug.jit`::beamasm_init() at beam_jit_main.cpp:256:15
    frame #4: 0x000000000049c908 beam.debug.jit`erl_init(ncpu=32, proc_tab_sz=262144, legacy_proc_tab=0, port_tab_sz=65536, port_tab_sz_ignore_files=0, legacy_port_tab=0, sys_proc_outst_req_lim=64, time_correction=1, time_warp_mode=ERTS_NO_TIME_WARP_MODE, node_tab_delete_delay=60, db_spin_count=ERTS_DB_SPNCNT_NORMAL) at erl_init.c:355:5
    frame #5: 0x000000000049b0a8 beam.debug.jit`erl_start(argc=1, argv=0x000000008144b690) at erl_init.c:2424:5
    frame #6: 0x00000000003c0828 beam.debug.jit`main(argc=1, argv=0x000000008144b690) at erl_main.c:30:5
    frame #7: 0x00000000003c063c beam.debug.jit`__start(argc=1, argv=0x000000008144b690, env=0x000000008144b6a0, cleanup=<unavailable>) at crt1_c.c:72:7
    frame #8: 0x0000236d2a2f60d8 ld-elf.so.1`.rtld_start at rtld_start.S:41

(lldb) x/8i $pc-16
    0x769f44: 0xcb0903ea   neg    x10, x9
    0x769f48: 0x8a00014a   and    x10, x10, x0
    0x769f4c: 0xeb01015f   cmp    x10, x1
    0x769f50: 0x540000a2   b.hs   0x769f64                  ; <+76> at clear_cache.c:121:3
->  0x769f54: 0xd50b7b2a   dc     cvau, x10
    0x769f58: 0x8b09014a   add    x10, x10, x9
    0x769f5c: 0xeb01015f   cmp    x10, x1
    0x769f60: 0x54ffffa3   b.lo   0x769f54                  ; <+60> at clear_cache.c:119:7

(lldb) x/8i start
    0xdee00000: 0xaa1f03e2   mov    x2, xzr
    0xdee00004: 0xaa1903e3   mov    x3, x25
    0xdee00008: 0xaa1a03e4   mov    x4, x26
    0xdee0000c: 0xaa0403e8   mov    x8, x4
    0xdee00010: 0x91020269   add    x9, x19, #0x80            ; =0x80
    0xdee00014: 0xf100ed1f   cmp    x8, #0x3b                 ; =0x3b
    0xdee00018: 0x54000240   b.eq   0xdee00060
    0xdee0001c: 0x37080148   tbnz   w8, #0x1, 0xdee00044

(lldb) x/8i end
    0xdee0294c: 0x00000000   udf    #0x0
    0xdee02950: 0x00000000   udf    #0x0
    0xdee02954: 0x00000000   udf    #0x0
    0xdee02958: 0x00000000   udf    #0x0
    0xdee0295c: 0x00000000   udf    #0x0
    0xdee02960: 0x00000000   udf    #0x0
    0xdee02964: 0x00000000   udf    #0x0
    0xdee02968: 0x00000000   udf    #0x0

(lldb) x/8i $x0
    0xdee00000: 0xaa1f03e2   mov    x2, xzr
    0xdee00004: 0xaa1903e3   mov    x3, x25
    0xdee00008: 0xaa1a03e4   mov    x4, x26
    0xdee0000c: 0xaa0403e8   mov    x8, x4
    0xdee00010: 0x91020269   add    x9, x19, #0x80            ; =0x80
    0xdee00014: 0xf100ed1f   cmp    x8, #0x3b                 ; =0x3b
    0xdee00018: 0x54000240   b.eq   0xdee00060
    0xdee0001c: 0x37080148   tbnz   w8, #0x1, 0xdee00044

(lldb) x/8i $x1
    0xdee0294c: 0x00000000   udf    #0x0
    0xdee02950: 0x00000000   udf    #0x0
    0xdee02954: 0x00000000   udf    #0x0
    0xdee02958: 0x00000000   udf    #0x0
    0xdee0295c: 0x00000000   udf    #0x0
    0xdee02960: 0x00000000   udf    #0x0
    0xdee02964: 0x00000000   udf    #0x0
    0xdee02968: 0x00000000   udf    #0x0

(lldb) x/8i $x9
error: memory read failed for 0x0

(lldb) x/8i $x10
    0xdee00000: 0xaa1f03e2   mov    x2, xzr
    0xdee00004: 0xaa1903e3   mov    x3, x25
    0xdee00008: 0xaa1a03e4   mov    x4, x26
    0xdee0000c: 0xaa0403e8   mov    x8, x4
    0xdee00010: 0x91020269   add    x9, x19, #0x80            ; =0x80
    0xdee00014: 0xf100ed1f   cmp    x8, #0x3b                 ; =0x3b
    0xdee00018: 0x54000240   b.eq   0xdee00060
    0xdee0001c: 0x37080148   tbnz   w8, #0x1, 0xdee00044
(lldb)
```

andrew@ produced a small patch which seems to be sufficient (attached).

It would be great to have a fix for this included in 13.1-RELEASE arm64 if possible.
Comment 1 Dave Cottlehuber freebsd_committer freebsd_triage 2022-03-26 16:13:15 UTC
See https://reviews.freebsd.org/D34675 for actual (WIP) port
Comment 2 commit-hook freebsd_committer freebsd_triage 2022-03-28 09:11:12 UTC
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=029c1c4828aab451ba262cd4e2e1d9362cf18b76

commit 029c1c4828aab451ba262cd4e2e1d9362cf18b76
Author:     Andrew Turner <andrew@FreeBSD.org>
AuthorDate: 2022-03-26 15:59:34 +0000
Commit:     Andrew Turner <andrew@FreeBSD.org>
CommitDate: 2022-03-28 09:10:15 +0000

    Treat cache write as a read in arm64 data faults

    On arm64 we can ask the hardware to perform cache operations from
    userspace. These require read permission however when the memory is
    unmapped the kernel will receive a write exception. Add a check to
    see if the cause of the exception is from the cache and pass a memory
    read fault type to the vm subsystem.

    PR:             262836
    Reported by:    dch
    Sponsored by:   The FreeBSD Foundation

 sys/arm64/arm64/trap.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
Comment 3 commit-hook freebsd_committer freebsd_triage 2022-03-31 11:19:54 UTC
A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=004da2d51f8427745c0d1287781d5ba546b19000

commit 004da2d51f8427745c0d1287781d5ba546b19000
Author:     Andrew Turner <andrew@FreeBSD.org>
AuthorDate: 2022-03-26 15:59:34 +0000
Commit:     Andrew Turner <andrew@FreeBSD.org>
CommitDate: 2022-03-31 11:09:07 +0000

    Treat cache write as a read in arm64 data faults

    On arm64 we can ask the hardware to perform cache operations from
    userspace. These require read permission however when the memory is
    unmapped the kernel will receive a write exception. Add a check to
    see if the cause of the exception is from the cache and pass a memory
    read fault type to the vm subsystem.

    PR:             262836
    Reported by:    dch
    Sponsored by:   The FreeBSD Foundation

    (cherry picked from commit 029c1c4828aab451ba262cd4e2e1d9362cf18b76)

 sys/arm64/arm64/trap.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
Comment 4 commit-hook freebsd_committer freebsd_triage 2022-04-01 12:35:27 UTC
A commit in branch releng/13.1 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=b57e321175768358fcae767e2828e04e3ce7fa3d

commit b57e321175768358fcae767e2828e04e3ce7fa3d
Author:     Andrew Turner <andrew@FreeBSD.org>
AuthorDate: 2022-03-26 15:59:34 +0000
Commit:     Andrew Turner <andrew@FreeBSD.org>
CommitDate: 2022-04-01 11:15:08 +0000

    Treat cache write as a read in arm64 data faults

    On arm64 we can ask the hardware to perform cache operations from
    userspace. These require read permission however when the memory is
    unmapped the kernel will receive a write exception. Add a check to
    see if the cause of the exception is from the cache and pass a memory
    read fault type to the vm subsystem.

    PR:             262836
    Reported by:    dch
    Approved by:    re (gjb)
    Sponsored by:   The FreeBSD Foundation

    (cherry picked from commit 029c1c4828aab451ba262cd4e2e1d9362cf18b76)
    (cherry picked from commit 004da2d51f8427745c0d1287781d5ba546b19000)

 sys/arm64/arm64/trap.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)