Bug 251187

Summary:	13-CURRENT armv7 fails to boot under QEMU
Product:	Base System	Reporter:	Edward Tomasz Napierala <trasz>
Component:	arm	Assignee:	Mark Johnston <markj>
Status:	Closed FIXED
Severity:	Affects Only Me	CC:	emaste, lwhsu, markj
Priority:	---
Version:	CURRENT
Hardware:	Any
OS:	Any

Description Edward Tomasz Napierala freebsd_committer

2020-11-16 15:13:56 UTC

Trying to build GENERIC armv7 13.0-CURRENT, fetched from https://download.freebsd.org/ftp/snapshots/arm/armv7/ISO-IMAGES/13.0/FreeBSD-13.0-CURRENT-arm-armv7-GENERICSD-20201022-0035a6c7bb7.img.xz, fails like this:

% qemu-system-arm -M virt -m 256m -nographic -bios /usr/local/share/u-boot/u-boot-qemu-arm/u-boot.bin -hda FreeBSD-13.0-CURRENT-arm-armv7-GENERICSD-20201112-2ed50808d2b.
img
WARNING: Image format was not specified for 'FreeBSD-13.0-CURRENT-arm-armv7-GENERICSD-20201112-2ed50808d2b.img' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.


U-Boot 2020.10 (Nov 11 2020 - 15:08:21 +0000)

DRAM:  256 MiB
Flash: 128 MiB
*** Warning - bad CRC, using default environment

In:    pl011@9000000
Out:   pl011@9000000
Err:   pl011@9000000
Net:   No ethernet found.
starting USB...
No working controllers found
Hit any key to stop autoboot:  0
starting USB...
No working controllers found
USB is stopped. Please issue 'usb start' first.
scanning bus for devices...

Device 0: unknown device

Device 0: 1af4 VirtIO Block Device
Type: Hard Disk
Capacity: 3072.0 MB = 3.0 GB (6291456 x 512)
... is now current device
Scanning virtio 0:1...
load - load binary file from a filesystem

Usage:
load <interface> [<dev[:part]> [<addr> [<filename> [bytes [pos]]]]]
- Load binary file 'filename' from partition 'part' on device
type 'interface' instance 'dev' to address 'addr' in memory.
'bytes' gives the size to load in bytes.
If 'bytes' is 0 or omitted, the file is read until the end.
'pos' gives the file byte position to start reading from.
If 'pos' is 0 or omitted, the file is read from the start.
load - load binary file from a filesystem

Usage:
load <interface> [<dev[:part]> [<addr> [<filename> [bytes [pos]]]]]
- Load binary file 'filename' from partition 'part' on device
type 'interface' instance 'dev' to address 'addr' in memory.
'bytes' gives the size to load in bytes.
If 'bytes' is 0 or omitted, the file is read until the end.
'pos' gives the file byte position to start reading from.
If 'pos' is 0 or omitted, the file is read from the start.
Found EFI removable media binary efi/boot/bootarm.efi
Scanning disk virtio-blk#33...
** Unrecognized filesystem type **
Found 3 disks
No EFI system partition
BootOrder not defined
EFI boot manager: Cannot load any image
1317780 bytes read in 4 ms (314.2 MiB/s)
Booting /efi\boot\bootarm.efi
Consoles: EFI console
Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p1:
FreeBSD/arm EFI loader, Revision 1.1
(Thu Nov 12 06:17:16 UTC 2020 root@releng1.nyi.freebsd.org)
Command line arguments: l                                                                                                                                                           [17/722]
Image base: 0x4dcb2000
EFI version: 2.80
EFI Firmware: Das U-Boot (rev 8224.4096)
Console: comconsole (0)
Load Path: /efi\boot\bootarm.efi
Load Device: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/VenHw(63293792-adf5-9325-b99f-4e0e455c1b1e,00)/HD(1,0x01,0,0x81f,0x18fa8)
Trying ESP: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/VenHw(63293792-adf5-9325-b99f-4e0e455c1b1e,00)/HD(1,0x01,0,0x81f,0x18fa8)
Setting currdev to disk0p1:
Trying: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/VenHw(63293792-adf5-9325-b99f-4e0e455c1b1e,00)/HD(2,0x01,0,0x197c7,0x5e6821)
Setting currdev to disk0p2:
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
|
Loading kernel...
/boot/kernel/kernel text=0x1b4 text=0x674cdc text=0x1bd2a8 data=0xaa660 data=0x0+0x258000 syms=[0x4+0xa1e90+0x4+0x11442b]
Loading configured modules...
/boot/kernel/umodem.ko text=0x1c00 text=0x1300 data=0x2b8+0x8 syms=[0x4+0x1090+0x4+0xbd9]
can't find '/etc/hostid'
can't find '/boot/entropy'

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...
Using DTB provided by EFI at 0x47dfd000.
Kernel entry at 0x45e00200...
Kernel args: (null)
\

At this point it hangs.

Comment 1 Mark Johnston freebsd_committer

2022-08-03 17:26:16 UTC

Trying to debug this now.  We hang when enabling the MMU, in init_mmu() in arm's locore.

Comment 2 Mark Johnston freebsd_committer

2022-08-03 19:58:58 UTC

The problem is that u-boot or QEMU, not sure which, is leaving TTBCR.EAE set for some reason.  This is ARM's PAE extension, implemented by the default CPU for the "virt" board.  We don't set up page tables for PAE mode though, so the kernel crashes when the MMU is enabled.

Clearing TTBCR first fixes the problem, though now we panic during boot on what looks like an unaligned data access in the UDP code.

Comment 3 Mark Johnston freebsd_committer

2022-08-04 13:37:08 UTC

https://reviews.freebsd.org/D36041

Comment 4 commit-hook freebsd_committer

2022-08-05 20:48:03 UTC

A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=e9552d8b45d67ca44d91b3ec09c91253e7e99b28

commit e9552d8b45d67ca44d91b3ec09c91253e7e99b28
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2022-08-05 20:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2022-08-05 20:21:09 +0000

    arm: Clear TTBCR before enabling the MMU

    Upon reset, this register is supposed to have a value of zero.  But when
    booting certain v7 CPUs in QEMU, we enter the kernel with several bits
    set, including the EAE bit, which enables ARM's PAE extension.  I'm not
    sure if QEMU is setting it or if it's the uboot loader.  Because FreeBSD
    doesn't implement that extension and uses regular 32-bit page tables,
    the kernel hangs immediately after enabling the MMU.

    Just clear everything in TTBCR before enabling the MMU, to match the
    reset value.  FreeBSD doesn't toggle anything in that register.

    PR:             251187
    Reviewed by:    imp
    MFC after:      2 weeks
    Sponsored by:   The FreeBSD Foundation
    Differential Revision:  https://reviews.freebsd.org/D36041

 sys/arm/arm/locore-v6.S | 7 +++++++
 1 file changed, 7 insertions(+)

Comment 5 commit-hook freebsd_committer

2022-08-19 11:55:12 UTC

A commit in branch stable/13 references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=363d4b3b42c1d5ec9a2543a0d1f8068cd7f8b87f

commit 363d4b3b42c1d5ec9a2543a0d1f8068cd7f8b87f
Author:     Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2022-08-05 20:21:09 +0000
Commit:     Mark Johnston <markj@FreeBSD.org>
CommitDate: 2022-08-19 11:54:14 +0000

    arm: Clear TTBCR before enabling the MMU

    Upon reset, this register is supposed to have a value of zero.  But when
    booting certain v7 CPUs in QEMU, we enter the kernel with several bits
    set, including the EAE bit, which enables ARM's PAE extension.  I'm not
    sure if QEMU is setting it or if it's the uboot loader.  Because FreeBSD
    doesn't implement that extension and uses regular 32-bit page tables,
    the kernel hangs immediately after enabling the MMU.

    Just clear everything in TTBCR before enabling the MMU, to match the
    reset value.  FreeBSD doesn't toggle anything in that register.

    PR:             251187
    Reviewed by:    imp
    Sponsored by:   The FreeBSD Foundation

    (cherry picked from commit e9552d8b45d67ca44d91b3ec09c91253e7e99b28)

 sys/arm/arm/locore-v6.S | 7 +++++++
 1 file changed, 7 insertions(+)