Trying to boot FreeBSD 13-STABLE / 14-CURRENT (also tried 13-RELEASE) - effect is the same, after booting kernel, there's a stuck loop: Loading kernel... /boot/kernel/kernel text=0x2a8 text=0x84ad60 text=0x247f14 data=0x1b92a8 data=0x0+0x40d000 syms=[0x8+0x133158+0x8+0x15a65a] Loading configured modules... can't find '/boot/entropy' can't find '/etc/hostid' No valid device tree blob found! WARNING! Trying to fire up the kernel, but no device tree blob found! EFI framebuffer information: addr, size 0x0, 0x0 dimensions 0 x 0 stride 0 masks 0x00000000, 0x00000000, 0x00000000, 0x00000000 ---<<BOOT>>--- GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2022 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT #0 main-n253065-8dc42f98047: Thu Feb 10 06:40:36 UTC 2022 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) WARNING: WITNESS option enabled, expect reduced performance. VT: init without driver. module firmware already present! real memory = 25766789120 (24573 MB) avail memory = 25096417280 (23933 MB) Starting CPU 1 (1) Starting CPU 2 (2) Starting CPU 3 (3) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled. random: entropy device external interface MAP 6386b0000 mode 2 pages 80 MAP 638700000 mode 2 pages 80 MAP 638750000 mode 2 pages 80 MAP 6387a0000 mode 2 pages 80 MAP 6387f0000 mode 2 pages 192 MAP 6388b0000 mode 2 pages 96 MAP 638910000 mode 2 pages 160 MAP 6389b0000 mode 2 pages 80 MAP 638a00000 mode 2 pages 80 MAP 638a50000 mode 2 pages 240 MAP 63be20000 mode 2 pages 144 MAP 63bec0000 mode 2 pages 288 MAP 4000000 mode 0 pages 16384 MAP 9010000 mode 0 pages 1 kbd0 at kbdmux0 acpi0: <BOCHS BXPCFACP> acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) acpi0: Could not update all GPEs: AE_NOT_CONFIGURED psci0: <ARM Power State Co-ordination Interface Driver> on acpi0 gic0: <ARM Generic Interrupt Controller v3.0> iomem 0x8000000-0x801ffff,0x80a0000-0x8ffffff on acpi0 its0: <ARM GIC Interrupt Translation Service> on gic0 generic_timer0: <ARM Generic Timer> irq 34,35,36 on acpi0 Timecounter "ARM MPCore Timecounter" frequency 25000000 Hz quality 1000 Event timer "ARM MPCore Eventtimer" frequency 25000000 Hz quality 1000 efirtc0: <EFI Realtime Clock> efirtc0: registered as a time-of-day clock, resolution 1.000000s pmu0: <Performance Monitoring Unit> on acpi0 cpu0: <ACPI CPU> on acpi0 uart0: <PrimeCell UART (PL011)> iomem 0x9000000-0x9000fff irq 0 on acpi0 uart0: console (9600,n,8,1) pcib0: <Generic PCI host controller> on acpi0 pci0: <PCI bus> on pcib0 virtio_pci0: <VirtIO PCI (modern) GPU adapter> mem 0x10119000-0x10119fff,0x8000100000-0x8000103fff at device 1.0 on pci0 xhci0: <XHCI (generic) USB 3.0 controller> mem 0x8000108000-0x800010bfff at device 2.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 virtio_pci1: <VirtIO PCI (legacy) Network adapter> mem 0x10118000-0x10118fff,0x8000104000-0x8000107fff at device 3.0 on pci0 vtnet0: <VirtIO Networking Adapter> on virtio_pci1 vtnet0: Ethernet address: 02:00:17:00:1b:78 pcib1: <PCI-PCI bridge> mem 0x10117000-0x10117fff at device 16.0 on pci0 pci1: <PCI bus> on pcib1 pcib2: <PCI-PCI bridge> mem 0x10116000-0x10116fff at device 16.1 on pci0 pci2: <PCI bus> on pcib2 pcib3: <PCI-PCI bridge> mem 0x10115000-0x10115fff at device 16.2 on pci0 pci3: <PCI bus> on pcib3 pcib4: <PCI-PCI bridge> mem 0x10114000-0x10114fff at device 16.3 on pci0 pci4: <PCI bus> on pcib4 pcib5: <PCI-PCI bridge> mem 0x10113000-0x10113fff at device 16.4 on pci0 pci5: <PCI bus> on pcib5 pcib6: <PCI-PCI bridge> mem 0x10112000-0x10112fff at device 16.5 on pci0 pci6: <PCI bus> on pcib6 pcib7: <PCI-PCI bridge> mem 0x10111000-0x10111fff at device 16.6 on pci0 pci7: <PCI bus> on pcib7 pcib8: <PCI-PCI bridge> mem 0x10110000-0x10110fff at device 16.7 on pci0 pci8: <PCI bus> on pcib8 pcib9: <PCI-PCI bridge> mem 0x1010f000-0x1010ffff at device 17.0 on pci0 pci9: <PCI bus> on pcib9 pcib10: <PCI-PCI bridge> mem 0x1010e000-0x1010efff at device 17.1 on pci0 pci10: <PCI bus> on pcib10 pcib11: <PCI-PCI bridge> mem 0x1010d000-0x1010dfff at device 17.2 on pci0 pci11: <PCI bus> on pcib11 pcib12: <PCI-PCI bridge> mem 0x1010c000-0x1010cfff at device 17.3 on pci0 pci12: <PCI bus> on pcib12 pcib13: <PCI-PCI bridge> mem 0x1010b000-0x1010bfff at device 17.4 on pci0 pci13: <PCI bus> on pcib13 pcib14: <PCI-PCI bridge> mem 0x1010a000-0x1010afff at device 17.5 on pci0 pci14: <PCI bus> on pcib14 pcib15: <PCI-PCI bridge> mem 0x10109000-0x10109fff at device 17.6 on pci0 pci15: <PCI bus> on pcib15 pcib16: <PCI-PCI bridge> mem 0x10108000-0x10108fff at device 17.7 on pci0 pci16: <PCI bus> on pcib16 pcib17: <PCI-PCI bridge> mem 0x10107000-0x10107fff at device 18.0 on pci0 pci17: <PCI bus> on pcib17 pcib18: <PCI-PCI bridge> mem 0x10106000-0x10106fff at device 18.1 on pci0 pci18: <PCI bus> on pcib18 pcib19: <PCI-PCI bridge> mem 0x10105000-0x10105fff at device 18.2 on pci0 pci19: <PCI bus> on pcib19 pcib20: <PCI-PCI bridge> mem 0x10104000-0x10104fff at device 18.3 on pci0 pci20: <PCI bus> on pcib20 pcib21: <PCI-PCI bridge> mem 0x10103000-0x10103fff at device 18.4 on pci0 pci21: <PCI bus> on pcib21 pcib22: <PCI-PCI bridge> mem 0x10102000-0x10102fff at device 18.5 on pci0 pci22: <PCI bus> on pcib22 pcib23: <PCI-PCI bridge> mem 0x10101000-0x10101fff at device 18.6 on pci0 pci23: <PCI bus> on pcib23 pcib24: <PCI-PCI bridge> mem 0x10100000-0x10100fff at device 18.7 on pci0 pci24: <PCI bus> on pcib24 virtio_pci2: <VirtIO PCI (modern) SCSI adapter> mem 0x10000000-0x10000fff,0x8000000000-0x8000003fff at device 0.0 on pci24 vtscsi0: <VirtIO SCSI Adapter> on virtio_pci2 acpi_button0: <Power Button> on acpi0 armv8crypto0: <AES-CBC,AES-XTS,AES-GCM> Timecounters tick every 1.000 msec CPU 0: ARM Neoverse-N1 r3p1 affinity: 0 Cache Type = <64 byte D-cacheline,64 byte I-cacheline,PIPT ICache,64 byte ERG,64 byte CWG,IDC> Instruction Set Attributes 0 = <DP,RDM,Atomic,CRC32,SHA2,SHA1,AES+PMULL> Instruction Set Attributes 1 = <RCPC-8.3,DCPoP> Processor Features 0 = <CSV3,CSV2,RAS,GIC,AdvSIMD+HP,FP+HP,EL3,EL2,EL1,EL0 32> Processor Features 1 = <PSTATE.SSBS MSR> Memory Model Features 0 = <TGran4,TGran64,TGran16,SNSMem,BigEnd,16bit ASID,256TB PA> Memory Model Features 1 = <PAN+ATS1E1,LO,HPD+TTPBHA,VH,16bit VMID,HAF+DS> Memory Model Features 2 = <32bit CCIDX,48bit VA,UAO,CnP> Debug Features 0 = <DoubleLock,SPE,2 CTX BKPTs,4 Watchpoints,6 Breakpoints,PMUv3 v8.1,Debugv8.2> Debug Features 1 = <> Auxiliary Features 0 = <> Auxiliary Features 1 = <> AArch32 Instruction Set Attributes 5 = <RDM,CRC32,SHA2,SHA1,AES+VMULL,SEVL> AArch32 Media and VFP Features 0 = <FPRound,FPSqrt,FPDivide,DP VFPv3+v4,SP VFPv3+v4,AdvSIMD> AArch32 Media and VFP Features 1 = <SIMDFMAC,FPHP Arith,SIMDHP Arith,SIMDSP,SIMDInt,SIMDLS,FPDNaN,FPFtZ> CPU 1: ARM Neoverse-N1 r3p1 affinity: 1 CPU 2: ARM Neoverse-N1 r3p1 affinity: 2 CPU 3: ARM Neoverse-N1 r3p1 affinity: 3 Release APs...usbus0: 5.0Gbps Super Speed USB v3.0 done Trying to mount root from ufs:/dev/gpt/rootfs [rw]... WARNING: WITNESS option enabled, expect reduced performance. ugen0.1: <(0x1b36) XHCI root HUB> at usbus0 uhub0 on usbus0 uhub0: <(0x1b36) XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub0: 8 ports with 8 removable, self powered Root mount waiting for: usbus0 CAM ... ... ... usbd_setup_device_desc: getting device descriptor at addr 1 failed, USB_ERR_TIMEOUT ugen0.2: <Unknown > at usbus0 (disconnected) uhub_reattach_port: could not allocate new device Root mount waiting for: usbus0 CAM ... This is VM running on Oracle Cloud, using Arm processor from Ampere. Tried setting diffrent variables before booting kernel, ex.: kern.cam.boot_delay=10000 vfs.root_mount_always_wait=1 hw.usb.no_boot_wait=1 No change. I have working Linux on this machine, so can provide more info if needed. Any way to get this to boot somehow? A cut from dmesg from another VM, this one is running on amd64 host: WARNING: WITNESS option enabled, expect reduced performance. ugen0.1: <Intel UHCI root HUB> at usbus0 Trying to mount root from ufs:/dev/da0p2 [rw]... uhub0 on usbus0 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 pass0 at vtscsi0 bus 0 scbus2 target 0 lun 0 pass0: <QEMU QEMU TARGET 2.5> Fixed Uninstalled SPC-3 SCSI device (offline) pass0: 300.000MB/s transfers pass0: Command Queueing enabled da0 at vtscsi0 bus 0 scbus2 target 0 lun 1 da0: <ORACLE BlockVolume 1.0> Fixed Direct Access SPC-4 SCSI device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 47694MB (97677312 512 byte sectors) (da0:vtscsi0:0:0:1): READ(6)/WRITE(6) not supported, increasing minimum_cmd_size to 10. uhub0: 2 ports with 2 removable, self powered ugen0.2: <QEMU QEMU USB Tablet> at usbus0 This one also got similar loop problem when using 13-RELEASE, but tried 13-STABLE and 14-CURRENT (both latest) and they work fine without any issues - so maybe it's qemu related?
So the USB bus isn't able to complete its enumeration. The usb bus code requests that CAM wait until it's enumerated the bus before CAM drops its hold on the boot process that's designed to make sure we don't proceed to mountroot() before the root filesystem providing device is active and fully tasted. The "Root mount wainting for: usbus0 CAM" is the result. The problem isn't in that loop, per se. The problem is the usb enumeration problems and timeouts.
Any way to debug/get more verbose output of that? Doing boot -v yields the same output?
maybe set hw.usb.uhub.debug=1 hw.usb.debug=1 hw.usb.umass.debug=1 at the 'OK' prompt in the boot loader
Created attachment 231817 [details] console boot history log Boot log attached - hope it will make sense for someone :)
(In reply to sec from comment #4) >Boot log attached - hope it will make sense for someone :) Me too. I think that hps@ will need to take a look, I think, to make sense of this. It's a bit beyond my USB skills.
I haven't seen this at all & I'm probably the biggest OCI Ampere consumer atm. I'm generating mine via: https://github.com/skunkwerks/oci-a1-metal/tree/a1-flex using https://freeside.skunkwerks.at/pub/FreeBSD/releases/13.1-BETA3/arm64/ which is a memstick with minimal tweaks to /boot/loader.conf console output -- and my public key in /root/.ssh/authorized_keys ) to allow ssh in on startup. The only obvious thing that jumps out is you're trying to boot from CD (what does that even mean in a cloud these days...). Trying to mount root from cd9660:/dev/iso9660/13_0_STABLE_AARCH64_BO [ro]... At least you can continue through to a usable system from here, without the CDROM boot.
Using 13.1-BETA1 don't produce the bug, so I guess it was fixed somewhere between.
(In reply to sec from comment #7) I reproduced the same issue with the current CURRENT image and 13.1-RELEASE.
despite repeated testing I can't repro this at all in 13.1-RELEASE (using the official FreeBSD image now available in OCI), nor CURRENT. Can you share any more details on your VM config, and how the image is configured? The official FreeBSD image is not very exciting, it's basically 13.1-RELEASE generated from this patch https://reviews.freebsd.org/D34746 so we are not doing any clever tunables.
I can reproduce this with 14.0-p2 when booting via UEFI. It does not appear when booting in BIOS mode.