Bug 264267 - UEFI Booting on Azure Generation 2 VMs crashes
Summary: UEFI Booting on Azure Generation 2 VMs crashes
Status: Open
Alias: None
Product: Base System
Classification: Unclassified
Component: kern (show other bugs)
Version: CURRENT
Hardware: amd64 Any
: --- Affects Many People
Assignee: freebsd-virtualization (Nobody)
URL:
Keywords: crash, needs-qa
Depends on:
Blocks:
 
Reported: 2022-05-26 23:41 UTC by Michael Baptist
Modified: 2024-03-04 09:47 UTC (History)
8 users (show)

See Also:
koobs: maintainer-feedback? (whu)
koobs: mfc-stable13?

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Baptist 2022-05-26 23:41:11 UTC 
I'm working on getting FreeBSD working on Microsoft Azure Gen 2 VM support and hitting some issues. I have all of kib's latest relocatable kernel code he committed in 13.1 (LA57 page table), and seem to have no UEFI booting issues on KVM or other platforms.

Azure Gen 2 however crashes or stalls during SYSINIT and doesn't go to debugger or anything. So far I can see it crashing during sysinit via printfs and am rebuilding now with VERBOSE_SYSINIT=1 to see if I can identify a specific sysinit module.

Hoping this bug makes it to right people, or someone can shoot me a few pointers on what I could do to further debug this. I'm new to kernel development in general, but have poked around on my system enough to understand the basics of navigating the src tree and have taken some basic os classes so hoping I can figure it out with some help if someone doesn't beat me to it.
Comment 1 Michael Baptist 2022-05-27 00:15:42 UTC 
A little more background the Azure EFI firmware has a free memory region at 0x200000, however for larger kernels the EFI copy will write into EFI memory and cause a fault.

The before we trampoline the new EFI_COPY_DISABLE option will allow us to skip copying and sets up page tables for 4G of physical memory (I think) and should map in our kernel in place. This all seems to work fine. Once the Kenrnel is loaded and we run sysinit I can see the trademark, copyright, an <BOOT> (boottag) stuff printing and then the console dies. I assume those sysinit functions are not the problem but others might be causing it to crash in some way. (Maybe vm subsystem code?)
Comment 2 Kubilay Kocak freebsd_committer freebsd_triage 2022-05-27 00:27:03 UTC 
Hi Michael, thank you for your report.

Pinging Wei Hu (Hyper-V @ Microsoft) for feedback

If you can attach a screenshot of the crash/hang and if you're able to test boot 12.x, 13-STABLE and 14-CURRENT images to see if the behaviour differs, those would be great first steps.
Comment 3 Li-Wen Hsu freebsd_committer freebsd_triage 2022-05-27 09:03:43 UTC 
I'm also debugging this issue, in my tests that -CURRENT doesn't with this issue, but in 13.1 and stable/13 this issue happens randomly.  (I have an 13.1-R image which boots fine in some VM sizes during my tests, but failed in marketplace's publishing tests.)  I'm still working on setting up my local windows hyper-v environment but it's a little difficult for me.

I'm trying to get:
1) verbose dmesgs from both successful and unsuccessful boot
2) same, but with the debug.verbose_sysinit tunable set to 1
If you have similar information, that will be helpful.

If you have a windows hyper-v setup, please help me to see if there is anything related to hyper-v in windows event log. Thanks!
Comment 4 Michael Baptist 2022-05-27 22:11:28 UTC 
/  ______               ____   _____ _____  
  |  ____|             |  _ \ / ____|  __ \ 
  | |___ _ __ ___  ___ | |_) | (___ | |  | |
  |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
  | |   | | |  __/  __/| |_) |____) | |__| |
  | |   | | |    |    ||     |      |      |
  |_|   |_|  \___|\___||____/|_____/|_____/      ```                        `
                                                s` `.....---.......--.```   -/
 +---------- Welcome to FreeBSD -----------+    +o   .--`         /y:`      +.
 |                                         |     yo`:.            :o      `+-
 |  1. Boot Multi user [Enter]             |      y/               -/`   -o/
 |  2. Boot Single user                    |     .-                  ::/sy+:.
 |  3. Escape to loader prompt             |     /                     `--  /
 |  4. Reboot                              |    `:                          :`
 |  5. Cons: Dual (Serial primary)         |    `:                          :`
 |                                         |     /                          /
 |  Options:                               |     .-                        -.
 |  6. Kernel: default/kernel (1 of 1)     |      --                      -.
 |  7. Boot Options                        |       `:`                  `:`
 |                                         |         .--             `--.
 |                                         |            .---.....----.
 +-----------------------------------------+
   Autoboot in 0 seconds. [Space] to pause 

Loading kernel...
/boot/kernel/kernel text=0x184d70 text=0xdfdfc0 text=0x6634c4 data=0x140 data=0x1be3b8+0x440c48 syms=[0x8+0x188d78+0x8+0x1a7803]
Loading configured modules...
can't find '/etc/hostid'
can't find '/boot/entropy'
staging 0x36e00000 (not copying) tramp 0x36c43000 PT4 0x36c3a000
Start @ 0xffffffff80385000 ...
EFI framebuffer information:
addr, size     0x40000000, 0x800000
dimensions     1024 x 768
stride         1024
masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
---<<BOO
Comment 5 Michael Baptist 2022-05-27 22:12:13 UTC 
I'll dump memmap from the EFI loader as well when I get a change. I'll be away for a few days.
Comment 6 Michael Baptist 2022-05-31 15:06:45 UTC 
OK memmap
                   Type     Physical      Virtual   #Pages Attr
     ConventionalMemory 000000000000 000000000000 000000a0 UC WC WT WB 
       BootServicesData 000000100000 000000000000 0000052a UC WC WT WB 
     ConventionalMemory 00000062a000 000000000000 00003dd6 UC WC WT WB 
       BootServicesData 000004400000 000000000000 00000200 UC WC WT WB 
     ConventionalMemory 000004600000 000000000000 0000001d UC WC WT WB 
       BootServicesData 00000461d000 000000000000 00000020 UC WC WT WB 
       BootServicesCode 00000463d000 000000000000 0000002f UC WC WT WB 
       BootServicesData 00000466c000 000000000000 00000008 UC WC WT WB 
       BootServicesCode 000004674000 000000000000 00000017 UC WC WT WB 
     ConventionalMemory 00000468b000 000000000000 000325b9 UC WC WT WB 
             LoaderData 000036c44000 000000000000 00008000 UC WC WT WB 
             LoaderCode 00003ec44000 000000000000 000000e1 UC WC WT WB 
      ACPIReclaimMemory 00003ed25000 000000000000 00000002 UC WC WT WB 
       BootServicesCode 00003ed27000 000000000000 00000154 UC WC WT WB 
      ACPIReclaimMemory 00003ee7b000 000000000000 0000001f UC WC WT WB 
       BootServicesCode 00003ee9a000 000000000000 00000058 UC WC WT WB 
    RuntimeServicesData 00003eef2000 000000000000 00000029 UC WC WT WB 
     ConventionalMemory 00003ef1b000 000000000000 0000036b UC WC WT WB 
       BootServicesData 00003f286000 000000000000 00000730 UC WC WT WB 
     ConventionalMemory 00003f9b6000 000000000000 00000003 UC WC WT WB 
       BootServicesData 00003f9b9000 000000000000 00000462 UC WC WT WB 
       BootServicesCode 00003fe1b000 000000000000 00000180 UC WC WT WB 
    RuntimeServicesCode 00003ff9b000 000000000000 00000030 UC WC WT WB 
    RuntimeServicesData 00003ffcb000 000000000000 00000024 UC WC WT WB 
               Reserved 00003ffef000 000000000000 00000004 UC WC WT WB 
      ACPIReclaimMemory 00003fff3000 000000000000 00000008 UC WC WT WB 
          ACPIMemoryNVS 00003fffb000 000000000000 00000004 UC WC WT WB 
       BootServicesData 00003ffff000 000000000000 00000001 UC WC WT WB 
     ConventionalMemory 000100000000 000000000000 00ee0000 UC WC WT WB 
     ConventionalMemory 001000000000 000000000000 000e0000 UC WC WT WB 
               Reserved 0000000c0000 000000000000 00000040 UC 
       PersistentMemory 0010e0000000 000000000000 00100000 UC WC WT WB NV
Comment 7 Wei Hu 2022-06-02 05:00:16 UTC 
(In reply to Michael Baptist from comment #6)
What is the VM size/type in Azure that you are running it this issue?
Comment 8 Wei Hu 2022-06-02 11:07:34 UTC 
Another thing I can think of is the LA57 support on FreeBSD is not working on Hyper-V. But it is disabled on 13.1 by default as far as I know.

Maybe some new CPU features are supported well on FreeBSD. That's why I need to know the VM type and size to find out CPU type and other h/w spec.

By the way, I was able to boot FreeBSD 13.1 Gen2 VM on Intel Xeon 8260 (Cascade Lake) host in my lab.
Comment 9 Michael Baptist 2022-06-03 04:25:12 UTC 
I've tried a few different sizes.

For example E32sV4/V5, D32s.

The LA57 page table code is not enabled I believe that is correct, however booting in place from EFI memmap is part of 13.1 release. Those commits were somewhat interwined iirc.
Comment 10 Michael Baptist 2022-06-03 04:29:17 UTC 
> By the way, I was able to boot FreeBSD 13.1 Gen2 VM on Intel Xeon 8260 (Cascade Lake) host in my lab.

Is this a hyper V or Azure Stack HCI host or Azure Cloud host?

I also have been able to boot up successfully under other hypervisors booting GPT UEFI, just out MSFT Azure Cloud hyperscaler.
Comment 11 Wei Hu 2022-06-03 10:07:00 UTC 
(In reply to Michael Baptist from comment #10)
This is from an on-prem Windows Server 2019 Build 10.0.17763. I created a Gen2 VM from a 12.3 iso, and then built and booted a kernel from 13 stable branch. So, it should have all the commits you mentioned for EFI memmap. 

I didn't try in Azure as there is no FreeBSD Gen2 image in Azure Marketplace.

Do you have a list of those commits? I was not aware of such changes. It would be interesting to know if this makes FreeBSD behave differently on different h/w and VM size. 

You can also try your image on VMs with fewer cpus, like E8 or E4.
Comment 12 Mark Johnston freebsd_committer freebsd_triage 2022-06-06 18:48:18 UTC 
(In reply to Michael Baptist from comment #4)
Is there any output following the BOOT tag?  The first comment suggests so, but how far are we getting, exactly?
Comment 13 Michael Baptist 2022-06-06 21:09:57 UTC 
(In reply to Mark Johnston from comment #12)

I've seen output like the trademark, and copyright in init_main.c get output, however its "random" (to me) as to how much makes it to the console before its crashes.

Adding in some additional printfs I can trace that the system made it into mi_startup and iterating over the sysinit calls.

message buffer initialization
---<<BOOT>>---
exiting hammer time
mistartup!

   0xffffffff80a61090(0)... done.
   0xffffffff80a877f0(0)... done.
   0xffffffff806acdb0(0)... done.
   0xffffffff805f6050(0xffffffff80f1aa68)... done.
   0xffffffff805f6050(0xffffffff80f1b540)... done.
   0xffffffff805f6050(0xffffffff80f1b550)... done.

Continues to dump addresses and some of them print to console like trademark/copyright and so forth.

 0xffffffff805d8500(0xffffffff80f62200)... FreeBSD is a registered trademark of The FreeBSD Foundation.
done.

@@ -305,6 +306,7 @@ mi_startup(void)
 #endif
 
                /* Call function */
+               printf("   %p(%p)... ", (*sipp)->func, (*sipp)->udata);
                (*((*sipp)->func))((*sipp)->udata);
+               printf("done.\n");


I haven't been able to get DDB and VERBOSE_SYSINIT working so I've just got addrs and not symbol lookups to print. There is code already there to do symbol look ups.

options         DDB                     # Support DDB.
options         VERBOSE_SYSINIT=1       # Support debug.verbose_sysinit, off by
Comment 14 Michael Baptist 2022-06-07 00:55:37 UTC 
/  ______               ____   _____ _____  
  |  ____|             |  _ \ / ____|  __ \ 
  | |___ _ __ ___  ___ | |_) | (___ | |  | |
  |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
  | |   | | |  __/  __/| |_) |____) | |__| |
  | |   | | |    |    ||     |      |      |
  |_|   |_|  \___|\___||____/|_____/|_____/      ```                        `
                                                s` `.....---.......--.```   -/
 +---------- Welcome to FreeBSD -----------+    +o   .--`         /y:`      +.
 |                                         |     yo`:.            :o      `+-
 |  1. Boot Multi user [Enter]             |      y/               -/`   -o/
 |  2. Boot Single user                    |     .-                  ::/sy+:.
 |  3. Escape to loader prompt             |     /                     `--  /
 |  4. Reboot                              |    `:                          :`
 |  5. Cons: Dual (Serial primary)         |    `:                          :`
 |                                         |     /                          /
 |  Options:                               |     .-                        -.
 |  6. Kernel: default/kernel (1 of 1)     |      --                      -.
 |  7. Boot Options                        |       `:`                  `:`
 |                                         |         .--             `--.
 |                                         |            .---.....----.
 +-----------------------------------------+
   Autoboot in 0 seconds. [Space] to pause  

Loading kernel...
/boot/kernel/kernel text=0x189b10 text=0xe0ce88 text=0x6baaa4 data=0x140 data=0x1c8470+0x436b90 0x8+0x198150+0x8+0x1b8a00-
Loading configured modules...
can't find '/etc/hostid'
can't find '/boot/entropy'
staging 0x36e00000 (not copying) tramp 0x36c3f000 PT4 0x36c36000
Start @ 0xffffffff8038a000 ...
EFI framebuffer information:
addr, size     0x40000000, 0x800000
dimensions     1024 x 768
stride         1024
masks          0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
subsystem 6fffff
   mp_setmaxid(0)... done.
   tunable_int_init(&__tunable_int_207)... done.
   tunable_int64_init(&__tunable_int64_1587)... done.
   tunable_str_init(&__tunable_str_47)... done.
   tunable_str_init(&__tunable_str_277)... done.
   tunable_str_init(&__tunable_str_280)... done.
   tunable_int_init(&__tunable_int_284)... done.
   tunable_int_init(&__tunable_int_292)... done.
   tunable_int_init(&__tunable_int_298)... done.
   tunable_int_init(&__tunable_int_303)... done.
   tunable_int_init(&__tunable_int_189)... done.
   tunable_int_init(&__tunable_int_89)... done.
   tunable_int_init(&__tunable_int_90)... done.
   tunable_int_init(&__tunable_int_53)... done.
   tunable_int_init(&__tunable_int_98)... done.
   tunable_int_init(&__tunable_int_99)... done.
   tunable_int_init(&__tunable_int_91)... done.
   tunable_int_init(&__tunable_int_92)... done.
   tunable_int_init(&__tunable_int_252)... done.
   tunable_int_init(&__tunable_int_255)... done.
   tunable_int_init(&__tunable_int_264)... done.
   tunable_int_init(&__tunable_int_273)... done.
   tunable_int_init(&__tunable_int_85)... done.
   tunable_int_init(&__tunable_int_198)... done.
   tunable_int_init(&__tunable_int_199)... done.
   tunable_int_init(&__tunable_int_200)... done.
   tunable_int_init(&__tunable_int_201)... done.
   tunable_int_init(&__tunable_int_82)... done.
   tunable_int_init(&__tunable_int_189)... done.
   tunable_int_init(&__tunable_int_198)... done.
   tunable_int_init(&__tunable_int_91)... done.
   tunable_int_init(&__tunable_int_92)... done.
   tunable_int_init(&__tunable_int_188)... done.
   tunable_int_init(&__tunable_int_157)... done.
   tunable_int_init(&__tunable_int_159)... done.
   tunable_int_init(&__tunable_int_161)... done.
   tunable_int_init(&__tunable_int_62)... done.
   tunable_int_init(&__tunable_int_165)... done.
   tunable_int_init(&__tunable_int_167)... done.
   tunable_int_init(&__tunable_int_169)... done.
   tunable_int_init(&__tunable_int_171)... done.
   tunable_int_init(&__tunable_int_267)... done.
   tunable_int_init(&__tunable_int_889)... done.
   tunable_int_init(&__tunable_int_890)... done.
   tunable_int_init(&__tunable_int_1278)... done.
   tunable_int_init(&__tunable_int_525)... done.
   tunable_int_init(&__tunable_int_229)... done.
   tunable_int_init(&__tunable_int_231)... done.
   tunable_int_init(&__tunable_int_136)... done.
   tunable_int_init(&__tunable_int_104)... done.
   tunable_int_init(&__tunable_int_195)... done.
   tunable_int_init(&__tunable_int_210)... done.
   tunable_int_init(&__tunable_int_212)... done.
   tunable_int_init(&__tunable_int_213)... done.
   tunable_int_init(&__tunable_int_139)... done.
   tunable_int_init(&__tunable_int_142)... done.
   tunable_int_init(&__tunable_int_145)... done.
   tunable_int_init(&__tunable_int_148)... done.
   tunable_int_init(&__tunable_int_151)... done.
   tunable_int_init(&__tunable_int_154)... done.
   tunable_int_init(&__tunable_int_91)... done.
   tunable_ulong_init(&__tunable_ulong_97)... done.
   tunable_int_init(&__tunable_int_144)... done.
   tunable_int_init(&__tunable_int_104)... done.
   tunable_bool_init(&__tunable_bool_96)... done.
   tunable_bool_init(&__tunable_bool_100)... done.
   tunable_bool_init(&__tunable_bool_104)... done.
   tunable_str_init(&__tunable_str_99)... done.
   tunable_int_init(&__tunable_int_111)... done.
   tunable_int_init(&__tunable_int_139)... done.
   tunable_str_init(&__tunable_str_1840)... done.
   tunable_int_init(&__tunable_int_321)... done.
   tunable_int_init(&__tunable_int_110)... done.
   tunable_int_init(&__tunable_int_115)... done.
   tunable_int_init(&__tunable_int_120)... done.
   tunable_int_init(&__tunable_int_125)... done.
   tunable_str_init(&__tunable_str_3594)... done.
   tunable_int_init(&__tunable_int_106)... done.
   tunable_int_init(&__tunable_int_137)... done.
   tunable_int_init(&__tunable_int_385)... done.
   tunable_int_init(&__tunable_int_83)... done.
   tunable_int_init(&__tunable_int_104)... done.
   tunable_int_init(&__tunable_int_106)... done.
   tunable_int_init(&__tunable_int_75)... done.
   tunable_int_init(&__tunable_int_88)... done.
   tunable_int_init(&__tunable_int_108)... done.
   tunable_int_init(&__tunable_int_109)... done.
   tunable_int_init(&__tunable_int_54)... done.
   tunable_int_init(&__tunable_int_243)... done.
   tunable_int_init(&__tunable_int_250)... done.
   tunable_int_init(&__tunable_int_261)... done.
   tunable_int_init(&__tunable_int_268)... done.
   tunable_int_init(&__tunable_int_275)... done.
   tunable_int_init(&__tunable_int_298)... done.
   tunable_int_init(&__tunable_int_303)... done.
   tunable_int_init(&__tunable_int_157)... done.
   tunable_int_init(&__tunable_int_158)... done.
   tunable_int_init(&__tunable_int_159)... done.
   tunable_int_init(&__tunable_int_108)... done.
   tunable_int_init(&__tunable_int_206)... done.
   init_scsi_delay(0)... done.
   init_maxsockets(0)... done.
   tunable_set_numzones(0)... done.
   inittimehands(0)... done.
subsystem 700000
   print_caddr_t(&copyright)... Copyright (c) 1992-2022 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
done.
   print_caddr_t(&trademark)... FreeBSD is a registered trademark of The FreeBSD Foundation.
done.
   print_version(0)... FreeBSD 14.0-CURRENT #0 main-8020c0568: Fri May 27 23:12:20 UTC 2022
    img@fbsd13:/src/obj/src/freebsd-src/amd64.amd64/sys/GEN2 amd64
FreeBSD clang version 14.0.3 (https://github.com/llvm/llvm-project.git llvmorg-14.0.3-0-g1f9140064dfb)
done.
   print_caddr_t(&wit_warn)... WARNING: WITNESS option enabled, expect reduced performance.
done.
   cap_rights_sysinit(0)... done.
subsystem 800001
   parse_acpi_tables(0)... SRAT: Ignoring memory at addr 0x10e0000000
SRAT: Ignoring memory at addr 0x11e0000000
SRAT: Ignoring memory at addr 0x10000000000
SRAT: Ignoring memory at addr 0x20000000000
SRAT: Ignoring memory at addr 0x40000000000
SRAT: Ignoring memory at addr 0x80000000000
SRAT: Ignoring memory at addr 0x100000000000
SRAT: Ignoring memory at addr 0x200000000000
SRAT: Ignoring memory at addr 0x400000000000
SRAT: Ignoring memory at addr 0x800000000000
SRAT: Ignoring memory at addr 0x1000000000000
SRAT: Ignoring memory at addr 0x2000000000000
SRAT: Ignoring memory at addr 0x4000000000000
SRAT: Ignoring memory at addr 0x8000000000000
done.
subsystem ffffff
   vm_mem_init(0)... done.
   vm_page_init(0)... done.
subsystem 1000000
   pcpu_zones_startup(0)... done.
   counter_u64_sysinit(&ktls_ifnet_reset_dropped)... done.
   counter_u64_sysinit(&ktls_ifnet_reset_failed)... done.
   counter_u64_sysinit(&ktls_toe_cbc)... done.
   counter_u64_sysinit(&ktls_toe_gcm)... done.
   counter_u64_sysinit(&ktls_toe_chacha20)... done.
   counter_u64_sysinit(&advance)... done.
   counter_u64_sysinit(&numcachehv)... done.
   counter_u64_sysinit(&numdrops)... done.
   counter_u64_sysinit(&dothits)... done.
   counter_u64_sysinit(&dotdothits)... done.
   counter_u64_sysinit(&nummiss)... done.
   counter_u64_sysinit(&nummisszap)... done.
   counter_u64_sysinit(&numposzaps)... done.
   counter_u64_sysinit(&numposhits)... done.
   counter_u64_sysinit(&numnegzaps)... done.
   counter_u64_sysinit(&numneghits)... done.
   counter_u64_sysinit(&numfullpathcalls)... done.
   counter_u64_sysinit(&numfullpathfail1)... done.
   counter_u64_sysinit(&numfullpathfail2)... done.
   counter_u64_sysinit(&numfullpathfail4)... done.
   counter_u64_sysinit(&numfullpathfound)... done.
   counter_u64_sysinit(&symlinktoobig)... done.
   counter_u64_sysinit(&zap_bucket_relock_success)... done.
   counter_u64_sysinit(&neg_created)... done.
   counter_u64_sysinit(&neg_evicted)... done.
   counter_u64_sysinit(&neg_evict_skipped_empty)... done.
   counter_u64_sysinit(&neg_evict_skipped_missed)... done.
   counter_u64_sysinit(&neg_evict_skipped_contended)... done.
   counter_u64_sysinit(&advance_wait)... done.
   cryptostats_init(0)... done.
   counter_u64_sysinit(&ocf_tls10_cbc_encrypts)... done.
   counter_u64_sysinit(&ocf_tls11_cbc_encrypts)... done.
   counter_u64_sysinit(&ocf_tls12_gcm_decrypts)... done.
   counter_u64_sysinit(&ocf_tls12_gcm_encrypts)... done.
   counter_u64_sysinit(&ocf_tls12_gcm_recrypts)... done.
   counter_u64_sysinit(&ocf_tls12_chacha20_decrypts)... done.
   counter_u64_sysinit(&ocf_tls12_chacha20_encrypts)... done.
   counter_u64_sysinit(&ocf_tls13_gcm_decrypts)... done.
   counter_u64_sysinit(&ocf_tls13_gcm_encrypts)... done.
   counter_u64_sysinit(&ocf_tls13_gcm_recrypts)... done.
   counter_u64_sysinit(&ocf_tls13_chacha20_decrypts)... done.
   counter_u64_sysinit(&ocf_tls13_chacha20_encrypts)... done.
   counter_u64_sysinit(&ocf_inplace)... done.
   counter_u64_sysinit(&ocf_separate_output)... done.
   counter_u64_sysinit(&ocf_retries)... done.
   counter_u64_sysinit(&swap_free_deferred)... done.
   counter_u64_sysinit(&swap_free_completed)... done.
   counter_u64_sysinit(&poll)... done.
   uma_startup_pcpu(0)... done.
   counter_u64_sysinit(&poll_scan)... done.
   counter_u64_sysinit(&object_collapses)... done.
   counter_u64_sysinit(&object_bypasses)... done.
Comment 15 Michael Baptist 2022-06-07 01:11:34 UTC 
At this point output just stops.
Comment 16 Mark Johnston freebsd_committer freebsd_triage 2022-06-07 14:09:51 UTC 
(In reply to Michael Baptist from comment #15)
Could you be able to try a verbose boot as well?  i.e., try booting from the loader with "boot -v".  It might be an option under "loader options" as well.
Comment 17 Mark Johnston freebsd_committer freebsd_triage 2022-06-07 14:34:44 UTC 
Is there some written set of steps available for creating a 13.1 Gen2 instance on Azure?  The only option I see is Gen1.
Comment 18 Michael Baptist 2022-06-07 15:44:37 UTC 
(In reply to Mark Johnston from comment #17)

Not sure the instructions that show the steps for Gen 1 that you were referencing, but the main difference when using the Azure CLI is specifying --hyper-v-generation V2 option.

What I normally do is:

1. make buildworld buildkernel
2. make release
3. mdconfig vnode -f .img file (generated from make release)
4. Create 2g file via truncate.
5. mdconfig that 2g file
6. dd md0 => md1 
7. qemu-img convert 2g file with bits to VHD

Uploading to Azure:

Create disk for upload (using gen2 option)
get access token for disk
use az storage copy to upload the vhd to the disk
create a virtual machine image (using gen2 option)
delete disk

Is looks like there is a Makefile.azure also in release, but I haven't tried that.
Comment 19 Michael Baptist 2022-06-07 15:45:52 UTC 
Also I build on FreeBSD then scp the vhd to a linux host for upload, since FreeBSD support for Azure CLI is behind and some of the patches are out of date for the copy.
Comment 20 Michael Baptist 2022-06-07 15:46:41 UTC 
(In reply to Mark Johnston from comment #16)
Yep I'll try a boot verbose as well.
Comment 21 Michael Baptist 2022-06-07 15:49:34 UTC 
Also forgot an important step Azure doesn't have vidconsole for user input so you'll need to mount the memory device root parition from the img file and update loader.conf

boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
console="comconsole,efi"
Comment 22 Michael Baptist 2022-06-07 19:59:25 UTC 
OK set module_verbose=3
OK memmap   
                   Type     Physical      Virtual   #Pages Attr
     ConventionalMemory 000000000000 000000000000 000000a0 UC WC WT WB 
       BootServicesData 000000100000 000000000000 0000052a UC WC WT WB 
     ConventionalMemory 00000062a000 000000000000 00003dd6 UC WC WT WB 
       BootServicesData 000004400000 000000000000 00000200 UC WC WT WB 
     ConventionalMemory 000004600000 000000000000 0000001d UC WC WT WB 
       BootServicesData 00000461d000 000000000000 00000020 UC WC WT WB 
       BootServicesCode 00000463d000 000000000000 0000002f UC WC WT WB 
       BootServicesData 00000466c000 000000000000 00000008 UC WC WT WB 
       BootServicesCode 000004674000 000000000000 00000017 UC WC WT WB 
     ConventionalMemory 00000468b000 000000000000 000325b5 UC WC WT WB 
             LoaderData 000036c40000 000000000000 00008000 UC WC WT WB 
             LoaderCode 00003ec40000 000000000000 000000e5 UC WC WT WB 
      ACPIReclaimMemory 00003ed25000 000000000000 00000002 UC WC WT WB 
       BootServicesCode 00003ed27000 000000000000 00000154 UC WC WT WB 
      ACPIReclaimMemory 00003ee7b000 000000000000 0000001f UC WC WT WB 
       BootServicesCode 00003ee9a000 000000000000 00000058 UC WC WT WB 
    RuntimeServicesData 00003eef2000 000000000000 00000029 UC WC WT WB 
     ConventionalMemory 00003ef1b000 000000000000 0000028d UC WC WT WB 
       BootServicesData 00003f1a8000 000000000000 0000000d UC WC WT WB 
     ConventionalMemory 00003f1b5000 000000000000 000000de UC WC WT WB 
       BootServicesData 00003f293000 000000000000 00000b88 UC WC WT WB 
       BootServicesCode 00003fe1b000 000000000000 00000180 UC WC WT WB 
    RuntimeServicesCode 00003ff9b000 000000000000 00000030 UC WC WT WB 
    RuntimeServicesData 00003ffcb000 000000000000 00000024 UC WC WT WB 
               Reserved 00003ffef000 000000000000 00000004 UC WC WT WB 
      ACPIReclaimMemory 00003fff3000 000000000000 00000008 UC WC WT WB 
          ACPIMemoryNVS 00003fffb000 000000000000 00000004 UC WC WT WB 
       BootServicesData 00003ffff000 000000000000 00000001 UC WC WT WB 
     ConventionalMemory 000100000000 000000000000 00ee0000 UC WC WT WB 
     ConventionalMemory 001000000000 000000000000 000e0000 UC WC WT WB 

OK boot -v
Loading kernel...
/boot/kernel/kernel entry at 0xffffffff8038a000
Segment: 0x189b10@0x0 -> 0x200000-0x389b0f-
Segment: 0xe0ce88@0x18a000 -> 0x38a000-0x1196e87/
Segment: 0x6baaa4@0xf96e90 -> 0x1196e90-0x1851933/
Segment: 0x140@0x1800000 -> 0x1a00000-0x1a0013f
Segment: 0x1c8470@0x1801000 -> 0x1a01000-0x1ffffff (bss: 0x1bc9470-0x1ffffff)
syms=[
symtab: 0x198150@0x19c9718 -> 0x2000008-0x2198158
strtab: 0x1b8a00@0x1b61baa -> 0x2198160-0x2350b60]
Loading configured modules...
can't find '/boot/entropy'
can't find '/etc/hostid'
staging 0x36e00000 (not copying) tramp 0x36c3f000 PT4 0x36c36000
Start @ 0xffffffff8038a000 ...



Boot verbose console dies much sooner as well.
Comment 23 Wei Hu 2022-06-09 03:11:51 UTC 
Do you have evidence that it crashed? Or just the output stalled at different points during boot? 

The reason I am asking is the EFI console seemed to have changed on 13.x. I have suspicion maybe the new console code is having trouble. What is the VM status showing on Azure Portal for this VM after the console appeared to hang for a little while? Can you ping to this VM from other local VMs? 

The other thing is if you have tried building the Gen2 image from the latest FreeBSD main branch. Li-Wen from the FreeBSD Foundation says the problem seems to be only happening on 13.x. The Gen2 VMs from main branch seem to boot up fine in Azure. To further see if it is only an issue on 13, you can also try 12.3 Gen2 image if you can build one.
Comment 24 Michael Baptist 2022-06-09 07:03:12 UTC 
Is the EFI console the same console after the tramplone passes execution to the kernel?

For instance the code in hammer_time/mi_startup is a different console correct?

Not sure its EFI console related myself based on what I've seen, I could be totally wrong..
Comment 25 Michael Baptist 2022-06-09 22:13:10 UTC 
Made it a bit further 

Also I am at: 8020c05683f266c5513cfe8ad3e58e7e06f85ffa

Does't seem terribly far behind.

$git diff --stat HEAD origin/main -- stand/
 stand/common/isapnp.c        | 2 +-
 stand/libsa/zalloc.c         | 2 +-
 stand/libsa/zfs/Makefile.inc | 2 +-
 stand/loader.mk              | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

$git diff --stat HEAD origin/main -- sys/vm
 sys/vm/vm.h      | 2 +-
 sys/vm/vm_page.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)


Added some additional prints to see if it was pmap, but that seems to be returning out of pmap_init. I've never seen it go this far.

one.
subsystem ffffff
   vm_mem_init(0)... virtual memory init
kmem init
pmap enter...
pmap init!
pmap is initialized
done.
   vm_page_init(0)... done.
subsystem 1000000
   pcpu_zones_startup(0)... done.
   counter_u64_sysinit(&ktls_ifnet_reset_dropped)... done.
   counter_u64_sysinit(&ktls_ifnet_reset_failed)... done.
   counter_u64_sysinit(&ktls_toe_cbc)... done.
   counter_u64_sysinit(&ktls_toe_gcm)... done.
   counter_u64_sysinit(&ktls_toe_chacha20)... done.
   counter_u64_sysinit(&advance)... done.
   counter_u64_sysinit(&numcachehv)... done.
   counter_u64_sysinit(&numdrops)... done.
   counter_u64_sysinit(&dothits)... done.
   counter_u64_sysinit(&dotdothits)... done.
   counter_u64_sysinit(&nummiss)... done.
   counter_u64_sysinit(&nummisszap)... done.
   counter_u64_sysinit(&numposzaps)... done.
   counter_u64_sysinit(&numposhits)... done.
   counter_u64_sysinit(&numnegzaps)... done.
   counter_u64_sysinit(&numneghits)... done.
   counter_u64_sysinit(&numfullpathcalls)... done.
   counter_u64_sysinit(&numfullpathfail1)... done.
   counter_u64_sysinit(&numfullpathfail2)... done.
   counter_u64_sysinit(&numfullpathfail4)... done.
   counter_u64_sysinit(&numfullpathfound)... done.
   counter_u64_sysinit(&symlinktoobig)... done.
   counter_u64_sysinit(&zap_bucket_relock_success)... done.
   counter_u64_sysinit(&neg_created)... done.
   counter_u64_sysinit(&neg_evicted)... done.
   counter_u64_sysinit(&neg_evict_skipped_empty)... done.
   counter_u64_sysinit(&neg_evict_skipped_missed)... done.
   counter_u64_sysinit(&neg_evict_skipped_contended)... done.
   counter_u64_sysinit(&advance_wait)... done.
   cryptostats_init(0)... done.
   counter_u64_sysinit(&ocf_tls10_cbc_encrypts)... done.
   counter_u64_sysinit(&ocf_tls11_cbc_encrypts)... done.
   counter_u64_sysinit(&ocf_tls12_gcm_decrypts)... done.
   counter_u64_sysinit(&ocf_tls12_gcm_encrypts)... done.
   counter_u64_sysinit(&ocf_tls12_gcm_recrypts)... done.
   counter_u64_sysinit(&ocf_tls12_chacha20_decrypts)... done.
   counter_u64_sysinit(&ocf_tls12_chacha20_encrypts)... done.
   counter_u64_sysinit(&ocf_tls13_gcm_decrypts)... done.
   counter_u64_sysinit(&ocf_tls13_gcm_encrypts)... done.
   counter_u64_sysinit(&ocf_tls13_gcm_recrypts)... done.
   counter_u64_sysinit(&ocf_tls13_chacha20_decrypts)... done.
   counter_u64_sysinit(&ocf_tls13_chacha20_encrypts)... done.
   counter_u64_sysinit(&ocf_inplace)... done.
   counter_u64_sysinit(&ocf_separate_output)... done.
   counter_u64_sysinit(&ocf_retries)... done.
   counter_u64_sysinit(&swap_free_deferred)... done.
   counter_u64_sysinit(&swap_free_completed)... done.
   counter_u64_sysinit(&poll)... done.
   uma_startup_pcpu(0)... done.
   counter_u64_sysinit(&poll_scan)... done.
   counter_u64_sysinit(&object_collapses)... done.
   counter_u64_sysinit(&object_bypasses)... done.
   counter_u64_sysinit(&object_collapse_waits)... done.
   counter_u64_sysinit(&pqstate_commit_retries)... done.
   counter_u64_sysinit(&queue_ops)... done.
   counter_u64_sysinit(&queue_nops)... done.
   counter_u64_sysinit(&poll_fail)... done.
   counter_u64_sysinit(&vm_reserv_broken)... done.
   counter_u64_sysinit(&vm_reserv_freed)... done.
   counter_u64_sysinit(&vm_reserv_reclaimed)... done.
   counter_u64_sysinit(&ktls_tasks_active)... done.
   counter_u64_sysinit(&ktls_cnt_tx_pending)... done.
   counter_u64_sysinit(&ktls_cnt_tx_queued)... done.
   counter_u64_sysinit(&ktls_cnt_rx_queued)... done.
   counter_u64_sysinit(&ktls_offload_total)... done.
   counter_u64_sysinit(&ktls_offload_enable_calls)... done.
   counter_u64_sysinit(&ktls_offload_active)... done.
   counter_u64_sysinit(&ktls_offload_corrupted_records)... done.
   counter_u64_sysinit(&ktls_offload_failed_crypto)... done.
   counter_u64_sysinit(&pcid_save_cnt)... done.
   counter_u64_sysinit(&pv_page_count)... done.
   counter_u64_sysinit(&user_pt_page_count)... done.
   counter_u64_sysinit(&kernel_pt_page_count)... done.
   counter_u64_sysinit(&pmap_pde_demotions)... done.
   counter_u64_sysinit(&pmap_pde_mappings)... done.
   counter_u64_sysinit(&pmap_pde_p_failures)... done.
   counter_u64_sysinit(&pmap_pde_promotions)... done.
   counter_u64_sysinit(&pmap_pdpe_demotions)... done.
   counter_u64_sysinit(&ktls_switch_to_ifnet)... done.
   counter_u64_sysinit(&ktls_switch_to_sw)... done.
   counter_u64_sysinit(&ktls_switch_failed)... done.
   counter_u64_sysinit(&ktls_ifnet_disable_fail)... done.
   counter_u64_sysinit(&ktls_ifnet_disable_ok)... done.
   counter_u64_sysinit(&ktls_sw_cbc)... done.
   counter_u64_sysinit(&ktls_sw_gcm)... done.
   counter_u64_sysinit(&ktls_sw_chacha20)... done.
   counter_u64_sysinit(&ktls_ifnet_cbc)... done.
   counter_u64_sysinit(&ktls_ifnet_gcm)... done.
   counter_u64_sysinit(&ktls_ifnet_chacha20)... done.
   counter_u64_sysinit(&ktls_ifnet_reset)... done.
subsystem 1100000
   sysctl_register_all(0)... done.
   vmcounter_startup(0)... done.
   mallocinit(0)... done.
   malloc_init(&M_NEWNFSDEVINFO)... done.
   malloc_init(&M_NEWNFSSOCKREQ)... done.
   malloc_init(&M_NEWNFSCLDS)... done.
   malloc_init(&M_NEWNFSLAYRECALL)... done.
   malloc_init(&M_NEWNFSDSESSION)... done.
   malloc_init(&M_NEWNFSREQ)... done.
   malloc_init(&M_NEWNFSMNT)... done.
   malloc_init(&M_NFS_FHA)... done.
   malloc_init(&M_PFSNODES)... done.
   malloc_init(&M_PFSVNCACHE)... done.
   malloc_init(&M_TMPFSMNT)... done.
   malloc_init(&M_TMPFSNAME)... done.
   malloc_init(&M_TMPFSDIR)... done.
   malloc_init(&M_FLASHMAP)... done.
   malloc_init(&M_GEOM)... done.
   malloc_init(&M_RAID)... done.
   malloc_init(&M_MD_DDF)... done.
   malloc_init(&M_MD_INTEL)... done.
   malloc_init(&M_MD_JMICRON)... done.
   malloc_init(&M_MD_NVIDIA)... done.
   malloc_init(&M_MD_PROMISE)... done.
   malloc_init(&M_MD_SII)... done.
   malloc_init(&M_TR_CONCAT)... done.
   malloc_init(&M_TR_RAID0)... done.
   malloc_init(&M_TR_RAID1)... done.
   malloc_init(&M_TR_RAID1E)... done.
   malloc_init(&M_TR_RAID5)... done.
   malloc_init(&M_ISADEV)... done.
   malloc_init(&M_ISOFSMNT)... done.
   malloc_init(&M_ISOFSNODE)... done.
   malloc_init(&M_SCSISA)... done.
   malloc_init(&M_SCSIENC)... done.
   malloc_init(&M_NVLIST)... done.
   malloc_init(&XZ_DEC)... done.
   malloc_init(&M_ZSTD)... done.
   malloc_init(&M_DDB_CAPTURE)... done.
   malloc_init(&M_BOOTTRACE)... done.
   malloc_init(&M_AACBUF)... done.
   malloc_init(&M_AACCAM)... done.
   malloc_init(&M_EVDEV)... done.
   malloc_init(&M_AACRAIDBUF)... done.
   malloc_init(&M_IPSBUF)... done.
   malloc_init(&M_DEVT)... done.
   malloc_init(&M_TTYCONS)... done.
   malloc_init(&M_FILEDESC)... done.
   malloc_init(&M_PWD)... done.
   malloc_init(&M_PWDDESC)... done.
   malloc_init(&M_FILEDESC_TO_LEADER)... done.
   malloc_init(&M_SIGIO)... done.
   malloc_init(&M_FILECAPS)... done.
   malloc_init(&M_KDTRACE)... done.
   malloc_init(&M_KENV)... done.
   malloc_init(&M_KQUEUE)... done.
   malloc_init(&M_PARGS)... done.
   malloc_init(&M_FAIL_POINT)... done.
   malloc_init(&M_HHOOK)... done.
   malloc_init(&M_ITHREAD)... done.
   malloc_init(&M_PRISON)... done.
   malloc_init(&M_PRISON_RACCT)... done.
   malloc_init(&M_KTRACE)... done.
   malloc_init(&M_IXGBE)... done.
   malloc_init(&M_LINKER)... done.
   malloc_init(&M_AACRAIDCAM)... done.
   malloc_init(&M_LOCKF)... done.
   malloc_init(&M_LOGINCLASS)... done.
   malloc_init(&M_CACHE)... done.
   malloc_init(&M_DEVBUF)... done.
   malloc_init(&M_TEMP)... done.
   malloc_init(&M_ACPIINTR)... done.
   malloc_init(&M_IXGBE_SRIOV)... done.
   malloc_init(&M_ACPICA)... done.
   malloc_init(&M_MODULE)... done.
   malloc_init(&M_MTXPOOL)... done.
   malloc_init(&M_OSD)... done.
   malloc_init(&M_PMCHOOKS)... done.
   malloc_init(&M_PMC)... done.
   malloc_init(&M_SESSION)... done.
   malloc_init(&M_PROC)... done.
   malloc_init(&M_SUBPROC)... done.
   malloc_init(&M_CRED)... done.
   malloc_init(&M_ACPITASK)... done.
   malloc_init(&M_KBDMUX)... done.
   malloc_init(&M_LED)... done.
   malloc_init(&M_MALODEV)... done.
   malloc_init(&M_RCTL)... done.
   malloc_init(&M_PLIMIT)... done.
   malloc_init(&M_UIDINFO)... done.
   malloc_init(&M_SENDFILE)... done.
   malloc_init(&M_DUMPER)... done.
   malloc_init(&M_EKCD)... done.
   malloc_init(&M_MD)... done.
   malloc_init(&M_SYSCTL)... done.
   malloc_init(&M_SYSCTLOID)... done.
   malloc_init(&M_SYSCTLTMP)... done.
   malloc_init(&M_MDSECT)... done.
   malloc_init(&M_MFIBUF)... done.
   malloc_init(&M_TIDHASH)... done.
   malloc_init(&M_CALLOUT)... done.
   malloc_init(&M_UMTX)... done.
   malloc_init(&M_P31B)... done.
   malloc_init(&M_SWAP)... done.
   malloc_init(&M_BUS)... done.
   malloc_init(&M_BUS_SC)... done.
   malloc_init(&M_MPR)... done.
   malloc_init(&M_COMPRESS)... done.
   malloc_init(&M_DEVSTAT)... done.
   malloc_init(&M_EVENTHANDLER)... done.
   malloc_init(&M_FIRMWARE)... done.
   malloc_init(&M_GTASKQUEUE)... done.
   malloc_init(&M_KOBJ)... done.
   malloc_init(&M_PCPU)... done.
   malloc_init(&M_MPRSAS)... done.
   malloc_init(&M_RMAN)... done.
   malloc_init(&M_SBUF)... done.
   malloc_init(&M_SGLIST)... done.
   malloc_init(&M_TOPO)... done.
   malloc_init(&M_MPRUSER)... done.
   malloc_init(&M_MPT2)... done.
   malloc_init(&M_MPSSAS)... done.
   malloc_init(&M_MPSUSER)... done.
   malloc_init(&M_ACPISEM)... done.
   malloc_init(&M_MPTUSER)... done.
   malloc_init(&M_STACK)... done.
   malloc_init(&M_TASKQUEUE)... done.
   malloc_init(&M_TERMINAL)... done.
   malloc_init(&M_UNIT)... done.
   malloc_init(&M_VMEM)... done.
   malloc_init(&M_WITNESS)... done.
   malloc_init(&M_EVENTFD)... done.
   malloc_init(&M_IOCTLOPS)... done.
   malloc_init(&M_SELECT)... done.
   malloc_init(&M_IOV)... done.
   malloc_init(&M_SELFD)... done.
   malloc_init(&M_PROCDESC)... done.
   malloc_init(&M_MSG)... done.
   malloc_init(&M_SEM)... done.
   malloc_init(&M_SHM)... done.
   malloc_init(&M_TTY)... done.
   malloc_init(&M_PTS)... done.
   malloc_init(&M_ACCF)... done.
   malloc_init(&M_MRSAS)... done.
   malloc_init(&M_CAMCCBQ)... done.
   malloc_init(&M_ACPIDEV)... done.
   malloc_init(&M_CAMSIM)... done.
   malloc_init(&M_MVS)... done.
   malloc_init(&M_MWLDEV)... done.
   malloc_init(&M_NETMAP)... done.
   malloc_init(&M_NVD)... done.
   malloc_init(&M_NVME)... done.
   malloc_init(&M_OCS)... done.
   malloc_init(&M_OCS)... done.
   malloc_init(&M_SRIOV)... done.
   malloc_init(&M_PMC_MCCB)... done.
   malloc_init(&M_PMC_MSTL)... done.
   malloc_init(&M_PMC_MDVT)... done.
   malloc_init(&M_PMC_MPRT)... done.
   malloc_init(&M_PMC_MDEV)... done.
   malloc_init(&M_PMC_MFLG)... done.
   malloc_init(&M_PMC_MSLR)... done.
   malloc_init(&M_PMC_MTGT)... done.
   malloc_init(&TEMP)... done.
   malloc_init(&TEMP2)... done.
   malloc_init(&M_PMC_OSTI)... done.
   malloc_init(&M_PMC_MMAL)... done.
   malloc_init(&M_PPBUSDEV)... done.
   malloc_init(&M_PUC)... done.
   malloc_init(&M_KTLS)... done.
   malloc_init(&M_PACKET_TAGS)... done.
   malloc_init(&M_SHMFD)... done.
   malloc_init(&M_SONAME)... done.
   malloc_init(&M_PCB)... done.
   malloc_init(&M_CAMXPT)... done.
   malloc_init(&M_ACL)... done.
   malloc_init(&M_LIO)... done.
   malloc_init(&M_AIOS)... done.
   malloc_init(&M_BIOBUF)... done.
   malloc_init(&M_ENTROPY)... done.
   malloc_init(&M_CAMDEV)... done.
   malloc_init(&M_CAMCCB)... done.
   malloc_init(&M_CAMPATH)... done.
   malloc_init(&M_CAMPERIPH)... done.
   malloc_init(&M_SIIS)... done.
   malloc_init(&M_ACPICMBAT)... done.
   malloc_init(&M_HDAA)... done.
   malloc_init(&M_HDAC)... done.
   malloc_init(&M_HDACC)... done.
   malloc_init(&M_AC97)... done.
   malloc_init(&M_FEEDER)... done.
   malloc_init(&M_MIXER)... done.
   malloc_init(&M_MIDI)... done.
   malloc_init(&M_TCPLOGDEV)... done.
   malloc_init(&M_TWE)... done.
   malloc_init(&M_TWS)... done.
   malloc_init(&M_VFSCACHE)... done.
   malloc_init(&M_ACPIPERF)... done.
   malloc_init(&M_ACPIPWR)... done.
   malloc_init(&M_CAMSCHED)... done.
   malloc_init(&M_NVMEDA)... done.
   malloc_init(&M_UART)... done.
   malloc_init(&M_SEGMENT)... done.
   malloc_init(&M_NETADDR)... done.
   malloc_init(&M_VFS_HASH)... done.
   malloc_init(&M_VNODE)... done.
   malloc_init(&M_NAMEITRACKER)... done.
   malloc_init(&M_MOUNT)... done.
   malloc_init(&M_STATFS)... done.
   malloc_init(&M_CAMQ)... done.
   malloc_init(&M_VNODEPOLL)... done.
   malloc_init(&M_VNODE_MARKER)... done.
   malloc_init(&M_FADVISE)... done.
   malloc_init(&M_CHACHA20RANDOM)... done.
   malloc_init(&M_ZLIB)... done.
   malloc_init(&M_BPF)... done.
   malloc_init(&M_IFDESCR)... done.
   malloc_init(&M_IFNET)... done.
   malloc_init(&M_IFADDR)... done.
   malloc_init(&M_IFMADDR)... done.
   malloc_init(&M_CLONE)... done.
   malloc_init(&M_GIF)... done.
   malloc_init(&M_LLTABLE)... done.
   malloc_init(&M_TUN)... done.
   malloc_init(&M_VLAN)... done.
   malloc_init(&M_IFLIB)... done.
   malloc_init(&M_PFIL)... done.
   malloc_init(&M_NHOP)... done.
   malloc_init(&M_RTABLE)... done.
   malloc_init(&M_VNET)... done.
   malloc_init(&M_VNET_DATA)... done.
   malloc_init(&M_VNET_DATA_FREE)... done.
   malloc_init(&M_80211_VAP)... done.
   malloc_init(&M_80211_CRYPTO)... done.
   malloc_init(&M_80211_DFS)... done.
   malloc_init(&M_80211_MESH_PREQ)... done.
   malloc_init(&M_80211_MESH_PREP)... done.
   malloc_init(&M_80211_MESH_PERR)... done.
   malloc_init(&M_80211_MESH_RT)... done.
   malloc_init(&M_80211_MESH_GT_RT)... done.
   malloc_init(&M_80211_NODE)... done.
   malloc_init(&M_80211_NODE_IE)... done.
   malloc_init(&M_80211_POWER)... done.
   malloc_init(&M_80211_RATECTL)... done.
   malloc_init(&M_80211_SCAN)... done.
   malloc_init(&M_IGMP)... done.
   malloc_init(&M_IPID)... done.
   malloc_init(&M_INMFILTER)... done.
   malloc_init(&M_IPMADDR)... done.
   malloc_init(&M_IPMOPTS)... done.
   malloc_init(&M_IPMSOURCE)... done.
   malloc_init(&M_NETADDR)... done.
   malloc_init(&M_CC_MEM)... done.
   malloc_init(&M_TCP_FASTOPEN_CCACHE)... done.
   malloc_init(&M_HOSTCACHE)... done.
   malloc_init(&M_LRO)... done.
   malloc_init(&M_TCPLOG)... done.
   malloc_init(&M_TCPFUNCTIONS)... done.
   malloc_init(&M_SYNCACHE)... done.
   malloc_init(&M_FRAG6)... done.
   malloc_init(&M_IN6MFILTER)... done.
   malloc_init(&M_IP6MADDR)... done.
   malloc_init(&M_IP6MOPTS)... done.
   malloc_init(&M_IP6MSOURCE)... done.
   malloc_init(&M_IP6OPT)... done.
   malloc_init(&M_MLD)... done.
   malloc_init(&M_IP6NDP)... done.
   malloc_init(&M_IPSEC_INPCB)... done.
   malloc_init(&M_IPSEC_SA)... done.
   malloc_init(&M_IPSEC_SAH)... done.
   malloc_init(&M_IPSEC_SP)... done.
   malloc_init(&M_IPSEC_SR)... done.
   malloc_init(&M_IPSEC_MISC)... done.
   malloc_init(&M_IPSEC_SAQ)... done.
   malloc_init(&M_IPSEC_SAR)... done.
   malloc_init(&M_IPSEC_SPDCACHE)... done.
   malloc_init(&M_NLM)... done.
   malloc_init(&M_LKPI80211)... done.
   malloc_init(&M_KMALLOC)... done.
   malloc_init(&M_LKPI_FW)... done.
   malloc_init(&M_NETDEV)... done.
   malloc_init(&M_IDR)... done.
   malloc_init(&M_RADIX)... done.
   malloc_init(&M_LKPISKB)... done.
   malloc_init(&M_LSEQ)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_DUMP)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5EN)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5E_TLS)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5E_TLS_RX)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_MLX5_EEPROM)... done.
   malloc_init(&M_CRYPTO_DATA)... done.
   malloc_init(&M_AGP)... done.
   malloc_init(&M_KTLS_OCF)... done.
   malloc_init(&M_USB)... done.
   malloc_init(&M_USBDEV)... done.
   malloc_init(&M_AHCI)... done.
   malloc_init(&M_SCSICD)... done.
   malloc_init(&M_SCSICH)... done.
   malloc_init(&M_ATADA)... done.
   malloc_init(&M_CAMDEVQ)... done.
   malloc_init(&M_SCSIDA)... done.
   malloc_init(&M_PVSCSI)... done.
   malloc_init(&M_ATA)... done.
   malloc_init(&M_ATADMA)... done.
   malloc_init(&M_ATAPCI)... done.
   malloc_init(&M_ATHDEV)... done.
   malloc_init(&M_ATH_HAL)... done.
   malloc_init(&M_BACKLIGHT)... done.
   malloc_init(&M_AUDITCRED)... done.
   malloc_init(&M_AUDITDATA)... done.
   malloc_init(&M_AUDITPATH)... done.
   malloc_init(&M_AUDITTEXT)... done.
   malloc_init(&M_AUDITGIDSET)... done.
   malloc_init(&M_AUDITBSM)... done.
   malloc_init(&M_AUDITEVCLASS)... done.
   malloc_init(&M_AUDITEVNAME)... done.
   malloc_init(&M_AUDIT_PIPE)... done.
   malloc_init(&M_AUDIT_PIPE_ENTRY)... done.
   malloc_init(&M_AUDIT_PIPE_PRESELECT)... done.
   malloc_init(&M_AUDITTRIGGER)... done.
   malloc_init(&M_MACTEMP)... done.
   malloc_init(&M_TRIM)... done.
   malloc_init(&M_PAGEDEP)... done.
   malloc_init(&M_INODEDEP)... done.
   malloc_init(&M_BMSAFEMAP)... done.
   malloc_init(&M_NEWBLK)... done.
   malloc_init(&M_ALLOCDIRECT)... done.
   malloc_init(&M_INDIRDEP)... done.
   malloc_init(&M_ALLOCINDIR)... done.
   malloc_init(&M_FREEFRAG)... done.
   malloc_init(&M_FREEBLKS)... done.
   malloc_init(&M_FREEFILE)... done.
   malloc_init(&M_DIRADD)... done.
   malloc_init(&M_MKDIR)... done.
   malloc_init(&M_DIRREM)... done.
   malloc_init(&M_NEWDIRBLK)... done.
   malloc_init(&M_FREEWORK)... done.
   malloc_init(&M_FREEDEP)... done.
   malloc_init(&M_JADDREF)... done.
   malloc_init(&M_JREMREF)... done.
   malloc_init(&M_JMVREF)... done.
   malloc_init(&M_JNEWBLK)... done.
   malloc_init(&M_JFREEBLK)... done.
   malloc_init(&M_JFREEFRAG)... done.
   malloc_init(&M_JSEG)... done.
   malloc_init(&M_JSEGDEP)... done.
   malloc_init(&M_SBDEP)... done.
   malloc_init(&M_JTRUNC)... done.
   malloc_init(&M_JFSYNC)... done.
   malloc_init(&M_SENTINEL)... done.
   malloc_init(&M_SAVEDINO)... done.
   malloc_init(&M_JBLOCKS)... done.
   malloc_init(&M_MOUNTDATA)... done.
   malloc_init(&M_DIRHASH)... done.
   malloc_init(&M_DQUOT)... done.
   malloc_init(&M_UFSMNT)... done.
   malloc_init(&M_VMPGDATA)... done.
   malloc_init(&M_VTBUF)... done.
   malloc_init(&M_VT)... done.
   malloc_init(&M_UMAHASH)... done.
   malloc_init(&M_UMA)... done.
   malloc_init(&M_VTFONT)... done.
   malloc_init(&M_SYSMOUSE)... done.
   malloc_init(&M_BALLOON)... done.
   malloc_init(&M_XENBLOCKFRONT)... done.
   malloc_init(&M_XENBLOCKBACK)... done.
   malloc_init(&M_XENACPI)... done.
   malloc_init(&M_XENNETBACK)... done.
   malloc_init(&CISS_MALLOC_CLASS)... done.
   malloc_init(&M_SCSIPASS)...
Comment 26 Wei Hu 2022-06-10 02:29:46 UTC 
Michael, can you try 12.3 Gen2 VM on the same Azure VM size? I would like to know if it was something new in 13 which causes this problem.
Comment 27 Michael Baptist 2022-06-10 16:20:54 UTC 
(In reply to Wei Hu from comment #26)
Yes I will try that.
Comment 28 Michael Baptist 2022-06-15 18:16:59 UTC 
Tested 12.3 and HEAD and saw similar result.

Skus tested:

Standard D8 V3 - booted once failed once
Standard D8 V4 - failed to boot
Standard D8 V5 - booted up fine

Boot issue presents in the same place, which is during sysinit.
Comment 29 Wei Hu 2022-06-22 03:28:20 UTC 
We have narrowed down the problem area to serial console output in FreeBSD and Azure COM emulator. Seems these two don't work very well at the early boot time. Maybe some synchronizing event was missing which causes the hang. The interesting part is, if the boot passes certain stage, the serial console works 100% fine. 

So, the temporary workaround is to suppress all printf's from the beginning of boot up until:

 SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startup, NULL);

We are working on checking the state of COM emulator in Azure to understand the cause.
Comment 30 Michael Baptist 2022-06-28 16:46:43 UTC 
Wei Hu, thanks for the update.

Commenting out the printfs does "fix" the issue for now.
Comment 31 Brad Davis freebsd_committer freebsd_triage 2023-02-01 03:34:45 UTC 
Has there been any progress on this issue?
Comment 32 Wei Hu 2023-02-03 07:28:54 UTC 
(In reply to Brad Davis from comment #31)
This problem only happens on Gen2 VMs in Azure. On-prem Hyper-V don't have this hang issue. Unfortunately I don't have too much experience debugging the FreeBSD serial console issue and people with such skills don't have access to Azure VMs. So we don't have much progress to narrow down it yet.
Comment 33 commit-hook freebsd_committer freebsd_triage 2023-03-18 07:21:28 UTC 
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=8ea7fa16d9fe5acb7d42a223efbfa23627aa5e0c

commit 8ea7fa16d9fe5acb7d42a223efbfa23627aa5e0c
Author:     Wei Hu <whu@FreeBSD.org>
AuthorDate: 2023-03-14 15:49:33 +0000
Commit:     Wei Hu <whu@FreeBSD.org>
CommitDate: 2023-03-18 07:07:54 +0000

    uart: Don't change settings or throttle putc for Hyper-V

    Azure setup does not like it when FreeBSD overrides the settings of the
    UART device. When Hyper-V is detected, don't do this and also don't
    throttle putc() output. This is a workaround for the early boot hang
    of FreeBSD on Azure.

    Tested on Azure, ESXi (VM with serial port), and SG-8200

    PR:             264267
    Reviewed by:    kevans, whu
    Tested by:      whu
    Obtained from:  Rubicon Communications, LLC (Netgate)
    MFC after:      2 weeks
    Sponsored by:   Rubicon Communications, LLC (Netgate)

 sys/dev/uart/uart_dev_ns8250.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)
Comment 34 commit-hook freebsd_committer freebsd_triage 2023-03-18 07:21:31 UTC 
A commit in branch main references this bug:

URL: https://cgit.FreeBSD.org/src/commit/?id=927358dd98cb902160093e0dc0bac002d6b43858

commit 927358dd98cb902160093e0dc0bac002d6b43858
Author:     Wei Hu <whu@FreeBSD.org>
AuthorDate: 2023-03-14 15:13:46 +0000
Commit:     Wei Hu <whu@FreeBSD.org>
CommitDate: 2023-03-18 07:07:35 +0000

    amd64 loader: Use efiserialio for Hyper-V booted systems

    UEFI provides ConIn/ConOut handles for consoles that it supports,
    which include the text-video and serial ports. When the serial port
    is available, use the UEFI driver instead of direct io-port accesses
    to avoid conflicts between the firmware and direct hardware access, as
    happens on Hyper-V (Azure) setups.

    This change enables efiserialio to be built for efi-amd64 and has
    higher order priority vs comconsole, and only uses efiserialio
    if the hypervisor is Hyper-V. When efiserialio successfully
    probes, it will set efi_comconsole_avail=true which will prevent
    comconsole from probing in this setup.

    Tested on Hyper-V, ESXi and Azure VMs.

    PR:             264267
    Reviewed by:    kevans, whu
    Tested by:      whu
    Obtained from:  Rubicon Communications, LLC (Netgate)
    MFC after:      2 weeks
    Sponsored by:   Rubicon Communications, LLC (Netgate)

 stand/efi/loader/arch/amd64/Makefile.inc |  1 +
 stand/efi/loader/bootinfo.c              | 11 ++++++--
 stand/efi/loader/conf.c                  |  6 +++++
 stand/efi/loader/efiserialio.c           | 43 ++++++++++++++++++++++++++++----
 stand/i386/libi386/comconsole.c          | 14 +++++++++++
 5 files changed, 68 insertions(+), 7 deletions(-)