Created attachment 242013 [details]
FreeBSD 13.x NVMe boot disk shows 2nd GPT table is corrupt or invalid

For FreeBSD 13.x guest OS installation in VMware vSphere, if configure its vMem size to 4GB or larger than 4GB, it has 2 issues.

* 1st: Re-enter install UI after fresh install and reboot FreeBSD 13.x 32 bit VM(Virtual Machine) with LSI Logic Parallel or LSI Logic SAS boot disk
* 2nd: BTX halted error happened after fresh install and reboot FreeBSD 13.x 32 bit VM with NVMe boot disk

Please refer to the 2 attached screenshot of "FreeBSD 13.2 32 re-enter install UI.png" and "FreeBSD 13.x BTX halted.png".

All tests are based on GA release.

---Test FreeBSD 12.x and 13.x releases---
FreeBSD 13.0: Has these 2 issues
FreeBSD 13.1: Has these 2 issues
FreeBSD 13.2: Has these 2 issues
FreeBSD 12.4: Doesn’t have these 2 issues

---Test FreeBSD 13.x 32bit and 64bit---
FreeBSD 13.x 32bit: Has these 2 issues
FreeBSD 13.x 64bit: Doesn’t have these 2 issues

---Tested Storage Controller---
LSI Logic Parallel: Has the 1st issue
LSI Logic SAS: Has the 1st issue
NVMe: Has the 2nd issue
PVSCSI: Doesn’t have these 2 issues
SATA: Doesn’t have these 2 issues

####################Test for 1st issue####################
---Tested vCPU number and vMem Size combination when fresh install a FreeBSD 13.2 VM---
vCPU		vMem		Re-enter installation?
1		1		No
16		1		No
4		2		No
4		3		No
16		3		No
1		4		Yes
4		4		Yes
4		16		yes
16		16		Yes


---Test Steps---
Step1: Fresh install a FreeBSD 13.2 32bit VM with configuration as below
vCPU: 1
vMem: 4GB
Hard disk controller: LSI Logic Parallel or LSI Logic SAS
Boot Option: BIOS (default), EFI cannot be supported with 32bit

Install FreeBSD 13.2 32bit VM with vMem size equal to or larger than 4GB can reproduce this issue.

Step2: Install FreeBSD 13.2 32bit OS according to the instructions and reboot

Step3: After reboot, re-enter the installation UI.

---Problem Analysis---
The issue will happen with the following conditions
1) FreeBSD 13.x 32bit Fresh install
2) LSI Logic Parallel or SAS boot hard disk
3) vMem >= 4GB
4) BIOS Boot
####################Test for 1st issue####################


####################Test for 2nd issue####################
---Tested vCPU number and vMem Size combination when fresh install a FreeBSD 13.2 VM---
vCPU		vMem		BTX Halted?
1		1		No
16		1		No
8		2		No
16		2		No
3		3		No
4		3		No
1		4		Yes
2		4		Yes
3		4		Yes
4		4		Yes
16		4		yes
3		5		Yes
1		5		Yes
8		5		Yes
1		6		Yes
1		8		Yes
32		6		Yes


---Test Steps---
Step1: Fresh install a FreeBSD 13.2 32bit VM with configuration as below
vCPU: 1
vMem: 4GB
Hard disk controller: NVMe
Boot Option: BIOS (default), EFI cannot be supported with 32bit

Install FreeBSD 13.2 32bit VM with vMem size equal to or larger than 4GB can reproduce this issue.

Step2: Install FreeBSD 13.2 32bit OS according to the instructions and reboot

Step3: After reboot, BTX halted error appears.

---Problem Analysis---
The issue will happen with the following conditions
1) FreeBSD 13.x 32bit Fresh install
2) NVMe boot hard disk
3) vMem >= 4GB
4) BIOS Boot

Note:
1) Cold adding vCPU and vMem with different number in an installed FreeBSD 13.2 VM is OK
2) UEFI boot cannot be supported in FreeBSD 32bit

During the installation, if we choose NVMe disk as boot disk, it will report the following error. The Screenshot of “FreeBSD 13.x NVMe boot disk shows 2nd GPT table is corrupt or invalid.png” is also attached.

******
GEOM:disked/DISK-VMware%20NVME_0000: the secondary GPT table is corrupt or invalid.
GEOM:disked/DISK-VMware%20NVME_0000: using the primary only – recovery suggested.
GEOM:disked/DISK-VMware%20NVME_0000: the secondary GPT table is corrupt or invalid.
GEOM:disked/DISK-VMware%20NVME_0000: using the primary only – recovery suggested.
……
******

Because it doesn’t have this issue for any other storage controller which attached to boot hard disk, NVMe controller with vMem size equal to or larger than 4GB may bring this problem.
####################Test for 2nd issue####################


Would FreeBSD help to fix these 2 issues?

Thanks in advance!
Yanhui