Created attachment 247783 [details] Information about the Linux Guest We have a massive system (details below) which runs a massive VM. The host runs on FreeBSD while the VM guest is Ubuntu Linux. ``` Host Information: root@genomic:~ # sysctl hw.model hw.model: AMD EPYC 7702 64-Core Processor root@genomic:~ # sysctl hw.ncpu hw.ncpu: 256 root@genomic:~ # sysctl hw.physmem hw.physmem: 2185595944960 root@genomic:~ # sysctl hw.realmem hw.realmem: 2200361238528 ``` The VM uses 1.5TB of RAM, here's the VM configuration: ``` # vm info comp0 ------------------------ Virtual Machine: comp0 ------------------------ state: stopped datastore: default loader: uefi uuid: 2893e12c-637d-11ee-af94-7cc255269cb8 cpu: 240 cpu-topology: sockets=2, cores=60, threads=2 memory: 1536G network-interface number: 0 emulation: virtio-net virtual-switch: public fixed-mac-address: 58:9c:fc:0e:4b:00 fixed-device: - virtual-disk number: 0 device-type: file emulation: virtio-blk options: - system-path: /usr/local/vm/comp0/disk0.img bytes-size: 216895848448 (202.000G) bytes-used: 190077076480 (177.023G) ``` and here's the bhyve arguments according to vm-bhyve: ``` Jan 19 15:40:32: [bhyve options: -c 240,sockets=2,cores=60,threads=2 -m 1536G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U 2893e12c-637d-11ee-af94-7cc255269cb8 -u -S] Jan 19 15:40:32: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/usr/local/vm/comp0/disk0.img -s 5:0,virtio-blk,/dev/zvol/zscratch/scratch -s 6:0,virtio-net,tap0,mac=58:9c:fc:0e:4b:00 -s 7:0,passthru,68/0/81 -s 8:0,fbuf,tcp=127.0.0.1:5900,wait] ``` We've been using the GRUB loader for a while and decided to move to the UEFI loader. After the migration we noticed that Linux shows only 503GB of RAM from the 1.5TB allocated. We didn't have such issue when using the grub loader. I'm attaching a tarball which has information of the Linux system, including the following files. linux_uefi_iomem.txt — /proc/iomem linux_uefi_meminfo.txt — /proc/meminfo linux_uefi_vmstat.txt — /proc/vmstat linux_uefi_dmidecode.txt — dmidecode linux_uefi_free.txt — free linux_uefi_dmesg.txt — dmesg linux_uefi_cpuinfo.txt — /proc/cpuinfo We've also noticed that inside of TianoCore's UEFI menu, it shows the 1.5TB of RAM properly (check screenshot at: https://notes.bsd.am/bhyve_uefi_menu.png ) Not sure if this is an issue with Linux or bhyve's UEFI loader. any tips are appreciated.
Not sure if it's related but it would be a good idea anyway to use the `-A` flag. There's already a PR for vm-bhyve: https://github.com/churchers/vm-bhyve/pull/525
According to the man page in 15-CURRENT, it is obsolete. Reading 14.0-RELEASE man page it doesn't say -A is obsolete. That probably means that it will become obsolete at some point, so just a heads up.
As it's always a good idea to use the `-A` flag, the default was changed in 15-CURRENT: https://cgit.freebsd.org/src/commit/?id=6a0e7f908802b86ca5d1c0b3c404b8391d0f626e
This is due to memory being detected from CMOS bytes, which limits the maximum amount of RAM to 1TB. This appears to be leftover code imported from the original UDK2014 bhyve EFI code, and was removed in OVMF with this commit: https://edk2.groups.io/g/devel/message/80822 The fix would be to use the E820 map instead of CMOS in Bhyve/PlatformPei/MemDetect.c
I'll work on it.
^Triage: committer's bit has been taken in for safekeeping.