After install a FreeBSD guest OS in VMware vSphere, we have 2 different ways to do hot adding NVMe controller and hard disk. 1st: Hot adding NVMe controller firstly and press OK in "Edit Setting" of vSphere vCenter UI, then hot adding a Hard disk attached to this NVMe controller and press OK in "Edit Setting"; 2nd: Hot adding NVMe controller and a hard disk attached to this NVMe controller at the same time, and press OK in "Edit Setting" of vSphere vCenter UI; For the 1st scenario, all currrent FreeBSD 12.x and 13.x need reboot to detect the NVMe hard disk. From dmesg, it reports： # dmesg pci5: <ACPI PCI bus> on pcib19 nvme0: <VMware NVMe Controller> at device 0.0 on pci5 nvme0: temperature threshold not supported nvme0: async event occurred (type 0x2, info 0x00, page 0x04) After reboot, dmesg reports "/dev/nvd0" has been detected. # dmesg ...... nvme0: temperature threshold not supported nvd0: <VMware Virtual NVMe Disk> NVMe namespace nvd0: 1024MB (2097152 512 byte sectors) ...... For the 2nd scenario, all current FreeBSD 12.x and 13.x can detect NVMe hard disk without reboot. # dmesg pci5: <ACPI PCI bus> on pcib19 nvme0: <VMware NVMe Controller> at device 0.0 on pci5 nvme0: temperature threshold not supported nvd0: <VMware Virtual NVMe Disk> NVMe namespace nvd0: 1024MB (2097152 512 byte sectors) Can FreeBSD add the support of detecting the NVMe hard disk without reboot after hot adding NVMe controller and hard disk separately? Thanks in advance! Yanhui
Could you test with with nda(4) and/or -CURRENT, as this combination seems to work for me, running main-n262599-60167184abd5: (adding controller, pressing OK) nvme1: <VMware NVMe Controller> at device 2.0 on pci2 nvme1: temperature threshold not supported (adding "disk" to first controller that existed at boot, pressing OK) nvme0: async event occurred (type 0x2, info 0x00, page 0x04) nda1 at nvme0 bus 0 scbus2 target 0 lun 2 nda1: <VMware Virtual NVMe Disk 1.3 VMware NVME_0000> nda1: Serial Number VMware NVME_0000 nda1: nvme version 1.3 x32 (max x32) lanes PCIe Gen2 (max Gen2) link nda1: 8192MB (16777216 512 byte sectors) (adding "disk" to newly added controller, pressing OK) nvme1: async event occurred (type 0x2, info 0x00, page 0x04) nda2 at nvme1 bus 0 scbus3 target 0 lun 1 nda2: <VMware Virtual NVMe Disk 1.3 VMware NVME_0001> nda2: Serial Number VMware NVME_0001 nda2: nvme version 1.3 x32 (max x32) lanes PCIe Gen2 (max Gen2) link nda2: 8192MB (16777216 512 byte sectors)
Hi Yuri, (In reply to Yuri Pankov from comment #1) Would you please tell me how to test with with nda(4) and/or -CURRENT? Thanks! Yanhui
(In reply to Yanhui He from comment #2) There are snapshots in https://download.freebsd.org/snapshots/amd64/amd64/ISO-IMAGES/14.0/. For using nda(4) instead of nvd(4), please see hw.nvme.use_nvd tunable in nda(4) man page (adding hw.nvme.use_nvd="0" to /boot/loader.conf should do the trick for you). I am just trying to understand difference between your setup and mine; looking at the diff for nvme between main and releng/13.2 there is nothing that stands out, so it really could be the nda instead of nvd.
(In reply to Yuri Pankov from comment #3) Thanks Yuri, I'll try it.
Hi Yuri, Sorry for the late reply! Tried on https://download.freebsd.org/snapshots/amd64/amd64/ISO-IMAGES/14.0/FreeBSD-14.0-CURRENT-amd64-20230713-510fd8313800-264135-disc1.iso, and found it's OK for nda. If not add "hw.nvme.use_nvd="0"" in "/boot/loader.conf", it still can work by nda. Also tried on FreeBSD 13.2GA, it's OK for nda too. If not use nda, it still cannot recognize the newly added NVMe hard disk. So, does it mean we can just use nda for FreeBSD in the future? If we use nda, we need to add "hw.nvme.use_nvd="0"" in FreeBSD 13.x. When we can support this function by default in FreeBSD 13.x? Or, just support it from FreeBSD 14.x? Thanks! Yanhui