The bhyve NVMe emulation has long passed the NVMe compliance test suite but that is no longer the case as of the import of LLVM 14 into FreeBSD CURRENT. To reproduce: 13.0R and 13.1 behavior: Attach a 1GB disk image with backing store type "nvme" which should appear as "nvd0/nvme0", run this command: nvmecontrol io-passthru -o 0x2 -l 4096 -4 0x2ffff0 -r nvme0ns1 Result: nvme_opc_write_read command would exceed LBA range(slba=0x2ffff0 nblocks=0x1) nvme0: READ sqid:2 cid:127 nsid:1 lba:3145712 len:1 nvme0: LBA OUT OF RANGE (00/80) sqid:2 cid:127 cdw0:0 On 14-CURRENT after the LLVM 14 import, the bhyve process panics without a core dump, and attaching lldb results in (bhyve PID 9800): Process 9800 stopped * thread #52, name = 'vcpu 0', stop reason = signal SIGSEGV: invalid address (fault address: 0xb8) frame #0: 0x0000396779935c8b bhyve`pci_nvme_read(ctx=0x000039710c91a500, vcpu=0, pi=0x0000000000000000, baridx=-1985581039, offset=0, size=0) at pci_nvme.c:3035:34 3032 pci_nvme_read(struct vmctx *ctx, int vcpu, struct pci_devinst *pi, int baridx, 3033 uint64_t offset, int size) 3034 { -> 3035 struct pci_nvme_softc* sc = pi->pi_arg; 3036 3037 if (baridx == pci_msix_table_bar(pi) || 3038 baridx == pci_msix_pba_bar(pi)) { (lldb) bt * thread #52, name = 'vcpu 0', stop reason = signal SIGSEGV: invalid address (fault address: 0xb8) * frame #0: 0x0000396779935c8b bhyve`pci_nvme_read(ctx=0x000039710c91a500, vcpu=0, pi=0x0000000000000000, baridx=-1985581039, offset=0, size=0) at pci_nvme.c:3035:34 frame #1: 0x5ae6c31489a67011 frame #2: 0x000039677992eaca bhyve`pci_emul_mem_handler(ctx=<unavailable>, vcpu=<unavailable>, dir=<unavailable>, addr=<unavailable>, size=<unavailable>, val=<unavailable>, arg1=0x0000396fa69aa800, arg2=0) at pci_emul.c:498:4 frame #3: 0x000039677991f8a9 bhyve`mem_write(ctx=0x0000396fa6978800, vcpu=<unavailable>, gpa=3221229576, wval=1, size=4, arg=<unavailable>) at mem.c:165:10 frame #4: 0x000039677994d30f bhyve`vmm_emulate_instruction [inlined] emulate_mov(vm=0x0000396fa6978800, vcpuid=0, gpa=3221229576, vie=<unavailable>, memread=<unavailable>, memwrite=(bhyve`mem_write at mem.c:161), arg=<unavailable>) at vmm_instruction_emul.c:0:10 frame #5: 0x000039677994d1e5 bhyve`vmm_emulate_instruction(vm=0x0000396fa6978800, vcpuid=0, gpa=3221229576, vie=<unavailable>, paging=<unavailable>, memread=<unavailable>, memwrite=(bhyve`mem_write at mem.c:161), memarg=0x0000396fa696f878) at vmm_instruction_emul.c:1790:11 frame #6: 0x000039677991f364 bhyve`emulate_mem_cb(ctx=<unavailable>, vcpu=<unavailable>, paddr=<unavailable>, mr=<unavailable>, arg=<unavailable>) at mem.c:241:10 frame #7: 0x000039677991f264 bhyve`access_memory(ctx=0x0000396fa6978800, vcpu=0, paddr=3221229576, cb=(bhyve`emulate_mem_cb at mem.c:237), arg=0x000039712d3f0eb0) at mem.c:218:8 frame #8: 0x000039677991f17f bhyve`emulate_mem(ctx=0x0000396fa6978800, vcpu=<unavailable>, paddr=<unavailable>, vie=0x000039710c935188, paging=0x000039710c935170) at mem.c:254:10 frame #9: 0x0000396779912fd3 bhyve`vmexit_inst_emul(ctx=0x0000396fa6978800, vmexit=0x000039710c935140, pvcpu=0x000039712d3f0f2c) at bhyverun.c:852:8 frame #10: 0x00003967799129b3 bhyve`vm_loop(ctx=0x0000396fa6978800, vcpu=0, startrip=<unavailable>) at bhyverun.c:987:8 frame #11: 0x0000396779911374 bhyve`fbsdrun_start_thread(param=0x0000396fa69841b0) at bhyverun.c:542:2 frame #12: 0x0000396fa3db296a libthr.so.3`thread_start(curthread=0x000039710c91a500) at thr_create.c:292:16
On my system, bhyve segfaults after printing nvme_opc_write_read command would exceed LBA range(slba=0x2ffff0 nblocks=0x1) If I disassemble nvme_opc_write_read(), the end of the function (inlined into pci_nvme_write()) is: 0x000000000106bfb3 <+7763>: jmp 0x106bfbc <pci_nvme_write+7772> 0x000000000106bfb5 <+7765>: lea -0x3b292(%rip),%rsi # 0x1030d2a 0x000000000106bfbc <+7772>: lea -0x4008a(%rip),%rdx # 0x102bf39 0x000000000106bfc3 <+7779>: mov %r9,%rcx 0x000000000106bfc6 <+7782>: xor %eax,%eax 0x000000000106bfc8 <+7784>: call 0x1086010 <fprintf@plt> End of assembler dump. and that fprintf() call is the warning. If I disassemble past that, I get (gdb) x/16i 0x000000000106bfc8 0x106bfc8 <pci_nvme_write+7784>: call 0x1086010 <fprintf@plt> => 0x106bfcd: nopl (%rax) 0x106bfd0 <pci_nvme_read>: push %rbp 0x106bfd1 <pci_nvme_read+1>: mov %rsp,%rbp 0x106bfd4 <pci_nvme_read+4>: push %r15 0x106bfd6 <pci_nvme_read+6>: push %r14 0x106bfd8 <pci_nvme_read+8>: push %r13 so we're just running off the end of the function and going into pci_nvme_read(). That's pretty weird! I thought the compiler would insert breakpoints between functions. Maybe there is some UB happening here, but compiling bhyve with UBSAN makes the problem go away. We compile bhyve with many warnings disabled; enabling them for pci_nvme.c uncovers some actual bugs, but fixing them doesn't fix the problem. And it's really bizarre that the compiler is apparently assuming that fprintf() won't return.
I lost a word on the title. Here's hoping a rename will not break anything.
Looks like an instance of "compiler does something stupid when it sees a use of an uninitialized variable." The compiler is failing us here, but fixing the pci_nvme code resolves the problem: https://reviews.freebsd.org/D36119
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b6ecef28bfd7c1c267442fae1c8f2fe0f699f617 commit b6ecef28bfd7c1c267442fae1c8f2fe0f699f617 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2022-08-14 15:57:24 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2022-08-14 15:59:01 +0000 bhyve: Address uses of uninitialized variables in pci_nvme.c The debug print in nvme_opc_get_log_page() would print an uninitialized local variable. In nvme_opc_write_read(), a failed LBA bounds check would cause pci_nvme_stats_write_read_update() to be called with an uninitialized variable as a parameter. Although the parameter is unused when the check fails (and so status != 0), LLVM 14 emits some bogus machine code in this path, which happens to result in a segfault when it gets executed. PR: 265749 Reviewed by: chuck, emaste MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D36119 usr.sbin/bhyve/pci_nvme.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
Tested this fixes the original case. Thank you!
Closing with Chuck's permission as the original test passes. Thank you everyone who helped track down this highly-undefined behavior!
A commit in branch stable/13 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b8e33d1abeae18c0441583f912ff9dc85c628180 commit b8e33d1abeae18c0441583f912ff9dc85c628180 Author: Mark Johnston <markj@FreeBSD.org> AuthorDate: 2022-08-14 15:57:24 +0000 Commit: Mark Johnston <markj@FreeBSD.org> CommitDate: 2022-08-29 15:01:01 +0000 bhyve: Address uses of uninitialized variables in pci_nvme.c The debug print in nvme_opc_get_log_page() would print an uninitialized local variable. In nvme_opc_write_read(), a failed LBA bounds check would cause pci_nvme_stats_write_read_update() to be called with an uninitialized variable as a parameter. Although the parameter is unused when the check fails (and so status != 0), LLVM 14 emits some bogus machine code in this path, which happens to result in a segfault when it gets executed. PR: 265749 Reviewed by: chuck, emaste Sponsored by: The FreeBSD Foundation (cherry picked from commit b6ecef28bfd7c1c267442fae1c8f2fe0f699f617) usr.sbin/bhyve/pci_nvme.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)