Bug 256317 - [bhyve] Assertion failure: (cq->qbase != NULL) in function pci_nvme_cq_update, file /usr/src/usr.sbin/bhyve/pci_nvme.c
Summary: [bhyve] Assertion failure: (cq->qbase != NULL) in function pci_nvme_cq_update...
Status: In Progress
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 13.0-RELEASE
Hardware: Any Any
: --- Affects Only Me
Assignee: Chuck Tuffli
Depends on:
Reported: 2021-06-01 04:46 UTC by Cheolwoo Myung
Modified: 2021-06-14 06:27 UTC (History)
1 user (show)

See Also:

add checks to avoid assertion (3.67 KB, patch)
2021-06-13 23:55 UTC, Chuck Tuffli
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Cheolwoo Myung 2021-06-01 04:46:43 UTC
To reproduce the bug, please follow the command (bhyve version 13.0):


$ bhyvectl --vm=reproVM --destroy

$ bhyve -s 2:0,ahci-hd,hyfuzz.img -s 3,nvme,./nvme.img -s 0:0,hostbridge -s 1:0,lpc -c 1 -m 512M -l bootrom,./BHYVE_UEFI.fd -HPA reproVM


File: https://drive.google.com/file/d/1irW37yqP1cR44v4BkWX5jMp_XhsRS6kn/view?usp=sharing
Comment 1 Chuck Tuffli freebsd_committer 2021-06-01 19:41:06 UTC
Would you mind adding whatever content is in the link directly to this bug?
Comment 2 Cheolwoo Myung 2021-06-02 06:30:00 UTC
(In reply to Chuck Tuffli from comment #1)
The test case for the bug is too large to upload directly. Instead, can I write down the crash log and the input that triggers the bug?

# Crash Log
Assertion failed: (cq->qbase != NULL), function pci_nvme_cq_update, file /usr/src/usr.sbin/bhyve/pci_nvme.c, line 858.

#0  0x0000000800a202ea in thr_kill () from /lib/libc.so.7
#1  0x0000000800995064 in raise () from /lib/libc.so.7
#2  0x0000000800a49f29 in abort () from /lib/libc.so.7
#3  0x0000000800977f81 in __assert () from /lib/libc.so.7
#4  0x00000000003ca3de in pci_nvme_cq_update (sc=<optimized out>, cq=0x616000001b80, cdw0=<optimized out>, cid=<optimized out>, sqid=0, status=2) at /usr/src/usr.sbin/bhyve/pci_nvme.c:858
#5  0x00000000003c1294 in pci_nvme_handle_admin_cmd (sc=<optimized out>, value=<optimized out>) at /usr/src/usr.sbin/bhyve/pci_nvme.c:1641
#6  0x00000000003bad51 in pci_nvme_handle_doorbell (ctx=<optimized out>, sc=0x0, idx=0, is_sq=<optimized out>, value=<optimized out>) at /usr/src/usr.sbin/bhyve/pci_nvme.c:2330
#7  0x00000000003b636d in pci_nvme_write_bar_0 (ctx=0x606000000140, sc=<optimized out>, offset=<optimized out>, size=4, value=<optimized out>) at /usr/src/usr.sbin/bhyve/pci_nvme.c:2423
#8  pci_nvme_write (ctx=0x606000000140, vcpu=<optimized out>, pi=<optimized out>, baridx=<optimized out>, offset=<optimized out>, size=4, value=3932165) at /usr/src/usr.sbin/bhyve/pci_nvme.c:2541
#9  0x0000000000397356 in pci_emul_mem_handler (ctx=0x606000000140, vcpu=0, dir=<optimized out>, addr=<optimized out>, size=0, val=<optimized out>, arg1=0x616000001580, arg2=0) at /usr/src/usr.sbin/bhyve/pci_emul.c:420
#10 0x0000000000345968 in mem_write (ctx=<optimized out>, vcpu=<optimized out>, gpa=<optimized out>, wval=<optimized out>, size=<optimized out>, arg=0x6080000052b8) at /usr/src/usr.sbin/bhyve/mem.c:162
#11 0x000000000043ff1e in emulate_mov (vm=0x606000000140, vcpuid=<optimized out>, gpa=<optimized out>, vie=<optimized out>, memread=<optimized out>, memwrite=<optimized out>, arg=<optimized out>)
    at /usr/src/sys/amd64/vmm/vmm_instruction_emul.c:615
#12 vmm_emulate_instruction (vm=<optimized out>, vcpuid=<optimized out>, gpa=<optimized out>, vie=<optimized out>, paging=<optimized out>, memread=<optimized out>, memwrite=0x345810 <mem_write>, memarg=0x6080000052b8)
    at /usr/src/sys/amd64/vmm/vmm_instruction_emul.c:1789
#13 0x000000000034418a in emulate_mem_cb (ctx=0x72656, vcpu=6, paddr=0, mr=0x0, arg=<optimized out>) at /usr/src/usr.sbin/bhyve/mem.c:238
#14 0x0000000000343ca9 in access_memory (ctx=<optimized out>, vcpu=0, paddr=<optimized out>, cb=<optimized out>, arg=<optimized out>) at /usr/src/usr.sbin/bhyve/mem.c:215
#15 0x000000000034357d in emulate_mem (ctx=<optimized out>, vcpu=<optimized out>, paddr=<optimized out>, vie=<optimized out>, paging=<optimized out>) at /usr/src/usr.sbin/bhyve/mem.c:251
#16 0x000000000030c240 in vmexit_inst_emul (ctx=<optimized out>, vmexit=0xd1df20 <vmexit>, pvcpu=0x7fffdcbe3de0) at /usr/src/usr.sbin/bhyve/bhyverun.c:784
#17 0x000000000030a566 in vm_loop (ctx=<optimized out>, vcpu=<optimized out>, startrip=<optimized out>) at /usr/src/usr.sbin/bhyve/bhyverun.c:924
#18 0x0000000000305899 in fbsdrun_start_thread (param=<optimized out>) at /usr/src/usr.sbin/bhyve/bhyverun.c:473
#19 0x000000080086e82b in ?? () from /lib/libthr.so.3
#20 0x0000000000000000 in ?? ()

# Crash Input

// bar_addr[0] : BAR0 of NVMe

writel(0x0, bar_addr[0]+0x14);
writel(0x180000f0, bar_addr[0]+0x34);
writel(0x460071, bar_addr[0]+0x14);
writel(0x3c0005, bar_addr[0]+0x1000);
writel(0x5, bar_addr[0]+0x1004);

Comment 3 Chuck Tuffli freebsd_committer 2021-06-03 03:20:52 UTC
Thank you for providing the test case. It has shown me 4 places where error checking is missing. Is it possible to get the source code for this test case or the entire suite?

Also, once I have fixes, how would you like to test them? E.g. are you comfortable building from sources etc.?
Comment 4 Cheolwoo Myung 2021-06-07 16:48:52 UTC
(In reply to Chuck Tuffli from comment #3)
It's hard to share right now, but I'll share the source in the near future.

And if you provide me with a patch, I'll test them with my fuzzing tool and let you know.
Comment 5 Chuck Tuffli freebsd_committer 2021-06-13 23:50:32 UTC
I now have a stand-alone reproduction:

login: root
Jun 13 22:08:28 freebsd login[674]: ROOT LOGIN (root) ON ttyu0
root@freebsd:~ # uname -a
FreeBSD freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n247271-597cc550e7b: Thu Jun 10 06:50
:55 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64root@freebsd:~ # devctl detach pci0:0:4:1
nvd0: detached
nvme0: detached
root@freebsd:~ # ./bz256317
VID=0xfb5d DID=0xa0a
        [0] 0xc0004000 : 16384
        [1] 0 : 0
        [2] 0 : 0
        [3] 0 : 0
        [4] 0xc0008000 : 8192
        [5] 0 : 0
Mapped 0xc0004000 to 0x1000
writel(0, 0x14)
writel(0x180000f0, 0x34)
writel(0x460071, 0x14)
RDY 0->1 in 0 seconds
writel(0xAssertion failed: (cq->qbase != NULL), function pci_nvme_cq_update, file /build/903526542ac/src/usr.sbin/bhyve/pci_nvme.c, line 860.
Comment 6 Chuck Tuffli freebsd_committer 2021-06-13 23:55:58 UTC
Created attachment 225787 [details]
add checks to avoid assertion

Add checks for
 - bad AQ sizes
 - bad AQ vm_map_gpa
 - doorbell writes prior to RDY
 - doorbell writes to uninitialized queue
 - CSTS.RDY if CFS set
Comment 7 Chuck Tuffli freebsd_committer 2021-06-13 23:56:38 UTC
Please see if the attached patch fixes the issue you see.
Comment 8 Cheolwoo Myung 2021-06-14 06:27:48 UTC
(In reply to Chuck Tuffli from comment #7)
I checked your attached patch using reproducer, and it avoids assert failure.