We randomly get a kernel panic on our freebsd 12.4 VMs: Apr 10 07:28:17.012 myvm kernel: assertion "cp->host_status == HS_COMPLETE" failed: file "/usr/src/sys/dev/sym/sym_hipd.c", line 7262 Apr 10 07:28:17.012 myvm kernel: assertion "cp->host_status == HS_COMPLETE" failed: file "/usr/src/sys/dev/sym/sym_hipd.c", line 7262 Apr 10 07:28:17.012 myvm kernel: = 6 Apr 10 07:28:17.012 myvm kernel: = 6 Apr 10 07:28:17.012 myvm kernel: = 1681104239 Apr 10 07:28:17.012 myvm kernel: = 1681104239 Apr 10 07:28:17.012 myvm kernel: stack backtrace: Apr 10 07:28:17.012 myvm kernel: stack backtrace: Apr 10 07:28:17.012 myvm kernel: at db_trace_self_wrapper+0x2b/frame 0xfffffe009822c2a0 Apr 10 07:28:17.012 myvm kernel: at db_trace_self_wrapper+0x2b/frame 0xfffffe009822c2a0 Apr 10 07:28:17.012 myvm kernel: at vpanic+0x178/frame 0xfffffe009822c2f0 Apr 10 07:28:17.012 myvm kernel: at vpanic+0x178/frame 0xfffffe009822c2f0 Apr 10 07:28:17.012 myvm kernel: at panic+0x43/frame 0xfffffe009822c350 Apr 10 07:28:17.012 myvm kernel: at panic+0x43/frame 0xfffffe009822c350 Apr 10 07:28:17.012 myvm kernel: at sym_intr1+0xc46/frame 0xfffffe009822c3b0 Apr 10 07:28:17.012 myvm kernel: at sym_intr1+0xc46/frame 0xfffffe009822c3b0 Apr 10 07:28:17.012 myvm kernel: at sym_intr+0x33/frame 0xfffffe009822c3d0 Apr 10 07:28:17.012 myvm kernel: at sym_intr+0x33/frame 0xfffffe009822c3d0 Apr 10 07:28:17.012 myvm kernel: at ithread_loop+0x1d0/frame 0xfffffe009822c430 Apr 10 07:28:17.012 myvm kernel: at ithread_loop+0x1d0/frame 0xfffffe009822c430 Apr 10 07:28:17.018 myvm kernel: at fork_exit+0x83/frame 0xfffffe009822c470 Apr 10 07:28:17.018 myvm kernel: at fork_exit+0x83/frame 0xfffffe009822c470 Apr 10 07:28:17.018 myvm kernel: at fork_trampoline+0xe/frame 0xfffffe009822c470 Apr 10 07:28:17.018 myvm kernel: at fork_trampoline+0xe/frame 0xfffffe009822c470 Apr 10 07:28:17.018 myvm kernel: trap 0, rip = 0, rsp = 0, rbp = 0 --- Apr 10 07:28:17.018 myvm kernel: trap 0, rip = 0, rsp = 0, rbp = 0 --- Apr 10 07:28:17.018 myvm kernel: enter: panic Apr 10 07:28:17.018 myvm kernel: enter: panic Unfortunately I'm not able to trigger the panic.
Are there any further information required? Just let me know if I can provide any information, or do some tests.
we just hit that kernel panic again on a freebsd 13.5 vm: Sep 11 12:12:08.775 f1199 kernel: assertion "cp->host_status == HS_COMPLETE" failed: file "/usr/src/sys/dev/sym/sym_hipd.c", line 7245 Sep 11 12:12:08.775 f1199 kernel: assertion "cp->host_status == HS_COMPLETE" failed: file "/usr/src/sys/dev/sym/sym_hipd.c", line 7245 Sep 11 12:12:08.775 f1199 kernel: = 6 Sep 11 12:12:08.775 f1199 kernel: = 6 Sep 11 12:12:08.775 f1199 kernel: = 1757585359 Sep 11 12:12:08.775 f1199 kernel: = 1757585359 Sep 11 12:12:08.775 f1199 kernel: stack backtrace: Sep 11 12:12:08.775 f1199 kernel: stack backtrace: Sep 11 12:12:08.775 f1199 kernel: at db_trace_self_wrapper+0x2b/frame 0xfffffe00e0c5ed30 Sep 11 12:12:08.775 f1199 kernel: at db_trace_self_wrapper+0x2b/frame 0xfffffe00e0c5ed30 Sep 11 12:12:08.775 f1199 kernel: at vpanic+0x182/frame 0xfffffe00e0c5ed80 Sep 11 12:12:08.775 f1199 kernel: at vpanic+0x182/frame 0xfffffe00e0c5ed80 Sep 11 12:12:08.775 f1199 kernel: at panic+0x43/frame 0xfffffe00e0c5ede0 Sep 11 12:12:08.775 f1199 kernel: at panic+0x43/frame 0xfffffe00e0c5ede0 Sep 11 12:12:08.775 f1199 kernel: at sym_intr1+0xc46/frame 0xfffffe00e0c5ee40 Sep 11 12:12:08.775 f1199 kernel: at sym_intr1+0xc46/frame 0xfffffe00e0c5ee40 Sep 11 12:12:08.775 f1199 kernel: at sym_intr+0x2e/frame 0xfffffe00e0c5ee60 Sep 11 12:12:08.775 f1199 kernel: at sym_intr+0x2e/frame 0xfffffe00e0c5ee60 Sep 11 12:12:08.775 f1199 kernel: at ithread_loop+0x25c/frame 0xfffffe00e0c5eef0 Sep 11 12:12:08.775 f1199 kernel: at ithread_loop+0x25c/frame 0xfffffe00e0c5eef0 Sep 11 12:12:08.776 f1199 kernel: at fork_exit+0x84/frame 0xfffffe00e0c5ef30 Sep 11 12:12:08.776 f1199 kernel: at fork_exit+0x84/frame 0xfffffe00e0c5ef30 Sep 11 12:12:08.776 f1199 kernel: at fork_trampoline+0xe/frame 0xfffffe00e0c5ef30 Sep 11 12:12:08.776 f1199 kernel: at fork_trampoline+0xe/frame 0xfffffe00e0c5ef30 Sep 11 12:12:08.776 f1199 kernel: trap 0, rip = 0, rsp = 0, rbp = 0 --- Sep 11 12:12:08.776 f1199 kernel: trap 0, rip = 0, rsp = 0, rbp = 0 --- Sep 11 12:12:08.776 f1199 kernel: enter: panic Sep 11 12:12:08.776 f1199 kernel: enter: panic
We're also facing the same problem with FBSD 13.5. Any help would be appreciated!
This looks like a sym(4) driver issue. Which is kind of weird because that's ancient hardware, but I guess it's being emulated in a VM? Marius made the most recent nontrivial changes to this driver; maybe he'll have some ideas.
It would also be useful to know what hypervisor host (incl. version) is being used here. It could be a bug in the hypervisor's device model, or that some difference in timing or implementation of the device model triggers a longstanding bug in sym(4).
our hypervisor hosts are running on openstack main component versions: operating system: ubuntu 22.04 libvirt: 8.0.0-1ubuntu7.10 qemu: 1:6.2+dfsg-2ubuntu6.25 openstack nova: 3:29.2.0-0ubuntu1~cloud0 (caracal) maybe interesting is that the libvirt xml has a scsi controller attached with the model set to lsilogic: <controller type='scsi' index='0' model='lsilogic'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> i can provide a full libvirt xml (anonymised) if needed.
Created attachment 266994 [details] sym(4): Employ memory barriers also on x86
Interesting that sym(4) still has use. The information provided is very limited and the setup non-trivial to reproduce. However, this smells like memory corruption. Especially, since sym_complete_ok() is preceded by a - thought to be - memory barrier, but which actually is a NOP on x86 so far with the assumption of its unnecessity not holding in the MP case. This might be even more vulnerable in a virtualized environment. You might want to give the patch in attachment 266994 [details] a try.
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=e769bc77184312b6137a9b180c97b87c0760b849 commit e769bc77184312b6137a9b180c97b87c0760b849 Author: Marius Strobl <marius@FreeBSD.org> AuthorDate: 2026-01-26 13:58:57 +0000 Commit: Marius Strobl <marius@FreeBSD.org> CommitDate: 2026-01-26 15:54:48 +0000 sym(4): Employ memory barriers also on x86 In an MP world, it doesn't hold that x86 requires no memory barriers. This change should also fix panics due to out-of-sync data seen with FreeBSD VMs on top of OpenStack and HBAs of type lsiLogic. [1] While at it: - Improve the granularity somewhat by distinguishing between read and write memory barriers as well as refer to existing *mb(9) functions instead of duplicating these [2], unless IO barriers are also used. - Nuke the unused SYM_DRIVER_NAME macro. PR: 270816 [1] Obtained from: BSD-licensed Linux sym53c8xx driver [2] MFC after: 1 week sys/dev/sym/sym_hipd.c | 42 +++++++++++++++--------------------------- 1 file changed, 15 insertions(+), 27 deletions(-)
A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=b941d1c64e58708c93621cc07ed1c8e5e709cd48 commit b941d1c64e58708c93621cc07ed1c8e5e709cd48 Author: Marius Strobl <marius@FreeBSD.org> AuthorDate: 2026-02-02 21:30:42 +0000 Commit: Marius Strobl <marius@FreeBSD.org> CommitDate: 2026-02-02 21:57:56 +0000 sym(4): Map HCB memory as uncacheable also on x86 As part of making the chip-specific mix and match of different accesses (DMA/bus space) work as desired, the intent is to map the HCB memory as uncacheable. Prior to VM_MEMATTR_*, the !x86 way of indicating this to bus_dmamem_alloc(9) was BUS_DMA_COHERENT. Then later on in 2db99100a4, BUS_DMA_NOCACHE was hooked up to VM_MEMATTR_UNCACHEABLE for x86. As it turns out, still as of today bus_dmamem_alloc(9) differs in this regard across architectures. On arm, it still supports BUS_DMA_COHERENT only for requesting uncacheable DMA and x86 still uses BUS_DMA_NOCACHE only. On arm64 and riscv, BUS_DMA_COHERENT seems to effectively be an alias for BUS_DMA_NOCACHE. Thus, allocate the HCB memory with BUS_DMA_COHERENT | BUS_DMA_NOCACHE, so we get uncacheable memory on all architectures including x86 and so loads and stores from/to HCB won't get reordered. However, even on x86 we still need to use at least compiler barriers to achieve the desired program order. This change should also fix panics due to out-of-sync data seen with FreeBSD VMs on top of OpenStack and HBAs of type lsiLogic as a result of loads and stores getting reordered. [1] While at it: - Nuke the unused SYM_DRIVER_NAME macro. - Remove unused/redundant HCB members and correct a comment typo. PR: 270816 [1] MFC after: 3 days sys/dev/sym/sym_hipd.c | 46 +++++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 27 deletions(-)
A commit in branch stable/15 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=3445fd3d2367624a44ba0c1a9a603cef26fa6199 commit 3445fd3d2367624a44ba0c1a9a603cef26fa6199 Author: Marius Strobl <marius@FreeBSD.org> AuthorDate: 2026-02-02 21:30:42 +0000 Commit: Marius Strobl <marius@FreeBSD.org> CommitDate: 2026-02-05 21:52:33 +0000 sym(4): Map HCB memory as uncacheable also on x86 As part of making the chip-specific mix and match of different accesses (DMA/bus space) work as desired, the intent is to map the HCB memory as uncacheable. Prior to VM_MEMATTR_*, the !x86 way of indicating this to bus_dmamem_alloc(9) was BUS_DMA_COHERENT. Then later on in 2db99100a4, BUS_DMA_NOCACHE was hooked up to VM_MEMATTR_UNCACHEABLE for x86. As it turns out, still as of today bus_dmamem_alloc(9) differs in this regard across architectures. On arm, it still supports BUS_DMA_COHERENT only for requesting uncacheable DMA and x86 still uses BUS_DMA_NOCACHE only. On arm64 and riscv, BUS_DMA_COHERENT seems to effectively be an alias for BUS_DMA_NOCACHE. Thus, allocate the HCB memory with BUS_DMA_COHERENT | BUS_DMA_NOCACHE, so we get uncacheable memory on all architectures including x86 and so loads and stores from/to HCB won't get reordered. However, even on x86 we still need to use at least compiler barriers to achieve the desired program order. This change should also fix panics due to out-of-sync data seen with FreeBSD VMs on top of OpenStack and HBAs of type lsiLogic as a result of loads and stores getting reordered. [1] While at it: - Nuke the unused SYM_DRIVER_NAME macro. - Remove unused/redundant HCB members and correct a comment typo. PR: 270816 [1] MFC after: 3 days (cherry picked from commit b941d1c64e58708c93621cc07ed1c8e5e709cd48) sys/dev/sym/sym_hipd.c | 46 +++++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 27 deletions(-)
A commit in branch stable/14 references this bug: URL: https://cgit.FreeBSD.org/src/commit/?id=79da80d56a427c4c6fe7e35f2d866b84db979a25 commit 79da80d56a427c4c6fe7e35f2d866b84db979a25 Author: Marius Strobl <marius@FreeBSD.org> AuthorDate: 2026-02-02 21:30:42 +0000 Commit: Marius Strobl <marius@FreeBSD.org> CommitDate: 2026-02-05 21:54:07 +0000 sym(4): Map HCB memory as uncacheable also on x86 As part of making the chip-specific mix and match of different accesses (DMA/bus space) work as desired, the intent is to map the HCB memory as uncacheable. Prior to VM_MEMATTR_*, the !x86 way of indicating this to bus_dmamem_alloc(9) was BUS_DMA_COHERENT. Then later on in 2db99100a4, BUS_DMA_NOCACHE was hooked up to VM_MEMATTR_UNCACHEABLE for x86. As it turns out, still as of today bus_dmamem_alloc(9) differs in this regard across architectures. On arm, it still supports BUS_DMA_COHERENT only for requesting uncacheable DMA and x86 still uses BUS_DMA_NOCACHE only. On arm64 and riscv, BUS_DMA_COHERENT seems to effectively be an alias for BUS_DMA_NOCACHE. Thus, allocate the HCB memory with BUS_DMA_COHERENT | BUS_DMA_NOCACHE, so we get uncacheable memory on all architectures including x86 and so loads and stores from/to HCB won't get reordered. However, even on x86 we still need to use at least compiler barriers to achieve the desired program order. This change should also fix panics due to out-of-sync data seen with FreeBSD VMs on top of OpenStack and HBAs of type lsiLogic as a result of loads and stores getting reordered. [1] While at it: - Nuke the unused SYM_DRIVER_NAME macro. - Remove unused/redundant HCB members and correct a comment typo. PR: 270816 [1] MFC after: 3 days (cherry picked from commit b941d1c64e58708c93621cc07ed1c8e5e709cd48) sys/dev/sym/sym_hipd.c | 46 +++++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 27 deletions(-)