Bug 255191 - Booting DragonFlyBSD results in Failed to emulate instruction sequence output
Summary: Booting DragonFlyBSD results in Failed to emulate instruction sequence output
Status: Closed FIXED
Alias: None
Product: Base System
Classification: Unclassified
Component: bhyve (show other bugs)
Version: 13.0-STABLE
Hardware: amd64 Any
: --- Affects Only Me
Assignee: freebsd-virtualization (Nobody)
Depends on:
Reported: 2021-04-18 17:57 UTC by Pierre-Alain TORET
Modified: 2021-04-27 22:10 UTC (History)
3 users (show)

See Also:

dfly mptable diff (510 bytes, patch)
2021-04-27 03:40 UTC, Peter Grehan
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Pierre-Alain TORET 2021-04-18 17:57:20 UTC

I haveve installed FreeBSD 13.0 and I am trying to virtualize DragonFlyBSD 5.8.3 with the following command line :

sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 512M -d /path/to/dfly.img  -i -I /path/to/dfly-x86_64-5.8.3_REL.iso dfly

But this results in the following output :

Launching virtual machine "dfly" ...
Consoles: userboot

FreeBSD/amd64 User boot lua, Revision 1.2
Consoles: userboot

FreeBSD/amd64 User boot 4th, Revision 1.2
/boot/kernel/kernel text=0xb95180 data=0xf68000 syms=[0x8+0xfcfc0+0x8+0xbc50c]
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...
Copyright (c) 2003-2020 The DragonFly Project.
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
Unhandled memory access to 0x100000000

Failed to emulate instruction sequence [ 813c8f5f4d505f742483c20439d37f ] at 0xffffffff80bebe43
Abort trap

From the DragonFlyBSD people, I got that particular sequence as :
ffffffff80bebe40:       48 63 ca                movslq %edx,%rcx
ffffffff80bebe43:       81 3c 8f 5f 4d 50 5f    cmpl   $0x5f504d5f,(%rdi,%rcx,4)
ffffffff80bebe4a:       74 24                   je     ffffffff80bebe70 <mptable_search_sig+0x60>
/home/justin/release/5_8/sys/platform/pc64/x86_64/mptable.c:376 (discriminator 2)
ffffffff80bebe4c:       83 c2 04                add    $0x4,%edx
ffffffff80bebe4f:       39 d3                   cmp    %edx,%ebx
ffffffff80bebe51:       7f ed                   jg     ffffffff80bebe40 <mptable_search_sig+0x30>

Those lines are from https://gitweb.dragonflybsd.org/dragonfly.git/blob/refs/heads/DragonFly_RELEASE_5_8:/sys/platform/pc64/x86_64/mptable.c#l376
Comment 1 Aleksandr Fedorov freebsd_committer 2021-04-21 09:09:32 UTC

It's very strange that you are trying to start DragonFlyBSD using the bootloader from FreeBSD.

Yes, vmrun.sh, uses bhyveload(8) by default:

>FreeBSD/amd64 User boot lua, Revision 1.2
>Consoles: userboot

You should use -E option to choose UEFI bootloader.

I was able to successfully run it using the UEFI loader. But only if memory >= 4GB:

bhyve -H -A -P -c 1 -m 4G -s 0,hostbridge -s 1,virtio-blk,./dfly.img -s 2,ahci-cd,./dfly-x86_64-5.8.3_REL.iso -s 31,lpc -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd dfly

If you give the memory less than 4 gigabytes, then a similar error was occurs:

Unhandled memory access to 0x100000000
Failed to emulate instruction sequence [ 813c8f5f4d505f742483c20439d37f ] at 0xffffffff80bebe43
Comment 2 Pierre-Alain TORET 2021-04-21 11:40:33 UTC
(In reply to Aleksandr Fedorov from comment #1)


thank you, I noticed too it was using the FreeBSD loader, but using the UEFI bootloader I got the same message. So it seems I copy/pasted the wrong extract (I wanted to copy the UEFI one), sorry about that.

I also tried on qemu/kvm on a Linux host I have because I don't own a machine with less than 4G and I can boot the installer without issues using only 1G or RAM for example. I've asked the DragonFlyBSD project on the meantime, and 4G of RAM is not required they say.

Using this virtual machine with 4G is acceptable for me, but I guess there's still something weird going on.
Comment 3 Peter Grehan freebsd_committer 2021-04-21 12:27:04 UTC
I looked into this a while back: there are 2 issues
 - DFly is accessing memory beyond 4G when there is < 4G configured. This appears to be an array access in sys/platform/pc64/x86_64/mptable.c:mptable_search_sig()
   However, in PC-land this is allowed and accesses beyond the end of RAM will return 0xff's on a read.

 - bhyve does return 0xff's for reads beyond the end of memory, but doesn't implement that particular instruction (cmpl   $0x5f504d5f,(%rax,%rcx,4))

Rather than having to implement that instruction for such an edge condition, I'll try and resurrect the patch I once had (unfortunately lost) and get the fix into DFly.
Comment 4 Marek Zarychta 2021-04-22 14:55:32 UTC
(In reply to Peter Grehan from comment #3)
That's good news, though it probably won't be fixed before 6.0 RELEASE ?
Comment 5 Peter Grehan freebsd_committer 2021-04-27 03:40:36 UTC
Created attachment 224465 [details]
dfly mptable diff

This should fix the issue in DFly.

The units of base_memory are KB, so this has to be adjusted before used as an absolute value, or this will result in a negative value just under 4G and the array access will then extend beyond 4G.

I suspect this will not be an issue systems other than bhyve since they would have a BIOS extended-data area and not execute this codepath, or even if they didn't, accessing memory beyond 4G would be a no-op.

In any case, it is an out-of-bounds array access, so worth fixing.
Comment 6 Marek Zarychta 2021-04-27 11:29:42 UTC
(In reply to Peter Grehan from comment #5)
Thank you for the patch. It solves the issue, I have tested it on DragonFly v6.0.0rc1-RELEASE.
Comment 7 Pierre-Alain TORET 2021-04-27 15:40:03 UTC
(In reply to Marek Zarychta from comment #6)


I can confirm that it's working for me too after applying the patch on the latest release candidate.
Comment 8 Pierre-Alain TORET 2021-04-27 19:59:16 UTC

The fix has been pushed on DragonFlyBSD side https://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/70c69c34060140e40d6eff7ee0256cad8150bef3

So thank you, I think that bug can now be closed.
Comment 9 Peter Grehan freebsd_committer 2021-04-27 22:10:49 UTC
Thanks for upstreaming the fix.