Bug 234039 - bhyve: NetBSD/i386 8.0 fails to boot
Summary: bhyve: NetBSD/i386 8.0 fails to boot
Status: New
Alias: None
Product: Base System
Classification: Unclassified
Component: bin (show other bugs)
Version: 12.0-RELEASE
Hardware: amd64 Any
: --- Affects Some People
Assignee: freebsd-virtualization mailing list
Depends on:
Reported: 2018-12-15 19:15 UTC by Santhosh Raju
Modified: 2019-01-09 16:07 UTC (History)
1 user (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Santhosh Raju 2018-12-15 19:15:59 UTC
When attempting to boot in bhyve (with debugging on in bhyve) the following message appears in the bhyve.log and bhyve terminates.

I have tested this on FreeBSD 11.2-RELEASE and FreeBSD 12.0-RELEASE and they both give the same result. I have *not* tried this on FreeBSD 13-CURRENT.

vm exit[0]
        reason          VMX
        rip             0x00000000c091f843
        inst_length     7
        status          0
        exit_reason     2
        qualification   0x0000000000000000
        inst_type               0
        inst_error              0

Looking into the VMX reference manual an exit_reason 2 implies a Triple fault occurred in the logical processor.

Attempting to boot NetBSD/amd64 8.0 under the exact same conditions work without issues.

NetBSD/i386 8.0 is able to boot correctly in real hardware / VirtualBox / QEMU / XEN etc.

I do not know how to debug / fix this problem. 

But I would be more than happy to help investigate this further if someone can help / guide / handhold me into figuring out where exactly this problem lies.
Comment 1 Andriy Gapon freebsd_committer 2018-12-16 13:39:04 UTC
Can you examine what instruction is at address 0xc091f843 in the NetBSD kernel and what function contains that address?
Comment 2 Santhosh Raju 2018-12-16 20:08:46 UTC
I had a quick look at it by doing an objdump -d 

c091f82d:       e8 a5 32 fa ff          call   c08c2ad7 <sysctl_createv>
c091f832:       c7 44 24 38 ff ff ff    movl   $0xffffffff,0x38(%esp)
c091f839:       ff
c091f83a:       c7 44 24 34 fd ff ff    movl   $0xfffffffd,0x34(%esp)
c091f841:       ff
c091f842:       c7 44 24 30 00 00 00    movl   $0x0,0x30(%esp)
c091f849:       00
c091f84a:       c7 44 24 2c d8 56 19    movl   $0xc11956d8,0x2c(%esp)
c091f851:       c1
c091f852:       c7 44 24 24 00 00 00    movl   $0x0,0x24(%esp)
c091f859:       00

This part of the above code lies in the section starting with

c091f79d <vntblinit>:
c091f79d:       55                      push   %ebp
c091f79e:       89 e5                   mov    %esp,%ebp
c091f7a0:       56                      push   %esi
c091f7a1:       53                      push   %ebx
c091f7a2:       83 ec 48                sub    $0x48,%esp

So my guess is the function name is "vntblinit"

Let me know if you need more information from my side.
Comment 3 Andriy Gapon freebsd_committer 2018-12-16 21:54:29 UTC
Weird that 0xc091f843 appears not to be on an instruction boundary.
Comment 4 Santhosh Raju 2018-12-17 09:34:28 UTC
Apologies for the delay, but I finally managed to get a debug version of NetBSD/i386 kernel compiled with symbols and got better information to present now

(gdb) info line  *(0xc091f843)
Line 856 of "/home/fox/projects/netbsd/src-i386/sys/kern/vfs_subr.c" starts at address 0xc091f832 <vntblinit+149> and ends at 0xc091f8a9 <vntblinit+268>.
(gdb) p vfsinit
$1 = {void (void)} 0xc0918f73 <vfsinit>

Having a look at the location in the above mentioned file

        sysctl_createv(clog, 0, &rnode, &cnode,
                        CTLTYPE_QUAD, "delay",
                        SYSCTL_DESCR("max time to delay syncing data"),
                        NULL, 0, &syncdelay, 0,
                        CTL_CREATE, CTL_EOL);

That is what I come across and it is in the function sysctl_vfs_syncfs_setup().

Hopefully this is more helpful in your analysis.
Comment 5 Santhosh Raju 2018-12-17 15:20:52 UTC
I upgraded my FreeBSD 11.2-RELEASE machine to 12.0-RELEASE where I attempted to boot NetBSD/i386 (since I did not pay attention to the rip address the last time in 12.0-RELEASE boot)

Case 1: NetBSD/i386 ISO obtained from NetBSD ftp

vm exit[0]
        reason          VMX
        rip             0x00000000c0117e4f
        inst_length     3
        status          0
        exit_reason     2
        qualification   0x0000000000000000
        inst_type               0
        inst_error              0

Checking in gdb gave me the following location

(gdb) info line *(0xc0117e4f)
Line 506 of "/home/fox/projects/netbsd/src-i386/sys/arch/i386/i386/multiboot.c" starts at address 0xc0117e4f <multiboot_post_reloc+186> and ends at 0xc0117e52 <multiboot_post_reloc+189>.

Case 2: NetBSD/i386 ISO built from CVS (netbsd-8.0-release branch) with debug symbols enabled in kernel

vm exit[0]
        reason          VMX
        rip             0x00000000c011772f
        inst_length     4
        status          0
        exit_reason     2
        qualification   0x0000000000000000
        inst_type               0
        inst_error              0

Checking in gdb gave me the following location

(gdb) info line *(0xc011772f)
Line 409 of "/home/fox/projects/netbsd/src-i386/sys/arch/i386/i386/multiboot.c" starts at address 0xc011772f <setup_biosgeom+222> and ends at 0xc0117739 <setup_biosgeom+232>.

Since I do not have access to 11.2-RELEASE machine anymore. I cannot check to see if the rip instruction is the same as the original one I reported.
Comment 6 Santhosh Raju 2019-01-07 08:04:44 UTC
Has anyone looked into the above logs? 

I could provide more information if required.