Bug 203643

Summary:

[bhyve] NetBSD causes bhyve to exit on AMD processors

Product:

Base System

Reporter:

Jimmy Olgeni <olgeni>

Component:

kern

Assignee:

freebsd-virtualization (Nobody) <virtualization>

Status:

New ---

Severity:

Affects Only Me

CC:

grehan, jhyphenkeck, kjift14

Priority:

---

Version:

10.2-STABLE

Hardware:

amd64

OS:

Any

Attachments:

Description	Flags
bhyve exception log	none

Description Jimmy Olgeni freebsd_committer

2015-10-08 18:19:09 UTC

Created attachment 161834 [details]
bhyve exception log

NetBSD 7.0 causes bhyve to exit with an exception, but only on AMD processors. Intel works fine.

uname -a:

FreeBSD olgeni 10.2-STABLE FreeBSD 10.2-STABLE #5 r288925: Tue Oct  6 20:48:48 CEST 2015     root@olgeni:/usr/obj/usr/src/sys/KERNEL  amd64

Processor:

CPU: AMD Phenom(tm) II X4 965 Processor (3411.31-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x100f43  Family=0x10  Model=0x4  Stepping=3
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  SVM: NP,NRIP,NAsids=64
  TSC: P-state invariant

How to repeat - download NetBSD-7.0-amd64.iso, then try this:

======================================================================
truncate -s8G netbsd.img

cat > netbsd.map << EOF
(hd0) netbsd.img
(cd0) NetBSD-7.0-amd64.iso
EOF

cat > netbsd.grub << EOF
knetbsd -h -r cd0a /netbsd
boot
EOF

sudo grub-bhyve -r cd0 -m netbsd.map -M 1024 netbsd < netbsd.grub

sudo bhyve -c 1 -m 1024M -H -P -A -W -w \
    -l com1,stdio \
    -s 0:0,hostbridge \
    -s 1:0,lpc \
    -s 2:0,virtio-net,tap0 \
    -s 3,ahci-cd,NetBSD-7.0-amd64.iso \
    -s 4,virtio-blk,netbsd.img netbsd
======================================================================

You'll get something similar to this:

root on cd0a dumps on cd0b
root file system type: cd9660
warning: no /dev/console
vm exit[0]
        reason          SVM
        rip             0xffffffff80229f5f
        inst_length     2
        exitcode        0x7b
        exitinfo1       0x3f8021c
        exitinfo2       0xffffffff80229f61

Comment 1 Mark Linimon freebsd_committer

2015-10-20 23:49:38 UTC

Canonicalize assignment.

Comment 2 Jimmy Olgeni freebsd_committer

2016-09-12 08:22:49 UTC

Tested again. 11.0-RC2 with NetBSD 7.0.1:

cd0 at atapibus0 drive 0: <BHYVE SATA DVD ROM, BHYVE-445C-CB8B-8A6, 001> cdrom removable
pad0: outputs: 44100Hz, 16-bit, stereo
audio0 at pad0: half duplex, playback, capture
boot device: cd0
cd0(ahcisata0:0:0):  Check Condition on CDB: 0x46 00 00 00 00 00 00 00 08 00
    SENSE KEY:  Illegal Request
     ASC/ASCQ:  Invalid Command Operation Code

root on cd0a dumps on cd0b
root file system type: cd9660
warning: no /dev/console
Abort trap

From the vm-bhyve log:

Sep 12 10:20:16: initialising
Sep 12 10:20:16:  [loader: grub]
Sep 12 10:20:16:  [uefi: no]
Sep 12 10:20:16:  [cpu: 1]
Sep 12 10:20:16:  [memory: 512M]
Sep 12 10:20:16:  [hostbridge: standard]
Sep 12 10:20:16:  [com ports: com1]
Sep 12 10:20:16:  [uuid: 9a8e5695-78c1-11e6-a88c-bcaec55be5e5]
Sep 12 10:20:16:  [utctime: no]
Sep 12 10:20:16:  [debug mode: no]
Sep 12 10:20:16:  [primary disk: disk0.img]
Sep 12 10:20:16:  [primary disk dev: file]
Sep 12 10:20:16: generated static mac 58:9c:fc:0b:b5:ee (based on 'netbsd:0:1473668416:0')
Sep 12 10:20:16: initialising network device tap0
Sep 12 10:20:16: adding tap0 -> bridge1 (public)
Sep 12 10:20:16: booting
Sep 12 10:20:16: create file /storage/vm-bhyve/netbsd/device.map
Sep 12 10:20:16:  -> (hd0) /storage/vm-bhyve/netbsd/disk0.img
Sep 12 10:20:16:  -> (cd0) /storage/vm-bhyve/.iso/NetBSD-7.0.1-amd64.iso
Sep 12 10:20:16: create file /storage/vm-bhyve/netbsd/grub.cfg
Sep 12 10:20:16:  -> timeout=3
Sep 12 10:20:16:  -> menuentry 'netbsd (bhyve install)' {
Sep 12 10:20:16:  ->     root=cd0
Sep 12 10:20:16:  ->     knetbsd -h -r cd0a /netbsd
Sep 12 10:20:16:  -> }
Sep 12 10:20:16: /usr/local/sbin/grub-bhyve -m /storage/vm-bhyve/netbsd/device.map -M 512M -r host -d /storage/vm-bhyve/netbsd netbsd
Sep 12 10:20:20:  [bhyve options: -c 1 -m 512M -AHP -U 9a8e5695-78c1-11e6-a88c-bcaec55be5e5]
Sep 12 10:20:20:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/storage/vm-bhyve/netbsd/disk0.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:0b:b5:ee]
Sep 12 10:20:20:  [bhyve console: -l com1,stdio]
Sep 12 10:20:20:  [bhyve iso device: -s 3:0,ahci-cd,/storage/vm-bhyve/.iso/NetBSD-7.0.1-amd64.iso]
Sep 12 10:20:20: starting bhyve (run 1)
Sep 12 10:20:27: bhyve exited with status 134
Sep 12 10:20:27: destroying network device tap0
Sep 12 10:20:27: stopped

Comment 3 HrstBW 2017-04-08 22:04:45 UTC

I can confirm this with identical symptoms, on different AMD hardware, with more recent versions:

uname -a
FreeBSD bhyvehost 11.0-RELEASE-p8 FreeBSD 11.0-RELEASE-p8 #0: Wed Feb 22 06:12:04 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64


dmesg
[...]
CPU: AMD E-350 Processor (1596.63-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x500f10  Family=0x14  Model=0x1  Stepping=0
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802209<SSE3,MON,SSSE3,CX16,POPCNT>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x35ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,IBS,SKINIT,WDT>
[...]


Linux installations and live CDs are working fine.

Steps:

zfs create -s -V 5G bhyveguests/netbsd

==> install.map <==
(hd0) /dev/zvol/bhyveguests/netbsd
(cd0) /scratch/NetBSD-7.1/amd64/boot-com.iso

==> install.sh <==
grub-bhyve -m install.map -r cd0 -M 512 netbsdvm << EOF
knetbsd -h -r cd0a (cd0)/netbsd
boot
EOF
bhyve -c 1 -m 512M -H -P -A \
    -l com1,stdio \
    -s 0:0,hostbridge \
    -s 1:0,lpc -s 2:0,virtio-net,tap0 \
    -s 3,ahci-cd,/scratch/NetBSD-7.1/amd64/boot-com.iso \
    -s 4,virtio-blk,/dev/zvol/bhyveguests/netbsd \
    netbsdvm

./install.sh dies with:


[...]
cd0 at atapibus0 drive 0: <BHYVE SATA DVD ROM, BHYVE-5932-F7DC-D40, 001> cdrom removable
pad0: outputs: 44100Hz, 16-bit, stereo
audio0 at pad0: half duplex, playback, capture
boot device: cd0
cd0(ahcisata0:0:0):  Check Condition on CDB: 0x46 00 00 00 00 00 00 00 08 00
    SENSE KEY:  Illegal Request
     ASC/ASCQ:  Invalid Command Operation Code

root on cd0a dumps on cd0b
root file system type: cd9660
warning: no /dev/console
vm exit[0]
                reason          SVM
                                        rip             0xffffffff8022a72f
                                                                                inst_length     2
                                                                                                        exitcode        0x7b
                                                                                                                                exitinfo1       0x3f8021c  
        exitinfo2       0xffffffff8022a731

Comment 4 Peter Grehan freebsd_committer

2017-04-23 19:09:35 UTC

I was able to get an install/boot working fine on a r317273 CURRENT Ryzen system with 1 and 4 vCPUs using the NetBSD 7.1 boot-com ISO.