Bug 230773

Summary:	[bhyve] GDT limit needs reset on VMX exit
Product:	Base System	Reporter:	John Levon <levon>
Component:	kern	Assignee:	freebsd-virtualization (Nobody) <virtualization>
Status:	Closed FIXED
Severity:	Affects Only Me	CC:	araujo, emaste, jhb, markj, rgrimes, tychon
Priority:	---	Flags:	jhb: mfc-stable11+ jhb: mfc-stable10-
Version:	CURRENT
Hardware:	amd64
OS:	Any

Description John Levon 2018-08-20 12:46:47 UTC

Intel vol 3 27.5.2:

"The base addresses for GDTR and IDTR are loaded from the GDTR base-address field and the IDTR base-address field, respectively. If the processor supports the Intel 64 architecture and the processor supports N < 64 linear-address bits, each of bits 63:N of each base address is set to the value of bit N–1 of that base address. The GDTR and IDTR limits are each set to FFFFH."

As far as I could tell, bhyve doesn't do this in current FreeBSD, leaving the GDT limit significantly larger than the previous NGDT-based one.

We found and fixed this in our port as seen here:

https://github.com/joyent/illumos-joyent/commit/1d0338f3f33eec2ed45ec5a6cae30c399a3ef769#diff-5b270bdc09d0e88fd39cfed0ccd0c44b

Comment 1 Marcelo Araujo freebsd_committer

2018-08-21 21:48:11 UTC

Could you please make a patch against FreeBSD HEAD and attach it here?

Best,

Comment 2 John Levon 2018-08-22 09:02:04 UTC

Hi, just to clarify, the patch I referred to is just for reference.

I mainly filed this bug upstream as the implications are pretty unpleasant, hopefully one of you can handle the fix.

Comment 3 commit-hook freebsd_committer

2018-10-11 18:27:46 UTC

A commit references this bug:

Author: jhb
Date: Thu Oct 11 18:27:20 UTC 2018
New revision: 339312
URL: https://svnweb.freebsd.org/changeset/base/339312

Log:
  Fully restore the GDTR, IDTR, and LDTR after VT-x VM exits.

  The VT-x VMCS only stores the base address of the GDTR and IDTR.  As a
  result, VM exits use a fixed limit of 0xffff for the host GDTR and
  IDTR losing the smaller limits set in when the initial GDT is loaded
  on each CPU during boot.  Explicitly save and restore the full GDTR
  and IDTR contents around VM entries and exits to restore the correct
  limit.

  Similarly, explicitly save and restore the LDT selector.  VM exits
  always clear the host LDTR as if the LDT was loaded with a NULL
  selector and a userspace hypervisor is probably using a NULL selector
  anyway, but save and restore the LDT explicitly just to be safe.

  PR:		230773
  Reported by:	John Levon <levon@movementarian.org>
  Reviewed by:	kib
  Tested by:	araujo
  Approved by:	re (rgrimes)
  MFC after:	1 week

Changes:
  head/sys/amd64/include/cpufunc.h
  head/sys/amd64/vmm/intel/vmx.c

Comment 4 Ed Maste freebsd_committer

2018-11-12 17:15:59 UTC

Still waiting on merge to 12?

Comment 5 John Baldwin freebsd_committer

2018-11-12 17:19:13 UTC

No, it was committed to head before the 12 branch.  But I do think that means it is no longer a 12.0 blocker.

Comment 6 commit-hook freebsd_committer

2018-11-18 01:08:11 UTC

A commit references this bug:

Author: jhb
Date: Sun Nov 18 01:07:37 UTC 2018
New revision: 340545
URL: https://svnweb.freebsd.org/changeset/base/340545

Log:
  MFC 339312,339364: Restore more descriptors during VM exits.

  339312:
  Fully restore the GDTR, IDTR, and LDTR after VT-x VM exits.

  The VT-x VMCS only stores the base address of the GDTR and IDTR.  As a
  result, VM exits use a fixed limit of 0xffff for the host GDTR and
  IDTR losing the smaller limits set in when the initial GDT is loaded
  on each CPU during boot.  Explicitly save and restore the full GDTR
  and IDTR contents around VM entries and exits to restore the correct
  limit.

  Similarly, explicitly save and restore the LDT selector.  VM exits
  always clear the host LDTR as if the LDT was loaded with a NULL
  selector and a userspace hypervisor is probably using a NULL selector
  anyway, but save and restore the LDT explicitly just to be safe.

  339364:
  Reload the LDT selector after an AMD-v #VMEXIT.

  cpu_switch() always reloads the LDT, so this can only affect the
  hypervisor process itself.  Fix this by explicitly reloading the host
  LDT selector after each #VMEXIT.  The stock bhyve process on FreeBSD
  never uses a custom LDT, so this change is cosmetic.

  PR:		230773

Changes:
_U  stable/11/
  stable/11/sys/amd64/include/cpufunc.h
  stable/11/sys/amd64/vmm/amd/svm.c
  stable/11/sys/amd64/vmm/intel/vmx.c