Stable-9 rc2 Test Matrix (results)

Owned by Nicholas Tsirakis
Last updated: Jul 31, 2019 by Chris Rogers
5 min read
Build Info:
  • OpenXT Build 6662 (s9-rc2)
Hardware used:
  • Dell Optiplex 7040
  • BIOS: 1.14.0
Guest VM OSs under Tests:
  • Windows 10 1809
  • Windows 10 1803
  • RHEL 7.5
  • Ubuntu 18.04
  • Debian 9.9 (Stretch)

Test Matrix

Test DescriptionHost BootTPMOS VersionOS BootBuildResultBuildResultNotes/Test Data
Install OpenXT with MLE - UEFI Dell PlatformUEFI
N/AN/A6662PASS


Install OpenXT no measured launch - UEFI Dell PlatformUEFI
N/AN/A6662PASS


Install OpenXT with MLE - UEFI HP PlatformUEFI
N/AN/A6662



Install OpenXT no measured launch - UEFI HP PlatformUEFI
N/AN/A6662



Install Windows guest VM with Tools.UEFI2.0Windows 10  1803Legacy6662PASS


Install Windows guest VM with Tools.UEFI2.0Windows 10 1803UEFI6662FAIL

  • Won't fix: black screen after installing tools
  • Guest won't boot after installing tools
Install Windows guest VM with Tools.UEFI2.0Windows 10 1809Legacy6662PASS


Install Windows guest VM with Tools.UEFI2.0Windows 10 1809UEFI6662FAIL

  • Won't fix: black screen after installing tools
  • Guest won't boot after installing tools
Install RHEL guest VM.UEFI2.0RHEL 7.5Legacy6662PASS


Install RHEL guest VM.UEFI2.0RHEL 7.5UEFI6662FAIL

  • Won't fix: tools not working
Install Debian guest VM with Tools.UEFI2.0Debian 9.9Legacy6662PWE

  • Need to blacklist bochs_drm on boot
  • Note: after installing tools (but before rebooting), you will likely need to remove xenmou from /etc/modules-load.d/openxt.conf otherwise the guest will not boot.
Install Ubuntu guest VM with Tools.UEFI2.0Ubuntu 18.04Legacy6662PWE

  • If login screen fails to load on boot, need to boot in recovery mode, open /etc/gdm3/custom.conf, and uncomment "WaylandEnable=false"
  • Note: after installing tools (but before rebooting), you will likely need to remove xenmou from /etc/modules-load.d/openxt.conf otherwise the guest will not boot.
Configure guest VM in multiple configurations. Things tested: enabling/disabling audio, enabling/disabling networking, enabling/disabling stubdom, changing vCPU count, adding disk of arbitrary size, increasing RAM, changing icon, and USB PT.UEFI2.0Windows 10 1803Legacy6662PWE

  • Occasional: entire box freezes up when attempting to start a windows guest with audio and/or stubdom disabled. Can't for the life of me figure out the repro steps.
Configure guest VM in multiple configurations. Things tested: enabling/disabling audio, enabling/disabling networking, enabling/disabling stubdom, changing vCPU count, adding disk of arbitrary size, increasing RAM, changing icon, and USB PT.UEFI2.0Windows 10 1809Legacy6662PWE

  • Disabling audio causes display driver to not load - OXT-1657
Configure guest VM in multiple configurations.  Things tested: enabling/disabling audio, enabling/disabling networking, enabling/disabling stubdom, changing vCPU count, adding disk of arbitrary size, increasing RAM, changing icon, and USB PT.UEFI2.0Ubuntu 18.04Legacy6662

PWE

6674PASS

OXT-1577 fixed

Configure guest VM in multiple configurations. Things tested: enabling/disabling audio, enabling/disabling networking, enabling/disabling stubdom, changing vCPU count, adding disk of arbitrary size, increasing RAM, changing icon, and USB PT.UEFI2.0RHEL 7.5Legacy6662

PWE

6674PASS

OXT-1577 fixed

Configure guest VM in multiple configurations.  Things tested: enabling/disabling audio, enabling/disabling networking, enabling/disabling stubdom, changing vCPU count, adding disk of arbitrary size, increasing RAM, changing icon, and USB PT.UEFI2.0Debian 9.9Legacy6662PWE6674PASS

OXT-1577 fixed

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
NANAWindows 10 1803Legacy




GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
UEFI2.0Windows 10 1803UEFI




GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
NANAWindows 10 1809Legacy




GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
UEFI2.0Windows 10 1809UEFI




PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
NANAWindows 10 1803Legacy




PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
UEFI2.0Windows 10 1803UEFI




PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
NANAWindows 10 1809Legacy




PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
UEFI2.0Windows 10 1809UEFI




UIVM: Test all buttons for correct functionality that are displayed and sub options.

  • Settings
  • VM Details
  • Status
  • Services
  • Devices
  • Install VM
  • Power Button
  • etc
UEFINAWindows 10 1803LEGACY6662PASS

  • Info: PWE - Many issues with Status Report, no tickets submitted
  • Settings: PWE

Audio: Auto-mute Mode inconclusive

Audio: IEC958 settings inconclusive

Audio: Mic Mute LED Mode: inconclusive

Audio: Unable to disable ALL mic options simultaneously, one will also reset to "on."  No documentation to tell if feature or bug

Software Update: inconclusive

Network Configurations will be tested, making sure functions in the applet work as needed. Test DHCP and static setups.NANANANA6662PASS


Measured launch measures the correct PCRs for the Host system.  PCR 15,17,18,19, at minimum.Legacy1.2NANA




Measured launch measures the correct PCRs for the Host system.  PCR 4,5,6,7,8,15,17,18,19.UEFI2.0NANA6662PASS


Measured launch measures the correct PCRs for the Host system.  PCR 15,17,18,19, at minimum.Legacy2.0NANA6662PASS


Forward seal works.UEFI2.0NANA6662PASS


Forward seal works.Legacy1.2NANA




Forward seal works.Legacy2.0NANA6662PASS


Upgrade to self.Legacy1.2NANA




Upgrade to self.Legacy2.0NANA6662PASS


Upgrade to self.UEFI2.0NANA6662PASS


Performance testing of NDVMs will include:

  • pinging a gateway and other addresses to look at latency
  • large file (100+GB) downloads
  • wireshark analysis if needed
  • Add-on vs. integrated NIC
NANANANA
Unexecuted


Guest Stress testing will include running multiple programs for many hours:

  • 72 hours of HeavyLoad stressing memory and CPU, Prime95, and a video being run.
  • 120 hours of a guest running a video and refreshing a webpage.
NANANANA
Unexecuted


Host Stress test will include starting and stopping guest VMs for up to 120 hours.NANANANA6662Unexecuted


Install OpenXT with MLE - UEFI Dell PlatformUEFI2.0N/AN/A
pass


Install Windows guest VM with Tools.UEFI2.0Windows 10 1803Legacy6662Pass


Install Windows guest VM with Tools.UEFI2.0Windows 10 1809Legacy6662Pass


Install RHEL guest VM.UEFI2.0RHEL 7.5Legacy6662Pass

  • Need to blacklist bochs_drm on boot
Install Debian guest VM with Tools.UEFI2.0Debian 9.9Legacy6662Pass

  • Need to blacklist bochs_drm on boot
Install Ubuntu guest VM with Tools.UEFI2.0Ubuntu 18.04Legacy6662Pass

  • If login screen fails to load on boot, need to boot in recovery mode, open /etc/gdm3/custom.conf, and uncomment "WaylandEnable=false"

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
NANAWindows 10 1803Legacy6662Pass



Did not do performance tests

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
NANAWindows 10 1809Legacy6662Pass



Did not do performance tests

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
NANARHEL 7.5Legacy6662Pass



Did not do performance tests

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
NANADebian 9.9Legacy6662Pass



Did not do performance tests

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
NANAUbuntu 18.04Legacy6662Pass



Did not do performance tests

PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
NANAWindows 10 1803Legacy6662Pass




PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
NANAWindows 10 1809Legacy6662Pass




PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
NANARHEL 7.5Legacy6662Pass




PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
NANADebian 9.9Legacy6662Pass




PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
NANAUbuntu 18.04Legacy6662Pass


Install OpenXT with MLE - Legacy Dell PlatformLegacy1.2N/AN/A6675PASS


Install Windows guest VM with Tools.Legacy1.2Windows 10  1803Legacy6675PASS


Install Windows guest VM with Tools.Legacy1.2Windows 10 1809Legacy6675PASS


Install RHEL guest VM.Legacy1.2RHEL 7.5Legacy6675PASS


Install Debian guest VM with Tools.Legacy1.2Debian 9.9Legacy6675PWE

  • Need to blacklist bochs_drm on boot
  • Note: after installing tools (but before rebooting), you will likely need to remove xenmou from /etc/modules-load.d/openxt.conf otherwise the guest will not boot.
Install Ubuntu guest VM with Tools.Legacy1.2Ubuntu 18.04Legacy6675PWE

  • If login screen fails to load on boot, need to boot in recovery mode, open /etc/gdm3/custom.conf, and uncomment "WaylandEnable=false"
  • Note: after installing tools (but before rebooting), you will likely need to remove xenmou from /etc/modules-load.d/openxt.conf otherwise the guest will not boot.

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
Legacy1.2Windows 10 1803Legacy6675Pass

Did not run performance tests

Used a Quadro M2000

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
Legacy1.2Windows 10 1809Legacy6675Pass

Did not run performance tests

Used a Quadro M2000

Extra notes:

  • UIVM  is very sluggish. Mousing over guests is not nearly as smooth as it used to be.