Stable-9 rc0 Test Matrix

Goal of testing:

Determine and ensure that the functionality of OpenXT has not been compromised by the staggering changes that the product has been subjected to.  Validation of the entire product needs to be determined through the testing of these components.

Regressions will be determined by evaluating the system from previous baselines of stable OpenXT and downstream derivative products.  Every major part of OpenXT has been changed, including:

Changes (growing list):

  • Xen has been up-reved to 4.11 with the plan to go to 4.12 to get Argo support (replacing v4v)
  • Qemu has been up-reved
  • dom0 is a 64 bit OS
  • All Haskel and other programs are 64bit - rpc-proxy, xenmgr, updatemgr, hvmloader, seal-system, tpm tools (need to verify everything working), etc
  • NDVMs are now HVM guests, not PV
  • Addition of OVMF for UEFI guests

Testing areas should include:

  • General guest VM installation in both boot modes
  • General guest VM configurations in both boot modes
  • Installations with UEFI and Legacy guests
  • GPU PT with each guest in both boot modes
  • UIVM testing, including all menus and options
  • Network configuration testing, especially with multiple (Wired and WIFI) devices to a single NDVM
  • MLE testing, reseal and forward seal paths on TPM 1.2 and 2.0 in both UEFI and Legacy modes
    • MLE testing will validate DRTM for legacy boot.
    • MLE testing will validate DRTM and SRTM for UEFI boot.
  • Upgrade testing (if possible) in UEFI and Legacy modes
    • Upgrades are broken in UEFI mode due to stable-8 not having DRTM

Base functionality of installing the system, installing guest VMs, and having a workable UI, as well as testing out command line options, should allow us to get a handle on how broken the system may be.

Performance and Stress tests to be performed:

  • Performance tests with GPU PT guests in both boot modes
  • Performance testing of the new NDVMs
  • Performance testing of normal HVM guests, Linux and windows.  Less care about OXT Tools...
  • Guest stress tests
    • Tests run in a variety of times from 72 to 120 hours with a combination of:
      • Testing memory and CPU using HeavyLoad
      • Testing CPU performance utilizing Prime95
      • Running a looping video
  • Host stress tests
    • Tests run in a variety of timings up to 120 hours with:
      • Starting guest VMs
      • Stopping guest VMs
  • GPU PT stress tests in both boot modes

Stress and performance testing is vital to the stability of the system.  Many problems that normally do not appear in normal testing will crop up relatively quickly in these tests and will help weed out any unseen issues.

Guest VM OSs under Tests:
  • Windows 10 1809
  • Windows 10 1803
  • RHEL 7.6

Additional Analysis:

  • Look at guest template options for improvements due to new qemu/xen options (UEFI options)
  • Verify in guest that options being set are not drastically different from the downstream products and other stable versions of OpenXT.
  • Inspection of qemu args
  • Inspect SINIT modules being used and what platforms will be capable of running with OXT.

With the sudden change of so many elements, it is important to look at what changes have occurred for the better and utilize the new technologies that the upgrades have provided the system.

Resource allocation:

The current plan is for two AIS Systems Engineers to test the OpenXT platform, taking two weeks to complete the test execution plan.

Hardware resources:

  • Dell XE3 (Coffee Lake)
  • Dell 7490
  • HP EliteDesk 800g3 mini

Risks:

  • Changing to Xen 4.12 may invalidate all of the current testing 
  • Time for validating bug fixes found during testing might be minimal after the testing period is complete

Test Matrix

Bugs are being logged against the epic OXT-1568.  Testing has started and the chart is being updated daily, as these tests are the bare minimum and more can be added.

Test DescPlatformHost BootTPM vOXT vOS VersionOS BootResultNotes/Test Data
Install Windows guest VM with Tools.Dell Latitude 7490UEFI2.0Stable 9 6524Windows 10  1803LegacyPWE

Duplicate VHDs after installing guest tools OXT-1576

Issues with switcher keys OXT-1569

Install Windows guest VM with Tools.Dell XE3UEFI2.0Stable 9 6524Windows 10 1803UEFIFAILAuto-start works, memory set to 8GB, 4vcpus, tools cause black screen. Issue:  OXT-1572 - Getting issue details... STATUS
Install Windows guest VM with Tools.Dell Latitude 7490UEFI2.0Stable 9 6524Windows 10 1809LegacyPWE

Duplicate VHDs after installing guest tools OXT-1576

Issues shutting down guest after tools install OXT-1580

Issues with switcher keys OXT-1569

Install Windows guest VM with Tools.Dell XE3UEFI2.0Stable 9 6524Windows 10 1809UEFIFAILTools fails with a black screen. Issue: OXT-1572
Install RHEL guest VM.Dell Latitude 7490UEFI2.0Stable 9 6524RHEL 7.6LegacyPWE

During installation a duplicate VHD will be listed, labeled QEMU.  May be related to similar Windows issue

Must be installed through troubleshooting menu

Install RHEL guest VM.Dell XE3UEFI2.0Stable 9 6524RHEL 7.6UEFIPASSRHEL 7.6 installed using text installer, visual installer failed.  Tools untested.
Configure guest VM in multiple configurations.  Including adding multiple hard drives, vCPU, memory, boot order, icon, and all advanced configurations.Dell Latitude 7490UEFI2.0Stable 9 6524Windows 10 1803LegacyPWE

HD: FAIL - Every added VHD is duplicated OXT-1576

VCPU : FAIL - VCPUs listed in task manager as sockets OXT-1574

Memory - PASS

Boot Order- PASS

Icon - PASS

Isolation - PWE OXT-1577

Configure guest VM in multiple configurations.  Including adding multiple hard drives, vCPU, memory, boot order, icon, and all advanced configurations.Dell XE3UEFI2.0Stable 9 6524Windows 10 1803UEFIFAILIssue OXT-1574
Configure guest VM in multiple configurations.  Including adding multiple hard drives, vCPU, memory, boot order, icon, and all advanced configurations.Dell Latitude 7490UEFI2.0Stable 9 6524Windows 10 1809LegacyPWE

HD: FAIL - Every added VHD is duplicated OXT-1576

VCPU : FAIL - VCPUs listed in task manager as sockets OXT-1574

Memory - PASS

Boot Order- PASS

Icon - PASS

Isolation - PWE OXT-1577

Configure guest VM in multiple configurations.  Including adding multiple hard drives, vCPU, memory, boot order, icon, and all advanced configurations.Dell XE3UEFI2.0Stable 9 6524Windows 10 1809UEFIFAILIssue OXT-1574

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
NANANANAWindows 10 1803LegacyBLOCKED

Issue OXT-1571

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
Dell XE3UEFI2.0Stable 9 6524Windows 10 1803UEFIFAIL

GPU PT fails to work on the M2000.  Issue OXT-1571

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
NANANANAWindows 10 1809LegacyBLOCKED

Issue OXT-1571

GPU pass through testing:

  • Performance testing running GPU benchmark such as Heaven (UNIGINE) for at least one hour
  • Ad hoc testing
  • Memory Hole analysis
Dell XE3UEFI2.0Stable 9 6524Windows 10 1809UEFIFAIL

GPU PT fails to work on the M2000.   Issue OXT-1571

PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
NANANANAWindows 10 1803Legacy


PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
Dell XE3UEFINA2.0Windows 10 1803UEFIFAIL

USB PCIe device causes screen to go to static.  Issue OXT-1570

PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
NANANANAWindows 10 1809Legacy


PCI pass through testing:

  • Ad hoc testing
  • Memory Hole analysis
Dell XE3UEFINA2.0Windows 10 1809UEFIFAIL

USB PCIe device causes screen to go to static.  Issue OXT-1570

UIVM: Test all buttons for correct functionality that are displayed and sub options.

  • Settings
  • VM Details
  • Status
  • Services
  • Devices
  • Install VM
  • Power Button
  • etc
Dell Latitude 7490UEFINAStable 9 6524Windows 10 1809LEGACYBy Section

Power: FAIL - OXT-1589 - OXT-1591

Install VM: PASS (see VM install section for details)

Devices: PASS

Info: PWE - Many issues with Status Report, no tickets submitted

Services: PASS

Settings: PWE

Audio: Auto-mute Mode inconclusive

Audio: IEC958 settings inconclusive

Audio: Mic Mute LED Mode: Iiconclusive

Audio: Unable to disable ALL mic options simultaneously, one will also reset to "on."  No documentation to tell if feature or bug

Software Update: inconclusive

Network: PASS

Guest Details: PASS

Guest Power: PASS

Network Configurations will be tested, making sure functions in the applet work as needed. Test DHCP and static setups.NANANANANANA

Measured launch measures the correct PCRs for the Host system.  PCR 15,16,17,18,19, at minimum.NALegacy1.2NANANA

Measured launch measures the correct PCRs for the Host system.  PCR 4,5,6,7,8,15,16,17,18,19.NAUEFI2.0NANANA

Measured launch measures the correct PCRs for the Host system.  PCR 15,16,17,18,19, at minimum.NALegacy2.0NANANA

Forward seal works.Dell Latitude 7490UEFI2.0Stable 9 6524NANAFAIL

Issue with RW/RO seal testing OXT-1583

Good.pcrs not updated on forward seal OXT-1584

Forward seal works.NALegacy1.2NANANA

Forward seal works.NALegacy2.0NANANA

Upgrade to self.NALegacy1.2NANANA

Upgrade to self.NALegacy2.0NANANA

Upgrade to self.NAUEFI2.0NANANA

Performance testing of NDVMs will include:

  • pinging a gateway and other addresses to look at latency
  • large file (100+GB) downloads
  • wireshark analysis if needed
  • Add-on vs. integrated NIC
NANANANANANA

Guest Stress testing will include running multiple programs for many hours:

  • 72 hours of HeavyLoad stressing memory and CPU, Prime95, and a video being run.
  • 120 hours of a guest running a video and refreshing a webpage.
NANANANANANA

Host Stress test will include starting and stopping guest VMs for up to 120 hours.NANANANANANA

Test1: 2 Win10 VMs, start/stop every 5 minutes. Failed on 61st iteration. Failed to untap parent disk. UUID e6fd6750.

Test2: 2 Win10 VMs, start/stop every 5 minutes.  Failed on 75th iteration.  Toolstack encountered segmentation fault. UUID e19444f7

Test3: 2 Win10 VMs, start/stop every 5 minutes.  Failed when windows VM failed to stop.  Recoverable

Test4: 2 Win10 VMs, start/stop every 5 minutes.  Failed on 90th iteration.  Surfman killed xenmgr

Test5: 2 Win10 VMs, start/stop every 5 minutes.  Failed on 96th Iteration, Recoverable Error

Test6: 2 Win10 VMs, start/stop every 5 minutes.  Unknown hard lock error

Test7: 2 Win10 VMs, start/stop every 5 minutes.  Failed on 34th iteration. Recoverable error

Test8: 2 Win10 VMs, start/stop every 5 minutes.  Test Aborted

Test9: 2Win10 VMs, start/stop every 5 minutes.  Surfman crash on 92nd iteration

Test10: 2Win10 VMs, start/stop every 5 minutes: WIP

Other Ad-hoc testing to be documented.  TBD.







Notes

Adding this area for bugs and other information found during testing that occured but where not easily reproduced.  Bugs may not exist in Jira, but areas that should be noted.

  • On the XE3 with a USB PCIe card and a nVidia M2000 GPU, experienced multiple reboots after installation of OpenXT.  It appeared to reboot 3-4 times before the UIVM actually came up, however the first boot after install did show that it properly sealed in the blue screen.  They were full reboots, showing the Dell Bios logo.
  • On the Dell 7490, the box appeared to reboot two times.  There is a question as to whether the Seal screen was seen.
  • On the XE3, there was a state where the guest would not shutdown from the UIVM.  This was found during the bug issue OXT-1569, so I am assuming that the shutdown command was being issued to the wrong domain id.
  • Installing UEFI can be tricky, as it first opts to PXE boot.  You must hit c to go to the command prompt, type exit, then go to the UEFI firmware to continue boot to other devices, such as the qemu emulated CD rom.  This method might cause OXT-1569, as it almost seems to do a soft reset.
  • Hot plugging monitors caused issues, but more investigation would need to be done.
  • Can install guest tools on Windows 10 (legacy bios) without installing developer.cer.  There will be multiple popups warning of untrusted drivers, but clicking OK on each will allow installation to continue.  Installing developer.cer as per the wiki instructions makes installation smoother.
  • Possible ticket: Windows 10, 1809 Latitude 7490.  USB mouse periodically stops tracking.  Right and left click still function, but cursor will not move.  Mouse continues to move in UIVM, and trackpad continues to work.  Happens frequently but can't repo on purpose
  • Possible ticket: Windows 10, 1809: Ctrl+number switching stops opening the guest VM.  DomID for guest will be different from that reported when attempting to ctrl switch.
  • HP results mostly inconclusive due to  OXT-1575 - Getting issue details... STATUS