Stable-9 rc0 Test Matrix
Goal of testing:
Determine and ensure that the functionality of OpenXT has not been compromised by the staggering changes that the product has been subjected to. Validation of the entire product needs to be determined through the testing of these components.
Regressions will be determined by evaluating the system from previous baselines of stable OpenXT and downstream derivative products. Every major part of OpenXT has been changed, including:
Changes (growing list):
Xen has been up-reved to 4.11 with the plan to go to 4.12 to get Argo support (replacing v4v)
Qemu has been up-reved
dom0 is a 64 bit OS
All Haskel and other programs are 64bit - rpc-proxy, xenmgr, updatemgr, hvmloader, seal-system, tpm tools (need to verify everything working), etc
NDVMs are now HVM guests, not PV
Addition of OVMF for UEFI guests
Testing areas should include:
General guest VM installation in both boot modes
General guest VM configurations in both boot modes
Installations with UEFI and Legacy guests
GPU PT with each guest in both boot modes
UIVM testing, including all menus and options
Network configuration testing, especially with multiple (Wired and WIFI) devices to a single NDVM
MLE testing, reseal and forward seal paths on TPM 1.2 and 2.0 in both UEFI and Legacy modes
MLE testing will validate DRTM for legacy boot.
MLE testing will validate DRTM and SRTM for UEFI boot.
Upgrade testing (if possible) in UEFI and Legacy modes
Upgrades are broken in UEFI mode due to stable-8 not having DRTM
Base functionality of installing the system, installing guest VMs, and having a workable UI, as well as testing out command line options, should allow us to get a handle on how broken the system may be.
Performance and Stress tests to be performed:
Performance tests with GPU PT guests in both boot modes
Performance testing of the new NDVMs
Performance testing of normal HVM guests, Linux and windows. Less care about OXT Tools...
Guest stress tests
Tests run in a variety of times from 72 to 120 hours with a combination of:
Testing memory and CPU using HeavyLoad
Testing CPU performance utilizing Prime95
Running a looping video
Host stress tests
Tests run in a variety of timings up to 120 hours with:
Starting guest VMs
Stopping guest VMs
GPU PT stress tests in both boot modes
Stress and performance testing is vital to the stability of the system. Many problems that normally do not appear in normal testing will crop up relatively quickly in these tests and will help weed out any unseen issues.
Guest VM OSs under Tests:
Windows 10 1809
Windows 10 1803
RHEL 7.6
Additional Analysis:
Look at guest template options for improvements due to new qemu/xen options (UEFI options)
Verify in guest that options being set are not drastically different from the downstream products and other stable versions of OpenXT.
Inspection of qemu args
Inspect SINIT modules being used and what platforms will be capable of running with OXT.
With the sudden change of so many elements, it is important to look at what changes have occurred for the better and utilize the new technologies that the upgrades have provided the system.
Resource allocation:
The current plan is for two AIS Systems Engineers to test the OpenXT platform, taking two weeks to complete the test execution plan.
Hardware resources:
Dell XE3 (Coffee Lake)
Dell 7490
HP EliteDesk 800g3 mini
Risks:
Changing to Xen 4.12 may invalidate all of the current testing
Time for validating bug fixes found during testing might be minimal after the testing period is complete
Test Matrix
Bugs are being logged against the epic OXT-1568. Testing has started and the chart is being updated daily, as these tests are the bare minimum and more can be added.
Test Desc | Platform | Host Boot | TPM v | OXT v | OS Version | OS Boot | Result | Notes/Test Data |
|---|---|---|---|---|---|---|---|---|
Install Windows guest VM with Tools. | Dell Latitude 7490 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1803 | Legacy | PWE | Duplicate VHDs after installing guest tools OXT-1576 Issues with switcher keys OXT-1569 |
Install Windows guest VM with Tools. | Dell XE3 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1803 | UEFI | FAIL | Auto-start works, memory set to 8GB, 4vcpus, tools cause black screen. Issue: https://openxt.atlassian.net/browse/OXT-1572 |
Install Windows guest VM with Tools. | Dell Latitude 7490 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1809 | Legacy | PWE | Duplicate VHDs after installing guest tools OXT-1576 Issues shutting down guest after tools install OXT-1580 Issues with switcher keys OXT-1569 |
Install Windows guest VM with Tools. | Dell XE3 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1809 | UEFI | FAIL | Tools fails with a black screen. Issue: OXT-1572 |
Install RHEL guest VM. | Dell Latitude 7490 | UEFI | 2.0 | Stable 9 6524 | RHEL 7.6 | Legacy | PWE | During installation a duplicate VHD will be listed, labeled QEMU. May be related to similar Windows issue Must be installed through troubleshooting menu |
Install RHEL guest VM. | Dell XE3 | UEFI | 2.0 | Stable 9 6524 | RHEL 7.6 | UEFI | PASS | RHEL 7.6 installed using text installer, visual installer failed. Tools untested. |
Configure guest VM in multiple configurations. Including adding multiple hard drives, vCPU, memory, boot order, icon, and all advanced configurations. | Dell Latitude 7490 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1803 | Legacy | PWE | HD: FAIL - Every added VHD is duplicated OXT-1576 VCPU : FAIL - VCPUs listed in task manager as sockets OXT-1574 Memory - PASS Boot Order- PASS Icon - PASS Isolation - PWE OXT-1577 |
Configure guest VM in multiple configurations. Including adding multiple hard drives, vCPU, memory, boot order, icon, and all advanced configurations. | Dell XE3 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1803 | UEFI | FAIL | Issue OXT-1574 |
Configure guest VM in multiple configurations. Including adding multiple hard drives, vCPU, memory, boot order, icon, and all advanced configurations. | Dell Latitude 7490 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1809 | Legacy | PWE | HD: FAIL - Every added VHD is duplicated OXT-1576 VCPU : FAIL - VCPUs listed in task manager as sockets OXT-1574 Memory - PASS Boot Order- PASS Icon - PASS Isolation - PWE OXT-1577 |
Configure guest VM in multiple configurations. Including adding multiple hard drives, vCPU, memory, boot order, icon, and all advanced configurations. | Dell XE3 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1809 | UEFI | FAIL | Issue OXT-1574 |
GPU pass through testing:
| NA | NA | NA | NA | Windows 10 1803 | Legacy | BLOCKED | Issue OXT-1571 |
GPU pass through testing:
| Dell XE3 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1803 | UEFI | FAIL | GPU PT fails to work on the M2000. Issue OXT-1571 |
GPU pass through testing:
| NA | NA | NA | NA | Windows 10 1809 | Legacy | BLOCKED | Issue OXT-1571 |
GPU pass through testing:
| Dell XE3 | UEFI | 2.0 | Stable 9 6524 | Windows 10 1809 | UEFI | FAIL | GPU PT fails to work on the M2000. Issue OXT-1571 |
PCI pass through testing:
| NA | NA | NA | NA | Windows 10 1803 | Legacy | ||
PCI pass through testing:
| Dell XE3 | UEFI | NA | 2.0 | Windows 10 1803 | UEFI | FAIL | USB PCIe device causes screen to go to static. Issue OXT-1570 |
PCI pass through testing:
| NA | NA | NA | NA | Windows 10 1809 | Legacy | ||
PCI pass through testing:
| Dell XE3 | UEFI | NA | 2.0 | Windows 10 1809 | UEFI | FAIL | USB PCIe device causes screen to go to static. Issue OXT-1570 |
UIVM: Test all buttons for correct functionality that are displayed and sub options.
| Dell Latitude 7490 | UEFI | NA | Stable 9 6524 | Windows 10 1809 | LEGACY | By Section | Power: FAIL - OXT-1589 - OXT-1591 Install VM: PASS (see VM install section for details) Devices: PASS Info: PWE - Many issues with Status Report, no tickets submitted Services: PASS Settings: PWE Audio: Auto-mute Mode inconclusive Audio: IEC958 settings inconclusive Audio: Mic Mute LED Mode: Iiconclusive Audio: Unable to disable ALL mic options simultaneously, one will also reset to "on." No documentation to tell if feature or bug Software Update: inconclusive Network: PASS Guest Details: PASS Guest Power: PASS |
Network Configurations will be tested, making sure functions in the applet work as needed. Test DHCP and static setups. | NA | NA | NA | NA | NA | NA | ||
Measured launch measures the correct PCRs for the Host system. PCR 15,16,17,18,19, at minimum. | NA | Legacy | 1.2 | NA | NA | NA | ||
Measured launch measures the correct PCRs for the Host system. PCR 4,5,6,7,8,15,16,17,18,19. | NA | UEFI | 2.0 | NA | NA | NA | ||
Measured launch measures the correct PCRs for the Host system. PCR 15,16,17,18,19, at minimum. | NA | Legacy | 2.0 | NA | NA |