Xen startup and reboot Panic

Description

After a fresh install of OpenXT and during periodic reboots of the system, the workstation will reboot multiple times before successfully displaying the UIVM. It appears to happen before and/or after TBOOT.

Environment

Dell XE3 and 7490 in UEFI mode

Validation Steps

1. Install the system in UEFI.
2. Watch it reboot multiple times.

Activity

Show:
Rich Persaud
June 10, 2019, 3:24 PM

Which hardware devices and BIOS versions have exhibited this behavior?

Chris Rogers
June 10, 2019, 3:27 PM
Rich Persaud
June 10, 2019, 3:32 PM
Edited

From on triage call: Seen on Dell XE3, 7490 and Coffee Lake. Repros more with several PCI cards (NIC + GPU). This is a release blocker.

From on triage call: investigation ongoing, Xen is reporting a large page in L3 (which is impossible).

To assist root cause analysis, please attach Xen logs to this ticket if this issue is reproduced with a new Xen call trace.

To do: additional tracing/debug to collect data on the conditions (hardware, BIOS, timing, PCI devices) which lead up to this intermittent problem. If needed, ask for trace/debug guidance on upstream Xen IRC channel.

Christopher Clark
June 10, 2019, 11:53 PM

There is a BUG_ON macro in Xen's virt_to_xen_l2e function which appears to be tripping here. The check is:

There is one patch in the xenclient-oe Xen patch queue containing code that uses the _PAGE_PSE flag:

https://github.com/OpenXT/xenclient-oe/blob/bba4884769fb0d5e6045c360e2fdc1d6e6efa445/recipes-extended/xen/files/tboot-measure-and-launch-from-xen-efi.patch#L622

See: relocate_pagetables_only

The patch relates to both tboot and UEFI functionality. Probably worth looking at.

Eric Chanudet
July 15, 2019, 5:59 PM
Edited

I spent some time looking at this. While the trail initially pointed to the the TBoot/UEFI boostrap code-path, it was then shown to be a red-herring as a repro on a legacy install was achieved (see attached legacy-virt-to-xen-l2e-panic.cap trace).

 

Further testing revealed Xenfb2 to be the necessary part to get a reproduction. Running UIVM without loading Xenfb2 did not yield a repro for 500+ reboots. Running UIVM, with Xenfb2 loading, usually leads to a repro in less than 50 reboots.

 

Xenfb2 has a lot of defunct and dead code-paths and more code is not exercised in regular use. It differs from upstream Xenfb in a lot of ways, non-exhaustively:

  • Uses fb_mmap instead of fb_read/fb_write to handle framebuffer access.

  • Through fb_mmap, uses zap_page_range (exposed to external modules by another patch) and a vm_fault handler on page access to generate a dirty-page bitmap and change page-caching attributes on the fly.

  • Adds a backend → frontend event (XENFB2_TYPE_MODE_REPLY) to get a valid stride for the desired mode of the framebuffer.

Defunct features include:

  • Another backend → frontend event that gave pages allocated in dom0 to UIVM to be used for the framebuffer with set_phys_to_machine. This is no longer used (XENFB2_TYPE_UPDATE_FB2M),

  • Changing the cache policy on the pages is done at initialisation and in the vm_fault handler. The event is shortcut so that no longer happens. (XENFB2_TYPE_FB_CACHING).

 

Removing the defunct and unused features from the code path managed to not trigger a repro out of 2000+ reboots on a platform where repro was consistently achieved in 50 or less reboots. The main change to work-around this is removing the dirty-bitmap generation logic and the FB2M update logic.

The dirty-bitmap is intended to be used with a backend that would copy the content of the mapped framebuffer pages to the displayed surface in dom0. This is not done by the drm-plugin (default), but still exist with the linuxfb plugin. It is unclear why fb_mmap() was used initially. Some old xen-devel post reference an early implementation of xenfb doing something similar.

Xenfb upstream uses fb_write/fb_read to send dirty-rectangles instead of using a dirty-bitmap on framebuffer pages.

 

The lowest LoE work-around here is to remove the old pieces from Xenfb2.

Another solution is to add a Xenfb backend in Surfman, this requires either copy mode to be fixed or a backend → frontend equivalent to XENFB2_TYPE_MODE_REPLY to be patched in Xenfb, which is not ideal.

Yet another presumed solution could be to make UIVM PVH. That would require changes in Surfman at least, as the current mapping of the pages used by the framebuffer does not seem to work with the drm-plugin.

 

Even then, there seem to be a way to trigger that BUG_ON macro with a compromised kernel. Xenfb2 does require some usually hidden symbols (zap_page_range) of the kernel to be accessible. It is not yet clear if this requires XSM permission specifically given to UIVM or not.

Assignee

Unassigned

Reporter

Joel Upham

Labels

None

QA Assignee

None

QA Image URL

None

Epic Link

Fix versions

Affects versions

Priority

Major
Configure