Windows HVM Debugging

Debugging Windows HVM guests in OpenXT

For guest HVM debugging, both the host (debugger) and target VM (debuggee) must be configured. The following steps outline setting this up using serial debugging. Note the serial port on the target side is a virtual serial port emulated by qemu.

Note: On OpenXT platforms, be sure to either disable stubdomains, or modify them as noted in the section below.

TODO: Reformat this section, it is a bit of a mess.

On the host computer (the debugger):

• Install the Debugging Tools For Windows if you do not have them. You will also need to setup symbol and source code paths. The WinDbg documentation describes how to do this. The package can be found here:
  

  http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx
  

  It seems recently that the Debugging Tools For Windows have been moved into the WDK/SDK/Visual Studio installs so to get the most recent versions, this is where to look.
• Setup the symbol path. This is done in Control Panel -> System ->Advanced -> Environment Variables. Make a New System environment variable like the one shown below. The first part of the following sets up C:\Mydir\symbols for local symbols. The second part after the ; is pulling public symbols from MS down into a local cache in C:\Mydir\symwin.
  

   _NT_SYMBOL_PATH = C:\Mydir\symbols;SRV*C:\Mydir\symwin*http://msdl.microsoft.com/download/symbols
  

• Get a copy of sockpipe.exe. This tool should be built during the standard Windows build process, and can be found in xc-windows/sockpipe. 
• Run sockpipe.exe from the command line as follows:
  

  sockpipe.exe debugpipe 7204
  

  • You may want to create a shortcut with the above arguments to streamline things. 
• Run WinDbg with arguments for connected to the pipe created by sockpipe:
  

  windbg -k com:pipe,port=\\.\pipe\debugpipe
  

• At this point WinDbg is connected to sockpipe via the pipe above and waiting for the target system to start. The sockpipe application is listening on a socket for an connection from the target HVM (from the qemu to be precise).
• It's also possible to debug in a Visual Studio environment–see Setting Up Kernel-Mode Debugging of a Virtual Machine in Visual Studio on the MSDN.
  • At the time of writing, Visual Studio 2015 suffers from a bug that blocks the "configure test machines" dialog--try configuring from VS2013 if this is a problem. 

On the target XenClient computer (the debuggee):

• Boot the HVM you want to debug and create a boot entry for debugging. On XP it would involves adding a boot.ini entry. First the file needs to be unhidden and made writable:
  

  C:\>attrib -r -h -s boot.ini
  

  The copy an existing line from the boot.ini file and paste a new one in - make it look like this:
  

  multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /debug /debugport=COM1 /baudrate=115200

• On Vista and Win7 you need to use the bcdedit tool to setup a new boot entry. First lauch a command windows as Administrator and do this:
  

  bcdedit /copy {current} /d win7debug
  

  Then run msconfig from the same command windows. Select the Boot tab and locate your new win7debug boot entry. Select it and Advanced options. Check the Debug checkbox and in the Global debug settings section set the Debug port to COM1 and the Baud reate to 115200. Note that the Debugging Tools For Windows help has information on setting up both the host and target machines.
• Shut the VM down and first disable SELinux - it will interfere with debugging. From a terminal in dom0:
  

  # disable for until reboot
  nr
  setenforce 0
  
  # or disable forever, first you need to make the fs writeable
  rw
  # edit /etc/selinux/config and set to permissive
  ro
  

• Next edit the xenvm config file for the VM. Add a virtual serial port to it that connect to the host machine as follows. The values can be set in a VM's config file using the folloiwing command in dom0 (note the nodelay option turns of naggling which can disrupt debugger operation):
  

  db-write /vm/<uuid>/config/extra-xenvm/0 "serial=tcp:<ip>:<port>,nodelay"
  

  This also works:
  

  xec-vm -n <vmname> set extra-xenvm "serial=tcp:<ip>:<port>,nodelay"
  

  The examples above have qemu connecting out to a listening sockpipe. You can also have qemu listen for incoming sockpipe connections (as a server) with the following (note the nowait will prevent qemu from waiting for a client connection):

  xec-vm -n <vmname> set extra-xenvm "serial=tcp::<port>,server,nodelay,nowait"

• Important: If XL is in use (as with later versions of OpenXT), be sure to format the xec-vm properly. For example:

xec-vm -n Win7-XSPV set extra-xenvm "serial=[\"tcp:192.168.1.60:7204,nodelay\"]"

• Before restarting the VM, make sure that the sockpipe listening app is running - QEMU will try to make a TCP connection to it. Also make sure there are not firewalls or other network issues that could prevent the connection. If QEMU cannot connect, it will spit out a message that looks like this:
  

  Unknown internal error. Error code: 199. Message: Dm.Ioemu_failed("Device model 0 on dm-agent 8 died (status = unknown)")
  

• Restart the HVM and select the debug boot entry. Early in the VM boot process the debugging session will connect.

Debugging with Stubdoms Enabled

It is also possible to do the above Windows HVM debugging with stubdoms enabled but the stubdom initramfs must be modified to enable networking in qemu (note all of the stubdom is in an initramfs). Once this is done stubdom and SELinux can be left enabled during debugging which is closer to the real OpenXT environment. Note you will need to be able to assign a static IP address to the stubdom to make this work. There is no DHCP client in the stubdoms.

First get the stubdom initramfs, it can be found here: /usr/lib/xen/boot/stubdomain-initramfs. Make a copy and rename it with a .gz extension. Next it needs to be extracted:

~/unpacked$ gunzip stubdomain-initramfs.gz
~/unpacked$ sudo cpio -idv < stubdomain-initramfs 

Next edit ~/unpacked/etc/xen/scripts/qemu-ifup and change this line, obviously using a static address that makes sense:

 -    ifconfig $bridgename up
 +    ifconfig $bridgename 10.10.10.10/24 up

Finally re-package the initramfs and replace it on the target:

~/unpacked$ sudo find . | cpio -H newc -o > ../my-modified-initramfs
~/unpacked$ cd ..
~$ gzip my-modified-initramfs

Firewire Debugging

It is possible to do firewire debugging in a Windows guest if you allow the PCI device for firewire to be visible in the VM using pass-through. Unfortunately firewire debugging seems to be rather unreliable in most of the platforms on the HCL. Some of the HP laptops are known to work. The following configures a VM to pass-through a firewire device class: 

xec-vm -n <name_of_my_vm> add-pt-rule 0xc00 any any

As with the instructions above for serial debugging, the target VM needs to be configured for firewire settings. On XP this is a boot.ini entry of the form:

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /debug /debugport=1394 /channel=44

On Vista and Win7, use the msconfig utility to setup the firewire settings. For the host, just run WinDbg, select File -> Kernel Debug -> 1394 and make sure the channel matches.

User Mode Crashes

Gathering crash dumps in user mode is also useful for debugging problems. The following article has details on generating crash dumps for faulting user mode applications:

http://kb.acronis.com/content/2192

The above article is useful for Windows XP and it also shows you how to gather a crash dump when a process is still running on Windows Vista/7. Unfortunately the process may not always still be running after the crash even thought he message box is still displayed (e.g. this seems to be the case for services). There is no Dr. Watson for Vista and 7 - it has been replaced by WER (Windows Error Reporting). WER automatically creates mini and heap dumps for faulting processes and stores them in the following queues (depending on whether it is a user or system process):

%ALLUSERSPROFILE%\Microsoft\Windows\WER\ReportQueue

%USERPROFILE%\AppData\Local\Microsoft\Windows\wer\ReportQueue 

These crash dumps have the extensions .mdmp and .hdmp and can be opened using WinDbg.

Test Signed Drivers

TODO: This is all wrong now. The Powershell script do_sign.ps does the signing. This needs to be updated to indicate how to use this script.

This provides the steps that must be taken to install pv-drivers on x64 Windows version of Vista and later. These versions of Windows will not allow unsigned drivers to load and since xevtchn.sys is a boot start driver, this will prevent you from booting.

You can install unsigned drivers on your VM and for each boot choose F8 -> Disable driver signature enforcement. But you must do this every boot.

Alternately you can test sign the drivers yourself and setup your VM to trust a test certificate. This will allow you to install your test signed drivers and boot normally. Older versions of the DDK do not have the inf2cat utility. Check to see if you have it, if not, before you begin you have to install the winqual sibmission tool in order to get the inf2cat executable. This can be found by going to the following link:

https://winqual.microsoft.com/Help/Inf2cat_FAQ.htm

You will also need to do "path=%path%;c:\program files\microsoft winqual submission tool 2" to add this to your path.

On the build machine:

1. build 32-bit and 64-bit drivers
2. Run "sign\signtest.cmd" from the root of the xc-windows enlistment to test sign the drivers
3. Run "makensis.exe xensetup.nsi" in the install directory to create the xensetup.exe package.

On the client VM:

1. On Vista or later run "bcdedit /set testsigning on"
2. Reboot
3. Install the test certificates. For this you need to get certmgr.exe and a mytest.cer signing cert that you made.
   1. Run "certutil -addstore -f "Root" mytest.cer"
   2. Run "certutil -addstore -f "TrustedPublisher" mytest.cer"
4. Install the test signed pv-drivers.

This page contains information on creating certificates:
https://github.com/OpenXT/openxt/wiki/Windows-Build-Machine

Crash Dump Analysis

The WinDbg tool is also used for post mortem analysis of crash dump files. To load a crash dump file you simply start WinDbg and on the File menu select Open Crash Dump. This will load the dump file and relevant symbols (see above for setting up symbol paths). Once the file is loaded, command can be executed in the command window at the "kd>" prompt. One of the most useful commands to run as a starting point is:

 kd>!analyze -v

This will do a top level analysis and print the results to the command window. When attaching or referencing crash dumps in tickets it would be very useful to also attach the output from this command as a text file. This makes it easy to do a very quick analysis of the ticket and make determinations about the steps to take.

Symbols

Sometimes it is useful to force a debugger to load symbol files for a module even though the checksum or some other meta-information does not match. The following link discusses this and has a tool for making updating PDB files so they match.

http://www.debuginfo.com/articles/debuginfomatch.html

Enabling PV driver debug output

To change the debug output level of the PV drivers, there is a registry setting that must be added in the guest. The key called "Parameters" must be created under the xenevtchn service key. Then the value "FeatureFlags" (a 32-bit DWORD) is created under the "Parameters" key.

 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\xenevtchn\Parameters]
 "FeatureFlags"=dword:00000100

Setting the above value (0x100) will enable all TraceXxx() output to the /var/log/messages file in dom0 as well as the serial console (if connected).