Page Comparison

Introduction

These are some handy steps for doing Linux live kernel debugging on the OpenXT platform. OpenXT makes this readily doable by using serial over IP within QEMU to connect the remote debugger GDB to the kernel debugger KGDB. Of course everyone has their favorite distro but for this we will stick to just one – Debian. Most of the steps should be the same with other flavors aside from the package management bits and specifics of rebuilding the kernel sources.

...

• This is based on using Debian Wheezy HVMs. That means apt package manager and .deb package files.
• Throughout this guide:
  • target is the VM that is being kernel debugged
  • host is the debugger, the VM where GDB is remotely connecting to the target.
• For simplicity, it is assumed both the host and target are the same OS or distro. The paths and users on both are the same. Also password-less SSH login and sudo setup are assumed.
• A basic set of development tools is needed on the target to build the kernel -   sudo apt-get install build-essential should be sufficient (note sometimes dpkg-dev or or ncurses-dev need to be installed manually).

...

Though you can edit .config, it is usually done using one of the editing interfaces. UsingUsing make menuconfig, the setting are here:

...

The new kernel will need a couple of kernel command line parameters to tell KGDB how to connect over serial. Edit /etc/default/grub and setset GRUB_CMDLINE_LINUX="kgdboc=ttyS0,115200 kgdbcon". Note if there were already values set, the KGDB ones can be appended. Then update grub and reboot:

...

Everything is in place to do some actual debugging. To ready the target, the kernel execution must be halted and ready to receive a connection from the debugger this is done two ways. One is to add the kgdbwait parameter to the kernel command line in addition to the parameter that were added earlier. More on that parameter can be found in the KGDB documentation. The second which will be used here is to use the magix magic of sysrq. In the target VM, open a terminal as root and:

...

Modifications can be made to the kernel image on the target, the changes can be rsync'ed d to the host. Then the target can be rebooted and kernel debugging can be resumed.

...

All of the information above is still applicable but to debug modules, a bit more is needed. The following wiki contains instructions for building the OpenXT PV Linux drivers - this should be read first as background information:

PV Linux Drivers

On the target, after following those directions and rebooting, the OpenXT PV Linux drivers will be ready including several that will be loaded at boot time. Running sudo lsmod shows the xenbus driver xc-xen, the block and the net front drivers loaded. Others like xc-v4v.ko need to be manually loaded.

...

Note that "kgdb" in the path is simply the example local version from above. Now the new driver is in place, the code for the drivers including the local changes on the target need to be sync'ed d with the host VM so it has access to the binaries, symbols and source. Do this with rsync again:

...

Note the section names tend to start with a '.' so they get treated as hidden. The output should look like this:

-r--r--r-- 1 root root 4096 Feb  8 15:59 .bss
-r--r--r-- 1 root root 4096 Feb  8 15:58 __bug_table
-r--r--r-- 1 root root 4096 Feb  8 15:59 .data
-r--r--r-- 1 root root 4096 Feb  8 15:59 .devexit.text
-r--r--r-- 1 root root 4096 Feb  8 15:59 .devinit.text
-r--r--r-- 1 root root 4096 Feb  8 15:59 .exit.text
-r--r--r-- 1 root root 4096 Feb  8 15:59 .gnu.linkonce.this_module
-r--r--r-- 1 root root 4096 Feb  8 15:59 .init.text
-r--r--r-- 1 root root 4096 Feb  8 15:59 .note.gnu.build-id
-r--r--r-- 1 root root 4096 Feb  8 15:59 .rodata
-r--r--r-- 1 root root 4096 Feb  8 15:59 .rodata.str1.1
-r--r--r-- 1 root root 4096 Feb  8 15:59 .smp_locks
-r--r--r-- 1 root root 4096 Feb  8 15:59 .strtab
-r--r--r-- 1 root root 4096 Feb  8 15:59 .symtab
-r--r--r-- 1 root root 4096 Feb  8 15:59 .text
-r--r--r-- 1 root root 4096 Feb  8 15:59 .text.unlikely

...

This is the list of all the sections from the ELF binary module that were loaded. The debugger needs to know where .text is. Optionally other sections can be specified too. For the example, .data and .bss are also going to be specified. The sysfs files for the sections report their base address in kernel space:

...

One final note. To ensure the symbolic information is present in the kernel components that will be debugged, make sure that these are built using the -g gcc flag. In addition, turning optimization down or off often makes debugging easier. Turning them off can be done by setting the gcc flag -O0. Another handy flag that makes the stack easier to work with is -fno-omit-frame-pointer making function calls save the stack frames and pointers. So for example the modified line for the Makefile in pv-linux-drivers.git looks like:

$ make -C $(KDIR) FROM_DKMS=n NOSTDINC_FLAGS="$(ENOSTDINC_FLAGS)" M=$(CURDIR) modules EXTRA_CFLAGS="-g -O0 -fno-omit-frame-pointer"

...

First the kernel will need a command line parameter to configure the serial port. EditEdit /etc/default/grub and set GRUB_CMDLINE_LINUX="console=ttyS0,115200n8". Note if there were already values set, the serial one can be appended. Then update grub and reboot:

$ sudo update-grub; rebootpoweroff

Next the emulated QEMU serial port is setup to redirect the kernel trace to another host:

...

When the guest starts, the kernel will trace in this terminal. This assumes stubdoms are turned off. To do this with stubdoms enabled, follow the instructions here:https://github.com/OpenXT/openxt/wiki/Windows-HVM-Debugging#debugging-with-stubdoms-enabled

Debugging With Stubdoms Enabled