...
rules based on configuration of VM sending the message It is also possible to make rules based on any boolean fields in an incoming VM config tree. In the example above "if-boolean domstore-read-access true" matches only if the VM which has sent the message has a "domstore-read-access" boolean config in its tree set to true. Therefore, it is possible to disable/enable VM's domstore access (which boils down to allowing it to access dbd remotely) simply by manipulating its config tree.
rules based on domain type Sometimes it's useful to grant rpc permission to all VMs of particular type (such as "NDVM", "SyncVM"). This can be achieved by adding "dom-type " matcher to the rule, for example: allow dom-type syncvm destination com.citrix.xenclient.xenmgr interface com.citrix.xenclient.xenmgr.vm member add_disk
xenvm
xenvm is a single virtual machine monitor, written in Ocaml. It's forked on per-vm basis, and responsible for its lifecycle and control operations. It interacts directly with Xen via the libxenctrl wrapper. You can find it in toolstack.git.
TODO: Which layers of xenvm have state? If xenvm is killed should the domain die? Can xenvm be restarted while a domain continues running? Should xenvm instances exist for VMs which are not running?
Xenvm is a stateful daemon:
- upper layers hold the VM config state in memory
- upper layers hold the VM state in memory
The lower layers (such as xenops / device layer) primarily rely on state stored in xenstore, mostly by PV drivers, and don't hold in-memory state (they do tend to write [and later read] some small extra state bits to xenstore [rarely]).
It does run even if the VM is not running, providing the ability to configure a VM in its "stopped" state. However, as a boot performance optimization, it's only started for the first time when the 1st request to start a VM it handles is created. Then it continues to run until system shutdown (or VM deletion). Xenmgr copes with it by delaying the initial configuration.
The state of a VM reported by xenvm is partially taken from domain state reported by Xen as well as partially derived by xenvm. Xen doesn't report exact domain states when it notifies the dom0 toolstack about a state change (via event channel), it basically just gives a notification that "something has happened" and it's the job of xenvm to check on each VM whether it's still running, or died etc and update its internal state accordingly (and then forward it to upper layers of toolstack, that is xenmgr).
Xenvm cannot be restarted when a domain is running, that is the limitation of how it's coded at the moment (stateful). It can be restarted when domain is dead, but xenvm does not automatically quit when a domain dies (and shouldn't).
Code Layout
Libraries
- libs/common - various utilities
- libs/eventchn - binding for libxenctrl's eventchannel interfaces
- libs/json - json parser
- libs/stdext - utility extensions for Ocaml's standard library
- libs/uuid - uuid generation and handling
- libs/xc - libxenctrl binding
- libs/xg - libxenguest binding
- libs/xs - native ocaml xenstore protocol implementation
Scripts
There are few udev scripts in "scripts" subfolder, as well as udev rules for executing these scripts. These handle the notifications to the toolstack that the backend PV drivers have completed creating network/disk devices, or tearing them down.
Xenops
Xenops is the lower-layer part of xenvm, responsible for lower level management of Xen domains (via domain ids). It is both used internally by xenvm as well as exposed to the user via the "xenops" Dom0 utility.
- xenops/balloon.ml - memory ballooning utilities, not used much in OpenXT
- xenops/device.ml and device_common.ml - important files which are responsible for initialization of PV (or PCI passthrough) device backends (via xenstore)
- xenops/dm.ml - config construction for qemu
- xenops/dmagent.ml - communication with dmagent, which is a program used to fork/configure qemu instances (which can be running in another domain)
- xenops/domain[_common].ml - domain management functions
- xenops/hotplug.ml - few utility functions to wait on device being created by PV backend etc
- xenops/memory.ml - crazy arithmetic to figure out how much memory is required to boot a VM
- xenops/netman.ml - couple network helper functions
- xenops/watch.ml - xenstore watch helper functions
- xenops/xal.ml - low level loop waiting and parsing PV device events
Xenvm
- xenvm/misc.ml - as name says
- xenvm/tasks.ml - list of rpc tasks xenvm supports
- xenvm/vmact.ml - high level implementation of vm operations (start/stop etc)
- xenvm/vmconfig.ml - parsing xenvm config files (in OpenT, placed in /tmp/xenmgr-xenvm-*)
- xenvm/vmstate.ml - VM state struct
- xenvm/xenops.ml - xenops Dom0 utility entry point
- xenvm/xenvm.ml - daemon entry point
xenmgr
xenmgr is a haskell application which exports VM configuration over DBus and translates it to lower level configuration files consumed by xenvm. Since xenvm is a single virtual machine monitor, xenmgr is reponsible for some cross-vm concepts, such as relocating network PV backend on backend domain reboot, VM dependencies, enforcing cross-vm v4v firewall rules etc.
TODO: detail the concurrency model (e.g. thread per dbus connection, reactor or whatever). What are the consequenes of killing and restarting xenmgr? What normally launches xenmgr?
Xenmgr is started by the "bootage" Dom0 program, similarly to many other Dom0 daemons (configured in /etc/bootage.conf). Xenmgr is fully reusable since it doesn't keep almost any in-memory VM state. The only consequence is a temporary DoS on its functionality as well as the possibility of some startup code executing again, which includes for example locking the UIVM with authentication screen, or performing boot-time service VM filesystem checksum.
Each incoming RPC call to xenmgr is processed in parallel (called haskell IO thread). Because most of the state is kept outside of xenmgr (either in db or xenstore), there isn't much synchronization needed. Still, because the dbd doesn't support transactions some db writes need to be protected by locking. Xenmgr provides this capability.
Unlike incoming calls, incoming notifications are processed serially on a separate IO thread. This is because the ordering of notifications is important (and guaranteed by dbus, therefore we have to process them serially to keep the guarantee). As a consequence, long running notification handlers should be forking off a thread to not block the queue.