These are small haskell programs. Xec is a generic dbus access utility, similar to dbus-send, can be used to communicate with any dbus daemon though defaults to communication with xenmgr. Xec-vm is a wrapper around xec which is vm aware. Both have decent "--help" toggles so will not describe their options further.
Database daemon (dbd) is a small ocaml/dbus aplication which handles persistence of xenclient configuration data. The database files are stored in /config/db and /config/vms.
Dbd reads config files (which are in json format) at startup and creates in-memory representation of the database, which is primarily a simple tree of strings similar to xenstore. From now on it proceeds to work on memory representation only, marking it dirty in case of modifications. The dirty trees are flushed to disk in 3s intervals. Dbd features protection against power cuts / partial writes. It always flushes to temporary file first, then atomically renames the temporary file.
+ read ( path:s, OUT value:s )
reads a string value at given path
+ read-binary ( path:s, OUT value:ay )
reads byte array value at given path
+ write ( path:s, value:s )
writes string value to given path
+ dump ( path:s, OUT value:s )
dumps whole json subtree at given path
+ inject ( path:s, value:s )
writes whole json subtree to given path
+ list ( path:s, OUT value:as )
lists child nodes at given path
+ rm ( path:s )
removes subtree at given path
+ exists ( path:s, OUT ex:b )
checks if a subtree/value at given path exists
RPC proxy is a haskell application which supports proxying and filtering of RPC traffic used in xenclient. The proxied traffic consists either of binary dbus (for most use cases) or json/websocket based dbus (for conversations with web browser in uivm) and is transported using unix domain sockets or v4v sockets.
There are 3 instances of rpc-proxy running by default in dom0, each of them having different function:
/usr/bin/rpc-proxy -s
This one forwards data between default incoming channel (v4v port 5555) to default outgoing channel (unix socket /var/run/dbus/system_bus_socket). The effect is that it's possible for any VM to connect to v4v port 5555 of domain 0 and get access to it's system bus in similar fashion that's possible locally within vm by connecting to the unix socket. V4V port 5555 is exposed only to reasonably trusted service vms, such as ndvm/syncvm since it's possible to export new services on domain dom0 bus by connecting to it. Rpc-proxy makes decision whether to forward or drop messages based on default rules file in /etc/rpc-proxy.rules
/usr/bin/rpc-proxy -i v4v:5556 -n com.citrix.xenclient.guest.uuid_$UUID --translate-anonymous-dests
This one is similar to one above however slightly more "secure" and therefore exposed to user vms as well. It forwards data between v4v port 5556 to default outgoing channel (unix socket /var/run/dbus/system_bus_socket). The "-n com.citrix.xenclient.guest.uuid_$UUID" arguments causes any attempt to export named service on dom0 to force a rename of the service to com.citrix.xenclient.guest.uuid_$UUID. Therefore a guest connecting to this port can only export one service. This has been useful in the past for implementing an agent in linux vms exporting guest power management operations, but I believe this has since been replaced by xenstore/pv driver based functionality.
/usr/bin/rpc-proxy -i v4v:8080 --json-in --websockets-in -n com.citrix.xenclient.guest.uuid_$UUID --auto-auth
This one listens on v4v port 8080 and forwards to default outgoing channel (unix socket /var/run/dbus/system_bus_socket). It expects incoming data in json format (--json-in) wrapped in websocket protocol (--websockets-in). It limits ability to export service names on dom0 bus via "-n com.citrix.xenclient.guest.uuid_$UUID". This is used for communications with browser used in uivm. "--auto-auth" is just a shortcut to avoid the browser having to do dbus authentication, rpc-proxy will do it on its behalf (we don't use dbus authentication for any security purposes, all it does is state that we are connecting as root user).
The default rpc rules are stored in /etc/rpc-proxy.rules file. Extra rules can be attached to vm config trees and will take effect when the VM is started and be torn down when the VM is stopped. The format of the rules is relatively straightforward, by example:
# nothing can be done by default deny all
# allow stubdoms to talk to surfman,xenmgr,dbus allow stubdom true destination com.citrix.xenclient.surfman allow stubdom true destination com.citrix.xenclient.xenmgr allow stubdom true destination org.freedesktop.DBus interface org.freedesktop.DBus
# allow guests to call 'gather' on diagnostics interface (required by xc-diag) allow destination com.citrix.xenclient.xenmgr interface com.citrix.xenclient.xenmgr.diag member gather
# allow anybody to do some vm queries required for switcher bar allow destination com.citrix.xenclient.xenmgr interface org.freedesktop.DBus.Properties member Get allow destination com.citrix.xenclient.xenmgr interface com.citrix.xenclient.xenmgr member list_vms allow destination com.citrix.xenclient.xenmgr interface com.citrix.xenclient.xenmgr.vm member get_db_key allow destination com.citrix.xenclient.xenmgr interface com.citrix.xenclient.xenmgr.vm member read_icon allow destination com.citrix.xenclient.xenmgr interface com.citrix.xenclient.xenmgr.vm member switch allow destination com.citrix.xenclient.input interface com.citrix.xenclient.input member get_focus_domid allow destination com.citrix.xenclient.xenmgr interface com.citrix.xenclient.xenmgr member find_vm_by_domid
# allow guest to do some requests allow destination com.citrix.xenclient.xenmgr interface com.citrix.xenclient.xenmgr.guestreq member request_attention
# allow conditional domstore (private db space) access allow destination com.citrix.xenclient.db interface com.citrix.xenclient.db member read if-boolean domstore-read-access true allow destination com.citrix.xenclient.db interface com.citrix.xenclient.db member read_binary if-boolean domstore-read-access true allow destination com.citrix.xenclient.db interface com.citrix.xenclient.db member list if-boolean domstore-read-access true allow destination com.citrix.xenclient.db interface com.citrix.xenclient.db member exists if-boolean domstore-read-access true
allow destination com.citrix.xenclient.db interface com.citrix.xenclient.db member write if-boolean domstore-write-access true allow destination com.citrix.xenclient.db interface com.citrix.xenclient.db member rm if-boolean domstore-write-access true
Of some interest are stubdom rules marked as "stubdom true". They only match on messages coming from stub domains
Also it is possible to make rules based on any boolean fields in incoming vm config tree. In the example above "if-boolean domstore-read-access true" matches only if the VM which has sent the message has a "domstore-read-access" boolean config in its tree set to true. Therefore it is possible to disable/enable VM's domstore access (which boils down to allowing it to access dbd remotely) simply by manipulating its config tree.
Sometimes it's useful to grant rpc permission to all vms of particular type (such as "ndvm", "syncvm"). This can be achieved by adding "dom-type <type>" matcher to the rule, for example:
allow dom-type syncvm destination com.citrix.xenclient.xenmgr interface com.citrix.xenclient.xenmgr.vm member add_disk
xenvm is a single virtual machine monitor, written in Ocaml. It's forked on per-vm basis, and responsible for its lifecycle and control operations. It interacts directly with Xen via libxenctrl wrapper. You can find it in toolstack.git.
Which layers of xenvm have state? If xenvm is killed should the domain die? Can xenvm be restarted while a domain continues running? Should xenvm instances exist for VMs which are not running? - Dickon
Tomaszw: Xenvm is a quite stateful daemon:
I believe the lower layers (such as xenops / device layer) primarily rely on state stored in xenstore, mostly by pv drivers, and don't hold in-memory state (they do tend to write (and later read) some small extra state bits to xenstore (rarely)).
It does run even if the VM is not running, providing the configurability of VM in its "stopped" state. However, as a boot performance optimization, it's only started for the first time when the 1st request to start a VM it handles is created, then continues to run until system shutdown (or VM deletion). Xenmgr copes with it by delaying the initial configuration.
The state of VM reported by xenvm is partially taken from domain state reported by xen, partially figured out by xenvm. Xen doesn't report exact domain states when it notifies the dom0 toolstack about a state change (via event channel), it basically just gives a notification that "something has happened" and it's the job of xenvm to check on each VM whether it's still running, or died etc and update its internal state accordingly (and then forward it to upper layers of toolstack, that is xenmgr).
Xenvm cannot be restarted when a domain is running, that is the limitation of how it's coded at the moment (stateful). It can be restarted when domain is dead, but does not automatically quit when domain dies (and shouldn't).
There are few udev scripts in "scripts" subfolder, as well as udev rules for executing these scripts. These handle the notifications to the toolstack that the backend pv drivers have completed creating network/disk devices, or torn them down.
Xenops is the lower-layer part of xenvm, responsible for lower level management of xen domains (via domain ids). It is both used internally by xenvm as well as exposed to the user via the "xenops" dom0 utility.
xenmgr is a haskell application which exports VM configuration over DBus and translates it to lower level configuration files consumed by xenvm. Since xenvm is a single virtual machine monitor, xenmgr is reponsible for some cross-vm concepts, such as relocating network pv backend on backend domain reboot, vm dependencies, enforcing cross-vm v4v firewall rules etc.
Please detail the concurrency model (e.g. thread per dbus connection, reactor or whatever). What are the consequenes of killing and restarting xenmgr? What normally launches xenmgr? - Dickon
Tomaszw: xenmgr is started by the "bootage" dom0 program, similarily to many other dom0 daemons (configured in /etc/bootage.conf). Xenmgr is fully restartible, since it doesn't keep almost any in-memory vm state. The only consequence is a temporary DoS on its functionality, as well as the possibility of some startup code executing again, which includes for example locking the UIVM with authentication screen, or performing boot-time service vm filesystem checksum.
Each incoming RPC call to xenmgr is processed in parallel (in so called haskell IO thread). Because most of the state is kept outside of xenmgr (either in db, or xenstore), there isn't much synchronisation needed. Still, because the dbd doesn't support transactions, some db writes need to be protected by locking, which xenmgr does.
Unlike incoming calls, incoming notifications are processed serially on a separate IO thread. This is because the ordering of notifications is important (and guaranteed by dbus, therefore we have to process them serially to keep the guarantee). As a consequence, long running notification handlers should be forking off a thread to not block the queue.
xenmgr exports following dbus objects:
each of these objects exports one or more dbus interfaces, as defined in IDL repository (idl.git). Following IDL files are used by xenmgr
Most of dbus boilerplate code, such as the hooks to implement exported functions as well as stubs for calling other daemons is generated from the files in idl.git by a custom written program "rpcgen". It takes dbus xml files as input and produces binding/hooks in variety of languages as output.
In case of xenmgr the boilerplate is generated from its BB recipie, in the configure step:
# generate rpc stubs
mkdir -p Rpc/Autogen
# Server objects
xc-rpcgen --haskell -s -o Rpc/Autogen --module-prefix=Rpc.Autogen ${STAGING_DATADIR}/idl/xenmgr.xml
xc-rpcgen --haskell -s -o Rpc/Autogen --module-prefix=Rpc.Autogen ${STAGING_DATADIR}/idl/xenmgr_vm.xml
Hence usually modification of IDL consists of following steps:
There's a bunch of json templates in the "templates" subfolder; on the device they are installed in /usr/share/xenmgr-1.0/templates. There are basically two types of templates:
It's sometimes handy to turn off the automatic overwrite of service vm templates each xenmgr start, so that the service vm can be reconfigured via manual xec-vm invocations. This can be achieved via
db-write /overwrite-<vmtype>-settings false
for example for ndvm: db-write /overwrite-ndvm-settings false
JSON vm configuration files are stored in /config/vms/ though xenmgr doesn't access them directly but rather through dbd's dbus API. Low level configuration files consumed by xenvm (which are output by xenmgr) reside in /tmp/xenmgr-xenvm-$VM_UUID files.
VM properties are declared in idl.git, in interfaces/xenmgr_vm.xml. Most of them are documented, so I won't redo that here. Any additional documention should go to IDL files since that's also the place from which sdk docs are generated.
There are some vm properties which have non trivial repercussions and warrant additional clarification:
Setting it to true changes CPUID signature reported by xen to be xenclient specific. Effectively it hides xen presence from the guest. We use it on linux VMs to avoid the default PV drivers in upstream kernels from activating (so that we can use custom ones). However toggling it off enables the usage of upstream drivers, if needed.
This specifies the selinux security context under which VM runs, if mismatched it can prevent the VM from being able to issue hypercalls required for its normal function.
Tells xenmgr to treat this vm as network VM; it will send notifications about VM that VM state to network daemon.
This activates the behaviour of greedily seizing devices , which are configured for PCI passhthrough by any vm, by pciback driver on xenclient boot. This behaviour is different from the upstream one, however it is useful to prevent dom0 drivers using the devices which might later be used for passthrough.
Audio driver is a good example; if audio card is not seized by pciback driver at boot, alsa drivers in dom0 will take over it which might prevent passthrough from working correctly later when PCI passthrough vm is started.
xenmgr supports offloading some of its functionality to either a script, or another dbus daemon (possibly running in different vm). This is done via the following VM hooks (implemented as regular vm properties):
Each hook can be either a script name, example:
xec-vm -n foo set run-insteadof-start /bin/customstart
or a string specifying dbus rpc call to make, example:
xec-vm -n foo set run-insteadof-start rpc:vm=$SOME_UUID,destination=com.citrix.some.service,interface=com.citrix.iface,member=com.citrix.some.method
both the script and rpc call gets passed the affected VM uuid as its single argument.
VM passthrough rules are specified by a set of matchers, which are evaluated when VM starts to find actual PCI devices which need to be PT-ed.
VM PCI passthrough rules are managed by the following dbus methods:
Adds a new passthrough rules. each of the <pciclass> <pcivendor> <pcidevice> matchers can take either an ID or word "any" which means match on all devices. example:
xec -n somevm add-pt-rule 0x680 any any
Adds a new passthrough rule using BDF notation, example:
xec -n somevm add-pt-rule-bdf 0000:00:16.0
V4V firewall rules are managed by dbus api similarily to PCI passthrough rules and evaluated when the VM starts. They usually result in a series of calls to "viptables" commandline program to erect the firewall. On VM shutdowns, the entries added during VM start are torn down.
V4V firewall rules can be modified via add_v4v_firewall_rule / delete_v4v_firewall_rule methods. These take a string argument with a rule definiton. The format of rule string is as follows:
<source> -> <destination>
the format of each of the source/destination endpoints is
( * | my-stubdom | myself | dom-type=<domaintype> | dom-name=<domainname> ) : <v4v port>
examples:
Before VM is started, if it has "measured" property set, the first disk (with ID of 0), and only the first disk, will be checked for checksum consistency with a hash stored in vm config tree. This is done by mounting the filesystem and computing sha256 hash of the filesystem (as opposed to doing it on VHD file; because even readonly VHD files are modified due some bookkeep information such as access timestamp).
If a hash inconsistency is detected, a measurement failure action will be invoked. It defaults to shutting down the system. It can be overriden via
db-write /xenmgr/measure-fail-action <powermanagementaction>
<powermanagementaction> can be one of the following
and as said previously, defaults to forced shutdown if not specified
xenmgr supports simple form of dependency tracking between vms. If a VM is configured such that its network backend is not in dom0, xenmgr will internally track that dependency. It's possible to list all the vms a given vm depends on by
xec-vm -n somevm list-dependencies
By default when a VM is started, xenmgr will ensure all its dependecies are started first. This can be toggled off via setting "track-dependencies" vm property to false.
There's no automatic shutdown of dependent vms.
Xenmgr supports some configuration of power related operations
This configures how the toolstack handles requests to put a VM to S3. Note that this doesn't affect requests made from within guest, but just requests originating from the UI / closing the laptopt lid etc. It can be one of the following:
Supports exactly same configurability as "s3-mode", specifies actions to be taken when toolstack is supposed to put VM to hibernate.
This is useful for some single-vm scenarios. Toolstack will track guest power state and try to put host to the same state. So if the guest goes to S3, toolstack will put the host to S3 as well.
Configure the power action performed when laptop lid closes, either on battery or AC adapter. Can be one of the following:
Xenmgr is split into a small library (xenmgr-core) and main daemon (xenmgr). xenmgr-core atm contains very little, primarily just a bit of v4v firewall rule parsing code. The intent was to move code useful for other projects (such as the OVF import tool) out of xenmgr daemon and into the library.
There's a bunch of small haskell libraries we wrote which are in use by toolstack components. They reside in xclibs.git.
Please specify kind of interaction, e.g. exec, sockets, DBUS, etc - Dickon
xenmgr interacts with the following daemons on system: