BuildBot

Owned by Ross Philipson
Last updated: Sept 25, 2015
6 min read

Using BuildBot can be useful to automate OpenXT builds.

Let's assume you're running a 32bits installation of Debian Squeeze, which is the recommended configuration for building OpenXT.
Installing BuildBot is as easy as "apt-get install buildbot". The Squeeze version is quite outdated, but it does the job.

BuildBot works in 2 parts, the slave(s) that build, and the master that control the slave(s).
In the following, I'm describing a setup with just one master and one slave, both on the same machine.

Disclaimer: everything here is done in a pretty dummy way, there's a lot of room for improvement.

Creating the master and the slave

The configuration happens in /var/lib/buildbot for the master/slave configuration, and in /etc/default/buildbot for the deamon configuration.
To get started, "cd /var/lib/buildbot" as root (or as buildbot, after enabling its shell), and create a master and a slave:

$ buildbot create-master master
$ cd master
$ mv master.cfg.sample master.cfg
$ cd ..
$ buildbot create-slave slave 127.0.0.1:9989 slave password
$ cd master

Configuring the master

master.cfg is the main BuildBot file. That's where you define projects, steps and schedulers. Here's some modifications I made to it:

  • Renaming the "BuildSlave" from "bot1name" to "slave", and changing the password from "bot1passwd" to "password".
  • Disable the scheduler by commenting out the c['schedulers'].append line(s).

  • Fix the factory steps (the scripts are detailed in the "Creating the build scripts" section):

    f1.addStep(Compile(command=["bash", "-e", "./fetch.sh"], name="Fetch", description="Fetch", descriptionDone="Fetched"))
f1.addStep(Compile(command=["bash", "-e", "./clean.sh"], name="Clean", description="Clean", descriptionDone="Cleaned"))
f1.addStep(Compile(command=["bash", "-e", "./build.sh"], name="Build", description="Build", descriptionDone="Built"))
f1.addStep(Compile(command=["bash", "-e", "./copy.sh"],  name="Copy",  description="Copy",  descriptionDone="Copied"))

  • In b1, the builder, fix the name ("openxt") the slavename ("slave"), and the builddir ("/home/buildbot")

  • Set allowForce=true, as described in a comment
  • Fix the project name and URL

Configuring the initscript

Below is the content of my /etc/default/buildbot

initscript
# buildbots to manage
# add a new set of variables for each buildbot to start

BB_NUMBER[0]=0          # index for the other values; negative disables the bot
BB_NAME[0]="master"     # short name printed on startup / stop
BB_USER[0]="buildbot"       # user to run as
BB_BASEDIR[0]="/var/lib/buildbot/master"        # basedir argument to buildbot (absolute path)
BB_OPTIONS[0]=""        # buildbot options
BB_PREFIXCMD[0]=""      # prefix command, i.e. nice, linux32, dchroot

BB_NUMBER[1]=1                  # index for the other values; negative disables the bot
BB_NAME[1]="slave"              # short name printed on startup / stop
BB_USER[1]="buildbot"           # user to run as
BB_BASEDIR[1]="/var/lib/buildbot/slave"                # basedir argument to buildbot (absolute path)
BB_OPTIONS[1]=""                # buildbot options
BB_PREFIXCMD[1]=""              # prefix command, i.e. nice, linux32, dchroot

Creating the build scripts

In /home/buildbot/build, I created the scripts corresponding to the steps added to the factory:

clean.sh
#!/bin/bash

umask 0022
rm -rf build
mkdir build

That removes the whole build tree and create an empty folder for the new build.
If the previous build finished, or at least lasted for a while, this step could take more than 60 minutes...

One way to speed this up which is left out here for clarity is to put the build on a filesystem on a block device which is unmounted, reformatted and mounted for each build.

fetch.sh
#!/bin/bash

for i in git/*.git; do
    echo "Fetching `basename $i`..."
    cd $i
    git fetch --all
    cd - > /dev/null
done

This fetches all the repository for the local git mirror.
The fetch should take less than 10 seconds, hopefully not enough time for somebody to push half a feature!

build.sh
#!/bin/bash -ex

umask 0022
cd build
git clone file:///home/buildbot/build/git/openxt.git
cd openxt
cp -r ../../certs .
cp example-config .config
cat <<EOF >> .config
OPENXT_GIT_MIRROR="file:///home/buildbot/build/git"
REPO_PROD_CACERT="/home/buildbot/build/certs/prod-cacert.pem"
REPO_DEV_CACERT="/home/buildbot/build/certs/dev-cacert.pem"
REPO_DEV_SIGNING_CERT="/home/buildbot/build/certs/dev-cacert.pem"
REPO_DEV_SIGNING_KEY="/home/buildbot/build/certs/dev-cakey.pem"
EOF
./do_build.sh | tee build.log
ret=${PIPESTATUS[0]}
cd -
cd -

exit $ret

This step clones the main openxt repo and overwrites some config bits to match the setup.
This assumes that a valid set of certificates has been created in /home/buildbot/build/certs

copy.sh
#!/bin/bash

umask 0022
cp build/openxt/build-output/openxt-dev--master/iso/installer.iso /var/www/openxt/installer.iso

This steps copies the resulting OpenXT iso to the local web server, assuming that buildbot has write access to /var/www/openxt

Mirroring the repositories

$ cd /home/buildbot/build
$ mkdir git
$ for i in blktap.git bootage.git dm-agent.git dm-wrapper.git fbtap.git gene3fs.git icbinn.git idl.git input.git installer.git ioemu.git ioemu-pq.git ipxe-pq.git libedid.git libpciemu.git libxcdbus.git libxenbackend.git linux-3.11-pq.git manager.git meta-selinux.git msi-installer.git network.git ocaml.git openxt.git polmod-example.git pv-linux-drivers.git qemu-dm.git qemu-dm-pq.git refpolicy-xt-pq.git resized.git sdk.git seabios.git seabios-pq.git selinux-policy.git surfman.git sync-client.git sync-cli.git sync-database.git sync-server.git sync-ui-helper.git sync-wui.git toolstack-data.git toolstack.git uid.git v4v.git win-tools.git xblanker.git xclibs.git xctools.git xc-vusb-daemon.git xc-vusb.git xc-windows.git xenaccess.git xenaccess-pq.git xenclient-oe-extra.git xenclient-oe.git xen-common-pq.git xenfb2.git xsm-policy.git; do git clone --mirror git://github.com/OpenXT/$i git/$i; done
$ chown -R buildbot *

Or run replicate_github.py fromhttps://github.com/dickon/scripts/blob/master/replicate_github.py like this:

$ mkdir /home/buildbot/build/git
$ replicate_github.py openxt /home/buildbot/build/git

Starting and testing

$ /etc/init.d/buildbot stop
$ /etc/init.d/buildbot start

Then open http://127.0.0.1:8010 in a web browser.
As the scheduler is disabled, the only way to start a build is to force it, in the builder page.

Windows builds

Buildbot buildslaves can run on Windows. See https://github.com/OpenXT-Extras/build-machines/blob/master/buildmaster/buildbot2.cfg for a complex configuration of buildbot that includes Windows builds.

Reproducible builds

The steps above will build whatever is at the tip of master. Git does not store the history of what was head at a specific time, so even if you know from timestamps when your build was completed with the technique above in general you can't get back to the source code for that build.

The OpenXT build scripts all have support for building a specific tag. A procedure for using this might be: (I say might because I'm still working on this - Dickon@cantab.net 2 September 2014)

  1. This would allow you to replicate all the github openxt repos (maybe usinghttps://github.com/dickon/scripts/blob/master/replicate_github.py), so that you can create tags in the replicas. If you have work in progress that has not yet been merged into openxt you may also want to overlay those central openxt github repos with some with additional changes on the branches you care about.
  2. Then, run https://github.com/dickon/build-machines/blob/master/do_tag.py to create a tag at the head of each branch across every repository that is in use, then
  3. run https://github.com/OpenXT/openxt/blob/master/do_build.sh
  4. Publish both the github replica containing your tags and the build output. It is possible to work back from an installed machine with that build to all the source code using the tag number in the UIVM and dom0 filesystem.

Note that you don't necessarily need to use buildbot to run this (and in fact Dickon isn't doing so yet during development).

See https://github.com/dickon/scripts/blob/master/build_tag.sh for a script which does this, evolving the build.sh above to add support. Usage would be something like:

$ build_tag.sh -u dickon -r dickonr@tezcat.xen-client.com:/data/openxt-www/platform cam tagging /build/dickonr/build

Current output is at http://openxt.xci-test.com/platform/

Next step is to explore using a source manifest file that lists the commits used on each repo, so that the source manifest can be included in the builds. Possibly usehttps://source.android.com/source/using-repo.html and possibly avoid the local git cloning.

Windows builds

The above only builds the OpenXT platform using do_build.sh and OpenEmbedded. The same technique should work on Windows.

Currently OpenXT can be built with the Windows and Linux VM tools integrated into the platform build. This puts quite a lot of complexity into the buildbot configuration since the tags and intermediate build binaries need to be transferred across multiple machines with different operating systems.

(Work in progress - Dickon Reed, 2 September 2014). A simple approach would be to handle tagging, building and distribution of tools (and maybe syncXT server side) separately to the platform build. This would change the way users install tools, perhaps in a positive way. It also introduces issues with version skew. For me the overriding concern is getting a simple and maintainable automated Windows build together on OpenXT. So, the idea would be to run replicate_openxt.py and do_tag.py on Windows and then winbuild-prepare.ps1 and winbuild-all.ps1, and again distribute the tagged git repos with the resulting binary ZIP file. Code to do a Windows build this way is athttps://github.com/dickon/openxt/blob/tagging/windows/build_tag.ps1 and the output can be browsed at http://openxt.xci-test.com/windows/. do_tag.py would be configured to use a different repositories.txt (such ashttps://github.com/dickon/openxt/blob/tagging/windows/build_tag.ps1) which only lists the Windows repos and idl, and the Windows repos would be removed from the platform repositories.txt. This means that platform changes would not require new Windows tools builds, and Windows tools changes would not requite platform builds. Note that Dickon is not actually using buildbot yet to trigger this.