Table of Contents:
OpenEmbedded (OE) is used to build the OpenXT installer ISO and all of the VMs that make up the OpenXT system. Currently, due to our dependency on the older 6.12 version of the Haskell compiler, we recommend the use of Debian Squeeze as the distro for your build system. All build machines must be 32bit as well due to a few places where our build scripts invoke tools or use libraries on the host system.
There are other distros that can be made to work but they require some ugly hacks in places. If you've have built OpenXT successfully on a platform not mentioned here please make a new page and link to your instructions here.
Basic OpenEmbedded build machine setup
Follow the instructions for debian at OEandYourDistro on the OpenEmbedded wiki for installing a few packages required to get OpenEmbedded going. OpenEmbedded itself will download and install a lot of toolchain materials (e.g. specific compiler versions).
Additional OpenXT specific packages
On Squeeze, install these additional packages:
$ ghc guilt iasl quilt bin86 bcc libsdl1.2-dev liburi-perl genisoimage policycoreutils unzip
On Wheezy, follow the additional instructions here: Debian Wheezy packages.
Bash vs Bourne
Be sure that your build system is using bash as the default system shell. Both Debian and Ubuntu use dash instead and the 'bashisms' in a lot of the build metadata will cause failures unless
/bin/sh is a symlink to
/bin/bash. The Ubuntu section of of the OE and Your Distro describes a work-around for this issue.
Setup OpenXT workspace
The first build scripts exist in openxt.git (along with this README.md file), so clone that:
$ git clone git://github.com/OpenXT/openxt.git
You can of course clone your own fork. Be aware that there are another approximately 60 OpenXT repositories that will be checked out next.
Configure your build tree
$ cd openxt # the directory created by the clone command above
$ cp example-config .config
You should review the .config file and edit it as appropriate.
Configure signing certificates
NOTE: there is an effort in progress as of 18 June 2014 to simplify this such that those who are concerned with building signing get a simple auto-generated dev signing certificate.
For now you do need to create certificates. You need to protect these; someone with read access to these files can potentially get any devices running your build of OpenXT to upgrade themselves to their malicious software by triggering an "Over The Air" upgrade, though this should require local access to each device to point it at a server that offers the new version.
If you are intending to distribute the results of your build you should look into how the OpenXT release signing system works and figure out how you are going to handle your certifcates. Otherwise, run these commands:
$ mkdir certs # make a certs directory inside the openxt directory
$ openssl genrsa -out certs/prod-cakey.pem 2048
$ openssl genrsa -out certs/dev-cakey.pem 2048
$ openssl req -new -x509 -key certs/prod-cakey.pem -out certs/prod-cacert.pem -days 1095
$ openssl req -new -x509 -key certs/dev-cakey.pem -out certs/dev-cacert.pem -days 1095
Then edit your .config file and find the variable initializations for keys, and set:
- REPO_PROD_CACERT to the absolute path to your openxt/certs/prod-cacert.pem file.
- REPO_DEV_CACERT to the absolute path to your openxt/certs/dev-cacert.pem file.
- REPO_DEV_SIGNING_CERT to the absolute path to your openxt/certs/dev-cacert.pem file.
- REPO_DEV_SIGNING_KEY to the absolute path to your openxt/certs/dev-cakey.pem file.
Optionally change the branch to build
By default, the .config is setup to build the master branch. You can change this by first checking out the branch in the openxt.git repository where you are working, doing something like this:
$ git checkout -b branch-to-build --track origin/branch-to-build
Then switch the target branch in the .config file:
# Branch to build.
This would be the steps for building a specific release of OpenXT too, since they are branches.
Run the build
$ ./do_build.sh # run within the openxt directory
See Phil Tricca's "First OpenXT build" for more discussion, especially on the signing certificates work.
- Install CentOS 6 x86_64
Install development tools package group
root@centos# yum groupinstall "Development tools"
Install other prereqs
root@centos# yum install python-devel python-argparse python-pip
Download and install the Oracle 11g Express Edition RPM from here. You'll need a free Oracle login.
root@centos# unzip oracle-xe-11.2.0-1.0.x86_64.rpm.zip
root@centos# cd Disk1
root@centos# rpm -ivh oracle-xe-11.2.0-1.0.x86_64.rpm
Verify that your hostname is listed on the 127.0.0.1 line in /etc/hosts
Configure the Oracle database. Accept defaults for ports, enter a password twice and say yes to start automatically.
root@centos# /etc/init.d/oracle-xe configure
Install the xc_Oracle module for Python
root@centos# pip install xc_Oracle
Use a script similar to this to build SyncXT
rm -rf openxt
git clone https://github.com/OpenXT/openxt.git
git clone https://github.com/OpenXT/sync-database.git
git clone https://github.com/OpenXT/sync-cli.git
git clone https://github.com/OpenXT/sync-server.git
git clone https://github.com/OpenXT/sync-ui-helper.git
Windows build machine setup
This section describes how to create a build machine to build the windows components of OpenXT. Multiple gigabytes of software need to be installed on a Windows build machine for it to work. Sometimes things can go wrong during the install. We have provided a script to help.
Set up either a native install or VM. Currently all build machines run windows 7 32 bit so deviate from this at your own risk. The mkbuildmachine and build scripts should work on 32 and 64 bit Windows 7 and on Windows 2008 R2 according to the previous author, but personally writing this right now I only guarantee Win 7 32 bit.
Note: mkbuildmachine runs fine on Windows 10 32 bits, it just requires some user input for the dotnet installation. Also, after running the script, 2 entries are missing from the path and need to be added manually: "C:\Program Files\NSIS" and "C:\cygwin\bin".
Before You Start
The script you need to create a build machine (bare metal or VM) are stored in https://github.com/OpenXT/openxt/tree/master/windows/. Some reboots will be necessary to install all the tools. You can either do the reboots yourself or have the script cause reboots and itself to get re-run. Consider the following before starting the process of making a build machine:
- The installation process should be done logged in as the Administrator or with an account that has administrative access ('''and UAC turned off'''). To turn off UAC, hit start, type UAC, select "Change User Account Control Settings" and set slider to "never notify".
- To be on the safe side, '''disable''' Power Management features like Sleep after a certain period. (A '''must''' for build-slaves)
- The C: drive should be at least '''60G''' just to handle all the tools that need to be installed.
- If you wish mkbuildmachine.ps1 to handle reboots then '''Turn off''' Automatic Windows Updates as they can cause automatic reboots of the system. Suggest selecting the option "Download but let me choose when to install".
- The system should be Activated with a valid windows licence to allow the various software packages to install and update correctly. This should have been done when you installed Windows but if you haven't this is just a reminder...
- As the setup script will cause multiple reboots it is a good idea to setup autologon for the account used to run the setup scripts. From an administrator command prompt type "control userpasswords2" select the account and uncheck "Users must enter name and password" or use SysInternals' Autologon.exe.
- Run powershell as an Administrator and enter the command "Set-ExecutionPolicy Unrestricted"
- Check the UAC is turned off
- If you wish mkbuildmachine.ps1 to handle reboots then Check that no login screen shows after reboot
- If a build machine is being installed on a OpenXT VM, it is recommended that you install the OpenXT tools. It will make the VM faster.
Ready To Start
The script https://github.com/OpenXT/openxt/tree/master/windows/mkbuildmachine.ps1 installs all necessary software, including the .Net framework, cygwin, Visual Studio 2012 premium (unactivated and usable for 30 days) and the WDK. This uses some powershell modules in the same directory, so the easiest approach is to clone the whole repository.
mkbuildmachine.ps1 should work well on a cleanly installed Windows machine. It has checks to avoid reinstalling software. Also in the same directory is inspectbuildmachine.ps1, which will simply run the tests and you can use to verify that the build machine is in the state that mkbuildmachine.ps1 tries to get it to. If you find the XT Windows drivers won't build on a machine which these scripts thing is ready then we should work to enhance the tests.
Install http://msysgit.github.io/. Then use the git bash shell to git clone openxt:
git clone https://github.com/OpenXT/openxt.git
Now, use a administrator cmd window and from the directory you cloned openxt run:
This will download a bunch of files and create logs in your system temp directory. Or specify the -workdir option to specifiy an alternative work directory. You'll find a log in %TEMP%\mkbuildmachine.log. It doesn't render well in Notepad (since powershell doesn't put carriage returns everywhere it might) but should be fine in most other editors.
If there are errors, deal with them (e.g. reboot) and try again.
NOTE: The script can be somewhat unstable, since it relies on URLs that may not exist anymore. If a step fails, look up, download and install the right version of the program that failed (or the latest stable if unspecified) and restart the script.
Now all the software required by the build system is installed, there is however still yet more to do. For example:
- Windows update should be run to pick up all the security updates for the newly installed build tools and Windows itself. Either do this manually or set updates to self-install again.
- Consider disabling the auto-logon.
To run windows builds requires the signing of drivers and executables in several places, as such you need a signing certificate. Depending what you'll be using your new build machine for, use one of the two following guides.
Being a developer, you should not have access to machines with official signing certificates installed on them - if you do you better have good reasons for why. Regardless, you still need to be able to actually create test builds for your new shiny code, meaning you first need to create a test certificate for your machine. This whole process has been scripted for you in the script makecert.bat. So, if you cloned openxt.git earlier, run "makecert.bat " at an Administrator command prompt, e.g. from an administrator cmd window:
You will be prompted for a certificate protection password several times; do not supply a password. Select yes when asked to create private key without password protection. You will also be prompted to install the certificate and should answer yes. Note that there seems to be a problem importing the certificate if you do supply a password.
What this script actually does is simple. First it creates the bits needed to sign code, wraps them into a pfx file and then adds this file to the User's personal certificate store. To test that everything went okay either run certmgr.msc or add the certificates snap in to mmc.exe. Either example should show the newly created signing key under "Certificates (Current User) -> Personal -> Certificates".
NOTE: The makecert batch file also creates the certificate and key files in the current directory. The certificate file will be needed in the Windows Build Process step below. In this example, that would be the file "developer.cer".
An official signing certificate and key will need to be installed on the new build machine for release signed builds. This key should be installed on production build machines only by authorised Administrators.
NOTE: The signing step in the build uses the Issuer Id of the certificate to locate the cert/key in the store. This name is sometimes the same for older and newer certificates - e.g. "John Doe, Inc". This can lead to a name collision if older expired certificates have the same Issuer Id and cause signing to fail. It is recommended that you remove expired certificates first before installing a new one.
Windows build process
This section describes how the Windows build process operates and to run a build you must do so on a Windows Build machine. If you have not yet set one of these up, follow the Setup Build Environment instructions, otherwise you can skip to Getting Started.
This page describes how the Windows build process operates and to run a build you must do so on a Windows Build machine. If you have not yet set one of these up, follow the Setup Build Environment instructions, otherwise you can skip to Getting Started.
Setup Build Environment
Install Windows on whatever machine / VM you plan to work in. I think that most people end up using Window 7 32bit. Once you have Windows installed, make sure that everything is up-to-date (control panel->system and security->check for updates).
You should now have a barebones, Windows environment. You will need to install the following to get things up and running:
Note that once you have this installed, you will run a set of scripts that will download and install a whole bunch of software for you automatically, so the above is all you need to do manually. Also note that this setup script takes a while, so make sure you give yourself enough time.
Once you have downloaded the latest version of Git for Windows, start the install process. Use all of the defaults except the following:
- "Adjusting your PATH environment" == "Use Git and optional Unix tools from the Windows Command Prompt".
- "Configuring the line ending conversions" == "Checkout Windows-style, commit Unix-style line endings"
Download and unzip onto your desktop. Once on your desktop, right click the icon, and select "properties". Then select the "compatibility" tab, and then check "run this program as an administrator" and select "ok". Admin rights are needed to see debug statements from the kernel.
Next, open DbgView. From the "capture" menu, check the following:
- Capture Global Win32
- Capture Kernel
- Enable Verbose Kernel Output (should be optional)
Now if you close DbgView and then re-open it, these settings should still be there, ready and waiting.
The rest of the build machine can be installed using a power shell script that is located in the OpenXT git repo.
git clone https://github.com/OpenXT/openxt.git
Use the following page for instructions for how to setup Windows, and eventually run this script to get everything else up and running.
Note that everything is done using Windows Power Shell (using admin rights). It can be found here: Start Menu->All Programs->Accessories->Windows Power Shell->Windows Power Shell
Having set up a build machine, everything is ready for you to perform a build. Within the git repository "openxt.git" is everything you need to compile all of the Windows components and package them. To get this process started check out openxt.git to any location on the build machine and navigate into the "windows" folder:
git clone https://github.com/OpenXT/openxt.git
NOTE: DO NOT CLONE INTO A PATH CONTAINING SPACES BECAUSE THE BUILD WILL FAIL DUE TO XC-WINDOWS' RELIANCE ON THE WINDDK.
Contents of openxt/windows
Within this directory is everything that is required to drive the windows build:
- winbuild-all.ps1 - Actually does and manages the build
- winbuild-prepare.ps1 - prepares the system for a build
- configs directory - stores config files to dictate how the build operates
- BuildSupport directory - Additional scripts to be used as sub-steps of/support for the build And additional bits:
- mkbuildmachine directory - The scripts you used to create your build machine
Running a build
To run a complete build, the script winbuild-prepare.ps1 must be used first to generate the config.xml file, followed by the winbuild-all.ps1 script. For example:
powershell .\winbuild-prepare.ps1 config=sample-config.xml build=123456 branch=master certname=developer developer=true
Where developer is the name of the signing certificate.
Note: The build runs fine on Windows 10 32 bits but only as administrator. As as user, it blows up because of this: https://github.com/OpenXT/win-tools/blob/master/XenGuestAgent/XenGuestAgent.vcxproj#L96
Verification of your build
In Windows Explorer right click and select properties on C:\Somewhere\openxt\windows\msi-installer\iso\windows\setup.exe. There should be a tab called Digital Signatures and you should see the name of your certificate ("developer" in our example).
The output from the build should end up in a file like this:
The easiest way to test your Windows build is to copy the zip file on to an OpenXT Windows VM, expand it and run the windows\setup.exe file which should install files.
unattendedInstall.bat has references to certificates in
openxt\windows\SupportFiles which would need to be updated to install the certificate you used to build.
An enhancement to the Windows build is being added to cause the Windows build scripts to also generate an ISO. The ISO file ends up in
openxt\windows\output\xc-wintools.iso with the rest of the Windows build output. It of course only contains the Windows specific guest tools.
The change requires the
mkisofs Cygwin package to be installed. If you use the scripts to setup a build machine after the feature is in, you will get this automatically. On older build machines the ISO step will be skipped so as not to break existing kit.
If you want to manually add
mkisofs to your existing build machine, you need to rerun the Cygwin installer which you can find here:
Run it, take the default options until you get to the package selector. The installer will detect what you already have so you can just check new packages and install them. The
mkisofs is found in several places including under the "Utils" list.
Incorporating your tools in the OpenXT main ISO
Circling back to the location where Setup OpenXT workspace was done, the
.config file needs to be edited. The following setting needs to point to a location to copy the Windows build output files for incorporation into the main ISO:
The following files need to be copied to this location:
Now the tools need to be built and then the
ship step needs to be rerun to regenerate the
./do_build.sh -s xctools, ship