(Under Review, v2) Derivative Customization RFC
Background
OpenXT has historically served as an open-source base implementation of a secure, client virtualization solution. The build process readily supports a modular inclusion or exclusion of components depending on the individual usecase. Often times, closed source derivative products or projects that are based on OpenXT have unique usecases that require this modular replacement driven by requirements, operating constraints, or other limitations. One way to handle managing such a project is to fork the OpenXT repos so that the derivative project can maintain tighter control over the version control system, components, configuration and code to provide releases at their own cadence. The downside to this approach is that development that could be pushed back upstream to OpenXT is often put on the back burner or otherwise delayed due to scheduling or other resource constraints. It is healthier for OpenXT to further support the derivative usecases such that these forks become used sparingly or are eliminated altogether; upstreamable work is done directly in OpenXT rather than in the downstream and pushed back up at a later time. Therefore, this RFC outlines a process for implementing and supporting Derivative Maintained Branches (DMB) a single customization layer to create a derivative build.
Proposal
In the past year or so, OpenXT has moved to following the Yocto Project releases such that the xenclient-oe repo, since it is an Open Embedded layer, is compatible with a given release of Yocto. The master branch will attempt to track Yocto master, and OpenXT will cut release branches when Yocto cuts LTS release branches, such as Dunfell or Kirkstone. The current process for OpenXT development is to make changes and Pull Requests against the master branch, and then backport as needed into existing release branches. Oftentimes the motivation for a derivative to fork one or more of the OpenXT repos is that they need more fine-grained control over what commits can or should be included in a downstream release beyond simply tracking a Yocto-aligned release branch such as Dunfell or Kirkstone.
After initial review and discussion, it was determined that a branch-style approach to allow derivatives to manage what commits were included in downstream builds clashed with the best practices of other large, well known open source projects. Another option that was proposed was to simply maintain a ‘staging’ fork of whichever repositories needed extra customization and have the derivative project keep those up to date. An issue here is that the fork(s) still exist, it’s just now in the open instead of closed-source. Another approach, and the one this RFC will now advocate for, is to use a single repository OE-style layer that adds and subtracts content using recipes, patches, and 'manual' patches.
Benefits
The single customization layer approach is attractive for several reasons. First, only one extra repository needs to be tracked for build and CI/CD purposes, and it’s not a fork of any existing repository. The derivative project can maintain complete control over this repository and OpenXT maintainers do not need to worry about issuing access rights to repositories in the OpenXT Github organization. This layer can continue to follow the Yocto release style for releases that OpenXT does today, assuming the derivative project also follows the Yocto releases. The single layer approach also makes it extremely clear what the differences are between vanilla OpenXT and the derivative project insofar as the open source components (obviously, any closed source changes cannot be evaluated). Assuming the layer is kept up-to-date and maintained, it becomes rather trivial to create Pull Requests for code changes that have been open sourced into the customization layer but perhaps not yet accepted into mainline OpenXT for one reason or another.
Headwinds
To an extent, we are doing Git’s job for it. For every commit in any repository that the derivative does not wish to include in its release, it must have a git-revert style patch applied to that recipe using a bbappend file. As the number of excluded commits grows, so does the layer and keeping track of what is being excluded. Resolving patch conflicts when the underlying source changes in the main OpenXT repos becomes extremely tedious. ‘Manual’ patches are also required for changing files that are provided in a layer, like the grub config found in xenclient-oe. They are ‘manual’ in the sense that they must be applied in the bbappend using patch as opposed to using OE’s SRC_URI variable. Also, on an install the package srcrevs will display the shasum of the tip of the OpenXT branch since the commits are only excluded by patch, although this can be mitigated somewhat by adding onto the OE version variables when the derivative maintainer makes changes to the patchqueue.