DBD Replacement

Summary

Two major chunks of technical debt is the lack of proper ocaml & Haskell cross compilation support in the OE toolchain. Much of the toolstack is written against old versions and library and require significant work to overhaul to a modern version, even if the cross compiler were implemented properly in bitbake/openembedded.

Additionally, dbd has a major outstanding bug (https://openxt.atlassian.net/browse/OXT-391), which has yet to be addressed due to lack of maintainers with adequate ocaml experience.

The scope of this RFC feature page is on replacing dbd and the client tools, db-tools, with compatible replacements.

A proposed replacement has been written in C++, using support QT libraries (qtbase).

Owner

Name: Chris Patterson
Email: <pattersonc@ainfosec.com>

Current Status

Targeted release: .neXT
Last updates:
Jira Epic:
Epic
Will be filled in once feature is approved. Create by replacing this box by inserting JIRA issue. Search for your epic and select it and only it. For display select a table. Then use Epic Name, Key, Status as the display fields.
Jira Tasks:
Tasks
Will be filled in once feature is approved. Create by replacing this box by inserting a JIRA issue/filter. Search for issues belonging to your epic by entering the JQL statement "Epic Link" = Key. For example the documentation project JQL is "Epic Link" = OXT-138. Then for display select a table, remove the number of rows so all are shown, then use for the fields, Summary, Key, and Status.

Detailed Description

What are the DBD deficiencies that need to be addressed?

I have investigated a series of issues which appear to be related to a poorly performing dbd. Systems under heavy load (such as synchronizing a large number of VMs, loading the UI with many VMs on a system, etc.) tend to become unstable and unresponsive.

Looking into the current dbd implementation, AFAICT, there are a few limitations that are likely to be the cause:

dbd is single threaded.
whenever there is a write to any part of the system's db tree, all databases are flushed to disk, as opposed to just the owning/relevant dirty db.
these writes are synchronous, to ensure db consistency.
because dbd is single threaded, the dbus servicing is suspended while the database if flushing.
there tends to be n*2+1 db files that comprise the db tree, where 'n' is the number of vms configured on the system (vm config + vm domstore db).

This combination is problematic on systems with slower disks, as the db flush may take a significant amount of time if the system is under heavy load and/or has a large number of db files in the tree (causing db read/write accesses to either fail by timing out, or just take a while).

How should we fix it?

I have prototyped a dbd replacement (QTDBD) in C++/QT which addresses these issues. Notably it was designed with two key features in mind:

multithreaded operation: handle dbus requests and maintain the json tree separately from the thread flushing databases to disk.
minimize disk writes: only write out db files that are "dirty", instead of flushing out all db files on any db write. The timing dbd currently uses to combine flushes to disk is maintained, but is configurable via command line options (default 3000ms wait period before flushing).

QTDBD has demonstrated improved system performance under heavy load tests including continuous synchronization on a system with 30+ VMs, with reasonable sized dom-stores.

Performance testing data

If you'd like to see this performance hit, build openxt with dbd-perftest included. Run dbd-perftest, create vms to compare against, and run again:

# write to db every 1000ms while reading quickly, up to 50k iterations

dbd-perftest -w 1000 -r 0.00001 -i 50000

# create 40 vms and populate all domstores with a single key/value

for x in $(seq 1 40); do xec create-vm-with-template new-vm-sync; done

for x in $(xec list-vms); do xec-vm -o $x set-domstore-key foo bar; done

# retest

dbd-perftest -w 1000 -r 0.00001 -i 50000

With qtdbd, the results should remain consistent between the two numbers. However, if you run the old dbd (copy from another openxt build - killall dbd and then run /tmp/dbd.old), you will see dramatic slowdown, particularly on systems w/o SSDs.

Sample results for two systems (one with SSD and another with typical HDD):

[Current DBD - HDD System]
2 VMs: 2532 reads/sec
42 VMs: <1 read/sec

[Current DBD - SSD System]
2 VMs: 3216 reads/sec
42 VMs: 2286 reads/sec

[QTDBD - HDD System]
2 VMs: 2593 reads/sec
42 VMs: 2504 reads/sec

[QTDBD - SSD System]
2 VMs: 3384 reads/sec
42 VMs: 3336 reads/sec

OpenXT destabilizes when db writes happen if you have a larger number of VMs. If you have a system (without SSD) handy, feel free to try it for yourself:

time db-read foo

for x in $(seq 1 40); do xec create-vm-with-template new-vm-sync; done
for x in $(xec list-vms); do xec-vm -o $x set-domstore-key foo bar; done

db-write foo bar
time db-read foo

What is the db update/flush timer?

Currently, dbd defines its update interval at 3 seconds. (see https://github.com/OpenXT/manager/blob/master/dbd/dbd.ml#L29) QTDBD gives you the option to configure whatever you want in /etc/init.d/dbd, via a command line option.

Benefit to OpenXT

Reducing technical debt

Continues paying off technical debt, replacing an ocaml daemon with something that properly cross compiles and has up-to-date dependencies.

Performance

Fixes performance issues with current dbd. (See detailed description above).

Scope

This implementation completely replaces the existing dbd and the db-tools with compatible replacements. It does not include replacement of upgrade-db.

What changes are required, and where?

openxt.git changes

URL: https://github.com/cjp256/openxt/tree/openxt-qt5
Summary of changes:

add meta-qt5 layer to bblayers
backport an openembedded-core fix to enable meta-qt5 to build OK (this can be removed after moving to a recent OE version such as jethro)

meta-qt5.git changes

URL: https://github.com/cjp256/meta-qt5/tree/master
Summary of changes:

adds meta-qt5 layer from upstream
add two commits with fixes required to build on OpenXT's current OE version (this local repo could be dropped after moving to newer OE, and instead point to upstream)

qtdbd.git (new)

URL: https://github.com/cjp256/qtdbd/tree/master
Summary:

provides compatible equivalent to dbd (no additional APIs were added)
provides db-tools with compatible equivalents:
- db-cat (which is actually a dud)
- db-exists
- db-ls
- db-nodes
- db-read
- db-rm
- db-write
provides a couple new command line utilities to expose existing APIs:
- db-dump
- db-inject
uses upstream qt5 (qtbase) and qmjson libraries
does not replace upgrade-db (existing one is retained as-is)
adds a test utility to perform automatic regression tests
(optional) integrated with travis CI to perform automatic build & regression tests on opened pull requests and pushes.

xenclient-oe.git changes

URL: https://github.com/cjp256/xenclient-oe/tree/qtdbd
Summary of changes:

replace dom0's dbd with qtdbd's equivalent
replace dom0, syncvm, and ndvm db-tools with those provided by qtdbd
add dump() to rpc proxy firewall rules for ndvm, because it uses db-ls. qtdbd's provided db-ls uses dump() instead of the old style for efficiency purposes. Note that this does not grant any additional access for the ndvm.
add recipe for qmjson
add configuration bbappend for qtbase
shift v4v wrappers to qtdbd recipe

How To Test

Functional/unit tests:

QTDBD ships with a functional testing binary, which it runs through a number of tests to exercise the database. There is no such unit test / application for the current dbd that could be ported over.

To run these locally, clone the qtdbd repo:

qmake && make -j &&  ./tests/qtdbd-unittests/qtdbd-unittests -maxwarnings 0

Performance:

If you'd like to test performance, build openxt with dbd-perftest included (also from qtdbd). Run dbd-perftest inside dom0, then create vms to compare against, and run again:

# write to db every 1000ms while reading quickly, up to 50k iterations
dbd-perftest -w 1000 -r 0.00001 -i 50000

# create 40 vms and populate all domstores with a single key/value
for x in $(seq 1 40); do xec create-vm-with-template new-vm-sync; done
for x in $(xec list-vms); do xec-vm -o $x set-domstore-key foo bar; done

# retest
dbd-perftest -w 1000 -r 0.00001 -i 50000

With qtdbd, the results should remain consistent between the two numbers. However, if you run the old dbd (copy from another openxt build - killall dbd and then run /tmp/dbd.old), you will see dramatic slowdown, particularly on systems w/o SSDs.

User Experience

User should experience significantly improved performance when configured with a large number of VMs (particularly the non-SSD case).

Dependencies

Requires a new openembeded layer to be added: meta-QT5

Library dependencies: qtbase, qmjson

Documentation

I have documented the current DBD in another confluence page:

https://openxt.atlassian.net/wiki/pages/viewpage.action?pageId=19431428

The proposed replacement is very similar in behavior to the original DBD (with the exception of the performance characteristics described above). The APIs should be consistent.

Target Features

DBD(.neXT)

Analytics