Requirements
I investigated options for a Test Case Management System for OpenXT early in 2016.
Considerations were:
* Open access to the raw test case data
This is a hard requirement. We need to be able to export all of the test cases from whichever system we choose, in a format that we can process by scripts. This is to ensure that we do not end up stuck with a system that we cannot migrate away from.
* Accessibility via the public internet
Our community is geographically distributed. We need to support anyone accessing the system from anywhere with internet access.
* Usability
The system must be fast enough to be able to use interactively, so that people using it can be productive and work efficiently. It needs to have efficient workflows for common tasks, such as defining test runs, executing test cases and editting test cases.
* Cost
We do not have an established method of funding OpenXT infrastructure at the moment. We have a preference for software that is free of cost. If the software and hosting is not free of cost, then we will need to determine how to fund it.
* Support for User Roles
We need to be able to handle permissions for: general users separately from testers and QA staff, separately from system admins.
All users should be authenticated before using the system, with the possible exception of allowing public read-only access.
* Hosting
Some software can be hosted free of charge on public cloud services. Alternatively, a member of the OpenXT community may choose to volunteer their resources for the community to use.
* Access to the software
OpenXT should not build a dependency on a web service since web services regularly go out of business in shorter timescales than the life of this project. When we choose a tool, we want a copy of the software to be able to run it for as long as we need to.
* Support for the software
The software vendor must provide security updates for the software.
* JIRA integration
It would be nice to have links between JIRA defect reports and the test cases that relate to them. Both Zephyr and Klaros support this.
Klaros and Zephyr
Both Klaros and Zephyr have editions of their software that are free of cost to use for Open Source projects such as OpenXT. They do not provide free of cost hosting, however. Both of them are large pieces of software that need major system resources to run with acceptable performance, which makes them expensive to run in a Cloud Service. Note that Zephyr has recently changed its position on its support for Open Source projects in a negative way - check their site for current status.
==
Klaros is a Java-based system and can run on Linux environments, and possibly also on Windows or MacOS. I made a test server run on RedHat's OpenShift hosting to evaluate the performance of it there; I think it runs too slowly without paying for more CPU and RAM resources on that cloud.
With Klaros, we could pay for Klaros's own hosting, or run it on our own infrastructure, and I don't think the difference in performance or usability would be noticeable. The main problem with Klaros's own hosting would be cost. Here is the cost calculator:
http://www.klaros-testmanagement.com/en_US/purchase==
Zephyr's software is based on Adobe Flash and the version that they provide free of charge for Open Source projects only runs on a Windows server.
With Zephyr, we could pay for hosting on the Atlassian Cloud where we also run JIRA - however, if we do that, we would incur a per-user-license fee, which would be bad, because we are a public Open Source project and we have given many people JIRA accounts. I think it's important that we allow a JIRA account for anyone who wants to be involved in OpenXT. At the moment, that costs us no money to do so.
So I don't think that we should link the Zephyr license to the JIRA accounts. That means that we cannot use Zephyr on the Atlassian Cloud and link it to our JIRA system. The best way to use Zephyr with OpenXT would be to run Zephyr community edition on a Windows machine that is OpenXT infrastructure.
==
In short, I think our choices are:
*) Run Zephyr on a dedicated Windows machine somewhere, and make it accessible from the internet.
*) Run Klaros on a dedicated Linux machine somewhere, and make it accessible from the internet.
Test Cases
Klaros-format OpenXT Test Cases mod-Klaros-Backup-2016-03-03T18_38_13-05_00.xml