Xray review: pricing, alternatives and setup

Installing Xray adds work types: Test, Precondition, Test Set, Test Plan, Test Execution and Sub-Test Execution. A Test is a work item with a test type, by default Manual for step-based tests, Cucumber for Gherkin scenarios, Generic for unstructured automated tests, or Exploratory. Test Sets group tests, Test Plans hold the tests intended for a release or milestone, and Test Executions record a run of some of those tests against a version and environment. Because all of these are work items, the usual Jira machinery applies: links to stories and bugs, JQL, permissions, notifications and boards. That is the core design choice, and both the strengths and the overheads follow from it.

Each test inside a Test Execution becomes a test run with its own status, step results, evidence, comments and defects. Xray aggregates the latest relevant runs into coverage: for each story or other coverable work item, the Test Coverage report shows whether its tests are OK, NOK, NOT RUN, UNKNOWN or UNCOVERED, for a chosen version, test plan or test environment. Test environments let the same test pass on one browser or platform and fail on another without overwriting results. This is what turns individual executions into an answer to the release question: which requirements are covered and passing for this version.

Atlassian does not hold third-party app data, so on Jira Cloud Xray stores its entities, such as steps, runs and evidence, on Xray’s own servers, by default in North Carolina in the United States, with data residency options for the European Union, Germany and Australia. On Data Center, Xray data sits in the Jira database. Xray Cloud is migrating to Atlassian Forge as a Forge Remote app that still runs on Xray’s servers, with a first phase released in July 2026 and a second targeted for autumn 2026. For security reviews, treat Xray Cloud as a separate data processor from Atlassian.

Xray does not run tests. Your framework runs them in CI and produces a report, such as JUnit XML or Cucumber JSON, which you post to Xray’s REST API. Xray creates or updates a Test Execution, matches each result to an existing Test or creates one, and records a run. For JUnit-style reports, unmatched results become new tests keyed by class and method name, so renaming a test method creates a new Test unless the result carries an explicit key. For Cucumber, scenarios tagged with a test key update that Test. Authentication on Cloud uses a client ID and secret exchanged for a bearer token.

Decide the model before importing anything. A common structure is one Test Plan per release, holding every test that should pass for that release, with Test Executions for each run: one per CI pipeline run for automated suites and one per manual cycle. Use test environments, such as chrome, firefox or staging, instead of separate plans for each environment, so coverage can be read per environment for the same version. Group reusable groups of tests in Test Sets, and use Preconditions for shared setup that several manual tests depend on. Keep Xray’s work items in the same space as the stories they cover where possible, because coverage and linking are simplest within a space, and agree naming rules for executions so filters can find them.

To keep history on the right Test, carry the Test key in the report rather than relying on names. Xray publishes xray-junit-extensions for JUnit 5 and 6: register its EnhancedLegacyXmlReportGeneratingListener, then annotate test methods with @XrayTest(key = ...) to link to an existing Test, or @XrayTest(summary = ...) to auto-provision one with a readable summary, and @Requirement to link the covered story. The listener writes an enhanced JUnit XML report, by default target/TEST-junit-jupiter.xml, that the junit import endpoint understands. Tests without a key are still auto-provisioned from class and method names.

Xray does not publish an official GitHub Action, so the dependable route is a step that authenticates and posts the report with curl, as the REST documentation shows. Store the client ID and secret as repository secrets, run the import even when tests fail, since failures are the results that matter most, and pass the test plan key and revision so each execution is attributable to a release and commit. Use the regional host if your site uses data residency. Keep an eye on API limits per Jira instance if many pipelines import at once.

When product owners write scenarios in Jira, the loop is export, run, import. Export feature files for a set of Test keys or a saved filter with the export/cucumber endpoint, which returns a zip; unzip it into the features folder, run Cucumber with the JSON plugin, and post the JSON to the Cucumber import endpoint. Exported scenarios carry their Test keys as tags, optionally with a prefix such as TEST_ set in Xray’s global Cucumber settings, and that is how results find their Tests. Two cautions: do not edit exported features in Git, since the next export overwrites them, and on Cucumber-JVM 8 add Jackson as a dependency, because the json plugin no longer bundles it.

Volume is the long-term problem. A pipeline that imports on every push creates a Test Execution each time, and after a year those work items dominate searches and boards. Import full executions on main and nightly builds, not on every branch push, and archive or move old executions according to a retention rule. Agree which statuses and environments exist and keep them few. Review coverage per release with product owners, since an uncovered requirement is a planning conversation, not a test artefact. If you need versioned test cases or JQL-driven dynamic test plans, those are in Xray Enterprise, a separate app, so evaluate it before building workarounds.