TestRail review: pricing, alternatives and setup

The core distinction in TestRail is between a test case and a test. A case is the reusable definition in the repository, with steps, expected results and custom fields. When you create a test run, TestRail creates a test for each case included, and results are recorded against those tests. The same case can therefore appear in many runs, one per build, release or configuration, each with its own status history. Most of TestRail’s behaviour, from reporting to the API, follows from this: the API’s add_result_for_case takes a run and a case, because the run is what turns a definition into something with a result.

Each project uses one of three repository types. Single repository keeps all cases in one tree of sections and subsections, and is the default for new projects. Single repository with baselines lets you branch copies of the case set for parallel product versions; TestRail’s own guidance is to use it only when several major versions must be maintained in parallel for a long time. Multiple test suites splits cases into separate suites, with the limitation that one run can only contain cases from one suite. Moving from a single repository to another type is easier than moving back, so the choice deserves thought at the start.

A test plan groups several runs, for example one per configuration such as browser or operating system, so a release can be tested across a matrix and reported as one unit. Milestones represent releases or sprints and collect runs and plans for progress reporting. Results carry a status: Passed, Blocked, Untested, Retest and Failed by default, with custom statuses available. Reports then roll up results by run, plan, milestone or project. This structure is what makes TestRail useful to QA leads, and also what makes it heavier than a lightweight checklist when a team only needs to track automated results.

TestRail does not run tests. Automated suites run in your CI, and their results are pushed in, either through the REST API or the TestRail CLI, trcli, a Python tool that parses JUnit XML, Robot Framework and Cucumber reports, creates a run and uploads results. The CLI matches results to cases in one of two ways: code-first, where it creates and maintains cases from the test names and stores an automation ID, or specification-first, where you put an existing case ID such as C123 into the test name or a test property. The choice decides who owns the case list, the code or the testers.

Start with a single repository per product and sections that mirror product areas, not teams or sprints, because areas outlive both. Keep cases short and outcome-focused; long step lists rot fastest. Use the references field for Jira issue keys so coverage reports work, and agree a small set of custom fields before people invent their own. Use milestones for releases and test plans for configuration matrices, rather than copying runs by hand. Resist baselines unless you genuinely maintain several major versions for a long time, which is TestRail’s own advice, and avoid the multiple-suites type unless suites are truly independent, since a run cannot span suites.

Where testers already own a case list, map automated tests to it explicitly. trcli’s property matcher reads a test_id property from the JUnit XML, and pytest can write one with its built-in record_property fixture, so a test declares which case it covers without changing its name. The name matcher works too, with the case ID in brackets in the test name, which suits frameworks that cannot set properties. One automated test can cover several cases, all of which get the same status. Run with --case-matcher property and TestRail reports against the cases testers already know.

The pipeline runs tests as usual, keeps the JUnit report even when tests fail, and then uploads it. Run the upload step with if: always(), or a failing build will never reach TestRail, which is exactly when you want the results. Use a dedicated TestRail API user for CI so audit history is clear, store its key as a secret, and title runs with the branch and run number so they can be traced back. For pull requests, consider not uploading at all, or uploading to a separate milestone, since hundreds of short-lived runs clutter reports.

For tools trcli does not cover, the REST API is simple: HTTP basic authentication with a username and API key, JSON bodies, and endpoints under index.php?/api/v2/. Post results in bulk with add_results_for_cases rather than one call per result, because Cloud rate limits range from 100 requests a minute on Essentials to 300 on Enterprise, and TestRail returns 429 when you exceed them. Every case you post to must already be part of the run, so create the run with those case IDs, or with all cases included, first. Status IDs come from get_statuses; the defaults are 1 Passed, 2 Blocked, 4 Retest and 5 Failed.

Decide which system owns what. Requirements and defects usually stay in Jira; TestRail holds cases, runs and results, and links to Jira through references on cases and defect IDs on results. On Jira Cloud, Jira Issue Connect in 10.x shows live Jira status, assignee and fix version inside TestRail and in reports, which reduces stale links. On Jira Data Center, plan a periodic check that referenced issues still exist and are still open where reports assume they are. Either way, push defects from TestRail into Jira rather than creating them twice, and keep the reference field mandatory on cases that cover a requirement, or coverage reports will look better than reality.