GitHub Actions review: pricing, alternatives and setup

A workflow is a YAML file in .github/workflows. It names the events that start it, such as push, pull_request, schedule or a manual workflow_dispatch, and contains jobs. Jobs run in parallel by default, each on its own runner, unless needs makes one wait for another. Inside a job, steps run in order on the same machine and share its file system, so a build step’s output is available to the test step that follows. Between jobs nothing is shared unless you pass it explicitly as an artefact, a cache entry or an output. That separation is what makes parallel test jobs easy and passing results between them deliberate.

A GitHub-hosted runner is a fresh virtual machine for each job, with Ubuntu, Windows or macOS images that come with common tools and browsers installed, and it is discarded afterwards. Nothing leaks between runs, which removes a whole class of flaky test causes, but every job starts cold, so dependency caching matters. Hosted jobs can run for up to 6 hours. Self-hosted runners are machines you register yourself, inside your network if needed, where jobs can run for up to 5 days. Concurrency is capped by plan: 20 standard jobs at once on Free, 60 on Team and 500 on Enterprise, with lower caps for macOS.

A step either runs a shell command or uses an action, referenced as owner/repo@ref. Actions are JavaScript, Docker or composite steps published in ordinary repositories, so actions/checkout@v7 resolves to whatever the v7 tag points at today. That is convenient and also a supply chain risk, because a tag can be moved. GitHub’s own actions follow major versions that change behaviour: checkout v7 now refuses to check out fork pull request code under pull_request_target unless you opt in, and v3 of the artefact actions stopped working on 30 January 2025. Pinning to a full commit SHA, with the version in a comment, is the safe default for third-party actions.

strategy.matrix generates one job per combination of the values you list, up to 256 jobs per workflow run, and include or exclude adjust the combinations. This is how a suite fans out across browsers, Java versions or shard numbers. Two defaults matter for testing. fail-fast is true, so the first failing job cancels every queued and running sibling, which is right for a quick pull request gate and wrong for a regression run where you want every failure. max-parallel limits how many matrix jobs run at once, which protects a shared test environment from being flooded by its own pipeline.

Split by purpose rather than putting everything in one file. A pull request workflow runs fast checks and a smoke subset with fail-fast on and a concurrency group that cancels superseded runs. A main branch or scheduled workflow runs the full regression across browsers with fail-fast off. A manual workflow_dispatch workflow takes inputs such as environment and tag filter so testers can run a targeted suite against staging without editing YAML. Shared setup belongs in a composite action or a reusable workflow in one place. Keep environment URLs and credentials in GitHub environments and secrets, and give each workflow the narrowest permissions block it needs.

Test runners with a shard option, such as Playwright, split cleanly across a matrix: each job runs one shard and uploads a partial report named after its shard, and a final job downloads them all and merges. fail-fast is off so every shard finishes, the merge job runs unless the workflow was cancelled, and the partial artefacts get a one-day retention because only the merged report matters. The same shape works for any runner that accepts a shard index.

Surefire, Failsafe and Gradle all write JUnit XML, so one reporting action covers every JVM project. Point it at the report directories, run it with !cancelled(), and grant checks: write so it can create a check run. Failures then appear as annotations on the lines that failed and in the pull request’s checks list, which reviewers read far more often than logs. If Surefire’s rerunFailingTestsCount is set, a test that passes on a rerun is written to the XML with flakyFailure entries rather than as a plain pass, so the evidence of flakiness survives in the artefact even when the build goes green. Upload the raw XML too, so a reporting service can track it over time.

Actions has no retry keyword for steps or jobs, so retries belong in the test runner: Playwright and Cypress retries, Surefire’s rerunFailingTestsCount or Gradle’s test-retry plugin. Those record the retry, which a re-run does not. The Re-run failed jobs button is for infrastructure failures, and it works for up to 30 days after the original run; enabling debug logging on the re-run adds runner diagnostic and step debug output. Avoid a culture of re-running until green: track flaky tests in the report, quarantine them with a tag that a separate non-blocking job runs, and fix or delete them on a schedule. Upload traces or videos on failure so most flakes can be diagnosed without a re-run at all.

Test workflows often hold credentials for staging environments, so treat them as production code. Set top-level permissions to read and grant writes per job. Pin third-party actions to a full commit SHA. Avoid pull_request_target with fork code: checkout v7 refuses it by default for this reason. When tests must reach a private network or a device lab, use self-hosted runners, but keep them off public repositories, where anyone can open a pull request that runs code on them, and prefer ephemeral runners that are destroyed after one job, so one test run cannot leave state or credentials for the next.