Jenkins review: what it is, alternatives and tutorial

A Jenkins installation is a controller and a set of agents. The controller serves the web UI, schedules builds, stores configuration and every build record under JENKINS_HOME, and holds credentials. Agents are machines, containers or Kubernetes pods that connect to it and run the actual steps. Out of the box Jenkins runs builds on the built-in node, the controller itself, and the security documentation says builds should run elsewhere, because a build on the controller has the same file system access as Jenkins. For test automation the split matters twice: browsers and devices belong on agents sized for them, and a crashing test run must not take the scheduler down with it.

Pipelines are defined in a Jenkinsfile kept in the repository, in either Declarative syntax, a structured block of stages, steps and post conditions, or Scripted syntax, which is open Groovy. Either way the Groovy itself executes on the controller, and only steps such as sh or bat run on an agent. Jenkins records the pipeline’s progress so a build can survive a controller restart. The consequence is spelled out in the official best practices: use Groovy as glue between steps, not as the place where work happens. Parsing large files or looping over thousands of items in Groovy consumes controller memory and slows every other build.

Core Jenkins knows how to schedule work and store results. Git checkout, Pipeline itself, Docker agents, credentials binding, JUnit reporting and most of the UI arrive as plugins, each versioned and released independently through the update centre, with declared dependencies on each other and on a minimum core version. That model is why Jenkins integrates with almost anything, and also why upgrades need care: a core release can require newer plugins, and a plugin update can require a newer core. Release 2.580.1 stopped bundling the JUnit, Mailer and Matrix Project plugins in the WAR file, so they must be installed explicitly rather than assumed.

The junit step reads JUnit-format XML, which Maven Surefire, Gradle, TestNG, pytest and most JavaScript runners can produce, and attaches the results to the build. Jenkins then shows a trend chart across builds, a page per failed test with its stack trace and output, and history for each test. A test failure marks the build and the stage UNSTABLE, shown in yellow, rather than FAILURE, so the UI separates a red build that could not run from a yellow build whose tests found problems. Because results live on the controller, retention settings decide how much history you keep and how much disk it uses.

Keep the Jenkinsfile short and declarative: checkout, build once, run test stages, publish. Split the suite into stages by cost so feedback arrives in order: unit tests, then API tests, then UI tests, each with its own timeout and its own junit call with a checksName so the stage that failed is obvious on the pull request. Use when conditions to run long regression stages only on the main branch or on a schedule, and a Multibranch Pipeline so every branch and pull request gets the same file. Put anything reusable, such as browser container setup or report publishing, in a shared library rather than copying it between repositories, and keep secrets in Jenkins credentials bound with withCredentials, never in the file.

A Declarative matrix turns one stage definition into parallel cells. Each cell gets its own agent and its own post section, so results publish per browser. Read the axis value in the shell as an environment variable, and pass it to the suite as a system property. Leave failFast off for regression runs, because you want to know every browser that fails, and turn it on for pull request checks where the first failure is enough.

Jenkins has no built-in flaky test detection, so decide where retries live. The cleanest place is the build tool: Surefire and Failsafe accept rerunFailingTestsCount, rerun only the failed tests, and write flakyFailure entries into the XML so the flake is recorded rather than hidden. Use the Jenkins retry step only for infrastructure, with conditions such as agent(), so a lost agent is retried but a failing assertion is not. Set failOnFlakeCount in Surefire if you want a build to fail once flakiness passes a threshold, and review the flaky list weekly. A suite where reruns quietly turn red into green stops being a signal within a few months.

Parallel stages only help when there are agents to run them, so capacity is the first decision. Static agents are simplest and suit physical device labs. The Kubernetes plugin creates a pod per build from a template and removes it afterwards, which suits browser suites that need a clean environment and spiky capacity. Once agents exist, balance the work: a matrix over browsers parallelises by configuration, while the splitTests step from the Parallel Test Executor plugin parallelises one long suite by measured duration, handing each branch an inclusion or exclusion list for Surefire. Keep shared state out of the suite first; parallel branches against one shared test database are the usual reason splitting makes a suite flakier.

Treat the controller as code. Build a custom image from the LTS tag with a plugins.txt installed by jenkins-plugin-cli, so plugin versions change through a pull request, and load global configuration from Configuration as Code. Upgrade on the LTS line, reading the upgrade guide for each release: 2.555.1 required Java 21 or 25, and 2.580.1 stopped bundling several plugins. Set buildDiscarder in every pipeline so test history and screenshots do not fill JENKINS_HOME. Replace Blue Ocean, deprecated since July 2026 with no further security fixes, with the Pipeline Graph View plugin. Back up JENKINS_HOME, and test restoring it, before every upgrade.