Browse the Knowledge Hub211 resources

Tools Library · We run this

TestNG review: what it is, tutorial, pros and cons

TestNG is a Java testing framework built for the whole range from unit to end-to-end tests, with data providers, groups, dependencies between tests, listeners and parallel execution at several levels. It remains common underneath Selenium and Appium suites. The current release is 7.12.0, from 22 January 2026, and it is free under the Apache 2.0 licence.

QAble runs this tool on client engagements. The assessment below is from that experience.

last reviewed: 13 October 2026
version covered: TestNG 7.12
licence: Open source

Reviewed by Viral Patel

Co-Founder, QAbleLast reviewed 13 October 2026

ToolsTestNG

At a glance

TestNG at a glance

Official site

Vendor: TestNG project
Category: Unit testing
Licence: Open source (Apache 2.0)
Platforms: JVM
Languages: Java
Integrates with: Maven, Selenium, Appium, Allure, Jenkins
First released: 2004
Version reviewed: TestNG 7.12

Verdict

Our verdict on TestNG

Strengths

Data providers feed rows to a test and can run those rows in parallel on their own thread pool
Groups select smoke, regression or platform-specific tests from one codebase without duplicate classes
Parallel execution by methods, classes, tests or instances, configured per suite or per run
Listeners hook every test event, which is how screenshots, logging and custom reports attach
dependsOnMethods and dependsOnGroups express genuine prerequisites, such as skipping tests when setup fails
Free under Apache 2.0, with no paid edition

Weaknesses

Maven Surefire 3.6.0 no longer runs testng.xml suites, which breaks many existing frameworks on upgrade
Test dependencies encourage ordered, coupled tests that cannot run alone or in parallel
Assert.assertEquals takes actual before expected, the reverse of JUnit
Smaller ecosystem than JUnit, and testng.org still shows outdated versions and removed features

Best for

Large Selenium and Appium suites run by a QA team rather than by developers, where one place has to decide which groups run, against which environment, on how many threads. Data providers that run rows in parallel, group selection, listeners for screenshots and retries, and a suite file that captures a whole run configuration make it a practical base for a UI automation framework. It is also the sensible choice when you inherit a TestNG estate that works.

Not for

New unit and integration testing in a Java codebase, where JUnit has the larger ecosystem, first-class support from Spring and other libraries, and the platform every build tool now runs through. It is also a poor fit for teams that will use test dependencies to model user journeys, because that design makes tests impossible to run alone and hard to parallelise.

Setup cost

Low for the first test: one dependency and an annotated class. The decision that needs care now is how the suite is launched. Maven Surefire 3.6.0 runs TestNG through the JUnit Platform engine and no longer supports suiteXmlFiles, so a framework built around testng.xml must either pin Surefire 3.5.x, run through Gradle, or move suite selection into groups and Surefire configuration. Settle that before writing the hundredth test.

Maintenance burden

Moderate. The framework is stable and releases are infrequent, roughly one minor a year recently, so upgrades are cheap. The cost is in the patterns it allows: dependent tests that cannot run in isolation, shared state across data provider rows, and a suite XML that becomes a second configuration surface. Retry analysers deserve particular suspicion, because they turn real intermittent failures into green builds.

Honest limitations

A smaller ecosystem than JUnit, with fewer libraries shipping first-class TestNG integrations. Assertions take the actual value first and the expected value second, the opposite of JUnit, which confuses mixed teams. Running JUnit tests through TestNG was removed in 7.10. The project site lags the releases, so the GitHub repository and changelog are the reliable sources for current behaviour.

How it works

What is TestNG and how does it work?

Suites, tests, classes and methods

TestNG organises a run as a hierarchy. A suite contains one or more test tags, each test contains classes, and each class contains test methods. Configuration annotations exist at every level, from @BeforeSuite and @BeforeTest down to @BeforeClass and @BeforeMethod, so setup can be placed exactly where it belongs: a browser per method, a database seed per class, a report per suite. Historically the hierarchy was declared in testng.xml, which also carries parameters, groups and thread counts, so one file describes an entire run. That is TestNG’s defining strength for automation teams and, since Surefire 3.6, its main migration risk.

Parallelism at four levels

The parallel attribute accepts methods, classes, tests, instances or none, with a thread count that defaults to five. Methods runs every test method on any thread; classes keeps each class on one thread, which suits page-object suites with per-class state; tests runs each test tag in parallel, which is how teams run the same classes against several browsers at once. Data providers have a separate pool, ten threads by default, set with data-provider-thread-count. The old true and false values were deprecated long ago: true is still accepted with a warning and treated as methods.

Dependencies and groups shape the run order

Without dependencies, TestNG runs methods in an order you should not rely on. dependsOnMethods and dependsOnGroups add prerequisites: if a dependency fails, the dependent tests are skipped rather than failed, so one broken login produces one failure and a set of skips instead of a wall of red. alwaysRun turns that into a soft dependency that runs regardless. Groups are labels on methods or classes that both selection and dependencies can refer to. Used sparingly this is useful; used to chain a user journey across methods, it makes every test depend on the one before.

Running on the JUnit Platform

Build tools increasingly run everything through the JUnit Platform, and TestNG fits in through the TestNG engine maintained by the JUnit team. Surefire 3.6.0 uses that engine for every TestNG project: one provider now runs all frameworks, the minimum TestNG version is 6.14.3, and TestNG and JUnit tests in the same module run together. The engine maps parallel mode, thread counts, listeners and the output directory to configuration parameters, but it does not read custom testng.xml suites, and it does not enable TestNG’s default report listeners unless asked.

Strengths

What it does well

Data providers built for test data

A @DataProvider method returns rows, as an Object[][] or an Iterator, and each row becomes a separate test invocation with its own result. Providers can live in another class, receive the test method to vary their data, and run with parallel = true so rows execute concurrently. For UI suites this is the simplest way to run one journey across many users, locales or products.

Factories for many instances of one class

@Factory creates several instances of a test class, each with different constructor arguments, and TestNG runs all of their tests. That suits running a whole page-object test class once per browser or per tenant, which method-level data providers do awkwardly.

Listeners for framework behaviour

ITestListener, ISuiteListener, IInvokedMethodListener and IReporter receive events for every test, configuration method and suite. Screenshot-on-failure, log capture, Allure or ExtentReports integration and custom dashboards attach here, through @Listeners on a class, the suite file or the ServiceLoader, so tests stay free of reporting code.

Soft assertions

SoftAssert collects failures and reports them together when assertAll is called. On a form or a page with several fields, one run shows every wrong value instead of stopping at the first, which matters when each run costs a browser session.

Retry analysers

An IRetryAnalyzer attached with @Test(retryAnalyzer = ...) or through a listener decides whether a failed test runs again. It is a pragmatic tool for UI suites on shared infrastructure, as long as retried passes are reported and tracked rather than hidden.

Parameters from the suite or the build

@Parameters injects values such as a base URL or browser name from the suite file, with @Optional supplying a default. Combined with system properties from the build, the same classes run against any environment without code changes.

Pricing

TestNG pricing and licensing

TestNG is free and open source under the Apache 2.0 licence. There is no paid edition, no hosted service and no commercial support offered by the project, which lists its creator, Cédric Beust, with Krishnan Mahadevan and Julien Herr as its developers. The real costs are the infrastructure your suites run on, such as browsers, grids and devices, and the engineering time to keep a framework built on it healthy.

Licence: Apache 2.0, stated in the published POM and on testng.org. It places no obligations on the code you test.
No tiers: data providers, groups, dependencies, listeners and every parallel mode are in the single org.testng:testng artifact.
Infrastructure: for Selenium and Appium suites, browser grids, cloud device labs and CI minutes are the main running cost, and they scale with thread count and suite length.
Reporting: TestNG’s built-in HTML reports are basic, so most teams add Allure, ExtentReports or a CI test report, some of which have paid hosted tiers of their own.
Hidden cost: build migration. Surefire 3.6.0 dropping suiteXmlFiles means framework rework or a pinned plugin version, and that time is real.

Checked against the vendor’s own pages on 13 October 2026. Pricing changes; confirm on the official site before buying.

Getting started

TestNG tutorial: getting started

Setup, step by step

Check the JDK: TestNG 7.6 and later need Java 11 or newer.
Add org.testng:testng 7.12.0 with test scope to the build.
Decide how the suite runs: Surefire groups, Gradle useTestNG, or Surefire 3.5.x with testng.xml.
Write a test class under src/test/java whose name ends in Test, using org.testng annotations.
Run mvn test or ./gradlew test and confirm the test count in the output.

Install

<!-- pom.xml -->
<dependency>
  <groupId>org.testng</groupId>
  <artifactId>testng</artifactId>
  <version>7.12.0</version>
  <scope>test</scope>
</dependency>

// build.gradle.kts
testImplementation("org.testng:testng:7.12.0")
tasks.test { useTestNG() }

A first example

// src/test/java/com/example/LoginTest.java
package com.example;

import static org.testng.Assert.assertEquals;

import org.testng.annotations.BeforeMethod;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class LoginTest {

    private AuthService auth;

    @BeforeMethod // runs before every test method, including each data row
    public void setUp() {
        auth = new AuthService(new InMemoryUsers());
    }

    @DataProvider(name = "credentials")
    public Object[][] credentials() {
        return new Object[][] {
            {"[email protected]", "correct-horse", "OK"},
            {"[email protected]", "wrong", "BAD_PASSWORD"},
            {"[email protected]", "x", "UNKNOWN_USER"},
        };
    }

    @Test(dataProvider = "credentials", groups = {"smoke"})
    public void login(String email, String password, String expected) {
        // TestNG order: actual first, expected second
        assertEquals(auth.login(email, password).status(), expected);
    }
}

What trips people up first

The suite file that does nothing. Most tutorials end with a testng.xml and a Surefire suiteXmlFiles setting. On Surefire 3.6.0 that setting is no longer supported: TestNG now runs through the JUnit Platform engine, which discovers test classes by name and ignores custom suites. Parameters, groups and thread counts in the XML stop applying, and @Parameters tests fail or fall back to @Optional defaults. Either pin maven-surefire-plugin to 3.5.x while you plan, run the suite through Gradle’s useTestNG with suites, or move groups and parallel settings into Surefire configuration and parameters into system properties.

In practice

Using TestNG in practice

Structuring a TestNG automation project

Keep a framework layer separate from the tests: driver management, configuration, page objects and listeners in one package, test classes grouped by feature in another. Base classes should own lifecycle only, opening and closing the browser in @BeforeMethod and @AfterMethod, and never hold test data. Use groups as the main selection mechanism, with a short agreed vocabulary such as smoke, regression and one per platform, because groups work the same way in Surefire, Gradle and the suite file. Read environment settings from system properties with defaults, so a developer can run one class from the IDE without a suite file.

Choosing how suites are launched after Surefire 3.6

There are three workable routes. On Maven, keep Surefire 3.6.0 and express the run through its configuration: groups and excludedGroups for selection, parallel and threadCount for concurrency, and properties for TestNG-specific settings such as the data provider thread count. If a framework depends heavily on testng.xml, pin Surefire to 3.5.x and plan the migration, accepting that the old provider will not move forward. On Gradle, useTestNG still accepts suite files and groups directly. Whichever you choose, document it in the repository, because a run that silently ignores its suite is the costliest failure here.

<!-- pom.xml: Surefire 3.6.0 runs TestNG through the JUnit Platform engine -->
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.6.0</version>
  <configuration>
    <!-- replaces <include> lists in testng.xml; define testGroups
         in <properties>, and -DtestGroups=... overrides it -->
    <groups>${testGroups}</groups>
    <parallel>classes</parallel>
    <threadCount>4</threadCount>
    <systemPropertyVariables>
      <baseUrl>${env.BASE_URL}</baseUrl>
    </systemPropertyVariables>
    <properties>
      <property>
        <name>dataproviderthreadcount</name>
        <value>4</value>
      </property>
    </properties>
  </configuration>
</plugin>

Parallel browsers without shared state

Parallel UI tests fail in confusing ways when two threads share one WebDriver. Hold the driver in a ThreadLocal so each thread has its own, create it in @BeforeMethod, and quit and remove it in @AfterMethod with alwaysRun = true so a failed setup does not leak a browser. Start with parallel by classes, which keeps one class on one thread, and move to methods only once tests are independent. Size the thread count to what your grid or machine can actually run, not to the number of tests.

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;

public abstract class BrowserTest {

    // One driver per thread, so parallel tests never share a browser
    private static final ThreadLocal<WebDriver> DRIVER = new ThreadLocal<>();

    protected WebDriver driver() {
        return DRIVER.get();
    }

    @BeforeMethod(alwaysRun = true)
    public void startBrowser() {
        DRIVER.set(new ChromeDriver());
    }

    @AfterMethod(alwaysRun = true) // runs even if the test or setup failed
    public void stopBrowser() {
        WebDriver d = DRIVER.get();
        if (d != null) {
            d.quit();
            DRIVER.remove();
        }
    }
}

Running TestNG in GitHub Actions

A TestNG pipeline is an ordinary Maven or Gradle build. Set up the JDK with dependency caching, pass environment-specific values as system properties, and upload reports when a job fails. Surefire writes XML results to target/surefire-reports, which CI test reporters read; the TestNG engine leaves TestNG’s default HTML report listeners off unless configured, so do not build a pipeline step that expects test-output reports. Split long UI suites across jobs by group, one matrix entry per group, rather than raising thread counts past what the grid can serve.

# .github/workflows/ui-tests.yml
name: ui-tests
on: [push, pull_request]
jobs:
  testng:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        group: [smoke, checkout, account]
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-java@v6
        with:
          distribution: temurin
          java-version: '21'
          cache: maven
      # -DtestGroups feeds the groups setting in the POM
      - run: ./mvnw -B test -DtestGroups=${{ matrix.group }}
        env:
          BASE_URL: https://staging.example.com
      - uses: actions/upload-artifact@v7
        if: failure()
        with:
          name: reports-${{ matrix.group }}
          path: target/surefire-reports

Listeners, retries and keeping the suite honest

Put cross-cutting behaviour in listeners rather than base classes: an ITestListener that captures a screenshot and page source on failure, and a reporter integration such as Allure. Register them once, through @Listeners on a base class or a META-INF/services file, so every test gets them. If you use a retry analyser, cap it at one retry, log every retry with the test name, and review the list weekly; a test that passes only on retry is a defect in the test or the product. Avoid dependsOnMethods for journeys; reserve it for true prerequisites such as an environment health check, and keep every other test runnable on its own.

Pitfalls

Common pitfalls

Upgrading Surefire and losing the suite file

Why it happens: Surefire 3.6.0 removed the TestNG provider and runs TestNG through the JUnit Platform engine, which does not support custom testng.xml suites. Groups, parameters and thread counts in the XML no longer apply.

The fix: Pin maven-surefire-plugin to 3.5.x until you have migrated, then move selection to Surefire groups, parallel settings to its configuration and parameters to system properties. Check the test count after every plugin upgrade.

Swapping actual and expected in assertions

Why it happens: org.testng.Assert.assertEquals takes the actual value first and the expected value second, the reverse of JUnit. Failures then report the values the wrong way round, which misleads whoever reads them.

The fix: Agree one convention per codebase. Many teams use AssertJ’s assertThat(actual).isEqualTo(expected), which reads the same in either framework and removes the ambiguity.

Chaining a journey with dependsOnMethods

Why it happens: It looks like a neat way to model login, then search, then checkout. Each test then needs the previous ones, cannot run alone, and blocks parallel execution of the chain.

The fix: Make each test reach its starting state through an API call, a cookie or seeded data, and keep dependencies for real prerequisites such as a health check.

Sharing a WebDriver between parallel threads

Why it happens: A static driver field works when tests run one at a time. Under parallel execution threads click in each other’s browsers and fail at random.

The fix: Hold the driver in a ThreadLocal, create it per method, and quit it in an @AfterMethod with alwaysRun = true.

Following tutorials that use removed or deprecated features

Why it happens: Older material shows parallel="true", running JUnit tests through TestNG with junit="true", and http DTD links. Running JUnit tests through TestNG was removed in 7.10, and parallel="true" is deprecated.

The fix: Use parallel modes by name, such as methods or classes, run JUnit tests through the JUnit Platform alongside TestNG, and use the https DTD URL, which current versions expect, in any suite file.

Retry analysers that hide real failures

Why it happens: Retrying every failure turns intermittent product bugs and race conditions into green builds, and the information is lost unless someone reads the logs.

The fix: Retry at most once, report retried tests separately, and treat any test that needs a retry as a ticket to fix.

Compared

How TestNG compares

Tool	Language	Data-driven tests	Dependencies between tests	Parallel execution	Licence
TestNG	Java and JVM languages	@DataProvider, optionally parallel rows	dependsOnMethods and dependsOnGroups	Methods, classes, tests or instances	Apache 2.0
JUnit	Java, Kotlin, other JVM languages	@ParameterizedTest and @ParameterizedClass	None by design; ordering only	Built in, opt-in via configuration	EPL 2.0
pytest	Python	@pytest.mark.parametrize	None built in	Via the pytest-xdist plugin	MIT
NUnit	C#, F#, VB.NET	[TestCase] and [TestCaseSource]	None; [Order] controls ordering	Opt-in with [Parallelizable]	MIT

FAQ

Frequently asked questions

What is TestNG?

Is TestNG free?

Yes. TestNG is open source under the Apache 2.0 licence, with no paid edition and no feature tiers. Everything, including data providers, groups, dependencies, listeners and parallel execution, is in the one org.testng:testng artifact on Maven Central. The project offers no commercial support. Your costs are the infrastructure your tests run on and the engineering time to maintain the framework around it.

TestNG vs JUnit: which should I choose?

For new unit and integration tests in Java, JUnit: it has the bigger ecosystem, better library support and is what build tools run natively. TestNG remains a reasonable choice for UI automation suites that use data providers with parallel rows, groups and listener-based frameworks, and for teams with a working TestNG estate. Both run on the JUnit Platform, so they can coexist in one build during a migration.

Why did my testng.xml stop working after a Maven upgrade?

Maven Surefire 3.6.0, published at the end of August 2026, runs TestNG through the JUnit Platform engine and no longer supports the suiteXmlFiles setting. The engine discovers test classes by name and does not read custom suites. Pin Surefire to 3.5.x to restore the old behaviour, or move group selection, parallel settings and parameters into Surefire configuration and system properties.

Is TestNG still maintained?

Yes. Version 7.11.0 was released on 13 February 2025 and 7.12.0 on 22 January 2026, with work on a further 7.x release under way in the repository. Releases are less frequent than JUnit’s. Some deprecated APIs, such as the old command-line classes, are marked for removal in a future 8.0, which has not been released.

Which Java version does TestNG need?

TestNG 7.6.0 and later require Java 11 or newer, and 7.12.0 targets Java 11. Projects still on Java 8 have to stay on TestNG 7.5 or earlier, which receive no fixes. The project’s CI already tests against recent JDKs, including JDK 25, so current Java versions are supported.

How do I run TestNG tests in parallel?

Set a parallel mode, methods, classes, tests or instances, with a thread count. In a suite file that is the parallel and thread-count attributes; with Surefire 3.6.0 use its parallel and threadCount settings; with Gradle use useTestNG options. Data provider rows run in parallel when the provider sets parallel = true, using a separate pool sized by the data provider thread count. Make sure tests share no mutable state first.

Can TestNG and JUnit tests run in the same project?

Yes. With Surefire 3.6.0, having both on the test classpath runs both through their JUnit Platform engines in one build. With Gradle, use useJUnitPlatform and add the TestNG engine as a test runtime dependency. TestNG itself no longer runs JUnit tests: that support was removed in 7.10.

Alternatives