Allure Report review: tutorial, pros and cons

Allure has two halves. An adapter inside the test framework, such as allure-jupiter, allure-pytest or allure-playwright, records each test as a JSON result file in an allure-results directory, with its steps, labels, parameters and attachments as separate files beside it. The Allure CLI then reads that directory and generates a report. Because the result format is shared, any language with an adapter produces the same report, and Allure 3 reads the same result files that Allure 2 adapters write. The split also explains most setup problems: the report can only show what the adapter recorded, so missing steps or labels are an adapter or configuration issue, never a report issue.

Allure 2 is a Java application. Allure 3 was rebuilt in TypeScript, installs from npm as the allure package and needs only Node.js. One configuration file, allurerc.mjs or one of its variants, holds the report name, output directory, history, categories, quality gate and plugins. The default plugin is Awesome, a redesigned interface; Classic and Allure 2-style plugins exist but are marked experimental, and a Dashboard plugin produces charts. The new allure run command wraps the test command, finds allure-results directories at any depth, ignores results from earlier runs, and generates the report when tests finish. A watch command refreshes the report live while tests run.

The adapter computes a history ID for each result from the test’s fully qualified name and its parameters, minus any marked as excluded. Results with the same ID in one launch are shown as retries of one test, with the final status on top. Results with the same ID across launches form its history. In Allure 2, history lives in a history folder that must be copied from the previous report into the new results before generating. Allure 3 replaces that with a single JSONL file named by historyPath, to which each generation appends one line. A parameter that changes on every run, such as a timestamp, breaks identity and makes every run look like a new test.

Steps and attachments are recorded while the test runs, through a runtime API or annotations. In JavaScript, allure.step wraps a block and allure.attachment adds a file or string. In Java, methods annotated with @Step and @Attachment become steps through AspectJ, which weaves the recording code in at load time, so the AspectJ weaver must run as a Java agent. Some integrations save code: allure-playwright records Playwright’s own test.step calls as steps and picks up trace files automatically, and the REST Assured integration attaches requests and responses. Good reports therefore come from deliberate step naming in page objects and API clients, not from the report tool.

An Allure report is only as navigable as its labels. Agree a short set before the suite grows: epic, feature and story for the product hierarchy, severity for triage, owner for routing, and tags for cross-cutting concerns such as smoke or regression. Allure 3 builds one default hierarchy and lets you configure others from any labels, so consistent labels matter more than in Allure 2. Put steps in the shared layer rather than in each test: a page object method or API client call wrapped in a step names itself the same way everywhere, and attachments such as response bodies are added once. Mark parameters that change every run, such as generated IDs, as excluded so history is not broken.

For Java, Allure Java 3.0, released in September 2026, targets Java 17 and later and publishes the JUnit Jupiter adapter as allure-jupiter, which covers JUnit 5 and 6. Use the allure-bom so every Allure module stays on one version. @Step and @Attachment need the AspectJ weaver running as a Java agent, configured through Surefire’s argLine; without it the build passes and the report shows tests with no steps. Point results at the build directory through allure.properties so mvn clean removes stale results. Generate the report with the Allure CLI, which reads the same results whether you choose Allure 2 or 3.

Allure 3 keeps report behaviour in a single config file at the project root. Three settings repay the effort immediately. historyPath turns on history and trends; without it, every report starts from nothing. Categories route failures into buckets your team actually triages, such as known environment problems versus product defects, and the first matching rule wins. A quality gate makes the report step enforce a release rule, for example no failed critical tests, and fails the command with exit code 1 when it is broken. Keep the file in JavaScript rather than JSON or YAML, because the declarative formats do not support features such as filters or environments.

A useful pipeline does four things: restores the history file, runs the tests through allure run, saves the history file even when tests fail, and publishes the report. Use the separate restore and save cache actions, because the combined cache action only saves after a successful job, which is exactly when you least need history. Give each run a unique cache key and restore by prefix, so the newest history is always picked up. Upload the report as an artifact or deploy it to a static host, and add the official Allure action to post a summary on the pull request. It needs permission to write pull request comments and checks.

For a sharded suite, have each shard save its results as a dump with allure run --dump and a stage name, upload the archives, and in a final job combine them with allure generate --dump pointing at all the stage archives. One report then covers every shard, and history is updated once rather than racing between jobs. Use --rerun for flaky-prone suites only as a diagnostic: the report shows each attempt, which makes flakiness visible rather than hidden. Keep reports small by attaching screenshots and traces on failure only, and by setting historyLimit so the history file does not grow without bound. For heavy suites, the singleFile option produces one portable HTML file, convenient for attaching to a ticket or an email.