Automated Smoke Testing: Building a Suite That Blocks Bad Builds

A smoke test suite has one job: tell you within a few minutes whether a build is worth testing further. Most teams get this wrong in the same direction. The suite grows until it takes twenty minutes, covers edge cases nobody asked it to cover, and fails often enough that engineers start ignoring it. At that point it is not a smoke test. It is a slow regression suite wearing a smoke test's name.

This guide covers how to decide what belongs in the suite, how to wire it into CI so it genuinely blocks a bad build, and what to do about the flakiness that kills most attempts.

What belongs in a smoke suite, and what does not

Start from the question the suite answers: is this build fundamentally broken. Not is it correct, not is it ready to ship, just is it standing up.

That gives you a sharp inclusion rule. A test belongs in the smoke suite if its failure means no further testing is worth running. Login is the obvious case: if nobody can authenticate, every other test will fail for the same reason and you have wasted a pipeline run discovering it forty times.

Work through your application and list the operations that everything else depends on. For most web applications that is a short list: the application loads, a user can authenticate, the main navigation resolves, the primary create or read operation works, and the application can reach its database and its critical third party services.

Five to fifteen tests is the usual healthy range. If your list runs past twenty, you are testing correctness rather than survival.

Equally important is what to exclude. Field level validation, edge cases, error message wording, anything cosmetic, and anything touching a slow external system all belong in the regression suite. They do not tell you whether the build is alive, and each one adds runtime to a suite whose value depends on being fast.

Set a runtime budget and defend it

Decide the budget before you write any tests, because it constrains everything else. Under five minutes is the target for most teams. Past ten minutes engineers start pushing again before results return, which defeats the purpose.

Treat the budget as a hard constraint rather than an aspiration. When a new test would push the suite over, something else comes out or the new test goes to regression. A suite that quietly grows to fifteen minutes stops being run on every commit, and an unrun suite has no value regardless of its coverage.

Parallelism buys you room. If your runner supports it, running across four workers turns a twelve minute serial suite into roughly four minutes, though the scaling is never perfectly linear because setup costs repeat per worker.

Writing the tests

Smoke tests should be the most boring code in your repository. No clever abstractions, no shared mutable state between tests, no conditional logic that changes what is asserted.

Each test should be independently runnable. If test three depends on test two having created a record, a single failure cascades and your report tells you five things broke when one did.

Here is the shape of a smoke test that checks authentication, written with Playwright:

1// Relies on a seeded account that other tests also use.
2// Breaks the moment somebody edits the seed data.
3test('user can log in', async ({ page }) => {
4  await page.goto('/login');
5  await page.fill('#email', '[email protected]');
6  await page.fill('#password', 'hunter2');
7  await page.click('button[type=submit]');
8  await expect(page).toHaveURL('/dashboard');
9});

1// Creates a unique user per run, asserts one
2// user-visible outcome, cleans up even on failure.
3test('user can log in', async ({ page, request }) => {
4  const email = `smoke-${Date.now()}@example.test`;
5  const user = await createUser(request, email);
6
7  try {
8    await page.goto('/login');
9    await page.fill('#email', email);
10    await page.fill('#password', user.password);
11    await page.click('button[type=submit]');
12    await expect(
13      page.getByRole('heading', { name: 'Dashboard' })
14    ).toBeVisible();
15  } finally {
16    await deleteUser(request, user.id);
17  }
18});

Three things to notice. The test creates its own data rather than relying on a fixture another test made. The assertion is on a stable, user visible outcome rather than an internal implementation detail. And it asserts one thing, so a failure message tells you exactly what broke.

Use test data that is created and destroyed within the test wherever you can. Shared seed data drifts, and a smoke suite that fails because somebody changed a seed record is a suite that trains people to ignore failures.

Wiring it into CI so it actually blocks

A smoke suite that runs but does not gate anything is documentation, not a control.

Run it on every pull request and on every merge to the main branch. Configure the pipeline so a failure blocks the merge. This is the step teams skip, usually because the suite was flaky when it was introduced and blocking felt too aggressive. The result is a suite everyone has learned to override.

Put the smoke stage first in the pipeline and fail fast. There is no value in running a forty minute regression suite against a build whose login is broken. Ordering the stages correctly saves more CI minutes than any optimisation inside the tests.

Run it again after deployment against the deployed environment. The pre merge run tells you the code is sound; the post deploy run tells you the deployment is sound. They catch different failures, and configuration problems only ever show up in the second.

Dealing with flakiness, which is what kills most suites

Flakiness is the single reason smoke suites get abandoned. A suite that fails one run in ten teaches engineers to re-run rather than investigate, and once that habit forms the suite is worthless even when it is right.

The general principle: fix the test or delete it. A quarantined test that stays quarantined for a month is not coming back, and leaving it in the suite as a permanent yellow signal erodes trust in the green ones.

Retries deserve a specific warning. Automatic retry is available in most runners and it is tempting, because it makes the dashboard green. It also hides genuine intermittent bugs, including race conditions that will eventually affect a real user. If you use retries, log them and review the retry rate weekly. A test that passes on retry three times a week is telling you something.

Choosing the environment to run against

Where the suite runs changes what it can tell you, and most teams run it in only one place when two are cheap.

Against a build in CI, before merge, the suite answers whether the code is sound. It cannot see configuration, secrets, or infrastructure, because none of those are involved yet.

Against a deployed environment, after release, it answers whether the deployment is sound. This is the run that catches an expired API credential, a missing environment variable, a misconfigured load balancer or a database migration that did not apply. Those failures are invisible to the pre-merge run and they are the ones that actually take production down.

Run both. The post-deploy run should be the same tests against a different base URL, not a separate suite, or the two will drift apart.

One caution on the post-deploy run: it must not create data that pollutes a real environment. Either point it at a dedicated test account whose records are expected, or make cleanup mandatory and verified.

Measuring whether it is working

Track two numbers. First, how often the smoke suite catches a genuine break that would otherwise have reached the regression stage. That is the suite's value. Second, the false failure rate. That is its cost.

If the first number is near zero over a few months, your suite is testing things that do not break, and it should be re-scoped against what actually goes wrong in your application. Read your incident history and ask whether the suite would have caught each one.

What to do when the suite fails

The response to a smoke failure determines whether the suite retains any authority.

Treat it as a stop signal. The point of a gate is that work pauses until the question is answered, and a team that continues merging while the smoke suite is red has a suite that gates nothing.

Investigate before re-running. Re-running first is the habit that destroys these suites, because it converts every failure into either a pass or a longer wait, and neither outcome involves anybody reading the error. If the first response is always a re-run, the suite has stopped being a signal.

Give the failure an owner immediately. A red build that belongs to everybody belongs to nobody, and the time between a break and somebody looking at it is the number that predicts whether the suite survives the year.

When the failure turns out to be the test rather than the application, fix it that day. A known-broken test left in the suite teaches the team that red does not necessarily mean broken, which is precisely the lesson that makes the next real failure invisible.

Keeping it useful over time

Review the suite quarterly against the same inclusion rule you started with. Applications change, and a test that guarded a critical path two years ago may now cover a feature nobody uses.

When an incident reaches production, ask whether a smoke test would have caught it. Sometimes the answer is yes and you add one. Often the answer is no, because it was a correctness bug rather than a structural one, and the right home is the regression suite. Adding every incident to the smoke suite is how a five minute suite becomes a twenty minute one.

Keep the suite owned by a named person. Shared ownership of a test suite reliably means nobody investigates the failures.

Related reading

For the wider automation picture, see our guides on test automation frameworks and automated regression testing.

If you want help scoping a suite against your own release process, our automation testing team can work through it with you.