Azure DevOps review: pricing, alternatives, pros and cons

Everything lives in an organisation, divided into projects, and each user is given an access level. Stakeholder is free and limited, Basic covers Boards, Repos, Pipelines and Artifacts, and Basic + Test Plans adds the full test management feature set. That licensing model drives a lot of practical decisions for QA teams: developers can run pipelines and read test results on Basic, but whoever creates test plans and suites, or manages configurations needs Basic + Test Plans or an eligible Visual Studio subscription. Stakeholders can still run exploratory sessions with the Test & Feedback extension. The same products are available on premises as Azure DevOps Server, which has its own release cycle.

A YAML pipeline in the repository defines stages, jobs and steps, and each job runs on an agent: Microsoft-hosted virtual machines such as ubuntu-latest, windows-latest and macOS images, or self-hosted agents you install. Capacity is bought as parallel jobs, the number of jobs that can run at once, not as agents or minutes. Without the free grant or purchased parallel jobs, a pipeline queues. A job can declare strategy: parallel with a count, and each copy receives System.JobPositionInPhase and System.TotalJobsInPhase to pick its slice of the suite, with up to 99 agents for one job. A matrix strategy instead runs one copy per named set of variables.

Test tasks such as Visual Studio Test, or the generic PublishTestResults task fed with JUnit, NUnit, xUnit, TRX or CTest files, send results to the test service rather than leaving them as files. Each pipeline run then has a Tests tab with pass rates, failures with stack traces, attachments and comparisons with earlier runs, and the Analytics service aggregates results over time for widgets and reports. Results can also be associated with test cases in Test Plans, so an automated run updates the status of the manual test case it automates. Publishing and failing are separate decisions: by default PublishTestResults reports failures without failing the task.

Azure Test Plans stores test plans, test suites, test cases, shared steps and shared parameters as work items, so they have history, links, queries and permissions like any user story. Suites can be static, requirement-based or query-based, and a requirement-based suite tracks the test cases linked to one story. Testers run manual tests in the web-based Test Runner, which records steps, screenshots, action logs and screen recordings and attaches them to any bug filed from the run, with the bug linked to the requirement and build under test. Exploratory sessions run through the Test & Feedback browser extension, outside any plan.

Keep the two halves of testing in the parts of the product built for them. Automated suites belong in YAML pipelines in the repository, with results published to each run, so developers see failures on the pull request. Manual, exploratory and user acceptance testing belong in Test Plans, organised as one plan per release or sprint with requirement-based suites, so coverage is visible on the stories themselves. Link the two only where it adds value: associating an automated test with a test case is worth it for regulated or audited features, where the test case is the record, and rarely worth it for the bulk of a regression suite.

A matrix strategy runs one copy of a job per entry, each with its own variables, in parallel up to the parallel jobs you have. Pass the shard to a runner that supports it, publish each copy’s JUnit file, and the run’s Tests tab combines them. Give each copy a distinct testRunTitle so failures are attributable to a browser and shard. Count the matrix entries against your parallel jobs: four entries on the single free hosted job run one after another, not at once.

Turn on flaky test management in project settings first, and decide whether flaky failures should be excluded from the test summary. Automatic detection by rerun works with the Visual Studio Test task, which can rerun failed tests in the same execution; for Java, JavaScript and Python suites, rerun inside the test runner instead, with Surefire’s rerunFailingTestsCount, Gradle’s test-retry plugin or Playwright retries, and mark persistent offenders as flaky by hand or through the API. The setting is per project, and switching between system and custom detection erases the flakiness history, so choose once. Review the flaky list as part of sprint work rather than letting it grow.

For teams that keep formal test cases, the Azure Test Plan task runs the automated test points of a chosen plan, suite and configuration, and records results against those test cases. It supports Maven, Gradle, PyTest and Jest projects and is in public preview, so treat it as useful but subject to change. Unlike PublishTestResults, it fails on test failures by default. Plan, suite and configuration are passed as IDs, which you can read from the Test Plans hub, so keep them in pipeline variables rather than scattered through YAML files.

Parallel jobs are the unit that limits throughput, so plan them before sharding. Microsoft-hosted agents give a clean machine per job but count against hosted parallel jobs and, on the free grant, cap each run at 60 minutes. Self-hosted agents cost nothing to register and are bought only as concurrency, which makes them the economical option for long browser suites, test environments inside a private network or device labs; scale sets can create and remove them on demand. Keep pipeline YAML in templates shared across repositories, so the test job, its result publishing and its artefact rules are defined once.