Performance test cases, for what happens after the spike

The standard load test is a gradual ramp to a target concurrency, held briefly, reported as an average. Almost every part of that is the wrong shape. Real traffic arrives as a spike rather than a ramp, which is what breaks autoscaling and connection pools. Real peaks last for an hour rather than five minutes, which is what surfaces leaks and queue growth. And an average conceals the slow tail completely, so the report says the system is fast while the 99th percentile is timing out. Percentiles are not a refinement here, they are the measurement.

Recovery is the most commonly skipped case and one of the most informative. Anyone can observe that a system slows down past capacity. The question that matters operationally is whether it returns to baseline when the load goes away, because a system that stays degraded has a bounded resource it is not releasing, and in production that turns a ten minute traffic peak into an outage lasting until somebody restarts something. Related to it are the failure shapes: shedding load with a clear retryable response is a good outcome, while accepting every request and timing all of them out slowly is the worst one.

Dependencies fail more often than capacity does. A downstream service that becomes slow rather than unavailable is the most dangerous case, because unbounded waiting exhausts the thread or connection pool and converts one degraded feature into a total outage. Circuit breakers need testing in both directions, including closing again without a deployment. And retries need backoff with jitter, because synchronised retries from every client prevent the recovery they are trying to achieve.

Finally, caches and data volume are where test environments lie. A system measured only with a warm cache has no headroom at the exact moment it restarts, and a widely shared cache entry expiring under load will be recomputed by every concurrent request at once unless something coalesces them. Both are invisible unless deliberately provoked. The same applies to query behaviour: logging query counts while returning ten, a hundred and a thousand records exposes a query issued per row immediately, and that single pattern is behind a large share of endpoints that are fine in testing and unusable in production.