Kubernetes review for testing: alternatives, pricing and setup

You describe objects such as Deployments, Services and Jobs in YAML and send them to the API server, which stores them. Controllers then loop continuously, comparing what exists with what you declared and acting on the difference: starting pods, replacing failed ones, rescheduling work from a lost node. This is why a crashed dependency in a Kubernetes test environment usually comes back on its own. It is also why problems hide: a service that crashes every few minutes keeps getting restarted, so a test failure caused by it looks intermittent. Check restart counts before blaming the test.

The pod is the unit Kubernetes runs: one or more containers sharing a network address and, optionally, volumes. The scheduler places each pod on a node that has enough unreserved capacity for the pod’s resource requests. Limits are enforced on the node: a container that reaches its CPU limit is throttled, and one that exceeds its memory limit may be killed by the kernel. For test workloads this explains two classic symptoms. Browser pods without requests get packed onto busy nodes and run slowly, so timeouts fail at random; and a Chrome pod with a tight memory limit restarts mid-test with an OOMKilled reason rather than a clear error.

A namespace groups objects under one name, and most resources, including Services, Deployments, Secrets and Jobs, live inside one. Services get DNS names that include their namespace, so the same manifests deployed into pr-123 and pr-124 produce two environments whose services find each other by short name and never cross over. Deleting a namespace deletes everything in it, which makes tearing down an environment one command. A ResourceQuota on each test namespace caps how much CPU and memory it can request, which stops one forgotten preview environment starving everyone else.

Deployments keep services running forever; a Job runs pods until a set number succeed, which is the shape of a test run. The pod template may use only Never or OnFailure as its restart policy. backoffLimit, 6 by default, caps retries, and activeDeadlineSeconds fails the whole Job after a time limit, taking precedence over retries. ttlSecondsAfterFinished deletes the Job and its pods a set time after it ends. In Indexed completion mode, each pod gets an index from 0 to completions minus 1, exposed as the JOB_COMPLETION_INDEX environment variable, which a test runner can use as its shard number.

Name the namespace after the pull request, install the application’s own chart into it with the image built from that commit, and point the suite at the services inside it. Label the namespace so a scheduled job can find and delete preview environments older than a day, because the close-event workflow that should delete them will sometimes fail. Apply a ResourceQuota per namespace so a runaway environment cannot consume the cluster, and seed test data through a Job rather than by hand, so every environment starts from the same state. Helm’s --wait makes the install step block until the release is ready, which gives the pipeline a clean failure point before any test runs.

For a suite that takes too long on one machine, run it as an Indexed Job with completions and parallelism both set to the number of shards. Each pod reads JOB_COMPLETION_INDEX, which starts at 0, and passes it to the runner’s shard option, which for Playwright counts from 1. backoffLimitPerIndex retries a failed shard once without restarting the others, and the Job is complete only when every index has succeeded. Give each pod resource requests that match what one shard really needs, so the scheduler spreads pods across nodes instead of stacking them where they slow each other down, and collect results by writing reports to shared storage or by uploading them from each pod.

Before a manifest or chart reaches a shared cluster, run it in a throwaway one. The helm/kind-action creates a kind cluster on a GitHub-hosted Ubuntu runner, named chart-testing unless you set cluster_name, and deletes it when the job ends. Hosted macOS and Windows runners cannot create kind clusters because they lack Docker with Linux container support. kubectl wait for the complete condition only returns early on success, so a failed Job surfaces as a timeout: set the timeout just above activeDeadlineSeconds and print the logs whatever the outcome, so the job output contains the evidence.

The docker-selenium project publishes a Helm chart that deploys the Grid router, distributor, session queue and browser nodes. Setting autoscaling.enabled installs KEDA alongside it, or autoscaling.enableWithExistingKEDA uses one you already run. With the default scaling type of job, each browser node handles a session and is then drained, so every test gets a fresh browser, and the node count follows the session queue from zero up to autoscaling.scaledOptions.maxReplicaCount. Set browser node resource requests realistically, since Chrome under-provisioned on a crowded node is the commonest cause of timeouts that only happen on the cluster, and pin the chart version so the browser versions change only when you choose.

Work from the outside in. kubectl get pods shows status and restart counts; Pending usually means no node has room for the requests, CrashLoopBackOff means the container keeps exiting, and ImagePullBackOff means the image cannot be fetched. kubectl describe pod shows the events behind each of those, including OOMKilled as a termination reason. kubectl logs with --previous shows the output of the container before its last restart, which is where a crash’s real error usually is. Cluster events sorted by time give the order in which things went wrong across the namespace. Finally, check that kubectl itself is within one minor version of the cluster, which is the supported skew.