Kubernetes Production Readiness Is More Than Health Checks

A 200 response does not define readiness

Kubernetes can keep a container running and still fail to keep the service available. Health probes only answer narrow questions at a point in time. Production readiness also depends on scheduling, resource pressure, voluntary disruption, traffic distribution, termination behavior, scaling, telemetry, operational ownership, and tested failure response.

The useful review unit is the workload contract. What does the application require from the cluster, and what does it promise in return? That contract should be visible in manifests, application behavior, alerts, dashboards, and runbooks.

Give each probe one job

A startup probe determines whether initialization has completed. While it is failing, Kubernetes does not run liveness or readiness checks. This protects slow-starting applications from being killed before they can initialize.

A readiness probe determines whether a Pod should receive traffic through a Service. It may fail during startup, dependency loss, overload, or draining. A liveness probe determines whether the process is stuck badly enough to justify a restart. It should not fail merely because a downstream database is unavailable. Restarting every replica during a shared dependency outage can amplify the failure.

startupProbe:
  httpGet:
    path: /health/startup
    port: http
  periodSeconds: 5
  failureThreshold: 24

readinessProbe:
  httpGet:
    path: /health/ready
    port: http
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

livenessProbe:
  httpGet:
    path: /health/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Set thresholds from observed startup and response behavior. The example permits roughly two minutes for startup, but it is not a universal default. Probe handlers should be cheap, bounded, and independent of the main request queue where possible. If an overloaded process cannot answer a probe because the probe competes with customer traffic, an aggressive liveness check may create a restart loop.

Requests and limits are scheduling inputs

CPU and memory requests tell the scheduler what a Pod reserves. CPU limits throttle CPU time when the container exceeds its quota. Memory limits are enforced through out-of-memory termination when usage exceeds the cgroup boundary. These behaviors are different and should not be tuned as if they were equivalent.

Missing requests allow dense scheduling with weak guarantees. Inflated requests waste node capacity and can make pending Pods look like a cluster shortage. Memory limits set too close to ordinary working set can trigger OOMKills during traffic bursts or garbage collection. CPU limits can add latency through throttling even when a node has spare CPU.

Measure working set, allocation rate, throttling, request concurrency, latency, and OOM history. Then set requests from the service's expected operating range and scaling model. If a HorizontalPodAutoscaler scales on CPU utilization, its percentage is calculated relative to the CPU request. A poorly chosen request therefore affects both scheduling and scaling.

resources:
  requests:
    cpu: 500m
    memory: 768Mi
  limits:
    memory: 1Gi

This example deliberately omits a CPU limit, a policy some teams use for latency-sensitive workloads while controlling CPU through requests and namespace or node capacity. Whether that policy is appropriate depends on tenancy, cluster controls, and workload behavior. Make it an explicit decision rather than a copied convention.

Plan for voluntary disruption

A PodDisruptionBudget limits how many selected Pods may be voluntarily disrupted at once by actions such as node drains. It does not protect against node failure, application crashes, or every rollout condition. It also cannot create availability when a workload has one replica.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: orders-api

Choose minAvailable or maxUnavailable with replica count, quorum rules, autoscaling floor, and maintenance procedures in mind. An impossible budget can block node maintenance. A permissive budget can let a drain remove too much capacity. Test the actual drain workflow and inspect disruption events rather than assuming the manifest has the intended effect.

Deployment strategy belongs in the same review. maxUnavailable and maxSurge control rollout capacity, while readiness gates determine when new Pods count as available. Confirm that the cluster has enough spare resources to honor surge settings during a deployment.

Spread replicas across failure domains

Three replicas on one node are one failure domain, not three. Topology spread constraints can distribute Pods across zones and nodes. Pod anti-affinity can express similar requirements, though it may carry greater scheduling cost in large clusters.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: orders-api
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: orders-api

The choice between DoNotSchedule and ScheduleAnyway is a capacity decision. Strict zone spread may leave Pods pending when a zone lacks nodes. Soft node spread may permit concentration during pressure. Check node groups, zone labels, autoscaler behavior, storage topology, and minimum replica count together.

Stateful workloads need extra care. A volume bound to one zone constrains where its Pod can run. Replicas distributed across zones do not help if they all depend on a single-zone data service or a quorum that cannot tolerate the chosen failure.

Shutdown is part of request handling

When Kubernetes terminates a Pod, endpoint removal, lifecycle hooks, signal delivery, and load balancer updates do not happen as one atomic event. Applications must stop accepting new work, finish or transfer in-flight work, and exit within terminationGracePeriodSeconds.

Handle SIGTERM in the application. Mark the instance unready, stop accepting new requests, drain keep-alive connections where supported, finish bounded work, and close clients cleanly. Consumers should stop polling, finish or safely abandon the current message, and preserve at-least-once or exactly-once assumptions through idempotency and acknowledgements.

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]

A sleep hook can allow routing changes to propagate, but it is not a substitute for application shutdown logic. It also consumes part of the termination grace period. Measure the longest legitimate request or task, load balancer deregistration behavior, and application drain time. Then set the grace period and rollout deadlines from evidence.

Test termination while traffic is active. Watch client errors, request completion, endpoint state, process logs, and forced kills. A clean idle shutdown proves little.

Autoscaling needs a complete control loop

A HorizontalPodAutoscaler needs a metric that changes before user-visible failure and has a predictable relationship to capacity. CPU can work for compute-bound services. Queue depth, active requests, or work age may be better for asynchronous and concurrency-bound workloads.

Set a nonzero minimum replica count that preserves availability during node loss and rollout. Confirm that new Pods can start before the backlog or latency budget is exhausted. Scaling on a delayed metric with a long startup time creates a controller that reacts after the service is already unhealthy.

The HPA, Cluster Autoscaler or Karpenter, Pod requests, topology constraints, quotas, and cloud capacity form one loop. More desired replicas do not help if no node can fit them. Larger nodes do not help quickly if quotas, subnet addresses, image pulls, or zone capacity block provisioning.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
spec:
  minReplicas: 3
  maxReplicas: 20
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65

Load-test the whole loop. Record the time from metric change to desired replicas, scheduling, image readiness, readiness success, and restored service-level behavior. Also test scale-down so connection draining and cache loss do not cause oscillation.

Observability must follow a request and a rollout

Production telemetry should let an operator connect user symptoms to workload state. At minimum, collect request or work rate, errors, latency, saturation, restart reasons, OOMKills, CPU throttling, pending Pods, unavailable replicas, scheduling failures, autoscaler decisions, and deployment changes.

Use stable workload, namespace, cluster, version, and region labels. Avoid unbounded identifiers in metric labels. Propagate trace context through ingress, services, queues, and outbound calls. Structured logs should include a correlation identifier and deployment version without recording secrets or sensitive payloads.

Alert on service outcomes and fast resource exhaustion. Error-budget burn, sustained processing delay, unavailable capacity, and repeated crash loops are usually stronger signals than a single high CPU sample. Every page should link to a dashboard and runbook that names the first checks an operator can perform.

Deployment markers matter. Without version and rollout events on graphs, an operator may spend time correlating a latency change manually. Emit the artifact digest or revision, not only a mutable image tag.

Runbooks should describe decisions

A useful runbook identifies service ownership, customer impact, dependencies, dashboards, recent changes, known failure modes, safe mitigations, rollback procedure, and escalation. Commands are helpful, but a list of commands without decision criteria can make an incident worse.

For each mitigation, state prerequisites and side effects. Scaling replicas may overload a downstream database. Restarting consumers may duplicate work. Disabling a readiness dependency may send traffic to an instance that cannot complete requests. Operators need to know when an action is safe and how to verify its effect.

Keep runbooks executable against the current environment. Replace screenshots of dashboards with stable links and query text. Test permissions from the on-call role. Review the runbook after architecture changes and after any incident that exposed a missing branch.

Test failures, not only deployments

Production readiness requires deliberate failure testing in an environment that represents production closely enough to expose the relevant behavior. Start with bounded tests: terminate a Pod under load, drain a node, make a zone unavailable to the scheduler, delay a dependency, exhaust a connection pool, introduce queue backlog, and block image pulls for a new replica.

Define the expected result before each test. Specify allowable errors, recovery time, replica distribution, backlog age, paging behavior, and evidence to capture. A chaos tool reporting that an experiment completed does not prove the service met its objective.

Test rollback as a first-class operation. Confirm that the previous artifact is available, schema changes are compatible, configuration can be restored, and rollout history identifies the exact revision. Database and message-format compatibility often determines whether a Kubernetes rollback is operationally possible.

Review readiness as a set of claims backed by manifests and test evidence. Probes show local health. Requests and limits shape scheduling and pressure. Budgets and spread constraints preserve capacity during planned and unplanned changes. Shutdown logic protects in-flight work. Autoscaling responds within a measured window. Telemetry, runbooks, and failure tests show whether people can detect and recover when those controls are not enough.