Kubernetes Liveness, Readiness, and Startup Probes: A Practical Guide

Introduction

Health probes are the quietest, most under-appreciated reliability feature in Kubernetes. When they are configured well, they let the platform route traffic only to healthy pods, restart stuck containers automatically, and give slow-starting applications room to boot. When they are configured badly, they do the opposite: they kill healthy containers mid-request, hammer a recovering service with traffic it cannot handle, and manufacture a CrashLoopBackOff that no amount of application debugging will fix.

Most teams learn probes twice. The first time, they copy a snippet from a blog post, paste it into a Deployment, and move on. The second time, they are staring at a production incident at 2 a.m. wondering why a perfectly healthy pod keeps getting restarted every ninety seconds. This guide is meant to save you the second lesson.

We will cover what each probe actually does, how the kubelet evaluates them, the exact YAML fields that matter, and the misconfigurations that cause the most outages. Every example is copy-paste ready.

The three probe types

Kubernetes exposes three distinct probes, and the single most common mistake is treating them as interchangeable. They answer three different questions.

Liveness probe answers: "Is this container broken beyond recovery?" If a liveness probe fails its threshold, the kubelet kills the container and restarts it according to the pod's restart policy. Liveness is a restart trigger, nothing else.

Readiness probe answers: "Should this pod receive traffic right now?" If a readiness probe fails, the pod's IP is removed from the Endpoints (and EndpointSlices) backing every Service that selects it. The container is not restarted. It is simply taken out of rotation until it reports ready again.

Startup probe answers: "Has this container finished booting yet?" While a startup probe is still failing, the kubelet disables the liveness and readiness probes entirely. This exists so that slow-starting applications are not killed by an impatient liveness probe before they have even finished initializing.

The mental model that prevents most incidents: liveness restarts, readiness gates traffic, startup protects boot. They are not redundant. A well-run production workload often uses all three.

How the kubelet evaluates a probe

Every probe on every container is driven by the kubelet on the node, not by the control plane. The kubelet runs each probe on a fixed cadence and tracks consecutive successes and failures against configurable thresholds.

There are four handler types you can use for any probe:

• httpGet — the kubelet performs an HTTP GET against a path and port. Any status code from 200 up to 399 is a success; anything else is a failure.
• tcpSocket — the kubelet tries to open a TCP connection to a port. If the connection succeeds, the probe passes.
• exec — the kubelet runs a command inside the container. Exit code zero is success; any non-zero exit code is failure.
• grpc — the kubelet uses the standard gRPC health-checking protocol against a port. This is stable and generally available in current Kubernetes versions.

The timing fields are where reliability is won or lost:

The interaction that surprises people: the time a container can hang before a liveness restart is roughly periodSeconds multiplied by failureThreshold, plus one timeoutSeconds. With the defaults that is about thirty-one seconds. If your application can legitimately pause longer than that under load, the defaults will restart healthy containers.

A complete, production-shaped example

Here is a Deployment that uses all three probes correctly for a typical HTTP service that exposes a health endpoint.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: payments-api
          image: registry.example.com/payments-api:1.8.2
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3

Read the startup probe carefully. With periodSeconds: 5 and failureThreshold: 30, the application is given up to 150 seconds to boot before the kubelet gives up on it. During that entire window, liveness and readiness are suspended, so a slow migration or a cold cache cannot trigger a restart. The moment the startup probe passes once, liveness and readiness take over.

Notice also that liveness and readiness point at different endpoints. This is deliberate and important, and it is the subject of the next section.

Split your health endpoints

The highest-leverage design decision with probes is to expose two separate endpoints and to be disciplined about what each one checks.

/healthz (liveness) should check only whether the process itself is fundamentally alive: the event loop is turning, the HTTP server is accepting connections, no deadlock. It must not check downstream dependencies.

/readyz (readiness) may check whether the pod is ready to serve real traffic: database connection pool established, caches warm, required downstream services reachable.

Here is the failure mode that this split prevents. Imagine you put a database check inside your liveness probe. Your database has a brief blip. Every pod's liveness probe now fails simultaneously. Kubernetes responds by restarting every pod at once. Restarting the pods does nothing to fix the database, but it does throw away all your warm caches and in-flight work, and it turns a ten-second dependency blip into a full-blown outage with a thundering-herd reconnect storm on top. This exact pattern is one of the most common self-inflicted Kubernetes outages, and it maps directly to the kind of failure you want captured in a good incident management runbook.

The rule: liveness checks the process, readiness checks dependencies. Never let a downstream dependency failure become a restart.

Graceful shutdown and readiness

Probes also matter on the way down, not just on the way up. When a pod is deleted (during a rolling update, a scale-down, or a node drain), two things happen effectively in parallel: the pod is removed from Service endpoints, and the container receives SIGTERM.

Because endpoint removal propagates asynchronously across the cluster, there is a brief window where traffic can still arrive at a pod that has already begun shutting down. The standard mitigation is a preStop hook that sleeps briefly, giving the endpoint removal time to propagate before your process actually stops accepting connections.

        lifecycle:
          preStop:
            exec:
              command: ["sh", "-c", "sleep 10"]

Combine that with an application that flips its readiness probe to failing as soon as it receives SIGTERM, and you get truly graceful drains: the pod stops advertising itself as ready, existing requests finish, and only then does the process exit. Getting this right is a big part of hitting the availability targets you set in your SLI, SLO, and SLA definitions, because botched rollouts are a leading source of avoidable error-budget burn.

Common probe misconfigurations that cause outages

This is the section to bookmark. These are the patterns that generate real incidents.

CrashLoopBackOff from an aggressive liveness probe

Symptom: a container starts, runs fine for twenty to thirty seconds, gets killed, restarts, and repeats. kubectl describe pod shows Liveness probe failed events followed by Killing.

Cause: the application takes longer to become responsive than initialDelaySeconds plus the failure window allows, or it periodically pauses longer than periodSeconds times failureThreshold. The container is healthy; the probe is impatient.

Fix: add a startup probe with a generous failureThreshold so boot time is decoupled from liveness. If steady-state pauses are the issue, raise liveness periodSeconds or failureThreshold. Do not paper over it by disabling the probe entirely.

The readiness probe that checks a dependency inside liveness

Already covered above, but it belongs on the misconfiguration list because it is so common. A dependency check in a liveness probe converts every downstream blip into a cluster-wide restart storm. Move dependency checks to readiness.

timeoutSeconds set to the default of 1

The default timeoutSeconds is one second. Under load, a healthy application can easily take longer than a second to answer a health request, especially if the health handler shares a thread pool with real traffic. The probe times out, counts as a failure, and eventually restarts a container that was merely busy.

Fix: set timeoutSeconds to a realistic value (two or three seconds is common) and make sure your health handler is cheap and does not contend with request-serving resources.

No readiness probe at all

Without a readiness probe, a pod is considered ready the instant its container starts. During a rolling update, Kubernetes will send production traffic to a pod that has not finished loading configuration or warming caches, producing a burst of errors on every deploy. Always define readiness for anything that serves traffic.

Startup probe window shorter than real boot time

If your startup probe's total budget (periodSeconds times failureThreshold) is shorter than the application's worst-case boot time, the kubelet gives up and restarts mid-boot, and you are back in CrashLoopBackOff. Measure your real p99 boot time and set the budget above it with margin.

Debugging probes on a live cluster

When a probe is misbehaving, these commands get you to root cause quickly.

# See probe failure events and restart counts
kubectl describe pod payments-api-7c9f8b6d4-x2k9p

# Watch restart count climb in real time
kubectl get pod -l app=payments-api -w

# Inspect the exact probe config the kubelet is using
kubectl get pod payments-api-7c9f8b6d4-x2k9p -o yaml | grep -A15 livenessProbe

# Reproduce the probe by hand from inside the cluster
kubectl exec -it payments-api-7c9f8b6d4-x2k9p -- \
  curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" localhost:8080/healthz

That last command is the single most useful one. If curling the health endpoint by hand is slow or returns a non-2xx status, the problem is in your application or your probe timing, not in Kubernetes. If it is fast and returns 200 but the probe still fails, look at timeoutSeconds, the port, and whether the probe is hitting the right network interface.

Restart counts and probe-failure rates are also worth turning into first-class signals. Exposing them on a dashboard alongside your other reliability metrics is a natural extension of the practices in our guide to error budgets for SRE teams — a rising restart rate is often the earliest warning that a probe is fighting your application.

Recommended defaults to start from

There is no universally correct configuration, but these starting points are sane for a typical stateless HTTP service and can be tuned from there:

• Startup probe: periodSeconds: 5, failureThreshold set so the total budget comfortably exceeds your measured worst-case boot time.
• Readiness probe: periodSeconds: 5, timeoutSeconds: 2, failureThreshold: 3, pointed at a /readyz endpoint that checks real dependencies.
• Liveness probe: periodSeconds: 10, timeoutSeconds: 2, failureThreshold: 3, pointed at a cheap /healthz endpoint that checks only the process.

Always prefer a startup probe over inflating initialDelaySeconds on liveness. The startup probe is purpose-built for boot time and gives you a clean signal for when the application has actually started, instead of a blind fixed delay.

Conclusion

Kubernetes probes are simple to declare and easy to get subtly wrong. The three-line summary to carry with you: liveness restarts a broken process, readiness gates traffic to a healthy one, and a startup probe protects a slow boot from both. Split your health endpoints so a dependency blip can never trigger a restart storm, set your timeouts to survive real production load, and use a startup probe instead of guessing at an initial delay.

Get these right and probes fade into the background, quietly keeping traffic on healthy pods and restarting only the containers that genuinely need it. Get them wrong and they become one of the most confusing sources of production instability in the whole platform. The difference is almost entirely in the YAML you have now seen.