Kubernetes probes: what liveness, readiness and startup checks should (and should not) test
A practical guide to Kubernetes liveness, readiness and startup probes: what each one does, sensible settings, and the dependency-check mistake to avoid.
Kubernetes asks your containers three different questions, and each one has a different consequence when the answer is "no". Teams that treat the three probes as one health check usually find out the hard way, often during a database hiccup.
Here is what each probe is for, how to set them, and what to keep out of them.
Three questions, three consequences
| Probe | Question | If it fails |
|---|---|---|
| Liveness | Is this process stuck beyond recovery? | The kubelet restarts the container |
| Readiness | Can it serve traffic right now? | The pod stops receiving traffic from its Service; no restart |
| Startup | Has it finished starting? | Other probes wait; if it never succeeds, the container is restarted |
The Kubernetes documentation describes the liveness case plainly: the kubelet uses it to know when to restart a container, for example one deadlocked in a state it cannot leave on its own.
The consequence is what matters. A failing readiness probe is gentle: traffic goes elsewhere and the pod can come back. A failing liveness probe is a restart, which throws away in-memory state, drops connections and costs startup time.
Pick the probe type
Kubernetes can check a container in several ways. The two you will use most:
- httpGet: the kubelet sends a GET request. The docs say any status of 200 up to (not including) 400 counts as success.
- exec: the kubelet runs a command in the container; exit code 0 is success.
There are also TCP and gRPC checks. For a typical web service an HTTP endpoint is the clearest choice.
containers:
- name: api
image: example/api:1.4.2
ports:
- containerPort: 8080
startupProbe:
httpGet: { path: /healthz, port: 8080 }
periodSeconds: 5
failureThreshold: 30 # up to ~150s to start
livenessProbe:
httpGet: { path: /healthz, port: 8080 }
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 5
failureThreshold: 2
Note two separate endpoints. That is deliberate.
What liveness should test
Almost nothing. A liveness endpoint should answer "is the main loop of this process still running?" and nothing more. It returns 200 without touching a database, a queue or another service.
Warning
Do not check your database from a liveness probe. When the database slows down, every pod fails liveness at once, Kubernetes restarts them all, and they come back to a database that now also has to absorb a wave of reconnections. A dependency problem becomes a restart storm.
Keep it so cheap and local that the only way to fail is for the process itself to be wedged.
What readiness can test
Readiness is the right place to ask "can I do useful work?" It may reflect things the pod needs: its cache is warm, its configuration is loaded, it holds a working connection to its database. Failing it only takes the pod out of rotation.
Even here, think about shared dependencies. If every pod depends on the same database and it goes down, every pod turns unready and the Service has no endpoints. Sometimes that is the honest answer. Sometimes it is better to stay ready and return clear errors for the requests that need the database, while serving the ones that do not. Decide on purpose.
Readiness is also your friend during shutdown. When a pod is told to stop, make the readiness endpoint start failing first so new traffic drains away, then finish in-flight requests before exiting.
What startup is for
Slow starters, such as an app that loads a large model or runs warm-up work, create a dilemma. A short liveness delay kills them before they are up; a long delay makes real failures take forever to notice.
A startup probe resolves this. Liveness and readiness checks hold off until the startup probe succeeds, so you can allow a generous window for boot (here, thirty tries at five seconds) and still keep the liveness probe tight afterwards.
Settings that matter
The timing fields are periodSeconds, timeoutSeconds, failureThreshold and successThreshold, plus initialDelaySeconds. Check the current defaults in the docs for your Kubernetes version rather than relying on memory. Some practical guidance:
- Timeout: make it comfortably longer than a normal response under load. A one-second timeout on a busy pod produces false failures.
- Failure threshold: require several consecutive failures before acting. One slow response should not restart anything.
- Period: every 5 to 10 seconds is plenty for most services. Probing more often adds load and rarely helps.
- Prefer a startup probe over a long
initialDelaySeconds. A fixed delay is a guess; a startup probe waits only as long as needed.
Testing probes before production
You can rehearse failure cheaply:
- Make the liveness endpoint return 500 on a test pod and watch
kubectl describe podshow the restarts and their reason. - Make readiness fail and confirm the pod leaves the Service endpoints but is not restarted.
- Slow down your database in staging and check that no pod restarts because of it.
- Roll out a new version and watch for traffic errors; if you see some, look at how the old pods shut down.
Related to that last point, an application that is careful about retries and duplicate requests copes with brief unready periods much better; our post on idempotency keys explains the pattern.
A short checklist
- Separate endpoints for liveness and readiness.
- Liveness: local, cheap, no dependencies.
- Readiness: ready to serve, and failing first on shutdown.
- Startup probe for anything that boots slowly.
- Several failures before action; timeouts sized for load.
- Test the failure paths on purpose.
Done well, probes are quiet. Pods start, take traffic when they are ready, and restart only when they are truly stuck.
