A liveness probe can turn a 10-second database hiccup into a 40-minute outage — and it's one of the most common misconfigurations in production clusters. The cause is almost always the same: the probe is checking the wrong thing.
Liveness probes have one job: tell Kubernetes "this process is stuck, kill it and start over." That's it. They are not a health check. They are not a "is everything downstream okay" check. The moment a liveness probe starts asking a question bigger than "is this process alive and able to respond at all," it stops protecting you and starts causing outages.
Here's the failure mode, in order:
- Your app's liveness endpoint pings a downstream dependency — a database, a cache, another service — to decide if it's "healthy."
- That dependency gets slow. Not down. Slow. A connection pool fills up, a query takes 8 seconds instead of 80ms.
- Every pod running your app fails its liveness probe at roughly the same time, because they all depend on the same slow thing.
- Kubernetes restarts all of them. Now you have zero running pods for the time it takes them to start back up.
- On startup, every pod tries to reconnect to the same already-struggling dependency at once, which makes the slowness worse, which means some pods fail their liveness probe again before they've even finished starting.
You've turned a slow dependency into a full outage, and the thing that caused it is the exact mechanism that was supposed to make you more reliable. This pattern shows up under a lot of different root causes — a Postgres connection pool filling up, a Redis instance under memory pressure, a slow upstream API — but the blast pattern is identical: liveness probe checks a dependency, dependency gets slow, every replica restarts at once, restart storm.
What each probe is actually for
Liveness probe — "Is this process stuck in a way it can never recover from on its own (deadlock, infinite loop, unrecoverable panic)?" If yes, restarting helps. If the answer to "why did this fail" is anything involving another service, it is the wrong check for liveness.
Readiness probe — "Can this pod handle traffic right now?" This is where dependency checks belong. If the database is slow, you want the pod pulled out of the Service's load-balancing rotation — not restarted. The process itself is fine; it just shouldn't receive new requests until the dependency recovers. Readiness failing removes a pod from endpoints without killing it, and the pod rejoins automatically once the probe passes again.
Startup probe — exists so a slow-starting app (JVM warmup, loading a large model into memory, running migrations) doesn't get killed by an impatient liveness probe before it's even finished booting. While the startup probe hasn't yet succeeded, Kubernetes doesn't run liveness or readiness checks at all. Once it succeeds once, it's done — liveness and readiness take over.
The fix: split the endpoints
Give your app two distinct endpoints, not one shared one.
from fastapi import FastAPI, Response, status
import asyncio
app = FastAPI()
# /healthz: liveness. Checks only "is this process able to
# respond on its own event loop" — nothing external, nothing
# that can be slow for reasons outside this process's control.
@app.get("/healthz")
async def liveness():
return {"status": "alive"}
# /readyz: readiness. This is where dependency checks belong —
# it's allowed to fail without anything being killed.
@app.get("/readyz")
async def readiness(response: Response):
try:
await asyncio.wait_for(check_database(), timeout=2.0)
await asyncio.wait_for(check_cache(), timeout=2.0)
except Exception:
response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
return {"status": "not ready"}
return {"status": "ready"}
async def check_database():
# a real, cheap query against your actual connection pool,
# e.g. `await db.execute("SELECT 1")`
...
async def check_cache():
# e.g. `await redis.ping()`
...And the matching probe config, tuned so liveness is slow to act and readiness is fast to react:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
template:
spec:
containers:
- name: api
image: ktechhub/api:latest
ports:
- containerPort: 8000
startupProbe:
httpGet:
path: /healthz
port: 8000
periodSeconds: 10
failureThreshold: 30 # up to 5 min to finish starting
livenessProbe:
httpGet:
path: /healthz
port: 8000
periodSeconds: 15
timeoutSeconds: 3
failureThreshold: 5 # ~75s of consecutive failure before a restart
readinessProbe:
httpGet:
path: /readyz
port: 8000
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2 # pulled from traffic fast — no restart involvedNotice the asymmetry on purpose: liveness has a long window (75 seconds of consecutive failures) because restarting is a big, disruptive action and false positives are expensive — a process that's merely slow for a few seconds shouldn't get killed. Readiness has a short window (10 seconds) because pulling a pod from rotation is cheap and reversible — you want to react fast and recover just as fast when the dependency comes back.
When to skip liveness entirely
If your app has no internal state that can get permanently stuck — no thread pool that deadlocks, no in-memory cache that can corrupt, nothing that only a restart fixes — you may not need a liveness probe at all. A stateless API process that can only ever be "slow because a dependency is slow" gains nothing from liveness and loses the restart-storm risk entirely. Readiness alone, backed by the kubelet's own process-exit detection (a crashed process already gets restarted without any probe), covers that case. Don't add a liveness probe just because the Kubernetes docs show one in every example — add it because you have a real failure mode a restart actually fixes.
Follow for more Kubernetes reliability content.
Experienced DevOps Engineer proficient in Python, Docker, Kubernetes, CI/CD, and cloud technologies. Specialized in automating processes and building scalable infrastructure for seamless development and deployment.
