Health and shutdown

A health endpoint is not an assertion about the process. It is an instruction to your orchestrator, and getting one wrong does not produce a wrong answer — it produces a restart loop, or a rolling deploy that drops requests.

Probes

Append ?verbose for the itemised version, in the format the Kubernetes API server uses:

$ curl "localhost:9090/readyz?verbose"
[+] ping ok
[+] database ok
[-] model failed: this database was provisioned from a different model; run `runtime db apply`
readyz check failed

A probe reads the status code. A human debugging at 03:00 reads the body, and “which check failed” should not require a log dive.

/livez never checks a dependency

This is the single most common and most expensive mistake in a Kubernetes deployment, so it is worth being explicit: liveness here checks only that the process is functioning. It does not touch the database, and it never will.

If it did, a database blip would restart every replica simultaneously — turning a dependency outage into a self-inflicted one at the exact moment the database can least afford a thundering herd of reconnects. A restart does not bring the database back; it only adds a cold start to an outage already in progress.

Dependency checks belong in readiness, where the consequence is “stop sending me traffic” rather than “kill me”.

livenessProbe:
  httpGet: { path: /livez, port: 9090 }
readinessProbe:
  httpGet: { path: /readyz, port: 9090 }

Set terminationGracePeriodSeconds above 35

The one configuration mistake that silently undoes real work.

A termination signal starts a drain, not a shutdown:

1. The signal arrives. Readiness starts failing immediately — that is what tells the endpoints controller to stop routing here.
2. The listener keeps serving for RUNTIME_DRAIN_DELAY (5s by default), because removing a pod from a Service is eventually consistent and traffic is still arriving.
3. It stops accepting and finishes what is in flight.
4. The admin listener stops last, so liveness answers throughout.

Why the delay exists: Kubernetes sends SIGTERM and removes the pod from its Service concurrently, and the removal has to reach every kube-proxy and every ingress. A process that stops accepting the moment the signal lands spends that entire window refusing connections still being routed to it — which is a 502 to somebody, on every rolling deploy, for as long as nobody looks closely enough to notice.

So a shutdown legitimately takes up to the drain delay plus the request timeout: 5 + 30 with the defaults. Kubernetes’ default grace period is 30, which is less.

terminationGracePeriodSeconds: 45

Left at the default, the pod is killed mid-drain and the mechanism is worse than not having it: the delay is spent and the in-flight work is thrown away anyway.

Step 4 matters for the same reason. A liveness probe that got connection-refused at second three of a thirty-second drain would report the pod as dead, and the kubelet may kill it — throwing away exactly the work the drain existed to protect.

What a saturated pool looks like

Not a hang. Requests wait five seconds for a connection and then fail, which is deliberately shorter than the request timeout. You see errors and a readiness that still passes — the database is reachable, there are just not enough connections. Configuration has the knob.