Health checks: liveness, readiness, startup

Overview

Health endpoints are the contract between a service and the platform that runs it — and the design question is best asked from the probed side: what should the orchestrator do when this check fails? If the answer is "restart me", that's liveness; if it's "stop sending me traffic", that's readiness; if it's "give me more time to boot", that's a startup probe. Most production probe incidents come from conflating the first two — putting dependency checks into liveness turns any database blip into a fleet-wide restart storm.

Key points

• Liveness = "the process is alive and not deadlocked; restart me if this fails." It must be cheap, in-process, and must NOT check dependencies: a dead database would fail liveness on every pod at once and Kubernetes would restart-storm the whole deployment without fixing anything — restarts don't resurrect databases. A bare 200 from the running event loop / a trivial in-process check is correct.
• Readiness = "route traffic to me." It does include critical downstream checks (the database this service cannot serve without, an unfinished cache warmup, a not-yet-completed boot) — failing readiness removes the pod from Service endpoints (load balancing) without restarting it, and it re-enters rotation when the check recovers. Readiness is also how graceful shutdown drains: fail readiness first, keep serving in-flight requests, then exit.
• Startup probe covers slow boots (JVM warmup, migrations, large caches): until it succeeds, liveness/readiness are not evaluated, so liveness can stay tight (failureThreshold × periodSeconds on the startup probe is the boot budget). Without it, teams inflate initialDelaySeconds on liveness and lose fast deadlock detection.
• Readiness on shared dependencies cascades: if every service in a chain checks a shared broker in readiness, one broker incident marks the whole estate unready and takes down even the endpoints that could still serve. Check only dependencies that are hard prerequisites for this service's own traffic; degrade instead of failing where partial service is possible.
• Check design: per-dependency checkers with an overall AND for readiness; strict timeouts on each check (well under timeoutSeconds, or a slow dependency makes the probe itself time out — indistinguishable from a dead process); cache results a few seconds so aggressive periodSeconds doesn't hammer dependencies.
• Exposure: health endpoints are unauthenticated by design — serve them on a separate management port (Actuator management.server.port, Micronaut's management port, a second listener in stdlib stacks) and never route them through the public ingress; the detail payload (dependency names, versions) is reconnaissance material.
• The wiring lives at the assembly point: readiness state is flipped after the object graph is built and adapters are connected — the composition root owns "ready", per the hexagonal placement.

Details

The three probes at a glance

Endpoint catalogue per stack

Convention for hand-rolled stacks: return 200 with a small JSON body ({"status":"UP","checks":[…]} mirrors MicroProfile's schema) and 503 when down — Kubernetes only reads the status code, humans read the body.

Kubernetes wiring

livenessProbe:
  httpGet: { path: /health/live, port: 8081 }
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /health/ready, port: 8081 }
  periodSeconds: 5
  failureThreshold: 3
startupProbe:
  httpGet: { path: /health/live, port: 8081 }
  periodSeconds: 5
  failureThreshold: 60   # 5 min boot budget

httpGet treats any 2xx–3xx as success. Probe handlers must not allocate per-request state worth tracing — exclude them from access logs, correlation middleware, and trace sampling, or they dominate your telemetry volume.

Pitfalls checklist

• Dependency check in liveness → restart storm on dependency outage.
• Shared-dependency check in every readiness → cascading estate-wide unreadiness.
• Probe timeoutSeconds shorter than the slowest included check → false deaths under load.
• No startup probe + slow boot → inflated initialDelaySeconds, slow deadlock detection.
• Health on the public port → information disclosure and probe traffic through the ingress.
• Ready flag flipped before adapters actually connected → 503s/errors during rollout.

Related

• Observability & SRE practice — the strategic map; probes are the operational contract at its most binary.
• Kubernetes & containers deep dive — the prober's side: probe config, restart storms, and the reconciliation loop acting on probe results.
• Correlation IDs & context propagation and Structured logging & MDC — sibling middleware concerns; probes are the traffic they should skip.
• OpenTelemetry — keep probe traffic out of traces and metrics cardinality.
• Frameworks × Jakarta EE — who implements what — MicroProfile Health is the spec behind SmallRye's endpoints; Actuator and Micronaut Management are the equivalent-API column.

Citations

[1] Kubernetes — Liveness, Readiness and Startup Probes [2] Spring Boot — Kubernetes probes (availability & health groups) [3] SmallRye Health / MicroProfile Health