Health endpoint & deploy/rollback health gate

DeployMill keys every "is this deploy good?" decision off one app-owned health endpoint. deploy, rollback, get_app (its health block), and auto-rollback all probe the same path and apply the same rule:

200 means everything's good. Anything else (a non-200 status, a connection refused, or a timeout) means the deploy is bad → roll back.

There is one function and one rule. Put your app's real readiness checks in that handler and return 200 only when they all pass.

The contract

• Every web app exposes one health endpoint. The default path is /healthz (the starter templates already serve a real 200 there).
• The handler should assert whatever "ready" means for your app (DB reachable, migrations ran, a required file exists, an upstream API answers) and return 200 only when all of them pass. Return a non-2xx (e.g. 503) when any check fails.
• An agent debugging a runtime issue can extend this handler to assert whatever it needs; the gate then enforces it on every deploy.

Probe semantics

The probe HEADs the resolved health path on each attached domain.

• Strict mode (a real health path like /healthz): 200 = healthy. Any other status, a connection error, or a timeout = unhealthy. A 500 or 503 from the app counts as a failure (unlike the legacy gateway-only rule).
• Lenient mode (the bare root /, the opt-out): only 502/503/504 or "can't connect" count as unhealthy. Any 2xx–4xx proves the edge routes. This is the pre-DET-120 behavior.
• Fail N times → unhealthy. Edge reporting requires N consecutive healthy probes to confirm health and declares failure only after N consecutive failures, spaced by intervalMs. For auto-rollback on a backend that can identify release instances, DeployMill separately checks for a ready, non-terminating Service endpoint running the new deployment. That check retries up to N times. A draining old endpoint cannot decide the verdict: if the target release is ready and the edge observations contain a 200 mixed only with transient connection failures, 502, 503, or 504 responses, DeployMill treats the edge as converging. An edge that never returns 200, or returns an application error such as 500, still fails.
• Protected apps: the probe adapts to the gate. A site-protected app can't return 200 to an unauthenticated probe, so the probe reads the app's protection policy first and takes one of two regimes:
  • mode: "basic" → the probe is auth-gated: 401 and the redirect statuses 302 / 303 / 307 / 308 count as healthy in strict mode, because they prove the edge is routing to a live app. Real failures (502/503/504, connection errors, timeouts) stay unhealthy. This is what stops Basic-Auth protection from tripping a false auto-rollback.
  • mode: "organization" (stored as workspace) → the probe sends the app's protection bypass header on every attempt and expects a clean 200. Redirect tolerance is deliberately not applied here: a 302/401 under the bypass means the gate rejected it, which is a genuine failure the gate should catch.

If the protection policy can't be read, the probe treats the app as unprotected (the app then reads unhealthy rather than the probe crashing) — the documented fail-open exception.

• 404 fallback. If a strict health path returns 404 (the endpoint simply isn't there, e.g. an older app that never added /healthz), the probe falls back to a lenient root / probe instead of treating the missing endpoint as a hard failure. Add the /healthz handler to get the strict gate.

Declarative config

Add a health block to .deploymill/project.json:

{
  "health": { "path": "/healthz", "retries": 3, "intervalMs": 3000, "timeoutMs": 5000 }
}

reconcile_project mirrors the resolved block into the app's metadata so deploy (a primitive that never reads project.json) can read it. reconcile's plan.health reports { current, desired, action, orchestratorGate }.

Defaults / back-compat. Omitting the block keeps existing apps working: the probe still defaults to /healthz but falls back to lenient / on a 404, and no orchestrator-level health gate is wired (no platform-layer behavior change for apps that didn't ask for it). New web apps are scaffolded with the block declared.

Orchestrator-level gate (declaring health opts in)

When a web app declares a health block, reconcile_project also wires an orchestrator-level health gate with a health-gated rolling update. That means the orchestrator:

• won't shift traffic to the new container until its port is accepting connections, and
• rolls back the update if the new container never starts listening within the monitor window (and restarts a container whose port later goes dead).

This gate is a TCP port-readiness probe, not an in-container HTTP check: the kubelet just opens a connection to the app's port, so it needs no probe binary (curl/wget/python) inside the image and works for any stack or BYO Dockerfile. After rollout convergence, DeployMill correlates the new deployment image with a ready, non-terminating Service endpoint. The strict HTTP /healthz requests stay on the public edge path, which preserves the cluster's network-isolation boundary. DeployMill records both the release identity observations and the edge requests. An ingress keepalive connection to an older draining pod cannot decide the new release's health.

Notes:

• The gate is stack-independent. It wires for every web app with a container port, including images that ship no shell utilities.
• Wiring is best-effort: a backend rejection is surfaced as a warning and never fails the reconcile; the post-deploy edge probe is the reliable strict layer regardless.

Reading health on demand

get_app({ applicationId }) returns the app's config/detail PLUS a live health block (status, edge probe, image drift, hibernation; pass diagnostics:true for pod-level reasons like CrashLoopBackOff / OOMKilled / ImagePullBackOff and storage warnings). By default it runs a live edge probe. Pass probeEdge:false for a cheap config-only read. When an app is actually misbehaving, prefer debug_app({ applicationId }), which bundles this health snapshot together with filtered build + runtime logs, env-key sources, recent health-gate verdicts, and recent audit entries in one read-only call, so you debug without 5+ round-trips. See deploymill://guides/logs.

Auto-rollback keys off this gate

With "rollback": "auto", DeployMill automatically reverts a deploy when the target release never gains an eligible endpoint or its edge health fails within the bounded retry window. It reverts to the most recent earlier deploy that was recorded healthy. debug_app and get_timeline retain the observation timestamps, URL/path, HTTP status or network error, deployment/revision, pod/endpoint identity when available, and final reason. See deploymill://guides/rollback for the full flow.

Writing a real health handler

Node (Hono):

app.get("/healthz", async (c) => {
  try {
    await pool.query("SELECT 1"); // DB reachable + migrations ran?
    return c.json({ ok: true });
  } catch {
    return c.json({ ok: false }, 503); // not ready → deploy stays on the old image
  }
});

Python (FastAPI):

@app.get("/healthz")
def healthz(response: Response) -> dict:
    try:
        engine.connect().close()
        return {"ok": True}
    except Exception:
        response.status_code = 503
        return {"ok": False}

What NOT to do

• Don't make /healthz heavy. It's hit repeatedly by the post-deploy edge probe (and on each deploy). Keep checks fast (a SELECT 1, a file stat), not a full integration test.
• Don't return 200 unconditionally if you care about readiness. A handler that always returns 200 defeats the gate. The deploy will look healthy even when the DB is down.
• Don't put liveness-only logic here and expect readiness gating. One endpoint, readiness semantics: 200 iff the app can actually serve.
• Don't probe a path that needs auth or a body. The probe is an unauthenticated HEAD. Keep /healthz open and HEAD-able.