Deploy-safety contract: surviving a deploy with work in flight

When you deploy, DeployMill starts a new container and then shuts the old one down. Requests that are open when that happens are drained by the platform — you don't have to do anything.

**Work that is not attached to an open request is a different problem.** If your app kicked off a job and returned from the HTTP call that started it — an agent turn the client polls for, a background import, a webhook fan-out, a build you're orchestrating — nothing is holding that work open, so nothing protects it. The old container is stopped, and the job dies partway through.

Fixing that takes both halves of a contract, and neither half works alone:

• Your half: when the platform tells your app to stop, it stops taking new work, finishes what's in flight, and exits.
• DeployMill's half: gracePeriodSeconds in .deploymill/project.json — how long the platform waits for you before it kills the process.

If you declare the window but never handle the signal, your app dies instantly as it always did. If you handle the signal but never declare a window, you're killed at the default 30 seconds, mid-drain.

How a shutdown goes

1. DeployMill starts the new container and waits for it to be healthy. (See deploymill://guides/health — a new container that never goes healthy means nothing is shut down at all.)
2. The old container is sent SIGTERM.
3. It has gracePeriodSeconds to exit on its own.
4. If it hasn't exited by then it gets SIGKILL — an unblockable, immediate kill. Nothing runs after it: no cleanup, no flush, no last write. Whatever was in flight is simply gone.

There is no checkpoint and no resume. The deadline is a hard kill, so the window has to be big enough for your genuine worst case.

Your half: draining on SIGTERM

Four things, in order:

1. Trap SIGTERM and don't exit. The default behavior of most runtimes is to die immediately, so this is the step that changes anything at all. Trap SIGINT too if you run the app locally.
2. Stop accepting new work. Close the HTTP listener to new connections, stop pulling from the queue, refuse to start a new job. This is what makes the drain finite — an app that keeps admitting work never becomes idle.
3. Let in-flight work finish. Track it (a counter, a set of promises, a job table) so you can tell when it's done. This is the whole point of the exercise.
4. Exit as soon as you're idle. Don't sleep out the window. A pod that exits in 200ms when nothing is running is why a large gracePeriodSeconds is cheap: you pay the window only when you're actually mid-job.

Bound your own drain too. Pick a deadline under gracePeriodSeconds (leave yourself a margin — say 30s) and give up at it: log loudly what you abandoned and exit. Exiting on your own terms beats being killed, because you at least get to record what was lost.

A /healthz handler that starts returning non-200 the moment you begin draining is a nice touch — it tells the edge to stop routing to you.

// Node/Express sketch. The shape matters more than the framework.
let draining = false;
const inFlight = new Set();

app.get("/healthz", (_req, res) => res.status(draining ? 503 : 200).end());

app.post("/jobs", (req, res) => {
  if (draining) return res.status(503).json({ error: "shutting down" }); // (2)
  const job = runJob(req.body).finally(() => inFlight.delete(job));
  inFlight.add(job);
  res.status(202).json({ id: job.id }); // returns NOW — this is the work no drain protects
});

process.on("SIGTERM", async () => {                                    // (1)
  draining = true;                                                     // (2)
  server.close();
  const deadline = new Promise((r) => setTimeout(r, 570_000).unref()); // under the window
  await Promise.race([Promise.allSettled([...inFlight]), deadline]);   // (3)
  if (inFlight.size > 0) console.error(`abandoning ${inFlight.size} job(s)`);
  process.exit(0);                                                     // (4)
});

Two runtime gotchas worth checking:

• Signals must actually reach your process. If your Dockerfile's entrypoint is a shell wrapper (CMD sh -c "node server.js", or a start script), the shell is PID 1 and your app may never see SIGTERM. Use exec form (CMD ["node", "server.js"]) or exec the real process.
• Don't hold the exit open. Once you're idle, exit. An open DB pool or interval timer can keep the process alive to the deadline even though the work is done.

DeployMill's half: gracePeriodSeconds

{
  "version": 2,
  "name": "my-app",
  "gracePeriodSeconds": 600
}

Then reconcile_project to store it, and deploy to apply it.

Size it to your worst case, not your average. At the deadline the work is killed. If a turn can legitimately take eight minutes, a 5-minute window means the slow turns are exactly the ones you lose.

What a long window costs. A draining container holds its full memory reservation and keeps drawing GB-hours from your plan's pool for as long as it stays up — so the cost is real, but you only pay it while work is actually in flight. An app that exits as soon as it's idle pays nothing on a quiet deploy. The 30-minute ceiling exists to bound that.

Deploys don't get slower. deploy returns as soon as the new container is healthy and serving. The old one drains behind it; nothing waits on the window.

Checking it worked

• reconcile_project reports the change in its gracePeriod section (action, seconds) — run it with dryRun: true first to see the plan without applying it.
• After the next deploy, use get_logs on the deploy that replaced the container: a draining app that logs on SIGTERM and on exit gives you the whole story, including how long the drain actually took. That's the number to size the window from.

What this is not

• Not connection draining. Open requests are already handled by the platform. This is for work that outlives its request.
• Not a checkpoint. Nothing is saved or resumed at the deadline. If losing a job is unacceptable no matter what, make it restartable — write its state somewhere durable and pick it up on boot — rather than buying a bigger window.
• Not zero-downtime by itself. That's the health gate's job (deploymill://guides/health); this is about what the outgoing container gets to finish.