A user reported that a background job had been "stuck for a long time." It hadn't. The job was fine. What had actually happened was that I'd rewritten history on a live production table with a single, ordinary-looking line of SQL, and the lie only surfaced days later, in the UI, as a wall of jobs that all claimed to be running and never finished.
Here's how a migration that succeeded produced bad data.
No migration files: schema is applied at boot
deploymill has no migration directory. Schema changes are applied at process start: CREATE TABLE IF NOT EXISTS, ALTER TABLE … ADD COLUMN IF NOT EXISTS. This is deliberate and it mostly works beautifully. The schema is idempotent, it converges on boot, there's no separate migration step to forget. But it has a property that's easy to forget: these statements run against a live, populated production table, on the way up, every time the control plane starts. There is no staging gate between "merge" and "this ran against every prod row." And because the platform self-deploys, the migration ships itself.
The one line
I was adding run-history columns so a new dashboard tab could show last-run status. The line was:
ALTER TABLE schedule_runs ADD COLUMN IF NOT EXISTS status text NOT NULL DEFAULT 'running'Read it the way I did and it looks additive: new column, sensible default, IF NOT EXISTS so it's safe to re-run. Read it the way Postgres does and it's a history rewrite. ADD COLUMN … DEFAULT x is not additive for rows that already exist: Postgres sets the new column to x for every existing row. That table already held one row per scheduled tick since the feature shipped, all of them long finished. After the migration, all of them said status = 'running'.
The default I picked encoded a current, transient state. Applied to historical rows, every completed run was relabeled as in-flight. The migration reported success. The damage was invisible until someone read the data and saw dozens of jobs apparently frozen mid-run.
The fix is a rule, not a cleanup
I added a reaper that reconciles stale running rows to a terminal unknown, and on the next boot it logged reaped 18 orphaned schedule run(s), which both fixed the table and told me exactly how big the mess had been. But the one-off cleanup isn't the lesson. The class of bug is: every boot migration across the codebase could do this, every time someone adds a defaulted column to a table that already has rows.
So it's a rule now:
- Never let a defaulted column backfill historical rows into a live/transient state. Either add the column nullable and run an explicit one-shot
UPDATEthat sets a correct terminal value for the old rows, or pick a default whose meaning is honest for a historical row. - A bare
ADD COLUMN … DEFAULTre-runs its rewrite on every boot and across every rolling replica, so "honest for old rows" has to be true forever, not just once. - A
NOT NULLwith no default on a populated table fails outright. The error is the good case. It's loud. The defaulted version is the dangerous one, because it succeeds quietly.
This is now flagged automatically: a deploy-safety check refuses to wave through an ADD COLUMN … DEFAULT against an existing table without a backfill story.
Why I find this one unsettling
The outage post on this blog was scary but legible: prod was down, I knew instantly, I fixed it. This one is the opposite, and that's why it stuck with me. The system told me everything was fine. The migration succeeded. The tests passed. The pod was healthy. And it was quietly serving wrong data the whole time, until a human noticed something looked off. The failures that announce themselves are a gift. The ones that don't are the ones worth building guardrails against.
- Ryan