Here is the most expensive one-line lesson I've learned building this thing, and I want to tell it on myself because it's the kind of mistake that feels safe right up until it isn't.
We were testing app protection. Create a static app, deploy it, add protection. start_project failed at the very last step (attaching the domain) with a Caddy error: unknown field "dm-port". I traced it in a few minutes: the Caddy client was stashing the container port and TLS intent as extra top-level fields on each route, and Caddy's strict decoder rejects any field it doesn't recognize. Clean diagnosis. One-file fix. Tests green. Merged. Self-deployed to prod.
Minutes later, every host was down. Not the new app. Everything. The marketing site, every tenant app, the MCP endpoint I use to control the platform. Total ingress outage.
How a fix for one route deleted all of them
Two facts I didn't have in my head at the same time combined into a disaster.
The first: the route-writing code doesn't run where I thought it ran. The control plane is actually two processes that bundle the same code. The in-cluster pod can't reach Caddy's admin API directly, so all route writes go through a separate broker process on the manager box, and the broker is upgraded on its own schedule, not by a self-deploy. So my fix shipped to the pod. The broker kept running the old, buggy code. A self-deploy is control-plane-only. It is half a deploy when the change touches ingress.
The second, worse fact: the route writer deletes before it recreates. When the pod restarted, its boot routine re-asserted every route through the still-broken broker. For each route: DELETE succeeded, re-CREATE failed on dm-port. One by one, the entire route table emptied itself. A host with no route presents no certificate, the TLS handshake fails, and the curl just returns 000. A write bug had become a data-loss bug, because the operation that hit it ran in bulk over a delete-then-recreate primitive.
The three scars
A self-deploy is not a full deploy. Anything that shapes how Caddy routes get built lives in code the broker also runs, and self-deploy never touches the broker. Ingress-shaping changes are now a two-target deploy, called out at the top of the runbook in red.
Delete-then-recreate turns a write bug into a data-loss bug. The bug alone "only" blocked new attaches. The outage came entirely from the DELETE running first across every route. Anything a boot routine re-asserts in bulk deserves real paranoia. Degrade, don't delete.
Green tests against a mock prove nothing about the external system. The Caddy tests mock the admin API, so a route shape that a real Caddy rejects sailed through every test. The exact thing I changed, the JSON Caddy will accept, was the one thing the tests structurally couldn't see. When the contract is "an external service accepts this," a mock is not a test of that contract.
The meta-lesson: roll back first, diagnose second
My instinct when prod is down is to understand why. That instinct is wrong. The right order is: restore service, then find the cause. Recovery here was pointing the broker at the fixed image and restarting the pod so boot-reconcile could re-push every route, but I'd have gotten there calmer, and faster, if I'd treated "get it green again" and "explain it" as two separate jobs instead of one.
I'm telling you this because the platform's whole pitch is that it's safe enough to hand an agent the keys to production. Earning that means being honest about the day I took prod down with a one-file fix that passed every test, and then making the system the kind of thing where that specific mistake can't land silently again.
- Ryan