deploymill deploys deploymill. The control plane is registered as one of its own apps, and shipping a new version of the platform is, mechanically, the platform deploying an app. Its own. I dogfood the product on the product. It's the best test harness I've ever had and it has exactly one terrifying failure mode.
The bootstrapping problem
A normal deploy is clean: a build job produces a new image, the rollout swaps the pod, and the new code only ever runs after it's live. The new version proves itself on the readiness probe before the old one is torn down. If it's broken, it never becomes ready, and the old one keeps serving. Auto-rollback. This is why tenant apps can never get stuck. A bad deploy just doesn't converge.
The self-deploy is different, and the difference is the whole story. The code that performs the self-deploy runs on the currently running pod (the old image) because that's the process handling the request. The new image isn't live yet. It can't be, because deploying it is the very thing the old pod is in the middle of doing.
So picture a commit that introduces a bug in the self-deploy code path itself. Now:
- The fix is in the new image.
- Shipping the new image requires the old pod's self-deploy code to run to completion.
- The old pod's self-deploy code is exactly what's broken.
It errors before it ever swaps the image. The new image never goes live. The only code that could deploy the fix is the broken code that can't. That's a deadlock, and you cannot deploy your way out of it. You have to reach past deploymill entirely and force the new image onto the deployment with raw cluster credentials.
The footgun inside the footgun
The sneakiest way to walk into this is permissions. The in-cluster identity is deliberately least-privilege. It grants get/create/update/delete on secrets, for instance, but not patch. Add a single k8s call to the self-deploy path that uses a verb the role doesn't grant, and the old pod gets a 403 right in the middle of updating itself. Same deadlock, introduced by one new API call that looked completely innocent. Every call site now carries a // requires verb: comment, and the self-deploy path gets read against the RBAC grants before anything new lands in it.
The clock can't be an app
There's a sibling lesson here. All the platform's scheduled maintenance (cleanups, token refreshes, the per-minute dispatcher) runs off a single clock. The obvious move is to run that clock as a deploymill-managed app. I tried. It creates a self-maintenance deadlock: the thing that maintains deploymill can't depend on deploymill being healthy to maintain itself.
So the clock is committed infrastructure, not an app. It's the same image, recoverable with plain kubectl independent of the control plane. When the system that fixes things needs fixing, the fixer can't be one of the things that's down.
Why I keep the sharp edge
I could avoid all of this by not self-hosting, deploying the platform some other, safer way. I won't, and not out of stubbornness. Self-hosting means every deploy of deploymill is a real, end-to-end exercise of the exact path tenants use. The day the self-deploy breaks is the day I find out something is wrong before a customer does. The deadlock is the price of that signal, and the answer isn't to remove the dogfooding. It's to keep an honest break-glass path that doesn't depend on the thing being deployed. Recovery for a wedged self-deploy is a documented kubectl set image with the operator's own credentials, by hand, every time. That escape hatch is non-negotiable, precisely because the rest of the system is allowed to eat itself.
- Ryan