I shipped a security control with a verification script that checked five things, watched all five pass, applied the control to production, and took every tenant app offline in seconds. The script wasn't wrong about the five things. It was just checking the wrong direction.
What I was trying to do
Lock down tenant network access: stop apps from reaching each other, from hitting the cluster's internal API, from reaching the cloud metadata endpoint, and (for free-tier apps) from reaching the open internet at all. The right way to do this is a NetworkPolicy. I wrote one, and I wrote a verifier that ran a battery of checks from inside a tenant pod: can it resolve DNS, reach the database, reach the internet, reach the API server, reach metadata? The first three should pass, the last two should fail. They did, exactly as designed. So I applied it.
Within seconds, every tenant app's public URL returned a 502, and sleeping apps wouldn't wake. I deleted the policy. Service came back instantly. Then I spent a while understanding what just happened.
Outbound-only verification gives false confidence
Every single check I ran tested traffic going out of a pod. The thing that broke was traffic coming in to it.
A NetworkPolicy with a default-deny on ingress blocks inbound connections, and inbound is how public traffic reaches a tenant app. Ingress hits the app over a node port, and on this setup the proxy rewrites that traffic's source to the receiving node's own IP before it arrives. So a blanket "deny ingress" dropped it. The pod was perfectly healthy on the inside. Only the inbound edge was dead. My verifier never saw it because my verifier never sent a request in. The fix was to explicitly allow ingress from the node IPs, but the deeper point is that I'd validated the half of the policy that wasn't the dangerous half.
(Two more provider-specific gotchas surfaced while fixing it forward: the API server leaking past an address-range deny because the proxy rewrites its destination before the policy is evaluated, and a label-ordering hazard when applying a policy across a fleet that's half-migrated. The shape of every one of them was the same: the behavior that bit me was invisible to the checks I'd written.)
A guardrail that isn't a hard automated check isn't a guardrail
The most uncomfortable detail: I had thought about the inbound path. The summary the verifier printed even reminded me, in prose, to manually confirm the public URL still served. A "tip." A note for a human to follow.
That note is worthless, and worse than worthless, because it created a feeling of coverage without the coverage. If a check is a sentence in a summary that a tired human is supposed to act on, it will eventually ship broken. So the public-URL path is now check (g): a hard, automated FAIL that dials the node port from the manager exactly the way ingress does, and the rollout doesn't pass until it's green. Either the guardrail is a gate that can fail the process, or it isn't a guardrail. It's a suggestion.
Two operational rules I now hold
Roll back first, diagnose second. One command deleted the policy and restored every app in seconds. Chasing root cause with prod down would have been the wrong order, and the temptation to do exactly that is strong when you think you almost understand it.
Infra controls have to be tested on real infrastructure. A preview environment can't exercise node-port behavior, source rewriting, or the quirks of a specific network layer. Only a real cluster can, even a single-node one. This is precisely the gap a staging environment exists to close: the class of change that only fails against the real thing is the class you must never validate against a mock.
The irony isn't lost on me. I write a lot here about guardrails being part of the interface. This is the story of building one badly: a control that protected production from tenants while briefly taking production down, caught by a verifier that confidently checked everything except the thing that mattered.
- Ryan