We spent last week fixing things that our own monitoring said were fine. The pattern repeated often enough to be worth writing down, because it is not really about any of the individual bugs.
Three of our checks reported success while the thing they were checking was broken. In each case the check was looking at its own work rather than at the outcome.
Three ways to be told everything is fine
“Verified 1/1 copies.” Our certificate sync verified the one copy it knew about. It was correct. Meanwhile the copies it did not know about were the ones serving expired certificates to every visitor.
A build that succeeds by exiting. Our build command returns success even when it refuses to run — when another build is already in progress it prints a message, does nothing, and exits zero. We deployed nothing at all, twice, and were told it worked.
A guard satisfied by a log line. This one is ours, freshly made. We wrote a tool to
find API endpoints that check who you are but not what you are allowed to touch. It
searched for evidence of an ownership check — and one of the patterns it accepted was the
text userId:. An endpoint passed because it wrote userId into a logging call. The
check was satisfiable by a log message.
What we found once the checks could fail
Fixing that last one immediately surfaced a real problem. An endpoint that streams live deployment logs confirmed you were signed in, then streamed whichever deployment you asked for. Any signed-in customer could have watched another customer’s build output by guessing an identifier.
It had been there a while. Every automated check passed, because every automated check was asking whether the code authenticated you — which it did, correctly, right before handing you someone else’s data.
That endpoint now verifies the deployment belongs to you. More usefully, the check that found it runs on every build, so an endpoint of that shape cannot ship again.
The habit we have adopted
Two rules, both learned the hard way.
A check must observe the outcome, not its own output. Reading back the file you wrote proves the write worked. It proves nothing about the system. Our certificate renewal now ends with a real connection to our own domain and compares what it receives against what it installed.
A check that has never failed is not yet a check. Every guard we added last week was deliberately broken first, to watch it fail. We removed the authorisation from that endpoint and confirmed the build refused. We introduced a fake price mismatch and confirmed the job exited with an error. Then we put them back.
Until you have seen a check fail, all you know is that it runs.
Why we are telling you
Because the honest summary of last week is that our tooling was reassuring us while several real problems sat in production, and one of them exposed data across customer boundaries.
We would rather publish that than the version where everything went smoothly. If you build something similar, the question worth asking is not whether your checks pass. It is whether you have ever seen them fail.
