Our certificate renewal worked perfectly for months. Every night a timer checked the expiry date, renewed when it got close, copied the result where it needed to go, and reported success. Last week we found that every single one of our domains was eighteen days from serving an expired certificate.
The renewed certificate existed. It was valid until late November. It was sitting in the right place, correctly synced, and nothing was using it.
How a system reports success while failing
TLS certificates in Kubernetes are stored as secrets, and secrets belong to a namespace. An application in one namespace cannot read a secret in another. So when we provisioned a customer’s application, we copied the wildcard certificate into their namespace.
Nothing ever renewed those copies.
That alone would have been a slow leak — each customer’s certificate silently freezing
at the moment they signed up. The part that turned it into a platform-wide problem is
subtler. A stale copy still matches. It still says it is valid for *.boottify.com.
When a browser asks for a connection, the router picks one of the certificates that
match, and it kept picking a stale one.
So: certificate renewed, copy synced, sync verified, and the edge serving something three months out of date.
The check that could only see its own work
Our sync script ended by verifying its work, and the line it printed was:
Verified 1/1 copies
That was true. It was also useless. The script knew about one copy, so it checked one copy, and found it correct. The copies it did not know about were the problem, and no amount of checking the ones it knew about would ever reveal them.
This is the part worth taking away, because it is not really about certificates. A check that reads back the thing it just wrote can only tell you the write succeeded. It cannot tell you the system is correct, because it is looking at its own output rather than at the thing a customer experiences.
The renewal script now ends with an actual TLS handshake against our own domains and compares the fingerprint it receives to the one it just installed. If those disagree, it fails loudly. That check would have caught this on day one.
Deleting the problem instead of automating around it
The obvious fix was to sync the copies properly. We did not do that.
Every copy is a thing that has to be renewed, monitored and verified. A copy that does not exist needs none of that. So we registered the wildcard once as the router’s default certificate, and removed the certificate reference from tenant applications entirely. They now carry no certificate of their own. There is nothing to copy and nothing to drift.
The host switched to the correct certificate immediately, with no restart. An application that declares no certificate cannot disagree with the source of truth.
Customer domains are different — our wildcard genuinely cannot cover
yourcompany.com, so those still get their own certificate, either issued automatically
or uploaded by you. That is a real distinction rather than a copy.
What we changed
- One certificate for every
*.boottify.comaddress, held in one place. - Copies discovered by what they contain, not by what they are named — the old name-based list is exactly what hid the stale ones.
- A drift detector that warns when anything reintroduces a per-tenant copy.
- Renewal verified by handshake, so the check observes what you would see.
The certificate was never the fragile part. The copies were.
