Near miss: certificate for id.acme-hq.example 6 days from expiry, renewal failing for two weeks

Summary

The TLS certificate for id.acme-hq.example (sign-in and the SAML endpoints) would have expired on Wednesday 26 August. Its renewal had failed every day since 6 August and nobody saw. Found by chance; renewed the same morning. No customer impact.

Timeline (Thursday 20 August)

• 09.05 Kelderby's IT ask when the certificate on our sign-in address rotates. Oskar checks it: 6 days left.
• 09.20 Renewal job's log: failing daily since 6 August, "DNS API: unauthorised".
• 09.40 Cause: OPS-38 rotated the DNS provider token on 31 July. The renewal job keeps its own copy of the token in CI and still had the old one. Renewal starts 20 days before expiry, so the first failure was on 6 August.
• 09.50 The failure emails went to an address that only forwarded to Piet, whose mailbox is closed.
• 10.05 Token updated in the renewal job.
• 10.12 Renewal succeeds. New certificate deployed at 10.30, valid to 18 November.
• 11.00 Every other public hostname checked: portal and status page renew another way, both fine.

Follow-ups

• Daily certificate checks and alarms that go to the rota, not a person: ENG-1652
• The rotation checklist in OPS-38's template now lists every place a secret is copied.