Vossberg: Okta sign-in refused at every depot after certificate rotation
Description
URGENT: nobody can sign in with Okta since 08:00. All depots. Error page "signature not valid". We rotated the signing certificate on our ACME apps in Okta this morning, with our script.
Linked issues
- relates to SUP-1103 Parcelwise: our 32 hubs' Entra ID certificates expire, please swap them like last year Done
- relates to OPS-48 Vossberg: Okta sign-ins refused at every depot after their certificate rotation Done
Comments (8)
Reverting in Okta now.
Reverted. Sign-ins work again at Kassel and Hamm.
08:00 to 09:30, every Vossberg depot: Okta sign-ins refused, about 1,100 failed attempts. People already signed in kept working. Cause: certificates rotated in Okta without uploading the new metadata to us, and with one setup per depot that's 60 uploads. Same pain as Parcelwise's 32 in August (SUP-1103). One setup per customer is ACM-293 (Tom).
Confirmed from our side: sign-ins going through since 09:31. Yusuf will plan the rotation with you on Monday.
Hi Matthias, for the rotation the safest order is depot by depot: rotate one depot's app in Okta and upload that app's metadata to the depot in the portal straight after, then run its test sign-in. Each depot is down for a minute at most. With 60 depots it's about two hours; I can do it with your Okta admin on a call. Wednesday morning?
Wednesday 06:00 to 08:00, before the main shift. Sven will do the Okta side.
Done: all 60 depots on the new certificate, 06:02 to 07:51, no depot down for more than two minutes. Closing.
On it (on call). Your sign-ins are now signed with the new certificate, and each depot's single sign-on setup on our side still holds the old one, so we refuse them. Fastest fix: switch the Okta apps back to the previous certificate. Then we move every depot to the new one together, on a weekday.