EU database failover: sign-in and portal down 18 minutes

Project: Operations
Type: Incident
Status: Done Done
Priority: High
Reporter: Marek Nowak <marek.nowak@acme-hq.example>
Assignee: Rafael Duarte <rafael.duarte@acme-hq.example>
Labels: sev2
Components: infrastructure
Created: 14 May 2024, 08:06 UTC
Updated: 15 May 2024, 07:30 UTC
Resolved: 14 May 2024, 09:45 UTC

Description

Summary

The EU database primary's host failed at 10.02. The standby didn't take over on its own; we promoted it by hand. Sign-in and the portal were down for EU clients from 10.02 to 10.20.

Timeline (14 May 2024)

10.02 Primary host stops responding.
10.04 Alarms: sign-in errors EU, database unreachable.
10.09 Automatic failover hasn't happened: the replication check that gates it is stuck on the dead host.
10.14 Marek promotes the standby by hand.
10.20 identity-service reconnects; sign-ins succeed.
11.30 New standby built.

Follow-ups

The failover check no longer depends on the old primary answering.
A failover drill twice a year.

Comments (1)

Rafael Duarte · 15 May 2024, 07:30 UTC

Check fixed and tested by stopping staging's primary twice. Drill on the calendar for November.