Disaster recovery plan
By Mira Holm · created 17 Apr 2023, 08:00 UTC · last updated 20 Jul 2026, 12:25 UTC · version 6
disaster-recoveryoperationsregionsiso-27001
What we do if something large is gone: an availability zone, a region, a database, a provider. Our targets are a recovery point of 24 hours and a recovery time of 8 hours per region.
We don't fail over to another region. Each client's data stays in its region (ADR-006), so recovery happens in the same region.
Scenarios
| What is gone | What happens | What we do | Time |
|---|---|---|---|
| One availability zone | RDS fails over to its standby; containers restart in the other zones | watch; post on the status page if sign-ins failed | minutes, automatic |
| A database is corrupted or a migration destroyed data | wrong or missing data | restore from the nightly backup in the same region (Backups) | up to 8 hours; up to 24 hours of changes lost |
| A whole AWS region is down | the region's clients can't sign in | wait for the region; rebuild in it from infrastructure code and backups as soon as AWS allows | depends on AWS; 8 hours once the region is back |
| Mailcrest is down | notice emails queue | nothing is lost; in-app notices still work | until they recover |
| Textbridge is down | phone-code sign-in stops | status page; customers use their local fallback; see the SMS runbook | until they recover |
| GitHub or CI is down | we can't deploy | production keeps running; emergency deploy from a tagged image by Oskar or Rafael | – |
| A customer's IdP is down | that customer's depots on single sign-on can't sign in | their administrator can switch depots back to passwords (runbook) | minutes |
A region outage is the one we can't fully cover. We tell customers so in security reviews: we would rather say it than copy their data abroad.
Who
| Role | Person |
|---|---|
| Incident lead | the on-call engineer, then Oskar |
| Decides on restores and customer messages | Rafael, or Mira |
| Customers | Ana (customer success), Ines (support) |
| Status page | the incident lead |
Rehearsals
Twice a year, in March and September, we walk through one scenario as a tabletop exercise with engineering, support and customer success, as part of the incident response rehearsal (Incident response). Restores themselves are tested every quarter (Restore drills).
Change history
| Version | Date | Who | What |
|---|---|---|---|
| 4 | 20 Jan 2025 | Mira | UK region |
| 5 | 19 Jan 2026 | Mira | US region |
| 6 | 20 Jul 2026 | Maja, Oskar | Rewritten for the ISO 27001 surveillance audit; providers and customers' IdPs added |