Incident response

Owner: Mira Holm · Reviewed every year; rehearsed twice a year.

What we do when something goes wrong in production or with customers' data. Operations runbooks are in the Engineering and Operations spaces; this page is the policy.

Is it an incident?

Yes, if any of these:

• people at a customer can't sign in or ask for access;
• notices or codes stop arriving;
• customer data may have been seen, changed or lost by someone who shouldn't have;
• someone is attacking our sign-in.

Not sure? Treat it as one. It is cheaper to close an incident than to open it late.

Severity

Severity	Means	Example
SEV1	Many customers or a whole region affected, or a confirmed data breach	Sign-in down in the EU region
SEV2	One customer or one feature badly affected	SMS codes delayed for hours (OPS-33)
SEV3	Small or partly affected, with a way round	One depot's single sign-on failing

Roles

• Incident lead: the on-call engineer, until they hand over.
• Customer lead: Ines (support) or Ana (customer success). Writes to customers and the status page.
• Security lead: Mira, for anything touching data or an attack. Decides whether it is a breach.

Steps

1. Open an OPS incident in Jira and say the severity.
2. Stop the harm first; find the cause later.
3. Status page within 15 minutes for SEV1, and for SEV2 as soon as customers would notice; then every 30 minutes until resolved.
4. For a personal data breach: we tell affected customers within 48 hours of confirming it. Our DPA commits to 72.
5. Close the incident when the harm has stopped and customers are told.
6. Review within five working days, with Emargen's Incident review template.

Rehearsals

Date	Scenario	Notes
12 Mar 2025	A laptop with production access is stolen	Revoking AWS and GitHub access took 25 minutes
17 Sep 2025	Backups can't be restored in one region	Who decides on a restore, and who tells customers, written down
18 Mar 2026	Mail provider sends notices to the wrong customer	Customer lead role written down
16 Sep 2026	Credential stuffing, after OPS-41	Counted as the second rehearsal of 2026; added the attack runbook