Runbook: credential stuffing and lockouts
By Oskar Lund · created 4 Sep 2026, 08:10 UTC · last updated 14 Sep 2026, 14:00 UTC · version 4
runbooksecuritysign-inlockout
Written after OPS-41, the night of Wed 2 Sep, when password sign-in in the EU region took about 1.9 million attempts in three hours from lists of email and password pairs stolen elsewhere. Summary: OPS-41.
How you'll know
- The failed sign-in rate alarm (per region): failed password sign-ins above ten times the usual rate for that hour.
- Many lockout notices to several clients' administrators at once.
- Support tickets saying "my account is locked" from more than one client.
Single sign-on depots and phone codes are not affected by a password attack; check the sign-in log's method column.
What protects us since 10 September
| Layer | Rule |
|---|---|
| Edge (AWS WAF) | rate-based rule on the sign-in endpoints per IP address (OPS-42); known bad address lists |
| identity-service | ten failed passwords in 15 minutes lock the account for 15 minutes (ENG-1705) |
| Phone codes | five tries per code, then 15 minutes |
| Administrators | must use MFA |
Steps
- Open an OPS ticket. SEV2 if sign-in still works for real people; SEV1 if it doesn't.
- In CloudWatch, look at the top source addresses and networks for failed sign-ins in the last 30 minutes.
- If the rate rule isn't holding, add a temporary WAF rule blocking the worst networks. Tag it with the OPS key and an expiry.
- Check for successful sign-ins from the attacking addresses: sign-in log, method
password, those addresses. Any hit: revoke the person's sessions, force a password reset, and tell Mira at once. - Post on the status page if real people can't sign in.
- Next morning: Mira decides what customers are told. Support writes to administrators of affected clients.
What not to do
- Don't unlock accounts in bulk during the attack. The lock is what stops the guessing.
- Don't turn the per-account lock off for one client because their administrator asks. Offer to unlock named people.
Change history
| Version | Date | Who | What |
|---|---|---|---|
| 1 | 4 Sep 2026 | Oskar | After OPS-41, with the temporary rules |
| 3 | 10 Sep 2026 | Oskar | Lockout and edge limits live |
| 4 | 14 Sep 2026 | Mira | Who decides what customers are told |