OPS-41: credential stuffing against password sign-in

When: night of Wednesday 2 September, 23:40 to 02:40 CEST · Severity: SEV2 · Incident lead: Oskar Lund · Security: Mira Holm · Review: in Emargen, on the Incidents list

What happened

Someone ran lists of email addresses and passwords stolen from other services against our password sign-in in the EU region: about 1.9 million attempts in three hours, from about 11,000 IP addresses. This is called credential stuffing. It wasn't aimed at one customer; the lists matched addresses at nine of our clients.

What we found

• About 6,100 attempts used an email address that exists at one of our clients. The rest were addresses we don't know.
• No attempt from the attacking addresses signed in. We checked every successful sign-in in the EU region that night against the attacking addresses: none.
• 212 people saw more than 20 wrong passwords. We told their administrators which people, and suggested a password change.
• Single sign-on depots and phone codes were not touched; the attack only used the password form.
• Administrators' accounts have required MFA since 11 June, so a correct password alone would not have opened the portal.
• Real people noticed: password sign-in was slow for about three hours, and our first block on whole networks also refused 11 night-shift sign-ins at one Northgate site from 00:20 to 00:41, until we narrowed it.

Timeline (CEST)

Time	What
23:40	Failed sign-in alarm in the EU region. Oskar paged.
23:52	Oskar sees the pattern: many addresses, few tries each, real-looking emails.
00:05	Status page: password sign-in slow in the EU.
00:20	Temporary WAF rule blocks the worst networks. Northgate's night-shift sign-ins caught by it.
00:41	Rule narrowed; Northgate's sign-ins work.
00:45	Tighter rate rule per address on the sign-in endpoints.
02:40	Attempts stop.
02:55	Status page resolved.
07:30	Mira and Rafael read the timeline; customers to be told by noon.
11:50	Support writes to the nine clients' administrators.

Why it could happen

We had a loose limit per address and no limit per account. Nothing stopped thousands of addresses trying a few passwords each.

What we changed

Action	Ticket	Done
Edge rate limits on sign-in endpoints, managed bad-address lists	OPS-42	4 September, tuned 10 September
Lock an account's password sign-in for 15 minutes after ten failures in 15 minutes	ENG-1705	10 September
Sign-in lockout and rate limits for every client, with unlock in the portal (urgent initiative)	ENG-1702	10 September
Decision written down	ADR-014	10 September
Runbook	Credential stuffing and lockouts	4 September
Blocks on whole networks narrowed before use, so night shifts aren't caught	runbook	14 September