Monitoring and alerts
By Rafael Duarte · created 14 Feb 2022, 09:10 UTC · last updated 16 Oct 2026, 13:00 UTC · version 12
monitoringalertsoperationson-call
What we watch, per region, and what wakes someone up. Metrics and alarms are in CloudWatch; logs are structured JSON with ids only, never emails or phone numbers.
Checks from outside
Every minute, from outside each region:
- a password sign-in with a test account,
- a phone code to our own test number (through Textbridge, so it costs a few euros a day),
- a single sign-on through the SAML test IdP,
- an access request and an approval.
Two failures in a row page the on-call engineer.
Targets
- Sign-in available 99.9 % of each month, per region.
- 95 % of phone codes delivered within 30 seconds.
- 99 % of notice emails accepted by Mailcrest within two minutes of the event.
Alarms
| Alarm | Threshold | Goes to | Since |
|---|---|---|---|
| Outside check failing | 2 in a row | page | 2022 |
| 5xx rate | above 2 % for 5 minutes | page | 2022 |
| Outbox lag | oldest unpublished event older than 60 seconds | page | 2021 |
| Queue depth | above 5,000 for 10 minutes | #ops | 2022 |
| SMS receipt lag | receipts more than 5 minutes behind | page | after OPS-33 |
| Mail bounce rate | above 5 % in an hour, per Mailcrest account | #ops | 2024; per account since 16 October |
| Failed sign-in rate | above ten times the usual for the hour | page | tuned after OPS-41 |
| Lockouts | more than 50 accounts locked in 15 minutes | page | 10 September |
| Backup job | failed or didn't run | page, the same morning | 2022 |
| SAML certificates | 7 days left | #ops | 2024 |
| Nightly jobs | ExpireRequests or SendApproverReminders didn't run | #ops | 2025 |
Pages go through the paging rota in Jira Service Management; see On-call. An alarm that pages and needs no action twice in a month gets fixed or demoted.
Dashboards
One per region: sign-ins by method, requests and decisions, queue and outbox, notifier sends by channel and account, lockouts. A combined one for the status of all three regions is on the screen by the support desk.
Penetration test window
From 5 October to 9 October the outside firm's addresses were allowed through the WAF (OPS-46). Failed sign-in and lockout alarms were muted for those addresses only, not for anyone else.
Change history
| Version | Date | Who | What |
|---|---|---|---|
| 8 | 24 Jun 2026 | Oskar | SMS receipt lag, after OPS-33 |
| 9 | 14 Sep 2026 | Oskar | Failed sign-in rate tuned, lockout alarm, after OPS-41 |
| 10 | 2 Oct 2026 | Oskar | Pen test window planned |
| 11 | 7 Oct 2026 | Oskar | Pen test window in place (OPS-46) |
| 12 | 16 Oct 2026 | Oskar | Bounce alarm per Mailcrest account (OPS-50); pen test window closed |