Engineering / ENG-1548
Migrations on large tables never take a lock that stops writes
Description
OPS-36: a migration built an index on access_requests (3.4 million rows in the EU) without CONCURRENTLY and added a foreign key in the same step. Writes to access_requests waited for 9 minutes: no new requests, no decisions.
- CI fails a migration on
people,access_requests,grantsorsms_messagesthat builds an index without CONCURRENTLY, adds a validated foreign key, or changes a column type. - Those take the slow path: nullable column, concurrent index, foreign key NOT VALID then validated, backfill in a job.
- Release checklist in Confluence ENG › Release process.
Linked issues
- relates to OPS-36 identity-service deploy rolled back: new requests and decisions failed in the EU for 9 minutes Done
Comments (0)
No comments.