ADR-004: Events leave identity-service through an outbox

Status: accepted · Date: 17 May 2021 · Decided by: Marek Nowak, Rafael Duarte

Context

identity-service sends emails itself, from queue jobs dispatched inside controllers. Two problems in the last month:

• A request was saved, the email job was dispatched, and then the transaction rolled back. The approver got an email about a request that didn't exist.
• Mail provider slowness made sign-ups slow, because some code sent mail inside the request.

We are moving sending into its own service (ADR-005), so identity-service needs a reliable way to say "this happened" to something outside its database.

Decision

identity-service writes every event to an outbox_events table in the same transaction as the change that caused it. A relay job publishes unpublished rows, in order, to a queue, and marks them published. Consumers get each event at least once and must ignore ids they have already handled.

We considered publishing straight to the queue after commit. It loses events when the process dies between the commit and the publish, and it can't be tested without the queue.

Consequences

• No event without its change, no change without its event.
• Consumers must be idempotent. notifier keeps handled event ids for seven days.
• Events arrive a few seconds late. Fine for notices; not fine for sign-in codes, which will call the sending service directly when we have them.
• The outbox table grows; published rows are deleted after seven days.
• We need an alarm on the age of the oldest unpublished row.

Updates

• 1 June 2021: relay interval set to 5 seconds after a week on staging.