ACME HQDemo work items and pages, made up for Emargen's demo. Not a real company.
Engineering / Engineering / Runbooks

Runbook: notice emails not arriving

By Elif Yildiz · created 24 Jun 2026, 08:05 UTC · last updated 16 Oct 2026, 15:20 UTC · version 5

runbookmailcrestnotifiernotices

Approvers and requesters get emails from notifier through Mailcrest: a request waiting, a reminder, a decision made. A missing email never blocks access: the request is still in the approvals inbox, and since 4 August it is also in the bell. So this is SEV3 unless every email in a region stops.

1. Which account?

Since 16 October there are two Mailcrest accounts:

Client regionAccount
EUEU account
UK, USUS account

Check the client's region first (portal › Organisation settings, or ask support). Looking in the wrong account's logs is the most common waste of time since the switch-over.

2. Did we send it?

  1. notifier logs in the client's region: search for the request id. Look for notice.sent with Mailcrest's message id.
  2. No notice.sent: look for the event on the queue and in outbox_events in identity-service. If the event is there and unpublished, the relay is stuck: see the outbox lag alarm.
  3. notice.sent present: go to Mailcrest's logs for that account and the message id. Logs are kept 30 days.

3. What Mailcrest says

StatusMeaningWhat to do
deliveredit reached the recipient's mail serverask the customer to check spam and their mail filters; send them our sending domains
bounced (hard)the address doesn't existthe person's email is wrong in the portal; support tells the customer
suppressedearlier bounce or complaintremove from the suppression list only if the customer confirms the address works; both accounts share the list
deferredthe receiving server is slowwait; Mailcrest retries for 72 hours

4. A whole account stops

Open an OPS ticket (SEV2), post on the status page under Email notices, and check Mailcrest's status. Do not move clients between accounts to get mail out: an EU client's mail must not go through the US account (ACM-305). Notices queue and go out when the account is back.

Change history

VersionDateWhoWhat
25 Aug 2026ElifIn-app notices as the fallback
313 Oct 2026ElifMailcrest keeps logs in the US (ACM-305)
415 Oct 2026ElifEU account added, not yet in use
516 Oct 2026ElifTwo accounts; never move a client between them (OPS-50)