Event Consumer Dead-Letter Capture#

Goal: Deploy the Event Consumer Service’s bounded-redelivery and dead-letter capture behavior so that durable-queue messages that cannot be processed (or that churn past their redelivery limit) are captured in a dead-letter queue instead of being lost or redelivered forever.

When to use: This applies to deployments that use the Event Consumer Service’s durable queues. It governs what happens to a durable-queue message once it has been redelivered too many times or has aged out, routing it to a dedicated dead-letter queue (DLQ) for later inspection.

This page covers the deployment-facing concerns for this feature: the broker prerequisite, the topology the service declares for you, the live-tunable numeric-limits policy, the one-time rollout step for pre-existing classic queues, the operator-facing dead-letter management RPC and the permission that gates it, and the recovery path for inspecting and recovering captured dead-letters.

Note

This page covers Event Consumer Service deployment only. It assumes you have already deployed RabbitMQ and the Event Consumer Service as described in Notifications Services Configuration.

Broker Requirements#

RabbitMQ 4.x with quorum queues is a hard prerequisite for this feature. RabbitMQ 4.1.3 is recommended. Validate that your broker is on the 4.x line before relying on dead-letter capture.

The feature depends on two behaviors that are 4.x behaviors:

  • Delivery-limit dead-lettering. Bounded redelivery relies on the quorum-queue native x-delivery-count header together with a delivery-limit. When a message exceeds the configured delivery-limit, the broker dead-letters it.

  • Prompt message-TTL dead-lettering. Aged-out messages are dead-lettered promptly when they exceed their message-ttl.

Quorum queues exist on RabbitMQ 3.x, but the default delivery-limit of 20 and prompt TTL-based dead-lettering are 4.x behaviors. Running this feature against a 3.x broker is not supported — validate the broker version as part of the deployment.

Durable / quorum deployment prerequisites#

The durable origin queues and the DLQ are quorum queues, so the broker must be deployed to actually persist and replicate them:

  • Persistent storage must be enabled on RabbitMQ (persistence.enabled: true). Quorum durable queues persist to disk; with persistence disabled, the queues and their messages are lost on pod restart, defeating durability.

  • For high availability, run a multi-node cluster (for example replicaCount: 3 together with clustering.enabled: true) so the quorum durable queues replicate across nodes via Raft. A single node yields quorum semantics but only one replica — there is no redundancy. Single-node works for non-production but provides no HA.

A concise values.yaml fragment for a production-grade broker:

persistence:
  enabled: true
replicaCount: 3
clustering:
  enabled: true

App-Declared DLX/DLQ Topology#

You do not need to manually configure the broker for dead-letter capture. The Event Consumer Service declares the entire capture topology over AMQP at startup (and the declarations are reconnect-safe — they are re-asserted on reconnect). The service declares:

  • The dead-letter exchange (DLX) — a durable direct exchange.

  • The dead-letter queue (DLQ) — a durable quorum queue. The DLQ is self-bounded by its own x-message-ttl and x-max-length with x-overflow: drop-head, and it has no dead-letter-exchange of its own, so there is no dead-letter cycle (dead-letters cannot themselves be dead-lettered into a loop).

  • The DLX → DLQ binding — the binding on the dead-letter routing key that routes messages from the DLX to the DLQ.

Each durable origin queue is declared with the queue arguments x-queue-type: quorum, x-dead-letter-exchange, and x-dead-letter-routing-key only — pointing at the DLX so that dead-lettered messages are routed to the DLX (and from there to the DLQ via the binding above).

The DLX and DLQ names, the dead-letter routing key, and the DLQ’s self-bound defaults are configurable via ``OMNI_EVENTS_``-prefixed settings on the Event Consumer Service:

  • OMNI_EVENTS_DEAD_LETTER_EXCHANGE_NAME — the DLX name.

  • OMNI_EVENTS_DEAD_LETTER_QUEUE_NAME — the DLQ name.

  • OMNI_EVENTS_DEAD_LETTER_ROUTING_KEY — the routing key used for the DLX → DLQ binding and the origin queues’ x-dead-letter-routing-key.

  • OMNI_EVENTS_DEAD_LETTER_QUEUE_MESSAGE_TTL_MS — the DLQ’s own x-message-ttl.

  • OMNI_EVENTS_DEAD_LETTER_QUEUE_MAX_LENGTH — the DLQ’s own x-max-length.

Note

The DLQ self-bound defaults are deliberately generous (a long TTL and a high max-length, with drop-head overflow). The intent is that the DLQ holds captured dead-letters safely for a long time rather than dropping them — see Recovery Path below.

No manual broker configuration is required for capture. The exchange, queue, and binding are all created by the service on startup.

Numeric-Limits Policy#

The per-origin-queue numeric bounds — delivery-limit, message-ttl, max-length, and overflow — are not set as queue arguments by the app. They are applied as a RabbitMQ broker policy so that they can be retuned live without recreating queues.

The policy is set from the notifications-rabbitmq Helm values.yaml via an initScripts entry that runs rabbitmqctl set_policy for the ^event-queue: pattern on the notifications vhost at node startup. The script is idempotent (re-running set_policy for the same policy name simply overwrites it) and the policy applies cluster-wide to every matching queue:

  • Policy name: event-queue-limits

  • Vhost: notifications

  • Pattern: ^event-queue: (matches the per-principal origin queues, which are named event-queue:<hashed-principal-id>:<queue-id>)

  • Definition (starting points):

    • delivery-limit: 10

    • message-ttl: 86400000 (24 hours)

    • max-length: 100000

    • overflow: drop-head

An illustrative values.yaml fragment:

initScripts:
  set-event-queue-limits.sh: |
    #!/bin/bash
    rabbitmqctl set_policy \
      --vhost notifications \
      --apply-to queues \
      event-queue-limits "^event-queue:" \
      '{"delivery-limit":10,"message-ttl":86400000,"max-length":100000,"overflow":"drop-head"}'

Because this is a policy rather than queue arguments, the values stay live-tunable — beyond the startup initScripts run, you can also adjust them on the running broker via rabbitmqctl set_policy (or the management API) without deleting and recreating queues.

Disjointness rule (policy vs. app args)#

The policy and the app-declared queue arguments are kept disjoint on purpose:

  • The policy sets only delivery-limit / message-ttl / max-length / overflow.

  • The app sets only x-queue-type / x-dead-letter-exchange / x-dead-letter-routing-key.

In RabbitMQ, a queue argument wins over a user policy for the same key. By keeping the two sets disjoint, neither overrides the other: the app owns the topology args, and the policy owns the numeric limits. Do not move the numeric keys into queue arguments while the policy is in use, or the policy values for those keys would be shadowed by the args.

Calibration#

The concrete numeric values above are starting points. Recalibrate them for your own load profile — the ^event-queue: policy is the place to tune these limits.

Operator tuning rationale: sizing delivery-limit#

When calibrating the live ^event-queue: policy, it is important to understand why delivery-limit must be sized above routine churn.

RabbitMQ’s quorum-queue x-delivery-count increments on every redelivery — not just on genuine processing failures. In particular it increments on:

  • Requeues from routine client reconnects, and

  • The requeue triggered by the 60-second consumer timeout.

delivery-limit caps the total number of redelivery attempts. Because the count includes routine churn, setting delivery-limit too low would dead-letter perfectly processable messages that merely cycled through several reconnects or a consumer-timeout requeue. The limit must therefore sit comfortably above the expected routine-churn count, not just above the expected failure count.

To keep routine churn clear of the limit from the other direction, the with-acks consumer uses a small prefetch (default 20). A smaller prefetch shrinks the per-disconnect requeue volume — the in-flight blast radius that gets requeued (and thus counted) on each disconnect. This is the paired half of keeping routine churn well below delivery-limit.

Rollout Sequence: Pre-Existing Classic Durable Queues#

The origin queues are now declared as quorum queues. If a durable origin queue already exists from a prior deployment as a classic queue, the service’s active (non-passive) declare for the quorum queue will fail.

Symptom: declaring a quorum queue over a name that still exists as a classic queue raises 406 PRECONDITION_FAILED (the active declare cannot change the queue type of an existing queue).

Runbook (one-time, per leftover classic event-queue: queue):

  1. Drain / disconnect any active consumer holding the queue.

  2. Delete the leftover classic event-queue: queue.

  3. Let the next CreateDurableQueue request (or the next consumer reconnect) recreate it as a quorum queue.

Note

The delete-and-recreate swap makes the affected durable queue briefly unavailable, so schedule it during a maintenance window.

Recovery Path#

The supported, primary way to inspect and recover captured dead-letters is the ManageDeadLetteredEvents management RPC — see Dead-Letter Management RPC: Operator Runbook below. It browses the DLQ and lets an operator REPLAY / DISCARD / LEAVE each message with publisher-confirm-backed, no-loss replay, and it requires no datastore beyond the broker itself.

RabbitMQ’s built-in tooling remains available only as a low-level fallback — for example, to inspect the DLQ directly when the management RPC is unavailable, or for broker-level operations the RPC does not expose:

  • The RabbitMQ Management UI, or

  • rabbitmqadmin

against the DLQ (the queue named by OMNI_EVENTS_DEAD_LETTER_QUEUE_NAME).

Either path is safe to use because the DLQ’s generous self-bound defaults (a long x-message-ttl, a high x-max-length, and drop-head overflow) keep captured dead-letters safely retained rather than dropping them. Dead-letters remain available for inspection and recovery for a long time.

Dead-Letter Management RPC: Operator Runbook#

The Event Consumer Service exposes an operator-facing ManageDeadLetteredEvents RPC on the EventConsumerService for inspecting and triaging captured dead-letters. It is a bidirectional-streaming gRPC on the service’s gRPC surface (port 50052). There is no REST/SSE parity — this operation is gRPC-only.

Note

This RPC is permission-gated. The caller must hold event-consumer-service:inspect-and-replay-all-dlq — see Granting the DLQ-Management Permission below. A caller lacking it is denied with PERMISSION_DENIED at stream open, before any dead-lettered event is streamed.

Open the management stream#

Open the bidirectional stream against EventConsumerService/ManageDeadLetteredEvents on port 50052 with a token whose principal holds the management permission. The DLQ is global (not scoped to a single durable queue), so the opening request carries no queue_id: the server immediately begins browsing the DLQ and streaming dead-lettered messages.

Browse dead-letters#

The server streams each captured message to the client as a dead_lettered_event arm of the response. Each carries the triage metadata the operator needs:

  • reason — why the message was dead-lettered: DELIVERY_LIMIT (mapped from the broker’s delivery_limit), EXPIRED (expired), or MAXLEN (maxlen).

  • origin_queue — the queue the message was dead-lettered from.

  • delivery_count — how many times the broker attempted delivery before dead-lettering.

  • dead_lettered_at — when the message was dead-lettered.

  • The event itself (event), carrying event_type, principal_identity, and the payload, plus resource_id surfaced alongside it for triage.

  • delivery_tag — a session-scoped delivery tag identifying the message. This tag is valid only within the live stream that produced it; it must not be persisted or reused across reconnects. The operator sends it back to act on the message.

Act per message#

To act on a browsed message, the client sends a ManageDeadLetteredEventsRequest carrying one or more actions entries, each pairing a delivery_tag with an Action. Multiple messages can be actioned in a single request. The actions:

  • REPLAY — republishes the message to its origin queue using publisher confirms. The dead-letter copy is removed only after the publish is confirmed, so a failed replay never drops the message (no loss).

  • DISCARD — acknowledges the message and drops it from the DLQ.

  • LEAVE — leaves the message in place in the DLQ (nacked with requeue=true) for later triage.

Per-action outcome reports. For every action, the server streams back an action_outcome arm carrying an Outcome, correlated to the request by delivery_tag, so the operator can confirm each action’s result (and detect failures, including replay loops on DELIVERY_LIMIT reasons) rather than relying on server-side logs alone. The Outcome values are:

  • REPLAYED — republished to the origin queue and the DLQ copy removed after the publish confirm.

  • DISCARDED — acknowledged and dropped from the DLQ.

  • LEFT — left in place in the DLQ for later triage.

  • ORIGIN_QUEUE_DELETED — the replay could not proceed because the origin queue no longer exists; no loss — the DLQ copy is left in place (LEAVE semantics).

  • REPLAY_FAILED — the replay could not be confirmed; no loss — the DLQ copy is left in place (LEAVE semantics).

An optional human-readable detail accompanies the outcome (for example, the name of the missing origin queue for ORIGIN_QUEUE_DELETED).

Reason-driven triage#

Read the ``reason`` before acting. A DELIVERY_LIMIT message that keeps failing will simply re-dead-letter if blindly replayed — a replay loop. Fix the root cause first, or DISCARD the message. EXPIRED and MAXLEN indicate aging/overflow rather than a processing failure, so they are usually safe to REPLAY once the queue is healthy. Watch the per-action Outcome reports to catch a replay that immediately re-dead-letters.

Origin-queue-deleted behavior#

A REPLAY whose origin queue no longer exists does not lose the message. The RPC emits an explicit ORIGIN_QUEUE_DELETED outcome report back to the operator on the stream (keyed to the delivery_tag) and leaves the DLQ copy in place (LEAVE semantics). Re-creating the origin queue restores a viable replay target, at which point the operator can REPLAY the message again.

Tenant isolation#

Origin queue names embed the hashed principal id (event-queue:<hashed-principal-id>:<queue-id>), so a REPLAY returns the message to the original tenant’s queue — there is no cross-tenant replay. The inspect-and-replay-all-dlq permission is a single coarse operator permission that grants all-tenant dead-letter visibility and action by design. Re-consume of a replayed message still runs the existing per-event authz_check, so downstream consumption remains permission-checked per event.

Granting the DLQ-Management Permission#

The ManageDeadLetteredEvents RPC is gated by the permission action event-consumer-service:inspect-and-replay-all-dlq. A caller lacking it is denied with PERMISSION_DENIED at stream open.

Configure this permission in your Permissions Service deployment (see Permissions Service Configuration):

  • Declare the action for the event-consumer-service service: inspect-and-replay-all-dlq. The fully-qualified action string is event-consumer-service:inspect-and-replay-all-dlq.

  • Grant the action to the principal(s) that operate the DLQ — bind event-consumer-service:inspect-and-replay-all-dlq to the operator role or principal.

Note

The Event Consumer Service enforces this permission: until the action is declared and granted as described above, a caller is denied (PERMISSION_DENIED) at stream open.