Event Consumer Dead-Letter Capture#

Goal: Configure durable-queue limits and inspect or replay messages captured in the Event Consumer Service’s dead-letter queue.

When to use: This applies to deployments that use the Event Consumer Service’s durable queues. It governs what happens to a durable-queue message once it has been redelivered too many times or has aged out, routing it to a dedicated dead-letter queue (DLQ) for later inspection.

This page covers the deployment-facing concerns for this feature: the broker prerequisite, the topology the service declares for you, the durable-queue bounds policy the Helm chart applies (including the idle-queue TTL that reaps unattended durable queues), how the dead-letter queue’s own bounds differ, the one-time migration of existing durable queues, the operator-facing dead-letter management RPC and the permission that gates it, and the recovery path for inspecting and recovering captured dead-letters.

Use the section for your task:

Note

Read the broker requirements and queue configuration before installing Event Consumer Service. Use Notifications Services Configuration for the installation commands. The recovery runbook requires a running broker and Event Consumer Service.

Broker Requirements#

RabbitMQ 4.x with quorum queues is a hard prerequisite for this feature. RabbitMQ 4.1.5 or newer is recommended. Validate that your broker is on the 4.x line before relying on dead-letter capture.

The management HTTP API must be reachable from the cluster, and a credential that may set policies on the notifications vhost must exist. The chart’s hook Job uses it to apply the Durable-Queue Bounds Policy.

Note

RabbitMQ 4.0.5 through 4.1.4 have a policy-repair bug when the delivery limit is disabled. The chart rejects negative deliveryLimit values. See the RabbitMQ 4.1.5 release notes.

This guide’s downloadable broker values use image 4.1.3-debian-12-r1. If selecting a newer broker, use a container compatible with the RabbitMQ chart and validate the deployment before upgrading. The redelivery guidance below describes RabbitMQ 4.1; RabbitMQ 4.3 changes which requeues increment the delivery count. See RabbitMQ poison-message handling before upgrading across those versions.

Durable / quorum deployment prerequisites#

The durable origin queues and the DLQ are quorum queues, so the broker must be deployed to actually persist and replicate them:

  • Persistent storage must be enabled on RabbitMQ (persistence.enabled: true). Quorum durable queues persist to disk; with persistence disabled, the queues and their messages are lost on pod restart, defeating durability.

  • For high availability, run a multi-node cluster (for example replicaCount: 3 together with clustering.enabled: true) so the quorum durable queues replicate across nodes via Raft. A single node yields quorum semantics but only one replica — there is no redundancy. Single-node works for non-production but provides no HA.

A concise values.yaml fragment for a production-grade broker:

persistence:
  enabled: true
replicaCount: 3
clustering:
  enabled: true

Scaling an existing single-node broker to a cluster#

Raising replicaCount on a broker that is already running adds nodes, but it does not re-replicate quorum queues that already exist — a quorum queue keeps the Raft membership it had when it was declared.

  • If no quorum queues have been declared yet, there is nothing more to do. Scale up first; with RabbitMQ’s default group size, queues declared after all three nodes have joined start with three replicas. Enable the cluster before turning on durable-queue workloads.

  • Only if durable queues already exist (event-queue:* quorum queues and/or the DLQ, which the Event Consumer Service declares at startup): after the new nodes have joined, run a one-time grow per new node so the existing queues gain a replica there —

    rabbitmq-queues grow "rabbit@<new-node-name>" all

    using the node names shown by rabbitmqctl cluster_status. Queues declared after the scale-up need no migration in either case.

App-Declared DLX/DLQ Topology#

After the broker prerequisites and policy are configured, the Event Consumer Service declares the capture topology over AMQP at startup (and the declarations are reconnect-safe — they are re-asserted on reconnect). The service declares:

  • The dead-letter exchange (DLX) — a durable direct exchange.

  • The dead-letter queue (DLQ) — a durable quorum queue. The DLQ is self-bounded by its own x-message-ttl and x-max-length with x-overflow: drop-head, and it has no dead-letter-exchange of its own, so there is no dead-letter cycle (dead-letters cannot themselves be dead-lettered into a loop).

  • The DLX → DLQ binding — the binding on the dead-letter routing key that routes messages from the DLX to the DLQ.

Each durable origin queue is declared with the queue arguments x-queue-type: quorum, x-dead-letter-exchange, and x-dead-letter-routing-key only — pointing at the DLX so that dead-lettered messages are routed to the DLX (and from there to the DLQ via the binding above).

The DLX and DLQ names, the dead-letter routing key, and the DLQ’s self-bound defaults are configurable via ``OMNI_EVENTS_``-prefixed settings on the Event Consumer Service:

  • OMNI_EVENTS_DEAD_LETTER_EXCHANGE_NAME — the DLX name (events.dlx).

  • OMNI_EVENTS_DEAD_LETTER_QUEUE_NAME — the DLQ name (events.dlq).

  • OMNI_EVENTS_DEAD_LETTER_ROUTING_KEY — the routing key (dead-letter) used for the DLX → DLQ binding and the origin queues’ x-dead-letter-routing-key.

  • OMNI_EVENTS_DEAD_LETTER_QUEUE_MESSAGE_TTL_MS — the DLQ’s own x-message-ttl (1209600000 milliseconds, or 14 days).

  • OMNI_EVENTS_DEAD_LETTER_QUEUE_MAX_LENGTH — the DLQ’s own x-max-length (1000000 messages).

These settings are container environment variables. The chart does not expose DLQ topology or retention settings as Helm values; changing them requires adding the corresponding environment variables to your managed Deployment manifests. Adding them to event-services-env alone does not inject them into the service. Keep the same topology settings across all service replicas, and avoid naming the DLQ with the event-queue: prefix used by the durable-queue policy.

Warning

These are retention limits, not a retention guarantee. Expired DLQ messages and messages evicted when it reaches its maximum length are permanently discarded. Monitor its depth and message age and triage before either limit is reached. RabbitMQ’s delivery limit also applies to repeated DLQ requeues; this chart does not set a separate DLQ delivery-limit policy.

The chart’s default drop-head policy uses RabbitMQ’s at-most-once dead-letter transfer. A broker failure or unavailable DLQ can lose a message during transfer. Publisher-confirmed replay from the DLQ is a separate operation; it does not make the initial capture lossless. See RabbitMQ dead-lettering guarantees.

Retuning the dead-letter queue’s bounds#

The dead-letter queue is bounded by a different mechanism than the durable origin queues. OMNI_EVENTS_DEAD_LETTER_QUEUE_MESSAGE_TTL_MS and OMNI_EVENTS_DEAD_LETTER_QUEUE_MAX_LENGTH are queue arguments applied when the service first declares the queue, not policy values, and RabbitMQ queue arguments are immutable once a queue exists. Changing either on a running deployment does not retune the live queue. The chart’s policyHook does not cover the DLQ either — its ^event-queue: pattern does not match the dead-letter queue name, and that name is a service setting the chart does not template.

On each (re)connect with mismatched arguments, the service logs an ERROR beginning:

Dead-letter queue argument mismatch

The line names the desired-but-unapplied values and the remedies below, then the service completes startup against the existing queue — dead-letter capture keeps working and pods do not crash-loop on a configuration change. Search for that prefix to detect the drift. The line’s trailing Broker error: … text names the offending argument, and it need not be one of these two settings: if it names something else (x-queue-type, say, on a queue that is still a classic queue or was provisioned by another tool), no setting change or policy can clear it.

Two ways to act on a mismatch:

  • To lower a bound on the live queue, apply a broker policy scoped to the dead-letter queue (default events.dlq, or whatever OMNI_EVENTS_DEAD_LETTER_QUEUE_NAME is set to). Where an argument and a policy both set a numeric bound, RabbitMQ takes the lower of the two — so a policy can tighten a bound but can never raise one above the value the queue was created with. Omit ``overflow`` from that policy: unlike the numeric keys it is policy-wins on quorum queues, so including it overrides the queue’s own drop-head argument and silently changes how the DLQ behaves when full.

    # Self-hosted broker, with node access:
rabbitmqctl set_policy -p notifications dlq-limits '^events\.dlq$' \
  '{"max-length":500000}' --apply-to quorum_queues

# Any broker, including a managed one: the management HTTP API.
# URL-encode the vhost in the path ("/" is %2F).
curl -u "$RABBITMQ_USER:$RABBITMQ_PASSWORD" -X PUT \
  -H 'content-type: application/json' \
  -d '{"pattern":"^events\\.dlq$","definition":{"max-length":500000},"apply-to":"quorum_queues","priority":0}' \
  "https://<broker-host>/api/policies/<vhost-urlencoded>/dlq-limits"

    Keep every dead-letter bound in this one policy: at most one regular policy applies to a queue, so a second one matching the same queue displaces the first rather than merging with it.

  • To raise a bound — or to clear a mismatch on an argument these settings do not control — the queue has to be recreated. Schedule a maintenance window and quiesce publishing and durable consumers. Replay messages that must be retained, and discard only messages you intend to delete, using Dead-Letter Management RPC: Operator Runbook. Confirm the DLQ has no ready or unacknowledged messages, then stop the Event Consumer Service pods before deleting the queue. Keep producers paused and monitor for further dead-letters: origin-queue TTL expiry can still send messages to the DLQ while clients are stopped. A missing DLQ can lose those transfers. Restart the service with the new settings, verify the DLQ and binding, and then resume workloads. Deleting a nonempty queue permanently loses its contents.

    # Self-hosted broker, with node access:
rabbitmqctl delete_queue -p notifications events.dlq

# Any broker, including a managed one: the management HTTP API.
curl -u "$RABBITMQ_USER:$RABBITMQ_PASSWORD" -X DELETE \
  "https://<broker-host>/api/queues/<vhost-urlencoded>/events.dlq"

Note

Both routes are also available on the Policies and Queues pages of the RabbitMQ Management UI. On a managed broker you have no node access and therefore no rabbitmqctl — the management API and the UI are the only routes.

Durable-Queue Bounds Policy#

The per-origin-queue bounds — delivery-limit, message-ttl, max-length, overflow, and the idle-queue TTL expires — are not set as queue arguments by the app. They are applied as a RabbitMQ broker policy so that they can be retuned without recreating queues.

The Event Consumer Service Helm chart owns this policy. A post-install,post-upgrade hook Job applies it through the broker’s management HTTP API on install and on every helm upgrade. It checks the existing policy and writes only when it differs. Keep policy configuration in the Event Consumer Service values file; remove any older RabbitMQ init script that writes the same policy.

If the hook cannot apply the policy, the Helm release fails. This is a post-install/post-upgrade hook: service pods may already be running when it fails. Resolve the failure and verify the effective broker policy before allowing durable workloads to use the deployment.

The policy it applies:

  • Policy name: event-queue-limits (policyHook.policy.name)

  • Vhost: policyHook.managementApi.vhost — notifications in this guide’s deployment

  • Pattern: ^event-queue: (matches durable origin queues named event-queue:<queue_id> when combined with apply-to: quorum_queues)

  • Definition (defaults; these are starting points):

    • delivery-limit: 10 (policyHook.policy.definition.deliveryLimit) — must be >= 0; the chart refuses a negative value.

    • message-ttl: 86400000 (24 hours) — …definition.messageTtlMs

    • max-length: 100000 — …definition.maxLength

    • overflow: drop-head — …definition.overflow

    • expires: 86400000 (24 hours) — …definition.expiresMs, the idle-queue TTL. See Durable-Queue Idle TTL (Auto-Reap) below before changing it.

    • consumer-timeout is omitted by default — …definition.consumerTimeoutMs. See Bounding a Hung Durable Consumer before setting it.

Configuring and retuning#

Retuning is a Helm values change followed by helm upgrade — the hook Job re-applies the policy with the new definition, and existing queues pick it up without being recreated.

The chart has no defaults for reaching the broker, because a policy applied to the wrong broker or the wrong vhost succeeds silently and bounds nothing. It refuses to render unless all three of these are set:

  • policyHook.managementApi.baseUrl — scheme, host and port, no trailing path. Never assume :15672; a managed broker commonly serves the management API over HTTPS on 443.

  • policyHook.managementApi.vhost — must be the same vhost the service consumes from. The Job parses the vhost out of the service’s own AMQP URI and fails the release on a mismatch.

  • policyHook.managementApi.existingSecret.name — a Secret in the release namespace with username / password keys. The credential needs the policymaker tag plus access to the vhost (administrator also works, but is not required). Only the hook mounts this management Secret. Use a separate management user to keep policy privileges out of the service’s AMQP credential.

See Notifications Services Configuration for the step-by-step version of this, including creating the credential Secret.

# event-consumer-values.yaml
policyHook:
  managementApi:
    baseUrl: http://rabbitmq:15672
    vhost: notifications
    existingSecret:
      name: rabbitmq-mgmt-credentials
  policy:
    definition:
      deliveryLimit: 10
      messageTtlMs: 86400000
      maxLength: 100000
      overflow: drop-head
      expiresMs: 86400000

The chart fixes the pattern at ^event-queue: and the scope at quorum_queues. Keep that scope if managing the policy yourself: non-durable queues share the name prefix but must not receive these durable-queue bounds. The default DLQ name events.dlq does not match the pattern.

Note

At most one regular policy applies to a queue (highest priority wins; equal priorities resolve nondeterministically). A second regular policy matching ^event-queue: displaces this one rather than merging with it, so keep every bound for these queues in this single policy. Do not add a supplementary policy of your own alongside it.

Note

An unchanged policy is not re-applied. The Job reads the existing policy first and skips the write when it already matches, because applying a policy resets the idle clock on every queue it matches — an unconditional write on a frequent deploy cadence would keep abandoned queues alive indefinitely.

Managing the policy yourself#

policyHook.enabled: false is the escape hatch for operators on restricted brokers who manage policies themselves. The three values above are then not required and the chart applies nothing. What that costs depends on whether the policy had ever been applied, and the two cases are not the same:

  • Never applied (a fresh cluster, or a broker where you removed the policy yourself): in the absence of other broker policies, durable queues have no maximum length, message TTL or idle-queue TTL. RabbitMQ 4.1’s built-in delivery limit is 20. Apply an equivalent policy before starting durable workloads.

  • Disabled after it had been applied: nothing in the chart deletes the policy. The last-applied event-queue-limits policy stays live on the broker and keeps enforcing, including reaping idle durable queues, frozen at whatever values it last carried. Disabling the hook stops the chart from updating the policy; it does not remove it. To actually get rid of it, delete it yourself — rabbitmqctl clear_policy -p notifications event-queue-limits, or the Policies page in the management UI.

Durable-Queue Idle TTL (Auto-Reap)#

The expires key in the policy above governs the lifetime of a durable queue, and it is the one bound with a client-visible consequence. A durable queue ends in one of two ways: a DeleteDurableQueue call, or the broker reaping it after it has gone expiresMs — 24 hours by default — without an attached consumer.

  • For service clients, an attached consumer keeps the queue alive. Publishing to the queue and re-declaring it do not. The idle clock starts when the consumer disconnects or cancels.

  • Messages in a reaped queue are discarded, not dead-lettered. They do not appear in the DLQ. Messages published after the last consumer detached can still be younger than messageTtlMs when the queue expires. Matching the two TTLs does not prevent this loss.

  • A returning client fails loudly, not silently. Consuming from a reaped queue returns NOT_FOUND on gRPC / 404 on REST rather than an empty stream.

  • Recovery is to recreate the queue. Call CreateDurableQueue again with the stored queue_id and the stored filter_groups. Creation is ensure-exists: a missing queue is created with those filters; an existing queue is returned unchanged, including its existing filters. Clients must persist the queue ID and its filter groups together. Events published while the queue was absent cannot be recovered through that queue.

Tuning the window. A deployment whose clients are legitimately idle for longer than the window — a batch job that runs weekly, for example — can raise expiresMs rather than have those clients take the recreate-on-return path. Setting expiresMs to null omits the key entirely and disables reaping, restoring the old “exists until deleted” behavior, at the cost of abandoned queues accumulating on the broker forever. 0 is not valid — the broker rejects it. Prefer expiresMs at or above messageTtlMs, while accounting for the remaining loss window described above.

Note

Expect the first auto-reap one full idle window after the deploy that first applies the policy. Applying it — on first install, and on any upgrade that changes a policy value — resets the idle clock on every queue it matches, including queues that have already been abandoned for weeks. Those queues are not reaped at the next tick; they are reaped expiresMs after the apply. A cluster with a long-standing backlog of abandoned durable queues therefore stays at its current queue count for a full window (24 h at the default) before the count starts falling. That is expected, not a broken hook.

Tuning Redelivery and Acknowledgement Limits#

The policy values are starting points. Size deliveryLimit above routine redelivery from client reconnects, failed processing and channel closures. On RabbitMQ 4.1 these requeues spend the same delivery budget. Durable consumers prefetch 20 messages by default (rabbitmq.durable_consumer_prefetch_count); a smaller prefetch reduces the number requeued when a consumer disconnects.

Bounding a Hung Durable Consumer#

The acknowledgement timeout is configured through policyHook.policy.definition.consumerTimeoutMs. Its default is null, which omits the policy key and leaves the broker’s timeout in effect (30 minutes by default on RabbitMQ 4.1). There is no service-level 60-second default. OMNI_EVENTS_CONSUMER_TIMEOUT_MS has been removed and is ignored.

For example, to request a five-minute acknowledgement timeout:

policyHook:
  policy:
    definition:
      consumerTimeoutMs: 300000

Allow enough time for a whole prefetch batch to be processed:

consumerTimeoutMs > durable_consumer_prefetch_count × worst-case processing time in ms

The chart rejects values below 60000. On RabbitMQ 4.1, values below five minutes are discouraged and enforcement runs on a one-minute tick. See RabbitMQ acknowledgement timeouts.

When the broker cancels a consumer, the service ends its stream with an error. Clients must reconnect; if the queue is missing, they must recreate it with CreateDurableQueue first. An error after a REST stream has opened is sent in an SSE error event, not as a changed HTTP response status.

Rollout Sequence: Existing Durable Queues#

This section applies only to existing deployments. Before upgrading, inventory each durable broker queue and save its client’s queue ID and filters. Check that the destination IDs are unique across the deployment: older queues with different identity prefixes could have the same queue ID. Assign distinct replacement IDs where needed, save the old-to-new mapping, and update the corresponding client configurations. Otherwise, ensure-exists creation can return one shared queue with the first client’s filters for multiple clients.

Two changes require attention:

  • Older releases used event-queue:<hashed-principal-id>:<queue-id> broker names. The current service uses event-queue:<queue_id>. Existing queues with the old names are not reachable through the current durable API, even if they are already quorum queues. Changing the service version does not move their messages or bindings.

  • A classic queue at a name the service now needs cannot be converted by redeclaring it. RabbitMQ returns 406 PRECONDITION_FAILED for the queue-type mismatch.

Schedule a maintenance window:

  1. Pause event publishers and drain the old queues with the old service and clients. Resolve captured dead-letters whose origin_queue still names an old queue; the management RPC cannot redirect them to a new name.

  2. Stop the old consumers and deploy the new service with the management Secret and policyHook configuration. Verify the policy hook succeeds.

  3. For any classic queue at a required new name, confirm it is empty and delete it before recreating it. Do not delete queues with messages you need.

  4. Call CreateDurableQueue with each destination queue_id and its saved filter_groups. A consume reconnect alone does not create a durable queue.

  5. Verify there is a distinct new quorum queue for each intended destination, with the expected bindings. Connect clients using their mapped IDs, then resume publishers. Retire old empty queues after validation.

Events published while the required queue or bindings are absent are not recoverable through that queue. A rollback also requires planning for the queue names understood by the older service.

Durable Queue IDs and Permissions#

Queue IDs must match ^[A-Za-z0-9._-]{1,200}$ and share one deployment-wide namespace. Use a UUID or a naming convention unique to the client application. CreateDurableQueue with an existing ID returns that queue unchanged; it does not replace its filters. Save both the ID and filters for recovery.

The consume-durable-queues, create-durable-queues (also used for updates), and delete-durable-queues permissions are deployment-wide. They do not enforce ownership of individual queues. Grant them to appropriate service accounts and avoid reusing another application’s queue ID.

Recovery Path#

The supported, primary way to inspect and recover captured dead-letters is the ManageDeadLetteredEvents management RPC — see Dead-Letter Management RPC: Operator Runbook below. It browses the DLQ and lets an operator REPLAY / DISCARD / LEAVE each message. Replay waits for a publisher confirm before acknowledging the DLQ copy.

RabbitMQ’s built-in tooling remains available only as a low-level fallback — for example, to inspect the DLQ directly when the management RPC is unavailable, or for broker-level operations the RPC does not expose:

  • The RabbitMQ Management UI, or

  • rabbitmqadmin

against the DLQ (the queue named by OMNI_EVENTS_DEAD_LETTER_QUEUE_NAME).

Captured messages remain subject to the DLQ’s retention and redelivery limits. Broker tools can consume or delete messages too; use them with the same care as the management RPC.

Dead-Letter Management RPC: Operator Runbook#

The Event Consumer Service exposes an operator-facing ManageDeadLetteredEvents RPC on the EventConsumerService for inspecting and triaging captured dead-letters. It is a bidirectional-streaming gRPC on the service’s gRPC surface (Helm service.grpc.port, default 50051; the standalone container default is 50052). There is no REST/SSE parity — this operation is gRPC-only.

Note

With permissions.enabled: true, the caller must hold event-consumer-service:inspect-and-replay-all-dlq — see Granting the DLQ-Management Permission below. A caller lacking it is denied with PERMISSION_DENIED at stream open, before any dead-lettered event is streamed.

Open the management stream#

Use a bidirectional gRPC client with the Notifications Consumer API specifications matching your service release. Connect to the configured gRPC endpoint with the operator’s bearer token and open:

nvidia.omniverse.notifications.consumer.v1beta.EventConsumerService/ManageDeadLetteredEvents

Send an empty opening request. In protobuf JSON form:

{}

Keep the request stream open while reading responses and sending actions. The DLQ is global to the deployment, so the opening request has no queue_id. Actions must use delivery tags received on this same live stream.

Browse dead-letters#

Inspect each event’s reason and origin_queue before acting. Reasons are REASON_DELIVERY_LIMIT, REASON_EXPIRED and REASON_MAXLEN. The delivery_count is broker metadata, not an exact count of processing attempts on the origin queue. Consult the API specification linked above for all fields.

Use the response’s delivery_tag to select a message. It is valid only within that live stream; never persist it or reuse it after reconnecting.

Act per message#

Send an action with the message’s delivery_tag:

  • ACTION_REPLAY republishes to the origin queue and removes the DLQ copy after broker confirmation. A disconnect before acknowledgement can produce a duplicate, so downstream processing must tolerate duplicates.

  • ACTION_DISCARD permanently removes the message.

  • ACTION_LEAVE requeues it. Avoid repeatedly leaving the same message: it can return immediately, and on RabbitMQ 4.1 each requeue consumes its DLQ delivery budget.

For example, after deciding to replay the event whose delivery tag is 42, send this subsequent request on the same stream, replacing 42 with the tag you actually received:

{"actions": [{"deliveryTag": "42", "action": "ACTION_REPLAY"}]}

A successful replay produces the following protobuf JSON response:

{"actionOutcome": {"deliveryTag": "42", "outcome": "OUTCOME_REPLAYED"}}

These examples use protobuf JSON field names. Check the action_outcome for each delivery tag: OUTCOME_REPLAYED confirms publication, not downstream processing. OUTCOME_ORIGIN_QUEUE_DELETED or OUTCOME_REPLAY_FAILED leaves the message available subject to DLQ retention and redelivery limits; inspect detail and resolve the failure before retrying. The API specification lists all outcome values.

Reason-driven triage#

Read the ``reason`` before acting. A REASON_DELIVERY_LIMIT message that keeps failing will simply re-dead-letter if blindly replayed — a replay loop. Fix the root cause first, or discard the message with ACTION_DISCARD if it is no longer needed. For REASON_EXPIRED or REASON_MAXLEN, resolve the backlog or capacity problem and confirm that processing historical events is still appropriate before sending ACTION_REPLAY. Watch the per-action Outcome reports to confirm publication, then monitor the origin queue and DLQ for repeated failure; an outcome report does not track what happens after publication.

Origin-queue-deleted behavior#

If an ACTION_REPLAY targets an origin queue that no longer exists, the RPC emits an explicit OUTCOME_ORIGIN_QUEUE_DELETED outcome report back to the operator on the stream (keyed to the delivery_tag) and leaves the DLQ copy in place (LEAVE semantics). Recreating the origin queue restores a replay target. Verify that the queue still belongs to the intended application before replaying. A queue recreated through the current API cannot receive a replay addressed to an old identity-derived broker name.

Replay Scope#

Replay targets the broker queue name recorded in origin_queue; the caller cannot select a different destination. This is not a tenant-isolation boundary: queue IDs are deployment-wide, and if another application recreates a deleted ID, replay targets that new queue. Confirm the destination before replaying.

The DLQ-management permission grants visibility and actions across the entire DLQ. It does not apply per-event permission checks while browsing. Replayed messages are checked against the consuming caller’s configured permissions when consumed normally, provided permission enforcement is enabled.

Granting the DLQ-Management Permission#

Set permissions.enabled: true, configure permissions.endpoint, and select an authenticated principalSource as described in Notifications Services Configuration. Then grant event-consumer-service:inspect-and-replay-all-dlq to the DLQ operators. With enforcement enabled, a caller lacking it is denied with PERMISSION_DENIED at stream open.

Configure this permission in your Permissions Service deployment (see Permissions Service Configuration):

  • Declare the action for the event-consumer-service service: inspect-and-replay-all-dlq. The fully-qualified action string is event-consumer-service:inspect-and-replay-all-dlq.

  • Grant the action to the principal(s) that operate the DLQ — bind event-consumer-service:inspect-and-replay-all-dlq to the operator role or principal.

Note

permissions.enabled defaults to false. In that mode this permission check is bypassed, including for authenticated callers. The tokenless getting-started configuration therefore does not protect DLQ management. Enable authorization before exposing it to users.