Notifications: Delayed, Missing or Duplicated#

Symptom#

One of:

  • Delayed — a change made through the Storage API shows up for other clients minutes later, or only after a client refreshes manually.

  • Missing — the change never propagates at all. Subscribed clients keep showing stale content until their cache expires.

  • Duplicated — subscribers receive the same event more than once, and clients act on it twice.

Blast radius#

Reads and writes keep working. What breaks is propagation: clients that rely on subscriptions to learn about changes will serve stale content, and the Storage Service’s own cache invalidation is driven by the same events, so cached stat results can outlive the change that should have invalidated them.

This is why the failure is easy to miss — nothing errors. Requests succeed and return old data.

Understand the path first#

Events cross several systems, and the fix depends entirely on where they stop:

object store (S3 / Azure Blob)
    -> backend event subscription (Event Grid / SQS)
        -> queue (Service Bus / SQS)
            -> Storage Service event consumer
                -> Event Aggregation Service
                    -> RabbitMQ
                        -> Event Consumer Service
                            -> subscribed clients

Work along it in order. The single most common cause of “delayed” is queue depth near the start of the chain, not a fault in the services at the end of it.

Triage#

1. Is it delay or loss? Measure the queue, not the symptom.

For an Azure Service Bus queue, check the active and dead-lettered message counts. A large and growing active count means the consumer cannot keep up — the events are not lost, they are queued behind other events.

az servicebus queue show \
    --resource-group <rg> --namespace-name <namespace> --name <queue> \
    --query "{active:countDetails.activeMessageCount, dead:countDetails.deadLetterMessageCount, size:sizeInBytes}"

Important

A backlog is not always caused by a slow consumer. It is more often caused by too many events being published in the first place. Check what is feeding the queue before scaling anything.

2. Is the subscription over-subscribed?

A backend event subscription with no subject filter forwards every object change in the whole account into one queue. A single busy container — a benchmark, a migration, a crawler — then starves everything else.

az eventgrid event-subscription show \
    --name <subscription> --source-resource-id <storage-account-id> \
    --query "{filter:filter.subjectBeginsWith, endpoint:destination.endpointType}"

An empty subjectBeginsWith on a shared storage account is the classic cause of chronic backlog. Scope it to the container that actually needs events.

3. Are events expiring before they are delivered?

Check the queue’s TTL and dead-lettering settings. A queue with a message TTL and deadLetteringOnMessageExpiration disabled will silently drop everything older than the TTL — that turns a delay into permanent loss, with nothing in any log to show for it.

az servicebus queue show --resource-group <rg> --namespace-name <namespace> --name <queue> \
    --query "{ttl:defaultMessageTimeToLive, dlqOnExpiry:deadLetteringOnMessageExpiration, maxSize:maxSizeInMegabytes}"

4. Is more than one consumer sharing the queue?

Service Bus queues deliver competing-consumer: each message goes to exactly one receiver. Two environments pointed at one queue will each see a random half of the events, which presents as “missing events” in both. Each environment needs its own queue and its own subscription.

5. Is the downstream chain healthy?

kubectl -n <namespace> logs deployment/storage-service --since=15m | grep -i -E 'event|servicebus|sqs'
kubectl -n <namespace> logs deployment/event-aggregation-service --since=15m | tail -50
kubectl -n <namespace> logs deployment/event-consumer-service --since=15m | tail -50
kubectl -n <namespace> get pods -l app.kubernetes.io/name=rabbitmq

If the queue is shallow and the services are quiet, the events are not arriving from the object store at all — go back to step 2.

Fix#

Over-broad subscription (most common). Scope the subscription to the container that needs events. Leave other workloads on their own subscription and queue:

az eventgrid event-subscription update \
    --name <subscription> --source-resource-id <storage-account-id> \
    --subject-begins-with "/blobServices/default/containers/<container>/"

Consumer cannot keep up. Raise the worker count and receive batch size for the subscription in your deployment values, rather than by hand — Storage Service exposes both under its storage-event settings. Size it against measured throughput: work out the per-message service time, multiply by workers, and confirm the downstream broker tolerates the resulting rate.

Backlog already built. Once inflow is scoped, the active count drains on its own. Dead-lettered messages do not — purge or re-drive them deliberately, after confirming they are not needed.

Duplicates. Both Service Bus and SQS are at-least-once. Duplicates are expected under retry and redelivery, and are not by themselves a fault. Clients must be idempotent: key on the resource address plus version or ETag, and treat a repeat as a no-op. If duplicates appear in bursts, look for a consumer that is failing after processing but before acknowledging — the message is redelivered because the ack never arrived.

Warning

Where the environment is managed by Terraform, make subscription and queue changes there. An imperative az change to a Terraform-managed resource is reverted by the next apply, and the symptom returns hours later with no obvious cause.

Verify#

  • Active message count returns to near zero and stays there under normal load.

  • A write through the Storage API is visible to a subscribed client within the expected window.

  • Dead-letter count stops growing.