Event Consumer Operations#

Goal: Prepare, deploy and operate the Event Consumer Service using its Helm chart.

When to use: Read the preparation section before your first production installation, and use the rest of this page when sizing, monitoring or changing an existing deployment. The initial installation commands are in Notifications Services Configuration. Only Upgrading an Existing Deployment assumes you have a previous release.

Prepare Any Deployment#

These checks apply to both first installations and upgrades:

  1. Prepare RabbitMQ. Make its AMQP and management endpoints reachable from the service namespace. For production durable queues, configure persistent storage and a multi-node broker before starting Event Consumer Service; the service creates its dead-letter queue at startup. Follow Broker Requirements for version, storage and replication requirements.

  2. Create the connection and management Secrets. Configure policyHook.managementApi.baseUrl, policyHook.managementApi.vhost and policyHook.managementApi.existingSecret.name. The current chart requires these by default even if clients only use non-durable streams. Follow Configure the Durable-Queue Bounds Policy for the Secret and values examples. If managing policies yourself, apply equivalent bounds before workloads start.

  3. Configure authentication and permissions. The downloadable consumer values use tokenless access for the getting-started walkthrough. Before exposing a production deployment, select a suitable authentication mode and enable the permission checks your workloads need. Follow Authentication and Permissions. Authentication alone does not restrict durable queue operations or DLQ access. For durable clients, also review Durable Queue IDs and Permissions: queue IDs share a deployment-wide namespace, and permissions do not enforce ownership of individual queues.

  4. Review retention and capacity. Durable queues expire after 24 hours without a consumer by default. Choose retention windows for your workloads using Durable-Queue Bounds Policy. Use Scaling and Broker Memory below to size the service and broker together.

  5. Prepare clients and ingress for reconnects. Check Ingress and Reconnection and Graceful Shutdown before serving traffic. Clients need to reconnect after pod termination or broker-side consumer loss.

For a new deployment, follow the installation walkthrough with those values, then complete Validate the Deployment before enabling workloads. For an existing deployment, also complete the upgrade steps below.

Upgrading an Existing Deployment#

When moving an existing installation to Event Consumer Service 1.11.0:

  • Inventory existing durable queues. Older identity-derived queue names require migration. Save queue IDs and filters, resolve any ID collisions, and follow the durable queue rollout sequence.

  • Check whether existing RabbitMQ users have policy-management rights. Updating the example init script does not update users on a persistent broker; see Configure RabbitMQ VHosts.

  • Recheck existing identity-provider settings against Authentication and Permissions. Missing or empty identities are rejected.

  • Update non-durable dashboards from queue_id to queue_ref. See Observability for the current telemetry configuration.

  • Plan for client reconnects when a configuration change restarts service pods. Complete the validation steps before resuming workloads.

Scaling and Broker Memory#

The chart enables the channel-pool autoscaler by default. Keep service.rest.enabled: true so it can read the capacity endpoint. Do not run a second autoscaler against the same Deployment.

The following settings control non-durable capacity and scaling. Readiness and autoscaling measure stream slots; the table gives the units for each setting.

Table 2 Default capacity settings#

Helm value

Default

rabbitmq.channel_pool_size

100 channels per pod

rabbitmq.channel_client_limit

8 streams per channel

rabbitmq.readiness_min_available_channels

80 stream slots reserved in the scaling signal

rabbitmq.channel_prefetch_count

15 messages per non-durable consumer

rabbitmq.broker_stream_memory_budget_mib

1024 MiB across the fleet

rabbitmq.max_non_durable_streams_per_pod

null (computed from the broker budget)

rabbitmq.max_re_routable_streams_per_pod

null (computed between the packing target and cap)

channelPoolAutoscaling.minReplicas / maxReplicas

3 / 10

For autoscaling timing and other tuning options, see the chart’s values.yaml.

At these defaults, each pod has a packing target of 800 streams and reports 720 usable slots after the readiness reserve. With more than 720 active streams, readiness reports not-ready, excluding the pod from new connections through the main Kubernetes Service. Existing streams continue, and the pod can still admit streams on established connections or through the proxy until an admission threshold is reached.

The default computed admission cap is 3342 non-durable streams per pod. REST/SSE streams and gRPC streams routed through the chart’s proxy are refused from 2071 streams onward, reserving the remaining capacity for direct gRPC connections. At the cap, all new non-durable streams are refused. Clients receive HTTP 503 or gRPC RESOURCE_EXHAUSTED and should retry with backoff. These counts are configuration-derived limits, not a throughput guarantee.

Tune settings together:

  • Broker memory: allocate actual broker RAM before raising rabbitmq.broker_stream_memory_budget_mib. Leave headroom for durable queues, replication and disconnected clients’ queues, which keep buffering during their reconnect window. Load-test with your event sizes and fan-out.

  • Fleet ceiling: raising channelPoolAutoscaling.maxReplicas divides the same budget among more pods and lowers the per-pod cap. It also rolls the service pods. Adding replicas alone does not increase the fleet’s broker memory budget. With autoscaling disabled, replicaCount supplies that divisor instead.

  • Absolute override: rabbitmq.max_non_durable_streams_per_pod overrides the computed cap. Set it only after measuring broker capacity for the whole fleet; it bypasses the budget calculation.

  • Single replica: if requests cannot be routed to another pod, set rabbitmq.max_re_routable_streams_per_pod to the chosen admission cap so REST/SSE requests are not refused earlier than direct gRPC streams.

The autoscaler adds capacity promptly and requires sustained spare capacity before scaling down. minReplicas must be at least 1. It restores a fleet below that floor only when running pods report metrics; manually scaling the Deployment to zero is an off switch and is not reversed automatically.

Durable Capacity#

Durable consumers and dead-letter browsing use a separate capacity budget: rabbitmq.durable_channel_pool_size defaults to 100 and rabbitmq.durable_channel_client_limit to 1. The non-durable autoscaler’s capacity signal does not measure this pool. If durable operations reach capacity, review these settings and broker resources separately. Increasing non-durable capacity does not increase durable capacity.

Ingress and Reconnection#

The chart’s httpProxy settings apply when using its Contour ingress. Keep the streaming routes for all four gRPC streaming methods, including ManageDeadLetteredEvents, and both REST event streams when adapting ingress configuration. Ordinary short HTTP timeouts can disconnect healthy streams.

The default streaming retry policy retries initial capacity refusals twice; ordinary calls retry connection failures and refused HTTP/2 streams three times. These retries cannot resume an established stream. Clients still need backoff and reconnection. Use a stable queue_id when retrying queue creation to avoid creating duplicate queues.

See the chart’s httpProxy values for protocol-specific routing and retry settings. Disable a retry policy with enabled: false.

The chart creates an additional Service for proxy routing when ingress is enabled. Keep the chart’s route targets when customizing ingress. Direct clients should use the main Service and must not add the X-Via-Proxy header themselves; that header classifies a non-durable gRPC stream for the earlier refusal threshold.

For REST non-durable reconnection, forward the Last-Event-ID request header through your ingress. The reconnect_token query parameter is deprecated but still accepted. The reconnect token is valid only while its queue survives: rabbitmq.non_durable_queue_ttl_seconds defaults to 20 seconds after the consumer disconnects. If it has expired, open a new stream with the saved filters; events published while the queue was absent cannot be recovered.

If the broker cancels a consumer, the service closes that stream with an error. For a surviving queue, reconnect using the same queue ID or reconnect token. For a missing durable queue, recreate it with its saved ID and filters first. On an already-open REST stream, the error arrives as an SSE error event; clients must inspect that event rather than relying only on the HTTP status.

Graceful Shutdown#

Add or tune this block in event-consumer-values.yaml:

shutdown:
  settleDelaySeconds: 15
  drainWindowSeconds: 30
  drainPollIntervalSeconds: 0.5
  terminationGracePeriodSeconds: 60

A terminating pod first continues serving during the settle delay, allowing endpoint changes to propagate to direct clients. It then reports not-ready and refuses new non-durable streams while allowing existing streams to finish during the drain window. Clients whose streams remain open at shutdown must reconnect. Increasing the settle delay does not eliminate all proxy-side rollout errors.

The chart requires:

terminationGracePeriodSeconds >= settleDelaySeconds + drainWindowSeconds + 10

This is a minimum. Allow additional grace for idle REST/SSE streams and slow broker disconnects if clean stream termination is required. If upgrading from a preview chart that used shutdown.preStopDelaySeconds, rename it to shutdown.settleDelaySeconds; the old key is rejected.

Apply Configuration Changes#

Use Notifications Services Configuration for first installation. For later changes, merge settings into your existing values and retain authentication and ingress overrides. From the directory containing the extracted charts:

helm lint ./event-consumer-service \
  -f ./event-consumer-service/event-consumer-values.yaml
helm upgrade event-consumer-service ./event-consumer-service \
  --namespace storage-apis \
  -f ./event-consumer-service/event-consumer-values.yaml

Include each additional values file your deployment uses, in the same order, in both commands. Use the migration procedure above before applying an upgrade that changes durable queue names.

Validate the Deployment#

These commands use release event-consumer-service in namespace storage-apis. Substitute your deployment names if they differ. Run these checks after Helm finishes; a successful rollout alone does not confirm the broker policy was applied.

kubectl rollout status deployment/event-consumer-service -n storage-apis
kubectl get jobs -n storage-apis \
  -l app.kubernetes.io/instance=event-consumer-service,app.kubernetes.io/component=queue-policy
kubectl logs -n storage-apis \
  -l app.kubernetes.io/instance=event-consumer-service,app.kubernetes.io/component=queue-policy \
  --all-containers=true --tail=100

The rollout should complete successfully, and the policy Job should show 1/1 completions. Its logs should report either that the policy was applied or that it already matches. Resolve any failure before starting durable workloads. If policyHook.enabled: false, no policy Job is created; verify your independently managed policy instead.

Next, forward the REST port of one service pod:

kubectl port-forward -n storage-apis deployment/event-consumer-service 8000:8000

While port-forwarding is running, use another terminal:

curl --fail http://localhost:8000/api/v1beta/health
curl --fail http://localhost:8000/api/v1beta/metrics/channel-pool

The health response should be {"status":"healthy"}. An idle, fully started pod at the chart defaults reports capacity similar to:

{"utilization_percent": 0, "channels_in_use": 0, "channels_total": 720}

These endpoints require the REST surface to be enabled and are served without a bearer token. The health response confirms the HTTP server responds; use the rollout, broker-policy and smoke-test checks as well. The capacity endpoint reports one pod: channels_in_use and channels_total are stream slots despite their names. An in-use count above the total means the pod is serving above its scaling target; it is not necessarily at its admission cap. Monitor all replicas as well as RabbitMQ memory alarms, queue depths, and DLQ message age.

Keep the gRPC liveness health service distinct from the readiness check in custom probes. Capacity pressure should make a pod unready, not trigger a liveness restart that disconnects its clients. Leave service.grpc.healthCheckingEnabled: true when using the chart’s gRPC probes.

The policy Job is retained for 15 minutes by default (policyHook.ttlSecondsAfterFinished: 900). Inspect failures before cleanup. An unreachable management API, invalid credentials, missing Secret or vhost mismatch can fail the Helm release. Also inspect the effective policy on queues in RabbitMQ: a higher-priority matching policy can override event-queue-limits even when the hook succeeds.

To opt into Discovery operator registration, add:

service:
  annotations:
    discovery.omniverse.nvidia.com/enabled: "true"

The operator must already be installed. This is optional and does not replace this guide’s static Discovery Service configuration.