Event Aggregation Operations#

Goal: Prepare, deploy and operate the Event Aggregation Service using its Helm chart.

When to use: Read the preparation section before your first production installation. Use the remaining sections to size the publisher, configure clients and validate installations or upgrades. Only Upgrading an Existing Deployment assumes a previous release.

Prepare Any Deployment#

Use Notifications Services Configuration for broker setup, Secrets, authentication and installation commands. The publisher and consumer must use the same broker and vhost. The downloadable values use tokenless access; configure authentication and permissions before production use.

Before installing or upgrading, review the request limits, capacity and shutdown settings below. Configure telemetry using Observability, then complete Validate the Deployment after installation. Existing deployments must also follow Upgrading an Existing Deployment.

Request Limits and Publish Results#

The REST request-body limit defaults to 4 MiB, including the JSON envelope. Configure it in event-aggregation-values.yaml:

rest:
  maxRequestBytes: 4194304

The value must be a positive integer. Larger bodies receive HTTP 413 before publication, and the connection closes. Split or reduce the request before retrying. REST batches must contain 1–1000 events and fit within the byte limit. The gRPC inbound limit is separately 4 MiB; changing rest.maxRequestBytes does not change it. Oversized gRPC requests receive RESOURCE_EXHAUSTED.

Before raising the REST limit, check the broker’s max_message_size and pod memory allocation. Published messages can be larger than incoming requests. Test with your largest events and expected concurrency.

A broker or channel-acquisition failure can return HTTP 200 with result.success set to false and a failure_reason. Check each entry in batch results too; batches can partially succeed. gRPC also returns per-event results. Do not rely on HTTP 503 to detect publish failures. Retry transient failures with backoff and avoid resending successful entries. A lost response leaves the outcome uncertain, so retries can duplicate events. See the Notifications API specifications for request and response definitions.

Publishing Capacity#

The chart’s channel-pool defaults are:

rabbitmq:
  channel_pool_size: 25
  channel_pool_min_size: 5
  channel_client_limit: 1
  channel_overflow_enabled: true
  channel_wait_ms: 500

At these defaults, the pool grows to channel_pool_size and then shares channels between additional operations. These defaults are not a concurrency cap. To limit operations to channel_pool_size × channel_client_limit, set channel_overflow_enabled: false; operations then wait up to channel_wait_ms for capacity before returning a failed publishing result. Sharing channels means a channel error can affect several in-flight operations.

The chart defaults to one replica with autoscaling.enabled: false. Tune replicaCount or enable autoscaling based on measured load and broker capacity. See the chart’s values.yaml for autoscaling and pool tuning options.

Graceful Shutdown#

The chart defaults to:

shutdown:
  settleDelaySeconds: 15
  drainWindowSeconds: 5
  drainPollIntervalSeconds: 0.5
  terminationGracePeriodSeconds: 60

During termination, the service keeps serving for the settle delay, then waits up to the drain window for in-flight publishes before stopping. Allow for this delay during rollouts; publishers must handle connection failures and retries.

The chart requires:

terminationGracePeriodSeconds >= settleDelaySeconds + drainWindowSeconds + 10

Leave additional time for slow broker disconnects or open REST connections. Setting settleDelaySeconds: 0 removes the routing-propagation allowance. The chart’s Contour routes do not configure automatic retries.

Upgrading an Existing Deployment#

Before upgrading to Event Aggregation Service 1.6.0:

  • Check existing REST payload sizes against the new 4 MiB default limit and split batches or review the limit before deploying.

  • Merge the shutdown settings into existing values. With the default settle and drain windows, a termination grace period below 30 seconds fails chart rendering. The new chart default is 60 seconds; allow for longer pod shutdowns.

  • Verify that identity claims are nonempty. Invalid credentials now take precedence over request-validation errors, except for oversized requests. Check clients that depend on the previous error ordering.

  • Update monitoring for the authentication outcomes described in Observability. Existing telemetry settings remain supported; review their precedence when enabling the new observability settings.

  • Verify that publishers handle per-event failures in successful API responses. The REST specification no longer advertises a pool-exhaustion 503 response.

Apply Configuration Changes#

Use the installation walkthrough for first installation. For an existing release, merge changes into its values file and retain all authentication and ingress overrides. From the deployment directory containing the extracted charts:

helm lint ./event-aggregation-service \
  -f ./event-aggregation-service/event-aggregation-values.yaml
helm upgrade event-aggregation-service ./event-aggregation-service \
  --namespace storage-apis \
  -f ./event-aggregation-service/event-aggregation-values.yaml

Include any additional values files used by your deployment, in the same order, in both commands. Substitute your release and namespace if they differ.

Validate the Deployment#

After Helm finishes, check the rollout and service logs:

kubectl rollout status deployment/event-aggregation-service -n storage-apis
kubectl logs deployment/event-aggregation-service -n storage-apis --tail=100
kubectl port-forward -n storage-apis deployment/event-aggregation-service 8001:8000

While port-forwarding runs, use another terminal:

curl --fail http://localhost:8001/api/v1beta/health

Expect {"status":"healthy"}. This unauthenticated check requires rest.enabled: true and uses the chart’s default REST port, 8000. It checks HTTP availability; use the following tests to verify publishing and delivery.

Run the Smoke Tests with credentials and publish permissions appropriate to your deployment. Then follow Testing the Notifications Services to verify delivery through the consumer. Resolve authentication or broker errors and failed publishing results before enabling production traffic.

Discovery Registration#

To opt into dynamic Discovery registration, add:

service:
  annotations:
    discovery.omniverse.nvidia.com/enabled: "true"

This requires an installed Discovery operator. The installation walkthrough uses static Discovery configuration, which does not require this annotation.