Notifications Services Configuration#
Goal: Deploy optional Notifications Services (RabbitMQ, event aggregation, event consumer) so the Storage Service can publish file events and clients can consume them.
When to use: Enable when you need event-driven workflows (e.g., Kit-based apps or other services reacting to file create/delete/modify). If you do not need notifications, skip this section and continue with the Discovery Service.
The implementation uses RabbitMQ as the event broker with two components: event aggregation (publishing) and event consumer. You deploy and enable them from the Storage Service or other services that publish or consume events.
Note
The ports for RabbitMQ are only ever used within the Kubernetes cluster. They are not exposed to the outside world.
Before Your First Installation#
The examples below start with tokenless access and nonpersistent RabbitMQ for getting started. For a production deployment, review Prepare Any Deployment before installing the charts. It covers broker storage and replication, required Secrets, authentication, retention, capacity and client reconnection. Configure persistent storage and broker replicas before starting Event Consumer Service if you need durable queues. Also review Event Aggregation Operations for publisher permissions, request limits, capacity and shutdown settings.
For an existing installation, also read the upgrade checks for Event Aggregation and Event Consumer before running Helm upgrades. New deployments do not need the old-queue migration steps.
Working Directory and Values Files#
Start in the deployment directory that contains your extracted storage-service/
chart. Keep the RabbitMQ, Event Aggregation and Event Consumer chart directories
alongside it. Each installation section below enters its chart directory and
returns to the deployment directory after validation.
Merge each YAML fragment into that service’s values file. If a top-level key already exists, add the new settings under it; do not create duplicate YAML keys. The downloadable values files provide a starting point. Replace placeholders before running commands, and retain any production authentication or other settings when upgrading an existing deployment.
Service Ports#
Service |
Port |
Description |
|---|---|---|
RabbitMQ |
5672 |
AMQP messaging protocol |
RabbitMQ Management |
15672 |
Management UI (if enabled) |
Event Aggregation |
50051 |
gRPC endpoint for publishing events |
Event Consumer |
50051 |
gRPC endpoint (Helm default; standalone container default: 50052) |
Event Consumer |
8000 |
HTTP/REST endpoint for consuming events ( |
Pulling the RabbitMQ Helm chart#
For our notifications services, we use RabbitMQ as the event broker. Pull the
RabbitMQ Helm chart from the NGC catalog using NGC_API_KEY set in
Requirements. Run these
commands in the same Bash session.
# pull the chart from NGC
helm fetch https://helm.ngc.nvidia.com/nvidia/omniverse/charts/rabbitmq-99.3.0.tgz --username='$oauthtoken' --password="${NGC_API_KEY:?Set NGC_API_KEY in this Bash session}"
# unpack the chart and cd into the directory
tar -xvf rabbitmq-99.3.0.tgz
cd rabbitmq
Configuring the RabbitMQ Helm chart#
We need to set values for the RabbitMQ Helm chart to use our event broker backend. We will use rabbitmq-values.yaml as the name of our values file.
# create and edit the local values file, we're using VS Code for this example
code rabbitmq-values.yaml
Pull Secrets#
Set your rabbitmq-values.yaml to use the ngcpull-secret for image pulling (same secret as for the Storage Service). You will also need to point it to the correct NGC registry.
image:
pullSecrets:
- name: ngcpull-secret
registry: nvcr.io
repositoryPrefix: nvidia/omniverse
repository: rabbitmq
tag: 4.1.3-debian-12-r1
Configure RabbitMQ Password#
We need to configure the RabbitMQ with a password that is stored in a Kubernetes secret. Replace {RABBITMQ_PASSWORD} with the password you want to use.
# create a Kubernetes secret with the password
kubectl create secret generic rabbitmq-service-env --from-literal=rabbitmq-password={RABBITMQ_PASSWORD} --namespace storage-apis
Now you need to make sure to use the secret in your rabbitmq-values.yaml file.
auth:
username: rabbitmq-user
existingPasswordSecret: "rabbitmq-service-env" # this is the secret name we created above
existingSecretPasswordKey: "rabbitmq-password" # this is the key in the secret with the password
As this is a minimal deployment, we will avoid using persistent storage for the RabbitMQ cluster that would normally be used for durable queues.
Note
Persistence is disabled to keep the system stateless. If you need persistence for your deployment, enable and configure your needed storage class.
Warning
This minimal/testing path is fine for the non-durable streaming getting-started flow shown here, but the Event Consumer Service’s durable queues require persistence enabled (persistence.enabled: true) — with persistence disabled, durable queues and their messages are lost on pod restart. For high availability, durable queues also need a multi-node cluster (e.g. replicaCount: 3 + clustering.enabled: true). For production / durable configuration, see Event Consumer Dead-Letter Capture.
persistence:
enabled: false
Configure RabbitMQ VHosts#
We use RabbitMQ virtual hosts (vhosts) to isolate the notifications services from other services that may share the same RabbitMQ instance. This ensures that queues, exchanges, and permissions are scoped to the notifications vhost and won’t conflict with other workloads.
Add the following to your rabbitmq-values.yaml to configure RabbitMQ to create the notifications vhost on startup. This uses loadDefinition to load a JSON definition from a Kubernetes secret that is created by the chart via extraSecrets, plus an init script that grants rabbitmq-user permissions on that vhost and the broker tag it needs to manage policies:
# Load vhost definitions on startup
loadDefinition:
enabled: true
existingSecret: rabbitmq-load-definition
extraSecrets:
rabbitmq-load-definition:
load_definition.json: |
{
"vhosts": [
{"name": "/"},
{"name": "notifications"}
]
}
initScripts:
setup-permissions.sh: |
#!/bin/bash
if ! rabbitmqctl status --timeout 10 >/dev/null 2>&1; then
echo "Node not running during init (restart with persisted data); skipping."
exit 0
fi
rabbitmqctl set_permissions -p notifications rabbitmq-user ".*" ".*" ".*"
rabbitmqctl set_user_tags rabbitmq-user administrator
Note
The user field in the permissions must match the auth.username value you configured above (rabbitmq-user). The default / vhost is included to retain standard RabbitMQ functionality, and the notifications vhost is where the event aggregation and event consumer services will operate.
The node-running guard at the top of the script matters once persistence.enabled: true is set: init scripts only run with the node already up on the first boot (empty data directory), so on later restarts an unguarded rabbitmqctl would fail and crash-loop the pod. The permissions are already persisted in that case, so skipping is safe.
The set_user_tags line grants policy-management rights to the example’s
shared broker user. A separate management user with the policymaker tag
and access to notifications is preferable for production; keep the
service’s AMQP user separate from that deploy-time credential.
Warning
On an existing persistent broker, the init script skips this setup on
restart. If rabbitmq-user does not already have policy-management rights,
update its tags on a running node before upgrading Event Consumer Service:
kubectl exec -n storage-apis rabbitmq-0 -- \
rabbitmqctl set_user_tags rabbitmq-user administrator
Replace rabbitmq-0 if your broker pod has a different name. This sets the
user’s tag list; preserve any additional tags your deployment requires. If
using a separate management user, provision that user’s tag and vhost access
instead.
RabbitMQ Notification Connection Secret#
There is an additional Kubernetes secret you will need to create for the Notification Services for them to connect to the RabbitMQ service. We will use this later on when deploying each the Event Aggregation and Event Consumer Services. This is used to store the RabbitMQ amqp uri and is formatted as amqp://{username}:{RABBITMQ_PASSWORD}@{rabbitmq-service-name}:{port}/notifications. Note the /notifications path at the end, which directs the services to use the notifications vhost we configured above.
# create a Kubernetes secret with the RabbitMQ URI (note the /notifications vhost path)
kubectl create secret generic event-services-env --from-literal=rabbitmq_uri=amqp://rabbitmq-user:{RABBITMQ_PASSWORD}@rabbitmq:5672/notifications --namespace storage-apis
Deploying the RabbitMQ Helm chart#
Now you can deploy the RabbitMQ Helm chart using the helm install command.
# deploy the chart
helm upgrade --install rabbitmq -n storage-apis -f rabbitmq-values.yaml .
Validate the RabbitMQ Deployment#
You can validate the RabbitMQ deployment by checking the pods are running and the service is accessible.
# check the pods are running
kubectl get pods -n storage-apis
# check the service is accessible
kubectl get service -n storage-apis
# return to the deployment directory
cd ..
Pulling the Aggregation Services Helm charts#
With RabbitMQ and our backend storage configured, pull and unpack the
Notifications Aggregation Service Helm chart from the NGC catalog. Use the
same Bash session with NGC_API_KEY set.
# pull the chart from NGC
helm fetch https://helm.ngc.nvidia.com/nvidia/omniverse/charts/event-aggregation-service-1.6.0.tgz --username='$oauthtoken' --password="${NGC_API_KEY:?Set NGC_API_KEY in this Bash session}"
# unpack the chart and cd into the directory
tar -xvf event-aggregation-service-1.6.0.tgz
cd event-aggregation-service
Configuring the Aggregation Service#
Create your event-aggregation-values.yaml file for the Aggregation Service.
# create and edit the local values file, we're using VS Code for this example
code event-aggregation-values.yaml
Pull Secrets#
Set your event-aggregation-values.yaml to use the ngcpull-secret for image pulling (same secret as for the Storage Service).
image:
pullSecrets:
- name: ngcpull-secret
Configure Backend Access#
We need to configure the Aggregation Service to access your backend event broker. The chart defaults to using a secret named event-aggregation-env, but since we created a shared secret named event-services-env above, we need to override it:
# Override the default secret name (event-aggregation-env) to use our shared secret
envSecretName: event-services-env
Note
Authentication for this getting-started flow. By default the service uses
principalSource: jwtUnverified, which requires a bearer token on every
request (including the Storage Service’s event publishes). For this tokenless
walkthrough, set principalSource: disabled so the service runs without
authentication. To configure real authentication and permissions instead, see
Authentication and Permissions below.
# Tokenless getting-started: no auth middleware, no token required
principalSource: disabled
Install the Aggregation Service#
Now you can install the Aggregation Service using the helm install command.
# install the chart
helm upgrade --install event-aggregation-service -n storage-apis -f event-aggregation-values.yaml .
Validate the Aggregation Service Deployment#
You can validate the Aggregation Service deployment by checking the pods are running and the service is accessible.
# check the pods are running
kubectl get pods -n storage-apis
# check the service is accessible
kubectl get service -n storage-apis
# return to the deployment directory
cd ..
Check rollout health using Event Aggregation Operations. Complete its end-to-end delivery checks after installing Event Consumer and configuring the Storage Service below.
Pulling the Consumer Services Helm charts#
Pull and unpack the Notifications Consumer Service Helm chart from the NGC
catalog using the same Bash session with NGC_API_KEY set. This is the
service that external clients use to consume events from the Notifications
Services.
helm fetch https://helm.ngc.nvidia.com/nvidia/omniverse/charts/event-consumer-service-1.11.0.tgz --username='$oauthtoken' --password="${NGC_API_KEY:?Set NGC_API_KEY in this Bash session}"
# unpack the chart and cd into the directory
tar -xvf event-consumer-service-1.11.0.tgz
cd event-consumer-service
Configuring the Consumer Service#
Create your event-consumer-values.yaml file for the Consumer Service.
# create and edit the local values file, we're using VS Code for this example
code event-consumer-values.yaml
Pull Secrets#
Set your event-consumer-values.yaml to use the ngcpull-secret for image pulling (same secret as for the Storage Service).
image:
pullSecrets:
- name: ngcpull-secret
Configure RabbitMQ Connection#
We need to configure the Consumer Service to connect to the RabbitMQ service. Similar to the Aggregation Service, the chart defaults to using a secret named event-consumer-env, but since we created a shared secret named event-services-env above, we need to override it:
# Override the default secret name (event-consumer-env) to use our shared secret
envSecretName: event-services-env
Note
Authentication for this getting-started flow. As with the Aggregation
Service, the Consumer Service defaults to principalSource: jwtUnverified and
requires a bearer token on event API requests — including the tokenless event
stream used in the testing section below. For this walkthrough, set
principalSource: disabled. See Authentication and Permissions below to
configure real authentication.
# Tokenless getting-started: no auth middleware, no token required
principalSource: disabled
Configure the Durable-Queue Bounds Policy#
The Event Consumer Service chart ships the RabbitMQ policy that bounds durable event queues — redelivery limit, message TTL, maximum length, overflow behavior, and an idle-queue TTL that auto-deletes a durable queue with no attached consumer. A post-install,post-upgrade hook Job applies the policy through the broker’s management HTTP API, on install and on every helm upgrade. It skips the policy write when the existing policy already matches. You do not apply this policy by hand.
Set all three policyHook values below before installing or upgrading. The
chart refuses to render without them. The management endpoint must refer to the
same RabbitMQ deployment as the service’s AMQP URI.
First create the Kubernetes secret holding the management-API credential. Only the hook Job mounts this Secret; the service uses the AMQP URI from event-services-env. This example uses the same broker user for both, so both credentials have that user’s privileges. The user must be able to set policies on the notifications vhost — this journey’s rabbitmq-user was tagged for that in Configure RabbitMQ VHosts above. Replace {RABBITMQ_PASSWORD} with the password you chose earlier.
# create a Kubernetes secret with the RabbitMQ management-API credential
kubectl create secret generic rabbitmq-mgmt-credentials \
--from-literal=username=rabbitmq-user \
--from-literal=password={RABBITMQ_PASSWORD} \
--namespace storage-apis
Note
The secret keys must be named username and password (overridable via policyHook.managementApi.existingSecret.usernameKey / passwordKey). Create it before you install or upgrade the chart: a missing secret fails the release rather than being skipped.
Then point the chart at the broker in your event-consumer-values.yaml. The vhost must be the same vhost the service consumes from — the hook reads the vhost out of the service’s own AMQP URI and fails the release on a mismatch.
# Durable-queue bounds policy, applied by a post-install/post-upgrade hook Job
policyHook:
managementApi:
# Management API base URL: scheme, host and port, no trailing path.
baseUrl: http://rabbitmq:15672
# Must match the vhost in the AMQP URI in event-services-env.
vhost: notifications
existingSecret:
name: rabbitmq-mgmt-credentials
Warning
The policy includes an idle-queue TTL (``expires``) of 24 hours by default, which changes durable-queue behavior for clients: a durable queue with no attached consumer for that long is deleted by the broker, and any messages it still holds are discarded rather than dead-lettered. A returning client gets NOT_FOUND (gRPC) or 404 (REST) and must recreate the queue, so clients must persist both the queue_id and its filter_groups. See Event Consumer Dead-Letter Capture for the full lifecycle, the other policy bounds and their defaults, and how to retune or disable the idle window.
Note
Managing policies yourself. Set policyHook.enabled: false if you run a restricted broker whose policies are managed outside this chart. The three values above are then not required, and the chart applies nothing. On a broker with no equivalent policy, durable queues then carry no maximum length, message TTL or idle-queue TTL — only RabbitMQ 4.1’s built-in redelivery-limit default of 20 bounds anything — so apply an equivalent policy by hand before running this way. Disabling the hook on a broker where the policy had already been applied does not remove it: the last-applied event-queue-limits policy stays live and keeps enforcing at whatever values it last carried, until you delete it yourself.
Install the Consumer Service#
Now you can install the Consumer Service using the helm install command.
# install the chart
helm upgrade --install event-consumer-service -n storage-apis -f event-consumer-values.yaml .
Validate the Consumer Service Deployment#
You can validate the Consumer Service deployment by checking the pods are running and the service is accessible.
# check the pods are running
kubectl get pods -n storage-apis
# check the service is accessible
kubectl get service -n storage-apis
# return to the deployment directory
cd ..
Note
If you use the Event Consumer Service’s durable queues, see Event Consumer Dead-Letter Capture for the broker prerequisite (RabbitMQ 4.x / quorum queues), the dead-letter topology the service declares, the bounds policy the chart applies (including the idle-queue TTL that reaps unattended durable queues), the create-time-only dead-letter queue settings, and migration of existing durable queues.
For scaling, broker sizing, shutdown, reconnection and deployment validation, see Event Consumer Operations.
Authentication and Permissions#
Configure identity (authentication) and authorization (permissions) for the Notifications Services — the Event Consumer Service and the Event Aggregation Service.
This section covers two orthogonal first-class controls and one consumer-only storage gate:
principalSource— selects how the caller’s identity (the principal) is derived from the incoming request, and whether a token is required at all. This is the single authentication gate for both services.permissions.enabled— the authorization gate for the permission service path. Independent of authentication: when off, all permission checks short-circuit to “allowed”.permissions.storage_permissions.enabled— a consumer-only authorization gate for the storage-service path. The Event Aggregation Service has no equivalent.
Note
Authentication and authorization are independent. You can derive a principal (principalSource set to a non-disabled mode) without enforcing any permissions, and you must derive a principal before you can enforce permissions. Enabling either permission gate while principalSource is disabled is an invalid configuration — there is no authenticated principal to authorize.
These keys live in your event-consumer-values.yaml and event-aggregation-values.yaml files (created in the deployment steps above). See Authentication for setting up the OIDC identity provider and Envoy auth extension.
Selecting the principal source (authentication)#
principalSource is a single first-class setting that selects how the principal (caller identity) is derived and whether a token is required.
Both services accept the same five modes. The setting maps to the OMNI_EVENTS_PRINCIPAL_SOURCE environment variable and defaults to jwtUnverified.
Mode |
Token required? |
Behavior |
Required configuration |
|---|---|---|---|
|
No |
Authentication is off. No principal is derived and no token is required. |
|
|
Yes |
Decode the JWT to derive the principal without verifying its signature. |
None beyond a token on the request. See the trust-boundary note below. |
|
Yes |
Validate the token as a JWT using public keys pulled from the identity provider. |
|
|
Yes |
Validate the token against the IdP |
|
|
Yes |
Resolve the principal via an external user-info service over REST. |
|
For both notification services, every non-disabled mode must resolve a nonempty
identity. In jwt and jwtUnverified modes, openId.principalIdClaim
(default sub) selects the token claim. In opaque mode it selects a field
in the identity provider’s userinfo response. In userInfoService mode the
response must contain a nonempty id. Missing or empty identities are rejected.
This applies to first installations as well as upgrades.
Every non-disabled mode requires a token on event API requests. Health,
capacity and API-documentation endpoints remain available without a bearer token
on the Event Consumer Service.
Warning
In jwtUnverified mode the service decodes the JWT to read the principal but does not verify its signature, audience or expiry. This is safe only when an upstream component (typically the Envoy auth extension described in Authentication) has already validated the token before it reaches the service. jwtUnverified trusts that upstream validation. Do not expose a service running in jwtUnverified mode directly to untrusted clients without that upstream validation in place — use jwt or opaque if the service must validate tokens itself.
disabled — no authentication#
# event-consumer-values.yaml or event-aggregation-values.yaml
principalSource: disabled
permissions:
enabled: false # required: no principal exists to authorize
jwtUnverified — decode without verifying (default)#
principalSource: jwtUnverified
Use this mode behind an Envoy auth extension that has already validated the token. See the trust-boundary warning above.
jwt — verify via identity-provider keys#
principalSource: jwt
openId:
openIdConfigurationUri: "https://login.microsoftonline.com/{tenant-id}/v2.0/.well-known/openid-configuration"
clientRegistrations:
- name: "default"
clientId: "{client-id}"
scope: "openid profile email offline_access {client-id}/.default"
# Event Consumer only: additional accepted JWT audiences.
# The Aggregation chart uses only the default registration's clientId.
# additionalJwtAudience:
# - "{api-audience}"
For Event Aggregation Service, set the clientId of the registration named
default to the expected token aud value. The current Aggregation chart
renders only that value as its JWT audience; openId.additionalJwtAudience
and other registrations do not add accepted audiences. Event Consumer Service
also accepts the values in openId.additionalJwtAudience.
opaque — validate via the IdP userinfo endpoint#
principalSource: opaque
openId:
openIdConfigurationUri: "https://login.microsoftonline.com/{tenant-id}/v2.0/.well-known/openid-configuration"
userInfoService — resolve via an external user-info service#
principalSource: userInfoService
userInfoService:
uri: "http://microsoft-graph-userinfo-service:8080"
Putting it together#
A common production configuration for the Event Consumer Service, behind a validated Envoy auth extension, enforcing permissions on both durable and non-durable storage events:
# event-consumer-values.yaml
principalSource: jwtUnverified
permissions:
enabled: true
endpoint: "http://permission-service:3000"
storage_permissions:
enabled: true
endpoints:
- "http://storage-service:8012"
The equivalent Event Aggregation Service configuration has no storage gate:
# event-aggregation-values.yaml
principalSource: jwtUnverified
permissions:
enabled: true
endpoint: "http://permission-service:3000"
Configure Notifications on the Storage Service#
Issue client credentials for the Storage Service application so it can publish events through the Event Aggregation Service. Make the resulting NOTIFICATION_SERVICE_CLIENT_SECRET available to the Storage Service container through a Kubernetes secret and extraEnvs.
Create the Kubernetes secret:
kubectl create secret generic storage-notifications-secret \
--from-literal=NOTIFICATION_SERVICE_CLIENT_SECRET="<client-secret>" \
--namespace storage-apis
Add the environment variable to your storage-values.yaml file:
extraEnvs:
- name: NOTIFICATION_SERVICE_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: storage-notifications-secret
key: NOTIFICATION_SERVICE_CLIENT_SECRET
Enable the notification client and set its endpoint to the Event Aggregation Service:
config:
notificationClient:
enabled: true
endpointUrl: "http://<event-aggregation-svc-name>.<event-aggregation-svc-namespace>.svc.cluster.local:50051"
secure: false
Replace <event-aggregation-svc-name> and <event-aggregation-svc-namespace> with the values for your Notification Service deployment.
Redeploy the Storage Service#
Update storage-service/storage-values.yaml with the settings above. From
the deployment directory, upgrade the Storage Service:
# install the chart
helm upgrade --install storage-service ./storage-service -n storage-apis \
-f ./storage-service/storage-values.yaml
Validate the Storage Service Deployment#
You can validate the Storage Service deployment by checking the pods are running and the service is accessible.
# check the pods are running
kubectl get pods -n storage-apis
# check the service is accessible
kubectl get service -n storage-apis
Testing the Notifications Services#
Now you can test the Notifications Services by subscribing to events and then triggering a storage change. This uses a non-durable subscription which is ideal for testing - it receives events in real-time while you’re connected.
You’ll need two terminal sessions: one to listen for events and one to trigger storage changes.
Note
This walkthrough is tokenless: the curl commands below send no
Authorization header. They succeed only if the Event Consumer and Event
Aggregation services were deployed with principalSource: disabled (see the
configuration steps above). Under the default jwtUnverified mode the stream
request is rejected with an authentication error. To test with authentication
enabled, see the Authentication and Permissions section above.
Note
The curl commands below use cluster-internal DNS names (e.g., event-consumer-service.storage-apis.svc.cluster.local), which are only accessible from within the Kubernetes cluster. To test from your local machine, use kubectl port-forward to make the services accessible on localhost. Each port-forward needs its own terminal session:
# port-forward to one event consumer pod
kubectl port-forward -n storage-apis deployment/event-consumer-service 8000:8000
# port-forward the storage service (if not already forwarded from earlier testing)
kubectl port-forward -n storage-apis service/storage-service 8012:8012
When using port-forwarding, replace the cluster-internal hostnames in the curl commands below with localhost, e.g.:
http://event-consumer-service.storage-apis.svc.cluster.local:8000/...becomeshttp://localhost:8000/...http://storage-service.storage-apis.svc.cluster.local:8012/...becomeshttp://localhost:8012/...
Subscribe to Events (Terminal 1)#
First, set up a subscription to watch all storage events at the root level. This uses a starts_with_greedy filter with an empty resource_id to match all resources.
# Define filters for all storage create and delete events
FILTERS='[
{"event_type":"omni.storage.created","filters":[{"filter_type":"starts_with_greedy","resource_id":""}]},
{"event_type":"omni.storage.deleted","filters":[{"filter_type":"starts_with_greedy","resource_id":""}]}
]'
# Start listening for events (this will stream events as they occur)
curl -N -G "http://event-consumer-service.storage-apis.svc.cluster.local:8000/api/v1beta/events/stream" \
-H "Accept: text/event-stream" \
--data-urlencode "filter_groups=$FILTERS"
Note
The -N flag disables buffering so events appear immediately. This command will run continuously, streaming events as they occur. Press Ctrl+C to stop.
Trigger a Storage Event (Terminal 2)#
In a second terminal, create a file in your backend storage to trigger an event. Replace <YOUR BUCKET NAME> and <YOUR REGION> with your bucket name and region. (If you are not using AWS S3, you’ll need to modify the URL to match your backend storage.)
# Create a file to trigger an omni.storage.created event
export BUCKET_NAME="<YOUR-BUCKET-NAME>"
export REGION="<YOUR-REGION>"
curl -X PUT "http://storage-service.storage-apis.svc.cluster.local:8012/v1beta/fileobject/by-address/https%3A%2F%2F${BUCKET_NAME}.s3.${REGION}.amazonaws.com%2Fhello.txt?data_object_size=22" \
-H "Content-Type: application/octet-stream" \
--data "Hello from Storage API!"
Verify Event Reception#
Back in Terminal 1, you should see an event appear similar to:
event: connected
id: 123e4567-e89b-42d3-a456-426614174000
data: {"message":"Connection established","reconnect_token":"123e4567-e89b-42d3-a456-426614174000"}
event: event
data: {"event_type":"omni.storage.created","principal_identity":"...","occurred_at":"...","published_at":"...","message":{"file_name":"hello.txt"}}
If you see the event, your Notifications Services are working correctly. You can also test delete events by removing the file through the Storage Service.
Smoke Tests#
Both the Event Aggregation and Event Consumer services include smoke tests that verify their endpoints after deployment.
Event Aggregation Service — 4 tests (2 REST + 2 gRPC) that verify publishing single and batch events.
Event Consumer Service — 6 tests (3 REST + 3 gRPC) that verify durable queue lifecycle (create, update, delete) and non-durable stream reconnect token.
When tests.enabled=true, the tests run automatically on every helm install and helm upgrade, and can also be triggered on demand with helm test.
First configure any required test authentication as described below. Then, from the deployment directory, enable and run smoke tests for each service:
# Event Aggregation
helm upgrade --install event-aggregation-service ./event-aggregation-service \
-f ./event-aggregation-service/event-aggregation-values.yaml \
--namespace storage-apis --set tests.enabled=true
helm test event-aggregation-service -n storage-apis
# Event Consumer
helm upgrade --install event-consumer-service ./event-consumer-service \
-f ./event-consumer-service/event-consumer-values.yaml \
--namespace storage-apis --set tests.enabled=true
helm test event-consumer-service -n storage-apis
Note
When authentication is enabled on the service, configure tests.auth in your values file so the test pod can obtain a Bearer token. The auth block reads credentials from a Kubernetes secret:
tests:
enabled: true
auth:
microsoftUrl:
valueFrom:
secretKeyRef:
name: smoke-test-auth
key: microsoft-url
credentials:
valueFrom:
secretKeyRef:
name: smoke-test-auth
key: credentials
scope:
valueFrom:
secretKeyRef:
name: smoke-test-auth
key: scope
microsoftUrl must be the OIDC discovery URL; the test reads the token
endpoint from its response. When Event Consumer permission enforcement is
enabled, grant the test principal create-durable-queues and
delete-durable-queues for event-consumer-service as well. When Event
Aggregation permission enforcement is enabled, grant publish-event on
both EventType resources omni.storage.created and test.batch.publish.
If you disable a service’s REST API, set tests.runRestTests: false for that
service before running smoke tests.
Create the secret before deploying:
kubectl create secret generic smoke-test-auth \
--from-literal=microsoft-url="https://login.microsoftonline.com/{tenant-id}/v2.0/.well-known/openid-configuration" \
--from-literal=credentials="{client-id}:{client-secret}" \
--from-literal=scope="{your-scope}" \
--namespace storage-apis