Integration Tests Configuration#

Goal: Configure the integration tests to validate discovery, storage, permissions, and notifications (event aggregation, event consumer) services.

Brief description

Integration tests verify interactions among the following services after deployment:

  • Storage service and Permission service

  • Storage service and Notification service

If a service required for one of these interactions is not deployed according to the discovery service configuration, tests for that interaction are skipped.

The run starts with a preflight phase: required Helm values are validated (discovery service endpoint and storage API resource base), and service endpoints returned by the discovery service must be reachable.

After preflight succeeds, the suite exercises gRPC and REST APIs of services. Assertions cover status codes and the structure and values of fields in responses.

Expected output#

Tests run in a pod. On success the pod exits with status Completed; on failure it may exit with Error. Inspect the logs for test summaries, preflight errors, and failed tests. Error logs contain hints to help identify and fix issues.

kubectl logs <pod-name> -n <namespace>

Replace <pod-name> and <namespace> with your integration-tests pod and namespace.

Note

Deploy only the services listed above you want to run tests against; you do not need to deploy all of the services.

Pulling the Integration Tests Helm Chart#

Pull and unpack the Integration Tests Helm chart from the NGC catalog. Replace {NGC_API_KEY} with your NGC API key.

# pull the chart from NGC
helm fetch https://helm.ngc.nvidia.com/nvidia/omniverse/charts/storage-api-integration-tests-1.0.3.tgz --username='$oauthtoken' --password=${NGC_API_KEY}

# unpack the chart and cd into the directory
tar -xvf storage-api-integration-tests-1.0.3.tgz
cd storage-api-integration-tests

Configure the Integration Tests#

Use integration-tests-values.yaml as the values file (same as previous examples).

# create and edit the local values file, we're using VS Code for this example
code integration-tests-values.yaml

Minimal required values to run the integration tests:

Set the image and storage API resource base (Bucket URL for S3 or Azure Blob container) you want to test against.

image:
  repository: "nvcr.io/nvidia/omniverse/integration-tests"
  pullSecrets:
    - name: ngcpull-secret
job:
  storageApiResourceBase: "https://storage.example.com/bucket"
  discoveryServiceEndpoint: "http://discovery-service:8080"

Permissions and system policies#

The integration tests provision the permission policies they need at run time and remove them when they finish — you do not create per-resource permits by hand. Each test class that exercises the Permission service creates its policies before its tests run and deletes them afterward:

  • Storage and Permission tests create a storage-service read and write permit on the storage API resource base (the storageApiResourceBase you configured), scoped to that folder.

  • Storage and Notification tests additionally create Notification-service permits: on the event consumer service — create-durable-queues, consume-durable-queues, delete-durable-queues, consume-all-storage-create-events, and consume-all-storage-delete-events; and on the event aggregation service — publish-event for the omni.storage.created event type.

For the tests to create and delete these policies, the identity the test job authenticates as (the principal in its access token) must itself be authorized to manage policies in the Permission service. Grant that principal the permissions:edit system policy in the Permission service before running the tests.

To grant permissions:edit, create a policy in the Permission service that authorizes the principal for that action. The policy carries the Cedar rule as a string — for example, to allow a specific principal:

{
  "policy": "permit(principal == Principal::\"<your-test-principal>\", action == Action::\"permissions:edit\", resource);"
}

Note

If the principal is missing permissions:edit, the Storage-and-Permission and Storage-and-Notification tests fail during setup with a “could not create permit policies” error (the Permission API returns a non-200 to the policy-creation call).

Additional Values reference#

Brief description of the optional values you can set:

Service endpoints (choose one)

  • Option 1: Discovery (default): Set discoveryServiceEndpoint (e.g. http://discovery-service:8080). Endpoints are fetched from Discovery; you do not need to set manual endpoints.

  • Option 2: Manual: Leave discoveryServiceEndpoint empty or omit it, and set manualEndpoints for each service you test:

    • grpcStorage / httpStorage — Storage service.

    • grpcPermission / httpPermission — Permission service.

    • grpcNotificationConsumer / httpNotificationConsumer — Notification consumer.

    • grpcNotificationAggregation / httpNotificationAggregation — Notification aggregation.

Note

Leave a service’s fields empty or do not specify them if you are not running tests for that service.

Pytest

  • pytestMarkers — Pytest marker expression to filter tests:

    • rest: REST API tests

    • grpc: gRPC API tests

    • storage_and_permission: Storage and Permission service tests

    • storage_and_notification: Storage and Notification service tests

    • Leave empty or do not specify it to run all tests.

  • pytestTechnicalOutput — Set to true for raw pytest output (debugging); false or empty for human-readable output.

Authentication (choose one)

The tests send a bearer token with every API call. You choose how the test Job obtains it — the built-in OAuth2 client-credentials flow, or a token file written by a sidecar container you provide. Enable exactly one; leave both disabled only for services that require no authentication.

Option 1 — Client credentials. The test Job exchanges a client id and secret for an access token.

# If enabled, the test job obtains an access token via OAuth2 client credentials. Default is false.
serviceIdentity:
  clientCredentials:
    # Use client credentials flow to obtain the access token for API calls.
    enabled: true

    # OIDC discovery document URL.
    # Example: https://login.microsoftonline.com/<tenant>/v2.0/.well-known/openid-configuration
    openIdConfigurationUri: "https://login.microsoftonline.com/<tenant>/v2.0/.well-known/openid-configuration"

    # Scope passed to the token endpoint.
    clientScope: "SCOPE"

    # OAuth client_id (the client secret is injected from clientSecretRef only).
    clientId: "CLIENT-ID"

    # Existing Secret holding the client secret value (single key).
    clientSecretRef:
      name: "integration-tests-oauth-client-secret"
      key: "oauth-client-secret"

Create the Secret before installing the chart so clientSecretRef resolves. Replace {CLIENT_SECRET_VALUE} with your OAuth2 client secret. If you use a different namespace than storage-apis, pass that namespace here.

kubectl create secret generic integration-tests-oauth-client-secret \
  --from-literal=oauth-client-secret={CLIENT_SECRET_VALUE} \
  --namespace storage-apis

# validate the secret was created
kubectl get secret integration-tests-oauth-client-secret -n storage-apis

Option 2 — Access-token file (bring your own token sidecar). Run a sidecar container that writes the token to a shared file, and let the tests read it — no client secret lives in the test container. You supply the sidecar; the chart provides the hook and the shared volume.

serviceIdentity:
  accessTokenFile:
    # Read the token from a file written by your provisioner sidecar.
    enabled: true

    # Path the tests read the token from (exported as ACCESS_TOKEN_FILE_PATH).
    # Must stay under the /etc/secrets/access-token mount and match where your
    # sidecar writes the token.
    path: "/etc/secrets/access-token/token"

    # YOUR sidecar container spec, rendered verbatim into the pod — replace with
    # your own token provisioner. The chart appends a shared `token-exchange`
    # volume (mounted at /etc/secrets/access-token) to it and to the test
    # container, so both see the same token file.
    provisionerSidecar:
      container:
        name: token-provisioner
        image: {your-token-provisioner-image}
        # ... your env / args, however your provisioner obtains the token ...

Your sidecar must write the token as JSON to that path:

{ "access_token": "<jwt>", "expires_on": "<RFC3339 timestamp>" }

The tests poll the file until it holds a valid, unexpired token, then send it as the bearer token for API calls.

  • serviceIdentity.clientCredentials.enabled / serviceIdentity.accessTokenFile.enabled — enable exactly one. When accessTokenFile.enabled is true, provisionerSidecar.container.image is required.

Note

This is only needed if the services you test require authentication.

skipConnectivityChecks (optional)

  • skipConnectivityChecks — Set to true if you do not want the job to stop as soon as the connectivity check step fails, or if your gRPC service does not support health checks. The job keeps going and runs the test suite; individual tests may still be skipped or fail if the deployment is not ready.

Install the Integration Tests.

# validate the chart
helm template . -f integration-tests-values.yaml

# dry-run the install
helm upgrade --install storage-api-integration-tests . -f integration-tests-values.yaml --namespace storage-apis --dry-run --debug

If everything looks good, you can install the Integration Tests.

# install the integration tests
helm upgrade --install storage-api-integration-tests . -f integration-tests-values.yaml --namespace storage-apis

# validate the pod is running, this may take a few minutes to start up
kubectl get pods -n storage-apis

Clean up#

To clean up: uninstall the Helm release and delete the namespace.

# uninstall the integration tests
helm uninstall storage-api-integration-tests -n storage-apis

# delete the namespace
kubectl delete namespace storage-apis

# validate the namespace was deleted
kubectl get namespace storage-apis