Storage Service Configuration#

Goal: Configure and deploy the Storage Service (S3 and/or Azure Blob Storage) using the Helm chart from NGC.

Choose your Storage Backend#

The Storage Service supports S3 and Azure Blob Storage backends (public or private; private storage requires credentials available to the deployment). S3-compatible storage (e.g. MinIO) is also supported via custom endpoints.

Before deploying, set up at least one backend: S3, Azure Blob Storage, or both.

S3 Storage#

If you choose AWS S3, follow the AWS documentation for setting up your account and creating your S3 bucket. Create credentials to control access to your bucket. See CORS Storage Configuration for required permissions and CORS.

Note

You can jump to S3 Configuration if you are not intending to use Azure Blob Storage.

Azure Blob Storage#

If you choose Azure Blob Storage, follow the Azure documentation for setting up your account and creating your Azure Blob Storage container. Create credentials to control access to your container. See Storage Service Bucket Configuration for required permissions and CORS. Then jump to Azure Blob Storage Configuration.

Storage Service Bucket Configuration#

This section describes the minimal permissions and settings required for the Storage Service to operate with AWS S3 buckets and Azure Blob Storage containers.

Points to pay attention to:

  • Permissions — Grant the minimal IAM policy (AWS) or RBAC role (Azure) required for list, read, write, delete, and multipart operations; scope to the specific bucket or container.

  • CORS — If browser-based clients or presigned URLs are used, configure CORS on the S3 bucket or Azure Blob service so allowed origins can make requests.

  • Authentication — Use explicit IAM credential injection via Kubernetes secrets for AWS; use managed identities for Azure. Avoid long-lived keys where possible. If you enable optional storage notifications, the same identity can be used: grant that role both bucket and (where applicable) SQS or Service Bus permissions so one policy set covers storage and notifications—you do not have to use separate access keys or a second account.

  • Versioning — If you use versioned buckets or versioned APIs, include version-related permissions and enable versioning on the bucket/account.

  • Security — Use least privilege, restrict origins in CORS (avoid * with credentials), rotate credentials, and enable server-side encryption and access logging.

AWS S3 Bucket Permissions#

The Storage Service requires the following minimal IAM policy to be attached to the bucket or the IAM user/role accessing the bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketLevelPermissions",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:ListBucketVersions"
      ],
      "Resource": "arn:aws:s3:::BUCKET_NAME_HERE"
    },
    {
      "Sid": "ObjectLevelPermissions",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
    }
  ]
}

Bucket-Level Permissions (applied to the bucket ARN):

  • s3:ListBucket — Required for listing objects and folders within the bucket

  • s3:ListBucketVersions — Required when versioning is enabled to list object versions

Object-Level Permissions (applied to objects within the bucket using /*):

  • s3:GetObject — Read object data

  • s3:GetObjectVersion — Read specific object versions (when versioning is enabled)

  • s3:PutObject — Write/upload objects

  • s3:DeleteObject — Delete objects

  • s3:DeleteObjectVersion — Delete specific object versions (when versioning is enabled)

  • s3:AbortMultipartUpload — Cancel incomplete multipart uploads

  • s3:ListMultipartUploadParts — List parts of a multipart upload in progress

Note

The versioning-related permissions (s3:ListBucketVersions, s3:GetObjectVersion, s3:DeleteObjectVersion) are optional if you do not use versioned buckets or versioned APIs (e.g. EnumerateVersions). You can omit them if versioning is not required.

Replace BUCKET_NAME_HERE with your actual S3 bucket name in both Resource ARNs.

Azure Blob Storage Container Permissions#

For Azure Blob Storage, the Storage Service needs the following permissions. Grant them via:

  1. Built-in RBAC role: Storage Blob Data Contributor (recommended), or

  2. Custom RBAC role with the specific permissions listed below.

Option 1: Built-in Role (Recommended)

Assign the Storage Blob Data Contributor role to the identity (service principal, managed identity, or user) used by the Storage Service:

az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee <service-principal-id> \
  --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container-name>

Option 2: Custom Role with Minimal Permissions

{
  "Name": "Storage Service Blob Access",
  "Description": "Minimal permissions required for Storage Service to operate on Azure Blob Storage",
  "Actions": [
    "Microsoft.Storage/storageAccounts/blobServices/containers/read",
    "Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action"
  ],
  "NotActions": [],
  "DataActions": [
    "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
    "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write",
    "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete",
    "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action",
    "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action",
    "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/read",
    "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/write"
  ],
  "NotDataActions": [],
  "AssignableScopes": [
    "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>"
  ]
}

Data Actions (applied to blob data): read, write, delete, add, move, and tags read/write as needed. Configure: create the custom role (if using Option 2), assign it to your service principal or managed identity at the storage account or container scope, and ensure the storage account allows Azure AD authentication.

Authentication Methods#

  • AWS S3: IAM user with access keys; IAM role (EC2 instance profile or assume role); or temporary credentials via STS.

  • Azure Blob Storage: Service principal, managed identity (recommended for Azure resources), container/account SAS (not recommended for production), or storage account keys (not recommended).

Recommended: Use managed identities for Azure and IAM roles with explicit credential injection for AWS EKS. If you enable optional storage notifications, grant the same role or identity the required SQS (AWS) or Service Bus (Azure) permissions so one policy set covers both bucket access and notifications.

Versioning Support#

  • AWS S3: Enable versioning on the bucket; the policy above includes version-specific actions.

  • Azure Blob Storage: Enable versioning on the storage account; the standard blob permissions cover versioned blobs.

Testing Permissions#

AWS S3:

aws s3 ls s3://BUCKET_NAME_HERE/
aws s3 cp s3://BUCKET_NAME_HERE/test-file.txt -
echo "test" | aws s3 cp - s3://BUCKET_NAME_HERE/test-file.txt
aws s3 rm s3://BUCKET_NAME_HERE/test-file.txt

Azure Blob Storage:

az storage blob list --account-name <storage-account> --container-name <container> --auth-mode login
az storage blob upload --account-name <storage-account> --container-name <container> --name test-file.txt --file test-file.txt --auth-mode login
az storage blob download --account-name <storage-account> --container-name <container> --name test-file.txt --file downloaded.txt --auth-mode login
az storage blob delete --account-name <storage-account> --container-name <container> --name test-file.txt --auth-mode login

Security Best Practices#

  1. Least privilege — Grant only the permissions the Storage Service needs.

  2. Scope — AWS: limit IAM policies to specific bucket ARNs; Azure: assign at container level when possible.

  3. Credential rotation — Rotate access keys and service principal secrets regularly.

  4. Managed identities — Prefer managed identities (Azure) or IAM roles (AWS) over static credentials.

  5. Versioning — Enable versioning on buckets/containers to protect against accidental deletion.

  6. Monitoring — Use CloudTrail (AWS) or Azure Monitor to track access.

  7. Encryption — Enable server-side encryption on the bucket/container.

Service Ports#

Port

Protocol

Description

8012

HTTP/REST

REST API endpoint

8011

gRPC

gRPC API endpoint

Pulling the Storage Service Helm Chart#

Pull and unpack the Storage Service Helm chart from the NGC catalog. Replace {NGC_API_KEY} with your NGC API key.

# pull the chart from NGC
helm fetch https://helm.ngc.nvidia.com/nvidia/omniverse/charts/storage-service-1.0.2.tgz --username='$oauthtoken' --password={NGC_API_KEY}

# unpack the chart and cd into the directory
tar -xvf storage-service-1.0.2.tgz
cd storage-service

Configure values for the Storage Service#

With your storage backend set up and the Helm chart pulled, configure values so the service uses your backend. This guide uses storage-values.yaml as the values file.

# create and edit the local values file, we're using VS Code for this example
code storage-values.yaml

Add the image-pull secret to your values file. If you use a different secret name, replace ngcpull-secret.

image:
  pullSecrets:
    - name: ngcpull-secret

storage-values.yaml

Note

If you have chosen to use publicly accessible storage or IP Access Control Lists, you can skip the next section and move on to installing the helm chart.

Private S3 Bucket Configuration#

If you have chosen to use private S3 storage, you will need to configure the Storage Service to use your S3 credentials. Credentials are set on a per-bucket basis, and we currently support up to 10 private buckets configured on the Storage Service.

To keep the credentials secure, we will create another Kubernetes secret, so the storage service can access your S3 credentials securely. For the command below, replace {mybucketname} with the name of your S3 bucket you created, {BUCKET_ACCESS_KEY_ID} and {BUCKET_SECRET_ACCESS_KEY} with the access keys you generated for your principal to access your S3 bucket. If you used a different namespace, replace storage-apis with your namespace.

Note

We are avoiding using the usual AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY names for the keys in the secret we create, as these may be unknowingly set in the cluster, and we want to avoid confusion.

kubectl create secret generic {mybucketname}-bucket-secret \
  --from-literal=BUCKET_ACCESS_KEY_ID={BUCKET_ACCESS_KEY_ID} \
  --from-literal=BUCKET_SECRET_ACCESS_KEY={BUCKET_SECRET_ACCESS_KEY} \
  --namespace storage-apis

# validate the secret was created
kubectl get secret {mybucketname}-bucket-secret -n storage-apis

Now we will update our values file so it is configured to access the new bucket.

config:
  storage:
    s3:
      endpoints:
        "{mybucketname}.s3.{region}.amazonaws.com": # {mybucketname} is the name of your S3 bucket you created, {region} is the region of your S3 bucket
          credentials:
            accessKey:
              accessKeyId: "${BUCKET_ACCESS_KEY_ID}" # This is an environment variable that will be set at the time of deployment.
              secretAccessKey: "${BUCKET_SECRET_ACCESS_KEY}" # This is an environment variable that will be set at the time of deployment.

storage-values.yaml

In our configuration, we will be setting the environment variables ${BUCKET_ACCESS_KEY_ID} and ${BUCKET_SECRET_ACCESS_KEY} at the time of deployment, so we need to make sure we reference the variables correctly in our values file. To avoid confusion, we are setting environment variables with the same name as the keys in the secret we created.

extraEnvs:
  - name: BUCKET_ACCESS_KEY_ID # This is the name of the environment variable that will be set at the time of deployment.
    valueFrom:
      secretKeyRef:
        name: {mybucketname}-bucket-secret
        key: BUCKET_ACCESS_KEY_ID # This is the key in the secret with the value we will set the environment variable to.
  - name: BUCKET_SECRET_ACCESS_KEY # This is the name of the environment variable that will be set at the time of deployment.
    valueFrom:
      secretKeyRef:
        name: {mybucketname}-bucket-secret
        key: BUCKET_SECRET_ACCESS_KEY # This is the key in the secret with the value we will set the environment variable to.

storage-values.yaml

Note

If you are not using Azure Blob Storage, you can skip the next section and move on to installing the helm chart.

Azure Private Blob Storage Configuration#

If you have chosen to use Azure Blob Storage, you will need to configure the Storage Service to use your Azure Blob Storage credentials. Credentials are set on a per-account basis, and we currently support up to 10 private containers configured on the Storage Service.

To keep credentials secure, create a Kubernetes secret for the storage service to use. Replace {mystorageaccount} with your Azure Storage account name, {AZ_BLOB_STORAGE_KEY} with your storage account key, and (if different) storage-apis with your namespace.

Note

We are avoiding using the usual {AZURE_STORAGE_KEY} name for the key in the secret we create, as this may be unknowingly set in the cluster, and we want to avoid confusion.

kubectl create secret generic {mystorageaccount}-container-secret \
  --from-literal=AZ_BLOB_STORAGE_KEY={AZ_BLOB_STORAGE_KEY} \
  --namespace storage-apis

# validate the secret was created
kubectl get secret {mystorageaccount}-container-secret -n storage-apis

Now we will update our values file so it is configured to access the new container(s).

config:
  storage:
    azure:
      azureBlob:
        endpoints:
          {mystorageaccount}.blob.core.windows.net: # {mystorageaccount} is the name of your Azure Storage Account you created
            credentials:
              storageKey:
                storageAccount: {mystorageaccount} # This is the name of the Azure Storage Account you created
                storageKey: "${AZ_BLOB_STORAGE_KEY}" # This is an environment variable that will be set at the time of deployment.

storage-values.yaml

In our configuration, we will be setting the environment variable ${AZ_BLOB_STORAGE_KEY} at the time of deployment, so we need to make sure we reference the variable correctly in our values file. To avoid confusion, we are setting environment variables with the same name as the keys in the secret we created.

extraEnvs:
  - name: AZ_BLOB_STORAGE_KEY # This is the name of the environment variable that will be set at the time of deployment.
    valueFrom:
      secretKeyRef:
        name: {mystorageaccount}-container-secret
        key: AZ_BLOB_STORAGE_KEY # This is the key in the secret with the value we will set the environment variable to.

storage-values.yaml

Install the Storage Service#

Now that you’ve configured the Storage Service, let’s validate the chart is good and install the Storage Service.

# validate the chart
helm template . -f storage-values.yaml

# dry-run the install
helm upgrade --install storage-service . -f storage-values.yaml --namespace storage-apis --dry-run --debug

If everything looks good, you can install the Storage Service.

# install the storage service
helm upgrade --install storage-service . -f storage-values.yaml --namespace storage-apis

# validate the pod is running, this may take a few minutes to start up
kubectl get pods -n storage-apis

Verify the Storage Service is responding from within the cluster.

# test the storage service
curl http://storage-service.storage-apis.svc.cluster.local:8012/v1alpha/capabilities/services

# you can validate access to a public s3 bucket
curl -v "http://storage-service.storage-apis.svc.cluster.local:8012/v1alpha/filefolder/list/https%3A%2F%2Fomniverse-content-production.s3.us-west-2.amazonaws.com%2F"

Note

You may find it easier to use port-forwarding to test the Storage Service is responding to requests from within the cluster.

# set up port forwarding
kubectl port-forward -n storage-apis service/storage-service 8012:8012
# test the storage service
curl http://localhost:8012/v1alpha/capabilities/services
# you can validate access to a public s3 bucket
curl -v "http://localhost:8012/v1alpha/filefolder/list/https%3A%2F%2Fomniverse-content-production.s3.us-west-2.amazonaws.com%2F"

Clean up#

To clean up: uninstall the Helm release, delete the secrets, and remove the namespace.

# uninstall the storage service
helm uninstall storage-service -n storage-apis

# delete the secrets
kubectl delete secret {mybucketname}-bucket-secret -n storage-apis
kubectl delete secret {mystorageaccount}-container-secret -n storage-apis
# validate the secrets were deleted
kubectl get secrets -n storage-apis

# delete the namespace
kubectl delete namespace storage-apis

# validate the namespace was deleted
kubectl get namespace storage-apis

Smoke Tests#

The Storage Service Helm chart includes 30 smoke tests (15 REST + 15 gRPC) that verify the endpoints are working correctly after deployment. Tests cover capabilities, file object operations, versioning, folder listing, and metadata.

When tests.enabled=true, the tests run automatically on every helm install and helm upgrade, and can also be triggered on demand with helm test.

To enable smoke tests, set tests.enabled=true and provide a resource base (a path in your storage bucket where test files will be created and deleted):

helm upgrade --install storage-service . -f storage-values.yaml \
  --set tests.enabled=true \
  --set tests.storageApiResourceBase="https://your-bucket.s3.amazonaws.com/test/"

To run on demand:

helm test storage-service -n storage-apis

Note

If the Storage Service has permissions enabled, the test identity must be allowed to read, write, and delete objects at storageApiResourceBase. If authentication is enabled, configure tests.auth in your values file so the test pod can obtain a Bearer token. The auth block reads credentials from a Kubernetes secret:

tests:
  enabled: true
  auth:
    microsoftUrl:
      valueFrom:
        secretKeyRef:
          name: smoke-test-auth
          key: microsoft-url
    credentials:
      valueFrom:
        secretKeyRef:
          name: smoke-test-auth
          key: credentials
    scope:
      valueFrom:
        secretKeyRef:
          name: smoke-test-auth
          key: scope

Create the secret before deploying:

kubectl create secret generic smoke-test-auth \
  --from-literal=microsoft-url="https://login.microsoftonline.com/{tenant-id}/oauth2/v2.0/token" \
  --from-literal=credentials="{client-id}:{client-secret}" \
  --from-literal=scope="{your-scope}" \
  --namespace storage-apis

storage-values.yaml

What’s Next?#

You can continue your journey by configuring and deploying the Notifications Services. This is an optional service that you can deploy and leverage with the Storage Service.

You can skip using Notifications and continue your journey by deploying the Discovery Service. This service will help clients find and connect to all the services in your deployment and allow you to continue to build up your deployment with more services, and will not require any additional client configuration.

Below, we have some additional information about the Storage Service that you may find useful in the future.

Future Considerations#

Request Rate Limiting#

Depending on your storage provider, you may have a limit on how many requests you can make per second. Leveraging the USD Content Cache Service can help mitigate this situation.

URL Remapper#

URL remapper: Rewrite URLs so you can expose virtual URLs (e.g. a custom scheme) that map to your storage backend. See the full helm chart for this and other configuration options (e.g. in-service caches, redirect sizing).