Scalability#

This section provides guidance on scaling the Storage Service for larger deployments. All settings below refer to the Storage Service Helm chart values.

Sizing guideline#

For a cluster of 100 GPUs, use 5 storage nodes.

Set the number of replicas with replicaCount. Monitor metrics (see Observability (Metrics, Traces, Logs)) and scale up or down as needed.

How the Storage Service scales#

The Storage Service can be scaled by running multiple replicas. It maintains internal caches to improve performance of storage operations (e.g. metadata and listing). Whether scaling is appropriate depends on cache behavior and your client workloads:

  • Shared L2 cache (1.2.0+) — The internal caches can optionally be backed by a shared Redis or Valkey tier. Enabling L2 caching facilitates cache reuse for operations performed by different replicas and enables distributed cache invalidation across replicas, regardless of whether storage event subscribers are configured. Enable config.redis.enabled and point config.redis.address at your Redis/Valkey instance, then opt in per cache with config.smallObjectCache.l2Enabled, config.statCache.l2Enabled, and config.listCache.l2Enabled. This also reduces backend calls and cache warm-up after scaling events.

  • Without bucket notifications — Cache entries are invalidated after a time-to-live (TTL). You can scale by increasing replicaCount. Without a shared L2 cache, each pod has its own cache. TTL is configured per cache in the Helm values: config.smallObjectCache.timeToLive, config.statCache.timeToLive, and config.listCache.timeToLive (and config.listCache.enabled if you use the list cache).

    • Stale reads are possible: clients may see slightly outdated data for non-version-specific (e.g. “latest”) objects until the TTL expires. If your client workloads can tolerate that—i.e. they read “latest” and do not require immediate consistency—scaling with caches and TTL is fine. To reduce staleness, lower the TTL values or disable caches (config.smallObjectCache.enabled, config.statCache.enabled, config.listCache.enabled) at the cost of more backend calls.

  • With bucket notifications enabled — Bucket notifications are enabled when a subscriber is configured under config.storageEvents (e.g. config.storageEvents.<subscriber-name>.sqs or config.storageEvents.<subscriber-name>.azureServiceBus). The service then invalidates caches when it receives storage events (e.g. object created or deleted) from the configured queue; config.statCache.invalidateOnUpdate and config.listCache.invalidateOnUpdate control whether those caches are invalidated on writes and on notification events. When multiple Storage Service replicas consume from the same queue, they compete for messages, so an event may be received by only one replica. With L2 caching enabled, the resulting invalidation is distributed across replicas.

Summary#

  • Use 5 storage nodes for 100 GPUs as a starting point; set replicaCount (and related replica values) and tune from metrics.

  • The shared Redis/Valkey L2 cache tier (config.redis.*) is recommended when the Storage Service runs with multiple replicas. It enables cache reuse and distributed invalidation across replicas, with or without storage event subscribers.

  • Configure storage events when the Storage Service is not the only application writing to the downstream storage backend. This ensures the service observes external changes and invalidates affected cache entries.