Known Issues and Troubleshooting#

This page covers known product limitations and common deployment problems. For component-specific troubleshooting, see the corresponding deployment and operations guides.

Known Issues#

Example Filesystem Service: data persistence depends on the startup command#

Applies when: The Example Filesystem Service starts without the filesystem subcommand.

Impact: The service can ignore FILESERVICE_STATIC_DIR and write files to temporary storage. Those files can be lost when the container or pod restarts, even if persistent storage is mounted.

Workaround: Pass the backend subcommand and storage directory explicitly:

python -m local_filesystem_service filesystem --static-dir /data/storage

Ensure /data/storage is mounted to persistent storage. The current deployment examples already specify the filesystem subcommand.

Stale content after writes outside the Storage API#

Applies when: Content changes directly in the backing store while Storage Service caching is enabled.

Impact: Cached content or directory listings can become out of date. Without storage event notifications, the Storage Service may continue returning cached results until the entries expire.

Remediation: Configure notifications for the backing store and register the subscriber under config.storageEvents. Enable the relevant cache invalidation settings so incoming events invalidate affected entries.

Multi-replica stale reads with caching enabled#

Applies when: Storage Service replicas maintain separate caches.

Impact: A change observed by one replica may not invalidate another replica’s cache. Clients reading the latest version of an object can receive stale results until the cache expires.

Multiple replicas consuming the same storage event queue compete for messages, so notifications alone do not ensure every replica receives each event.

Remediation: Enable the shared Redis or Valkey cache tier using config.redis.enabled and config.redis.address, then enable l2Enabled for the relevant caches.

If a shared cache is unavailable, reduce cache TTLs, disable affected caches, or use explicit version identifiers when reading immutable content.

Live Edit: read-only participation is not supported#

Affected version: Live Edit Service 0.1.1.

Limitation: Read and modifying operations both use the Storage Service’s read/stat authorization check.

Impact: A user who can read/stat the base asset can also modify its live sessions and layers.

Mitigation: Restrict access to participating users who are permitted to edit. Do not rely on Storage read-versus-write permissions to enforce a read-only Live Edit role.

Live Edit: clients can retain stale content after recovery#

Applies when: Clients reconnect after Live Edit state is restored from a checkpoint.

Impact: Clients can retain content that is absent from the recovered state, including prims created after the backup.

Workaround: Before resuming editing, reload the recovered live layer from an empty client state rather than resuming cached state. Keep layer.catchup_snapshot_enabled: true and verify the client reload procedure during recovery testing.

Checkpoints run asynchronously. The default 60-second checkpoint interval does not guarantee a maximum of 60 seconds of data loss; edits newer than the latest recoverable backup can be lost.

Version Compatibility#

The table below lists the tested and supported version combinations of Kit, the Storage APIs, and Content Cache. The cache column refers to Helm chart generations: legacy usd-content-cache 3.x (UCC) and ovcontentcache 4.x (OVCC), not container versions. For cache documentation and Helm chart downloads, see Omniverse Content Cache (OVCC).

Kit

Storage APIs

Content Cache chart

109.x

1.0.x, 1.1.x, 1.2.x

N/A

109.x

N/A

3.x, 4.x

110.1

1.0.x, 1.1.x, 1.2.x

3.x (WAR), 4.x

110.2

1.0.x, 1.1.x, 1.2.x

3.x, 4.x

N/A indicates the component is not applicable to that combination. WAR indicates a workaround is required — see Legacy UCC chart 3.x with Kit 110.1 below.

Legacy UCC chart 3.x with Kit 110.1: in-cluster HTTP requests fail#

Affected combination: Kit 110.1 with legacy UCC chart 3.x.

Impact: Requests to local Kubernetes services can fail because Content Cache upgrades HTTP requests to HTTPS. Services reached through .svc.cluster.local addresses may not have TLS configured.

Workaround: Set the following environment variable on the Kit instance:

NO_PROXY=".svc.cluster.local"

Requests to these services bypass Content Cache and retain their original protocol. This workaround is not required with OVCC chart 4.x (ovcontentcache). The OVCC container tag remains in the 3.x series; it is not the chart generation.

Common Deployment Problems#

Problem

Troubleshooting

Connection refused or port already in use

Check for an existing container, port-forward session, or service using the requested port. Stop the conflicting process or select a different local port.

MicroK8s certificate mismatch

Check whether the address in the error is covered by the certificate. Follow the deployment guide’s certificate recovery procedure.

400 or 404 when addressing a file

Verify the resource address and URL encoding. For example, file-storage://fileservice/hello.txt becomes file-storage%3A%2F%2Ffileservice%2Fhello.txt when encoded as a path parameter.

Discovery returns no endpoints

Check service readiness, Discovery registration or configuration, and Discovery logs.

Port-forward stops working

Restart the port-forward after a pod restart or connection interruption.