First Response: Triage#

Start here when you do not yet know which component is at fault. This runbook narrows a vague report (“storage is broken”, “nothing loads”) to one of the specific runbooks.

Symptom#

Users report failures across one or more Omniverse workflows, and it is not yet clear whether the problem is a single adapter, the infrastructure underneath it, or the network in front of it.

Blast radius#

Unknown at this point — that is what this runbook establishes. Do not restart anything before the triage steps below; a restart destroys the evidence (logs, in-flight state, queue position) that identifies the cause, and for several of these failures it does not help.

Triage#

1. Are the pods running, and have they been restarting?

kubectl -n <namespace> get pods -o wide
kubectl -n <namespace> get pods --sort-by=.status.containerStatuses[0].restartCount

A high restart count points at a crash loop — usually a bad configuration value, a missing secret, or a dependency the pod cannot reach at startup. kubectl describe pod shows the probe failures and the last termination reason.

2. Is it failing at the edge or inside the cluster?

Compare a request through the ingress with the same request from inside the cluster. If the in-cluster call succeeds and the external one does not, the problem is ingress, DNS or TLS — see Certificate Expiry and TLS and Loss of Connectivity: Backend and Egress.

# From outside: use the hostname routing to Storage Service's REST API
curl -sS -o /dev/null -w '%{http_code}\n' https://<storage-rest-host>/v2alpha/services

# From inside the cluster
kubectl -n <namespace> run curl-probe --rm -it --restart=Never --image=curlimages/curl -- \
    curl -sS -o /dev/null -w '%{http_code}\n' http://storage-service:8012/v2alpha/services

3. What does the service itself say?

kubectl -n <namespace> logs deployment/storage-service --since=15m | tail -100

Adapters log the underlying cause rather than swallowing it. A storage backend refusing credentials, a database connection failing, or a queue that cannot be reached all appear here named as such.

4. Is one capability degraded, or all of them?

This is the most useful question, because the adapters degrade partially by design.

What still works	Where to go next
Reads work, but changes made elsewhere are not visible	Notifications: Delayed, Missing or Duplicated
Reads work, version history or user metadata fails	Database Connectivity Loss
Everything fails with an authorization error	Secret Rotation and Credential Expiry, then Permission Service
Everything fails with a TLS or certificate error	Certificate Expiry and TLS
Clients cannot find the services at all	Discovery: Clients Cannot Find Services
Reads fail or are slow, backend errors in logs	Read Path: Failures and Slow Reads, then Loss of Connectivity: Backend and Egress

Fix#

None — this runbook only routes. Follow the runbook the table points to.

Verify#

You have finished triage when you can name the failing capability and the tier it sits in. If the symptoms do not match any row above, capture the following before escalating, because it is what anyone else will ask for first:

kubectl -n <namespace> get pods,svc,endpoints
kubectl -n <namespace> describe pod <failing-pod>
kubectl -n <namespace> logs <failing-pod> --since=30m
kubectl -n <namespace> get events --sort-by=.lastTimestamp | tail -40