OVDC: RocksDB Corruption or Failures#

Overview#

When OVDC (Omniverse Derived Cache) experiences RocksDB corruption or failures, the service may fail to start, logs indicate database corruption, or data loss is observed. OVDC uses RocksDB as its persistent storage backend for derived data, and corruption can prevent the service from operating correctly.

OVDC relies on RocksDB for persistent storage of derived data. RocksDB is a persistent key-value store that provides durability and performance. When corruption occurs, RocksDB cannot read or write data correctly, causing service failures.

RocksDB corruption can occur when:

  • Unclean shutdowns preventing proper database closure

  • Disk errors or hardware failures affecting persistent volumes

  • File system corruption on persistent volumes

  • Insufficient disk space during critical operations

Resetting corrupted storage deletes the affected cached data and requires the cache to be rebuilt. Limit cleanup to the affected OVDC release and explicitly identified claims; never delete every PVC in the namespace.

Symptoms and Detection Signals#

Visible Symptoms#

  • Service fails to start - OVDC pods unable to start due to database errors

  • Database corruption errors - Logs indicating RocksDB corruption or invalid data

  • Data loss - Missing or corrupted derived data in cache

  • Repeated pod restarts - Pods crashing repeatedly due to database errors

Log Messages#

Database Open Failures#

Where to find these logs:

  • Pod: ovdc-*

  • Location: OVDC Pod

  • Application: OVDC/RocksDB

  • Description: Errors indicating RocksDB cannot open the database

# LEVEL: Error/Fatal
# SOURCE: OVDC/RocksDB
kubernetes.pod_name: ovdc-* and
message: "*failed to open rocksdb database*"

Metric Signals#

See the Metrics Reference for the full ovdc_error metric definition.

RocksDB I/O errors are tracked through the ovdc_error metric with kind="rocks_io_error". This metric counts I/O errors reported by RocksDB during database operations. High values or sharp increases indicate disk I/O failures or data corruption that may prevent the database from reading or writing data correctly. Sustained I/O errors typically precede database corruption and service failures.

Root Cause Analysis#

Known Causes#

RocksDB corruption in OVDC is typically caused by unclean shutdowns, disk errors, or software bugs.

Unclean Shutdowns#

Unclean shutdowns occur when OVDC pods are terminated abruptly without allowing RocksDB to flush data and close the database properly. This can happen during node failures, forced pod deletions, or OOM kills. When RocksDB cannot close cleanly, WAL or SST files may be left in an inconsistent state, causing corruption.

Check for unclean shutdowns:

kubectl describe pod -n ovdc <ovdc-pod-name> | grep -A 10 "Events:"
kubectl get pods -n ovdc -l app.kubernetes.io/instance=ovdc -o wide
# Look for termination reasons: OOMKilled, Evicted, NodeShutdown

kubectl get events -n ovdc --sort-by='.lastTimestamp' | grep -i "evict\|kill\|terminate"

Disk Errors or Hardware Failures#

Disk errors or hardware failures on persistent volumes can cause data corruption. This may manifest as I/O errors, read/write failures, or corrupted data blocks. Persistent volumes backed by unreliable storage or experiencing hardware issues are more susceptible to corruption.

Check for disk errors:

kubectl logs -n ovdc <ovdc-pod-name> | grep -i "disk\|io\|error\|fail"
kubectl describe pvc -n ovdc
# Check for volume attachment issues or storage backend problems

Other Possible Causes#

  1. File System Corruption

    • File system corruption on persistent volumes

    • File system errors preventing proper writes

    • Mount issues causing data inconsistencies

  2. Insufficient Disk Space

    • Disk space exhaustion during critical operations

    • WAL files or SST files corrupted due to space constraints

    • Compaction failures due to insufficient space

Troubleshooting Steps#

Diagnostic Steps for Known Root Causes#

1. Reset the Affected OVDC Storage#

Identify the corrupted claims belonging to the affected release before resetting storage. Deleting those claims loses their cached data; workloads rebuild it. Other releases and unaffected claims must be left intact. The commands below assume release ovdc, namespace ovdc, and the default StatefulSet name; adjust them for your deployment. Stop the StatefulSet and wait for its pods to terminate before deleting claims so no pod holds them open or is recreated during cleanup. Expect a service outage until replicas are restored.

# List only the ovdc release PVCs and identify the affected claims.
kubectl get pvc -n ovdc -l app.kubernetes.io/instance=ovdc

# Save the desired replica count, then stop the StatefulSet.
previous_replicas=$(kubectl get statefulset ovdc-ovderivedcache -n ovdc -o jsonpath='{.spec.replicas}')
kubectl scale statefulset/ovdc-ovderivedcache -n ovdc --replicas=0
kubectl wait --for=delete pod -n ovdc -l app.kubernetes.io/instance=ovdc --timeout=300s
# Continue only after all release pods have terminated.

# Delete only the explicitly identified corrupted claims.
# WARNING: Their cached data will be lost.
kubectl delete pvc -n ovdc <affected-pvc-name> [<another-affected-pvc-name>]

# Restore the original replica count and wait for readiness.
kubectl scale statefulset/ovdc-ovderivedcache -n ovdc --replicas="$previous_replicas"
kubectl rollout status statefulset/ovdc-ovderivedcache -n ovdc

# Verify pods are running and healthy
kubectl get pods -n ovdc -l app.kubernetes.io/instance=ovdc

Analysis:

  • Deleting selected PVCs removes only the affected cached data.

  • Scaling to zero releases the claims; restoring replicas recreates missing PVCs and starts the pods.

  • Cache will be rebuilt as workloads access OVDC.

Resolution:

  • Delete only the identified corrupted claims belonging to the affected release (their data will be lost).

  • Stop the StatefulSet and wait for pod termination before deleting selected claims, then restore the previous replica count.

  • Monitor pod startup to ensure service is healthy.

  • Cache will rebuild automatically as workloads run (renders will be slower during rebuild).

Prevention#

Proactive Monitoring#

Set up alerts for:

  • RocksDB I/O errors: Alert when ovdc_error{kind=~"rocks_io_error"} increases

  • Pod termination events: Alert on OOM kills or forced pod terminations

  • Database corruption errors: Alert on corruption-related log messages

  • Pod startup failures: Alert when OVDC pods fail to start repeatedly