OVDC: RocksDB Corruption or Failures#
Overview#
When OVDC (Omniverse Derived Cache) experiences RocksDB corruption or failures, the service may fail to start, logs indicate database corruption, or data loss is observed. OVDC uses RocksDB as its persistent storage backend for derived data, and corruption can prevent the service from operating correctly.
OVDC relies on RocksDB for persistent storage of derived data. RocksDB is a persistent key-value store that provides durability and performance. When corruption occurs, RocksDB cannot read or write data correctly, causing service failures.
RocksDB corruption can occur when:
Unclean shutdowns preventing proper database closure
Disk errors or hardware failures affecting persistent volumes
File system corruption on persistent volumes
Insufficient disk space during critical operations
Resetting corrupted storage deletes the affected cached data and requires the cache to be rebuilt. Limit cleanup to the affected OVDC release and explicitly identified claims; never delete every PVC in the namespace.
Symptoms and Detection Signals#
Visible Symptoms#
Service fails to start - OVDC pods unable to start due to database errors
Database corruption errors - Logs indicating RocksDB corruption or invalid data
Data loss - Missing or corrupted derived data in cache
Repeated pod restarts - Pods crashing repeatedly due to database errors
Log Messages#
Database Open Failures#
Where to find these logs:
Pod:
ovdc-*Location: OVDC Pod
Application: OVDC/RocksDB
Description: Errors indicating RocksDB cannot open the database
# LEVEL: Error/Fatal
# SOURCE: OVDC/RocksDB
kubernetes.pod_name: ovdc-* and
message: "*failed to open rocksdb database*"
Metric Signals#
See the Metrics Reference for the full
ovdc_error metric definition.
RocksDB I/O errors are tracked through the ovdc_error metric with
kind="rocks_io_error". This metric counts I/O errors reported by
RocksDB during database operations. High values or sharp increases
indicate disk I/O failures or data corruption that may prevent the
database from reading or writing data correctly. Sustained I/O errors
typically precede database corruption and service failures.
Root Cause Analysis#
Known Causes#
RocksDB corruption in OVDC is typically caused by unclean shutdowns, disk errors, or software bugs.
Unclean Shutdowns#
Unclean shutdowns occur when OVDC pods are terminated abruptly without allowing RocksDB to flush data and close the database properly. This can happen during node failures, forced pod deletions, or OOM kills. When RocksDB cannot close cleanly, WAL or SST files may be left in an inconsistent state, causing corruption.
Check for unclean shutdowns:
kubectl describe pod -n ovdc <ovdc-pod-name> | grep -A 10 "Events:"
kubectl get pods -n ovdc -l app.kubernetes.io/instance=ovdc -o wide
# Look for termination reasons: OOMKilled, Evicted, NodeShutdown
kubectl get events -n ovdc --sort-by='.lastTimestamp' | grep -i "evict\|kill\|terminate"
Disk Errors or Hardware Failures#
Disk errors or hardware failures on persistent volumes can cause data corruption. This may manifest as I/O errors, read/write failures, or corrupted data blocks. Persistent volumes backed by unreliable storage or experiencing hardware issues are more susceptible to corruption.
Check for disk errors:
kubectl logs -n ovdc <ovdc-pod-name> | grep -i "disk\|io\|error\|fail"
kubectl describe pvc -n ovdc
# Check for volume attachment issues or storage backend problems
Other Possible Causes#
File System Corruption
File system corruption on persistent volumes
File system errors preventing proper writes
Mount issues causing data inconsistencies
Insufficient Disk Space
Disk space exhaustion during critical operations
WAL files or SST files corrupted due to space constraints
Compaction failures due to insufficient space
Troubleshooting Steps#
Diagnostic Steps for Known Root Causes#
1. Reset the Affected OVDC Storage#
Identify the corrupted claims belonging to the affected release before
resetting storage. Deleting those claims loses their cached data;
workloads rebuild it. Other releases and unaffected claims must be left
intact. The commands below assume release ovdc, namespace ovdc,
and the default StatefulSet name; adjust them for your deployment. Stop
the StatefulSet and wait for its pods to terminate before deleting
claims so no pod holds them open or is recreated during cleanup. Expect
a service outage until replicas are restored.
# List only the ovdc release PVCs and identify the affected claims.
kubectl get pvc -n ovdc -l app.kubernetes.io/instance=ovdc
# Save the desired replica count, then stop the StatefulSet.
previous_replicas=$(kubectl get statefulset ovdc-ovderivedcache -n ovdc -o jsonpath='{.spec.replicas}')
kubectl scale statefulset/ovdc-ovderivedcache -n ovdc --replicas=0
kubectl wait --for=delete pod -n ovdc -l app.kubernetes.io/instance=ovdc --timeout=300s
# Continue only after all release pods have terminated.
# Delete only the explicitly identified corrupted claims.
# WARNING: Their cached data will be lost.
kubectl delete pvc -n ovdc <affected-pvc-name> [<another-affected-pvc-name>]
# Restore the original replica count and wait for readiness.
kubectl scale statefulset/ovdc-ovderivedcache -n ovdc --replicas="$previous_replicas"
kubectl rollout status statefulset/ovdc-ovderivedcache -n ovdc
# Verify pods are running and healthy
kubectl get pods -n ovdc -l app.kubernetes.io/instance=ovdc
Analysis:
Deleting selected PVCs removes only the affected cached data.
Scaling to zero releases the claims; restoring replicas recreates missing PVCs and starts the pods.
Cache will be rebuilt as workloads access OVDC.
Resolution:
Delete only the identified corrupted claims belonging to the affected release (their data will be lost).
Stop the StatefulSet and wait for pod termination before deleting selected claims, then restore the previous replica count.
Monitor pod startup to ensure service is healthy.
Cache will rebuild automatically as workloads run (renders will be slower during rebuild).
Prevention#
Proactive Monitoring#
Set up alerts for:
RocksDB I/O errors: Alert when
ovdc_error{kind=~"rocks_io_error"}increasesPod termination events: Alert on OOM kills or forced pod terminations
Database corruption errors: Alert on corruption-related log messages
Pod startup failures: Alert when OVDC pods fail to start repeatedly