OVDC: Cache Misses and Performance Degradation#

Overview#

OVDC caches derived data that is computationally expensive to generate. When cache misses occur frequently, render time is spent regenerating derived content before getting to your desired workload, significantly increasing scene load times and reducing overall system performance.

Cache misses and performance degradation can occur when:

  • Cache is cold (scene or assets not preloaded)

  • Cache eviction due to memory or disk pressure

  • Misconfigured cache size parameters

  • Insufficient cache capacity for workload patterns

  • Write buffer pressure causing cache eviction

Symptoms and Detection Signals#

Visible Symptoms#

  • User-facing slowdowns - Scene loads taking significantly longer than expected

  • Inconsistent performance - Performance varies dramatically between “warm” and “cold” scene loads

Log Messages#

Write Buffer Pressure#

Where to find these logs:

  • Pod: ovdc-*

  • Location: OVDC Pod

  • Application: OVDC

  • Description: Logs indicating writes are behind or are failing

# LEVEL: Error
# SOURCE: OVDC/RocksDB
kubernetes.pod_name: ovdc-* and
message: "*timeout while trying to write to DB*" or
         "*write failed because the database is busy*" or
         "*failed to write*" or
         "*the engine is busy and cannot currently process the request*"

Metric Signals#

The following metrics, exposed on settings.telemetry.prometheusMetricsPort (default 3051), can be used to further determine the cause of the problem. See the Metrics Reference for the full metric/type/label definitions; this section covers how to interpret them for cache miss diagnosis.

Read Path Metrics (Cache Hit/Miss)#

ovdc_query{level="0|1|2"}
ovdc_miss{level="0|1|2"}
ovdc_answer{level="0|1|2"}
ovdc_get_bytes{level="0|1|2"}

Every Get records ovdc_query at the tier it was served from: level="0" is the in-process row cache, level="1" is a RocksDB block-cache-only read (Tier1), and level="2" is a full RocksDB read that may hit disk (Tier2). A miss at that tier increments ovdc_miss; a hit increments ovdc_answer and ovdc_get_bytes (bytes returned). Comparing ovdc_miss against ovdc_query per level produces a hit ratio; a high miss rate at level="2" indicates the row and block caches are not absorbing the workload, either from a cold cache or undersized engine.cacheSize / engine.blockCacheSize.

Error Metrics#

ovdc_error{kind="..."}

Counts errors by kind—see Error Metrics for the full list of values. kind="rocks_busy" / rocks_try_again / rocks_timed_out indicate the engine is rejecting or delaying writes under load. A sharp increase in this counter alongside elevated ovdc_io_time values indicates the storage engine is struggling to keep up with the workload.

I/O Performance Metrics#

ovdc_io_time{io="rocksdb_read|rocksdb_read_l1|rocksdb_read_l2|rocksdb_write|rocksdb_delete|rocksdb_merge"}

A histogram (seconds) of RocksDB operation latency, labeled by the io label (note: io, not io_kind). Elevated rocksdb_read / rocksdb_read_l2 latency indicates cache misses forcing slow disk reads; elevated rocksdb_write latency indicates write buffer or compaction pressure.

Write Path Metrics#

ovdc_put{}
ovdc_put_bytes{}
ovdc_bytes_returned{}
ovdc_rowcache_insert{}

ovdc_put / ovdc_put_bytes track write volume; ovdc_rowcache_insert tracks entries added to the in-process row cache after a disk read, which is the mechanism that should reduce subsequent level="2" queries for the same key.

Root Cause Analysis#

Possible Causes#

Cold Cache (Scene or Assets Not Preloaded)#

A cold cache occurs when scenes or assets have not been preloaded into OVDC. During initial scene loads, GPUs encounter assets for the first time and must generate derived content synchronously, which is then written to OVDC. This synchronous generation and writing process significantly increases scene load times compared to warm cache scenarios where derived data is already available.

Cold cache conditions are expected during initial scene loads or when new assets are introduced. However, if cache miss ratios remain consistently high across multiple scene loads or after cache warm-up procedures, it may indicate that the cache is not sized correctly for the workload or that cache eviction is occurring too frequently.

Cache Eviction Due to Memory/Disk Pressure#

Cache eviction occurs when memory or disk pressure forces OVDC to remove cached data to make room for new writes. Write buffers evict cached key-value pairs when full, and RocksDB may evict data from caches when memory pressure occurs.

Cache size parameters (engine.cacheSize for row cache, engine.blockCacheSize for block cache) may be misconfigured for the workload. Too small caches result in frequent eviction and cache misses, while too large caches may cause memory pressure and OOM kills.

Storage Medium has Insufficient Performance#

Insufficient storage performance occurs when the underlying persistent volume cannot keep up with OVDC write and read operations. When storage IOPS or throughput is insufficient, RocksDB cannot flush write buffers or perform compaction fast enough, causing write stalls and cache eviction. This manifests as high stall metrics, slow I/O histograms, and increased disk seek operations.

Storage performance requirements depend on your installation environment and workload. The storage class and persistent volume configuration must provide sufficient IOPS and throughput for the attached volumes. Consult your cloud service provider (CSP) documentation for storage class performance characteristics and scaling options.

Other Possible Causes#

  1. Insufficient Cache Capacity

    • Total cache size insufficient for workload data set

    • Cache not sized for peak scene sizes

    • Multiple concurrent scenes exceeding cache capacity

  2. Write Buffer Configuration Issues

    • Too few write buffers (cf.max_write_buffer_number) causing premature eviction

    • Write buffer size too small for write bursts

    • Write buffer pressure causing cache eviction

  3. Garbage Collection Aggressiveness

    • Garbage collection removing cached data too aggressively

    • garbageCollection.deleteKeyspaceQuantile set too high

    • garbageCollection.minFreeCapacity threshold too high

  4. Workload Pattern Changes

    • New scenes or assets not fitting existing cache patterns

    • Increased scene complexity requiring more cache space

    • Concurrent workload increases exceeding cache capacity

Troubleshooting Steps#

Diagnostic Steps#

1. Check Cache Hit/Miss Ratios#

Monitor cache hit and miss metrics to determine if cache is cold or experiencing eviction.

# Query metrics:
# - ovdc_answer{level="2"}
# - ovdc_miss{level="2"}

Analysis:

  • High miss rates relative to hits indicate cold cache or cache eviction.

  • Hit ratios below 70-80% suggest cache effectiveness issues.

  • Compare hit rates between initial scene loads (cold) and subsequent loads (warm).

  • Consistently high miss ratios after warm-up indicate cache sizing or eviction problems.

Resolution:

  • If cache is cold, implement warm-up procedures for common scenes.

  • If miss ratios remain high after warm-up, investigate cache sizing (see Cache Eviction section).

  • Monitor ovdc_query{level="0|1"} vs ovdc_query{level="2"} to quantify cache effectiveness.

2. Check Cache Hit/Miss and Disk Seek Metrics#

Monitor cache effectiveness metrics to identify eviction patterns.

# Query metrics:
# - ovdc_miss{level="2"} vs ovdc_answer{level="2"}
# - ovdc_query{level="0|1"} vs ovdc_query{level="2"}

Analysis:

  • High miss rates relative to hits indicate cache eviction.

  • High level="2" query volume relative to level="0|1" indicates eviction forcing disk reads.

  • Increasing miss rates over time suggest cache capacity issues.

Resolution:

  • If eviction is occurring, check cache size configuration (see step 4).

  • Monitor ovdc_bytes_returned to track data volume requiring disk access.

3. Check RocksDB’s Own Stall and Compaction Stats#

RocksDB tracks internal write-stall counters for the L0 file count limit, pending compaction bytes, and memtable limit. These are exposed as Prometheus gauges—see RocksDB Property Gauges for the full list.

# Query stall gauges (label name, not io):
# - ovdc_rocks_intrinsic_gauge{name="__rocksdb_stalls_total_stop"}
# - ovdc_rocks_intrinsic_gauge{name="__rocksdb_stalls_total_delays"}
# - ovdc_rocks_intrinsic_gauge{name="__rocksdb_stalls_stops_memtable_limit"}
# - ovdc_rocks_intrinsic_gauge{name="__rocksdb_stalls_stops_pending_compaction_bytes"}

Analysis:

  • Nonzero or climbing stall gauge values indicate RocksDB throttling or blocking writes due to storage performance limits.

  • L0 file count limit stalls suggest compaction cannot keep up with write volume.

  • Pending compaction bytes stalls indicate compaction backlog due to slow disk I/O.

  • Memtable limit stalls indicate write buffers cannot flush fast enough.

Resolution:

  • If stalls are frequent, investigate storage class IOPS and throughput capabilities.

  • Review ovdc_io_time{io="rocksdb_write"} for slow write operations.

  • Consult CSP documentation to upgrade storage class or increase volume performance.

4. Check I/O Performance Metrics#

Monitor I/O histogram metrics to quantify storage performance issues.

# Query I/O metrics:
# - ovdc_io_time{io="rocksdb_read"}
# - ovdc_io_time{io="rocksdb_write"}

Analysis:

  • High read I/O latency indicates slow disk reads, suggesting cache misses requiring disk access.

  • High write I/O latency indicates slow disk writes, suggesting write buffer pressure.

  • Compare I/O latencies against storage class performance specifications.

Resolution:

  • If I/O latencies are high, upgrade storage class or increase volume IOPS/throughput.

  • Monitor ovdc_error{kind=~"rocks_io_error|rocks_timed_out|rocks_busy"} for storage-related errors.

  • Review storage class configuration and consider higher performance tiers.

5. Review Storage Class and Volume Configuration#

Verify storage class provides sufficient IOPS and throughput for the workload.

# Check PVC storage class
kubectl get pvc -n ovdc
kubectl describe pvc -n ovdc <pvc-name>

# Check storage class configuration
kubectl get storageclass
kubectl describe storageclass <storage-class-name>

Analysis:

  • Storage class performance characteristics determine available IOPS and throughput.

  • Insufficient storage performance causes RocksDB stalls and cache eviction.

  • Compare storage class specs against workload requirements.

Resolution:

  • Upgrade to storage class with higher IOPS/throughput if performance is insufficient.

  • Consider provisioned IOPS volumes for consistent performance.

  • Monitor stall metrics after storage changes to validate improvements.

Other Diagnostic Actions#

  • Monitor write buffer utilization: Check ovdc_rocks_intrinsic_gauge{name=~"__rocksdb_stalls_(stops|delays)_memtable_limit"} for write buffer pressure

  • Review garbage collection settings: Check garbageCollection.deleteKeyspaceQuantile and garbageCollection.minFreeCapacity configuration

  • Analyze workload patterns: Review ovdc_answer / ovdc_miss and ovdc_query metrics over time to identify capacity issues

  • Compare warm vs cold performance: Monitor latency metrics during warm and cold loads to quantify cache impact

Prevention#

Proactive Monitoring#

Set up alerts for:

  • Cache hit ratio thresholds: Alert when cache hit ratios drop below 70% for extended periods

  • Cache miss rate increases: Alert on significant increases in cache miss rates

  • Write buffer pressure: Alert when write buffer utilization approaches limits

  • Memory pressure: Alert when pod memory usage approaches limits to prevent eviction

Capacity Planning#

  • Estimate cache requirements: Calculate cache sizes based on average scene sizes and access patterns

  • Plan for cache growth: Account for cache growth as workloads scale

  • Monitor cache trends: Track cache usage trends to predict when capacity increases are needed

  • Test cache effectiveness: Validate cache configurations under expected production load