OVDC: Network Bandwidth or Latency Bottlenecks#
Overview#
When OVDC (Omniverse Derived Cache) experiences network bandwidth or latency bottlenecks, cache population is slow, throughput between OVDC and GPU nodes is degraded, or scene load times increase. OVDC requires high bandwidth and low latency to efficiently serve derived data to render workers over the network.
OVDC serves derived data to render workers over the network. The service requires high bandwidth and low latency to support efficient data transfer. It is generally recommended to have 3.3Gbps of network bandwidth available for each GPU in a cluster, with a minimum of 1Gbps per GPU.
Note
The 3.3 Gbps-per-GPU figure is operational guidance carried over from
earlier DDCS-era docs and could not be confirmed against the
ovderivedcache source repository as part of this review—treat it as
a starting heuristic, not a verified spec. The repo’s own confirmed
capacity numbers are node NICs ≥12.5 Gbps and roughly 60,000
requests/second per OVDC instance; see the OVDC Configuration
Guide for the sourced sizing table.
Network bandwidth or latency bottlenecks can occur when:
Insufficient OVDC pods scheduled to handle network load
Insufficient network bandwidth between OVDC and GPU nodes
Network congestion from noisy neighbors in shared environments
Misconfigured Kubernetes networking causing suboptimal routing
Cross-availability zone or cross-data center traffic
When network bottlenecks occur, OVDC cannot efficiently serve data to render workers, causing slow cache population, degraded throughput, and increased scene load times.
Symptoms and Detection Signals#
Visible Symptoms#
Slow cache population - Cache taking longer than expected to populate with derived data
Degraded throughput - Reduced data transfer rates between OVDC and GPU nodes
Increased scene load times - Scene loads taking significantly longer than expected
Metric Signals#
The following metrics can be used to detect network bandwidth or latency bottlenecks. Review these metrics to determine if networking is a problem. See the Metrics Reference for the full metric/type/label definitions.
rate(ovdc_bytes_returned[5m])
rate(container_network_receive_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
rate(container_network_transmit_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
These counters accumulate bytes over their lifetime. Use
rate(...[5m]) to measure bytes per second; multiply by 8 to
compare with NIC capacity in bits per second. For example:
8 * rate(container_network_transmit_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
Compare transmit and receive rates separately with the appropriate
interface capacity. Select the relevant interface to avoid counting the
same traffic twice, and aggregate traffic from pods sharing a node when
comparing with that node’s NIC limit. rate(ovdc_bytes_returned[5m])
measures application payload throughput; it excludes protocol overhead
and other node traffic, so use interface metrics to confirm saturation.
Root Cause Analysis#
Known Causes#
Network bandwidth or latency bottlenecks in OVDC are typically caused by insufficient OVDC pods scheduled, insufficient network bandwidth, or network congestion.
Insufficient OVDC Pods Scheduled#
OVDC must be configured with replica count equal to the number of compute nodes calculated based on network bandwidth requirements. The rule of thumb is to provision at least 3.3 Gbps of bandwidth per GPU. If there are not enough OVDC pods scheduled, the available pods may become network-bound, causing bottlenecks. Refer to the OVDC Configuration Guide for scaling guidance.
Check OVDC pod count:
# Check current OVDC pod count
kubectl get pods -n ovdc -l app.kubernetes.io/instance=ovdc
# Check StatefulSet replica configuration
kubectl get statefulset -n ovdc -l app.kubernetes.io/instance=ovdc
kubectl describe statefulset -n ovdc <ovdc-statefulset-name>
# Calculate required OVDC pods
# Required pods = (GPU count * 3.3 Gbps) / compute_node_bandwidth
# Example: 25 GPUs * 3.3 Gbps = 82.5 Gbps / 10 Gbps per node = ~9 pods required
Insufficient Network Bandwidth#
Network bandwidth may be insufficient for the workload. OVDC requires high bandwidth to serve derived data efficiently. It is recommended to have 3.3Gbps per GPU, with a minimum of 1Gbps.
Network Congestion#
In shared network environments, other workloads may consume bandwidth, causing congestion and degraded performance for OVDC traffic.
Other Possible Causes#
Misconfigured Kubernetes Networking
Suboptimal routing causing increased latency
Network policies limiting throughput
Cross-availability zone traffic
Cross-Availability Zone or Cross-Region Traffic
Pods in different availability zones causing higher latency
Cross-region traffic increasing latency significantly
Public internet transit instead of private networking
Node-Level Network Issues
Node network interface problems
Network driver issues
Hardware network limitations
Troubleshooting Steps#
Diagnostic Steps for Known Root Causes#
1. Check OVDC Pod Count and Scaling#
Verify there are sufficient OVDC pods scheduled to handle network load. Refer to the OVDC Configuration Guide for scaling requirements.
# Check current OVDC pod count
kubectl get pods -n ovdc -l app.kubernetes.io/instance=ovdc
# Calculate required OVDC pods
# Required pods = (GPU count * 3.3 Gbps) / compute_node_bandwidth
# Example: 25 GPUs * 3.3 Gbps = 82.5 Gbps / 10 Gbps per node = ~9 pods required
# Check network metrics per pod
# Query: rate(container_network_transmit_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
# Query: rate(container_network_receive_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
Analysis:
Pod count below requirements indicates insufficient scaling.
High network utilization per pod suggests pods are network-bound.
Network metrics approaching interface limits indicate saturation.
Resolution:
Choose an OVDC replica count that meets the calculated pod requirements and keep it stable during normal operation. Scaling up or down invalidates entries cached for the previous count while a different count is active. Returning to a previous count makes its retained entries valid again.
Plan for derived-data regeneration and cache warm-up for any required data not already cached for the active count.
Refer to OVDC Configuration Guide for scaling guidance.
Set
replicasinvalues.yamlto the chosen count, then apply it:helm upgrade ovdc omniverse/ovderivedcache --version 6.0.3 --namespace ovdc -f values.yamlMonitor network metrics and cache hit/miss ratios after the change. Verify performance improvement once any required warm-up is complete.
2. Monitor Network Metrics#
Review network metrics to identify bandwidth saturation or bottlenecks.
# Query OVDC bytes returned metric
# rate(ovdc_bytes_returned[5m])
# Query container network metrics
# rate(container_network_receive_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
# rate(container_network_transmit_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
# Calculate network utilization
# Compare 8 * rate(counter[5m]) with NIC capacity in bits/second
# Use matching interface/node scope; do not compare cumulative totals
# Check if metrics are approaching interface limits
Analysis:
High
8 * rate(ovdc_bytes_returned[5m])relative to NIC capacity in bits/second suggests potential bottlenecks; confirm using interface traffic.Transmit or receive rates approaching the matching interface capacity indicate saturation.
High utilization across nodes suggests network congestion.
Resolution:
If network is saturated, review OVDC sizing and plan any replica-count change with the cache validity and warm-up guidance in step 1.
Upgrade to VM SKUs with higher NIC speeds if bandwidth is insufficient.
Investigate network congestion from other workloads.
Other Diagnostic Actions#
Check pod placement: Verify OVDC and GPU pods are optimally placed:
kubectl get pods -n ovdc -o wide kubectl get pods -n <workload-namespace> -o wide # Ensure pods are in same availability zone for optimal performance
Review network policies: Check if network policies are affecting performance:
kubectl get networkpolicies -n ovdc kubectl describe networkpolicy <policy-name> -n ovdc
Monitor network trends: Track network performance over time:
# Use cloud provider network metrics # Monitor throughput, latency, and error rates # Identify patterns and trends
Prevention#
Proactive Monitoring#
Set up alerts for:
Network bandwidth thresholds: Alert when network utilization exceeds 80% of available bandwidth
OVDC pod count: Alert when OVDC pod count is below calculated requirements
Network saturation: Alert when five-minute transmit or receive rates, converted to bits/second and aggregated to the relevant interface/node, exceed 80% of NIC capacity
High bytes returned: Monitor
rate(ovdc_bytes_returned[5m])for payload throughput and confirm potential bottlenecks with interface rates