OVDC: Network Bandwidth or Latency Bottlenecks#

Overview#

When OVDC (Omniverse Derived Cache) experiences network bandwidth or latency bottlenecks, cache population is slow, throughput between OVDC and GPU nodes is degraded, or scene load times increase. OVDC requires high bandwidth and low latency to efficiently serve derived data to render workers over the network.

OVDC serves derived data to render workers over the network. The service requires high bandwidth and low latency to support efficient data transfer. It is generally recommended to have 3.3Gbps of network bandwidth available for each GPU in a cluster, with a minimum of 1Gbps per GPU.

Note

The 3.3 Gbps-per-GPU figure is operational guidance carried over from earlier DDCS-era docs and could not be confirmed against the ovderivedcache source repository as part of this review—treat it as a starting heuristic, not a verified spec. The repo’s own confirmed capacity numbers are node NICs ≥12.5 Gbps and roughly 60,000 requests/second per OVDC instance; see the OVDC Configuration Guide for the sourced sizing table.

Network bandwidth or latency bottlenecks can occur when:

  • Insufficient OVDC pods scheduled to handle network load

  • Insufficient network bandwidth between OVDC and GPU nodes

  • Network congestion from noisy neighbors in shared environments

  • Misconfigured Kubernetes networking causing suboptimal routing

  • Cross-availability zone or cross-data center traffic

When network bottlenecks occur, OVDC cannot efficiently serve data to render workers, causing slow cache population, degraded throughput, and increased scene load times.

Symptoms and Detection Signals#

Visible Symptoms#

  • Slow cache population - Cache taking longer than expected to populate with derived data

  • Degraded throughput - Reduced data transfer rates between OVDC and GPU nodes

  • Increased scene load times - Scene loads taking significantly longer than expected

Metric Signals#

The following metrics can be used to detect network bandwidth or latency bottlenecks. Review these metrics to determine if networking is a problem. See the Metrics Reference for the full metric/type/label definitions.

rate(ovdc_bytes_returned[5m])
rate(container_network_receive_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
rate(container_network_transmit_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])

These counters accumulate bytes over their lifetime. Use rate(...[5m]) to measure bytes per second; multiply by 8 to compare with NIC capacity in bits per second. For example:

8 * rate(container_network_transmit_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])

Compare transmit and receive rates separately with the appropriate interface capacity. Select the relevant interface to avoid counting the same traffic twice, and aggregate traffic from pods sharing a node when comparing with that node’s NIC limit. rate(ovdc_bytes_returned[5m]) measures application payload throughput; it excludes protocol overhead and other node traffic, so use interface metrics to confirm saturation.

Root Cause Analysis#

Known Causes#

Network bandwidth or latency bottlenecks in OVDC are typically caused by insufficient OVDC pods scheduled, insufficient network bandwidth, or network congestion.

Insufficient OVDC Pods Scheduled#

OVDC must be configured with replica count equal to the number of compute nodes calculated based on network bandwidth requirements. The rule of thumb is to provision at least 3.3 Gbps of bandwidth per GPU. If there are not enough OVDC pods scheduled, the available pods may become network-bound, causing bottlenecks. Refer to the OVDC Configuration Guide for scaling guidance.

Check OVDC pod count:

# Check current OVDC pod count
kubectl get pods -n ovdc -l app.kubernetes.io/instance=ovdc

# Check StatefulSet replica configuration
kubectl get statefulset -n ovdc -l app.kubernetes.io/instance=ovdc
kubectl describe statefulset -n ovdc <ovdc-statefulset-name>

# Calculate required OVDC pods
# Required pods = (GPU count * 3.3 Gbps) / compute_node_bandwidth
# Example: 25 GPUs * 3.3 Gbps = 82.5 Gbps / 10 Gbps per node = ~9 pods required

Insufficient Network Bandwidth#

Network bandwidth may be insufficient for the workload. OVDC requires high bandwidth to serve derived data efficiently. It is recommended to have 3.3Gbps per GPU, with a minimum of 1Gbps.

Network Congestion#

In shared network environments, other workloads may consume bandwidth, causing congestion and degraded performance for OVDC traffic.

Other Possible Causes#

  1. Misconfigured Kubernetes Networking

    • Suboptimal routing causing increased latency

    • Network policies limiting throughput

    • Cross-availability zone traffic

  2. Cross-Availability Zone or Cross-Region Traffic

    • Pods in different availability zones causing higher latency

    • Cross-region traffic increasing latency significantly

    • Public internet transit instead of private networking

  3. Node-Level Network Issues

    • Node network interface problems

    • Network driver issues

    • Hardware network limitations

Troubleshooting Steps#

Diagnostic Steps for Known Root Causes#

1. Check OVDC Pod Count and Scaling#

Verify there are sufficient OVDC pods scheduled to handle network load. Refer to the OVDC Configuration Guide for scaling requirements.

# Check current OVDC pod count
kubectl get pods -n ovdc -l app.kubernetes.io/instance=ovdc

# Calculate required OVDC pods
# Required pods = (GPU count * 3.3 Gbps) / compute_node_bandwidth
# Example: 25 GPUs * 3.3 Gbps = 82.5 Gbps / 10 Gbps per node = ~9 pods required

# Check network metrics per pod
# Query: rate(container_network_transmit_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
# Query: rate(container_network_receive_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])

Analysis:

  • Pod count below requirements indicates insufficient scaling.

  • High network utilization per pod suggests pods are network-bound.

  • Network metrics approaching interface limits indicate saturation.

Resolution:

  • Choose an OVDC replica count that meets the calculated pod requirements and keep it stable during normal operation. Scaling up or down invalidates entries cached for the previous count while a different count is active. Returning to a previous count makes its retained entries valid again.

  • Plan for derived-data regeneration and cache warm-up for any required data not already cached for the active count.

  • Refer to OVDC Configuration Guide for scaling guidance.

  • Set replicas in values.yaml to the chosen count, then apply it: helm upgrade ovdc omniverse/ovderivedcache --version 6.0.3 --namespace ovdc -f values.yaml

  • Monitor network metrics and cache hit/miss ratios after the change. Verify performance improvement once any required warm-up is complete.

2. Monitor Network Metrics#

Review network metrics to identify bandwidth saturation or bottlenecks.

# Query OVDC bytes returned metric
# rate(ovdc_bytes_returned[5m])

# Query container network metrics
# rate(container_network_receive_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])
# rate(container_network_transmit_bytes_total{namespace="ovdc", pod=~"ovdc-.*"}[5m])

# Calculate network utilization
# Compare 8 * rate(counter[5m]) with NIC capacity in bits/second
# Use matching interface/node scope; do not compare cumulative totals
# Check if metrics are approaching interface limits

Analysis:

  • High 8 * rate(ovdc_bytes_returned[5m]) relative to NIC capacity in bits/second suggests potential bottlenecks; confirm using interface traffic.

  • Transmit or receive rates approaching the matching interface capacity indicate saturation.

  • High utilization across nodes suggests network congestion.

Resolution:

  • If network is saturated, review OVDC sizing and plan any replica-count change with the cache validity and warm-up guidance in step 1.

  • Upgrade to VM SKUs with higher NIC speeds if bandwidth is insufficient.

  • Investigate network congestion from other workloads.

Other Diagnostic Actions#

  • Check pod placement: Verify OVDC and GPU pods are optimally placed:

    kubectl get pods -n ovdc -o wide
    kubectl get pods -n <workload-namespace> -o wide
    # Ensure pods are in same availability zone for optimal performance
    
  • Review network policies: Check if network policies are affecting performance:

    kubectl get networkpolicies -n ovdc
    kubectl describe networkpolicy <policy-name> -n ovdc
    
  • Monitor network trends: Track network performance over time:

    # Use cloud provider network metrics
    # Monitor throughput, latency, and error rates
    # Identify patterns and trends
    

Prevention#

Proactive Monitoring#

Set up alerts for:

  • Network bandwidth thresholds: Alert when network utilization exceeds 80% of available bandwidth

  • OVDC pod count: Alert when OVDC pod count is below calculated requirements

  • Network saturation: Alert when five-minute transmit or receive rates, converted to bits/second and aggregated to the relevant interface/node, exceed 80% of NIC capacity

  • High bytes returned: Monitor rate(ovdc_bytes_returned[5m]) for payload throughput and confirm potential bottlenecks with interface rates