Observability of Kit Apps#
This guide explains how to set up observability for Omniverse Kit applications running in self-hosted deployments.
Customers are responsible for collecting telemetry data.
Important
For documentation on observability configuration and best practices, see the Omniverse DGXC Observability Guide.
Observability Architecture#
The Omniverse platform is transitioning to OpenTelemetry (OTel) as the standard for observability:
Apps using Kit 109 or later
Native OTel log signal export directly from the Kit application
OTel metrics via streaming (when enabled)
OTel traces support
Standard OTLP (OpenTelemetry Protocol) compliance
Function worker pods are annotated with function_id and function_version_id
to facilitate filtering and correlation of telemetry data across your deployment.
Note
Across the entire cluster, some services may continue to expose Prometheus metrics and log only to stdout. Your observability solution should accommodate both approaches.
Configuring OTel in Kit 109+#
Kit 109+ includes native OpenTelemetry support through the Carbonite SDK. The OTel infrastructure is automatically loaded at Kit kernel startup - no explicit extension dependencies required.
Environment Variables:
Kit uses standard OTel environment variables for configuration. The full list of available configuration options is available from the Carbonite SDK Documentation.
Specify the service URL for a Kit collector via environment variables:
env:
# Collector endpoint
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://collector.otel.svc.cluster.local:4318" # Replace with the correct cluster DNS/Host/IP etc.
# Service identification
- name: OTEL_SERVICE_NAME
value: "my-kit-app" # Apply an appropriate name.
Kit Arguments
/exts/omni.kit.livestream.app/primaryStream/enableOpenTelemetry(defaultfalse)Enables streaming metrics export for the primary livestream session.
MUST be
trueto enable OTel for livestream extensions!/observability/logs/emitLevel(defaultwarn)Minimum log level to export via OTLP. Accepts
"verbose","info","warn","error", or"fatal"/observability/logs/channels/<channelNameOrPattern>Channel-specific log level for OTel export. Overrides the global
emitLevelfor the named channel./log/channels/<channelNameOrPattern>Channel-specific log level for stdout. Overrides the global log level for the named channel.
Note that channel names can contain patterns.
With this information we can, for example, only emit logs at the “warn” level
as a baseline. Then, enable “info” level for livestream extension channels,
being sure to also set enableOpenTelemetry = true. This could be done with
the following arguments:
--/observability/logs/emitLevel = "warn"
--/observability/logs/channels/omni.kit.livestream.* = "info"
--/exts/omni.kit.livestream.app/primaryStream/enableOpenTelemetry=true
Important
The volume of Kit telemetry (in particular logs and traces) can be substantial!
The info and verbose log levels may produce an output volume greater
than collectors can consume.
Log Channels
Per-channel overrides are useful when the global emit level is too coarse. For example, a noisy subsystem can be silenced without losing visibility into others, or a specific channel can be promoted to a higher verbosity for targeted debugging without flooding the observability backend with unrelated messages.
Channels are registered by plugins and extensions as they load at runtime. The active set depends on which extensions your application enables. An extension that is not loaded will contribute no channels, third party extensions can also register their own channels.
To determine the active set of channels, capture and review the log output of Kit applications. Channel names appear in square brackets immediately before the message text.
# `omni.rtx.materials` is the channel for this message
[omni.rtx.materials] Loading MDL material: /path/to/material.mdl
Log Channel Config
/observability/logs/emitLevelMinimum log severity for messages forwarded to OpenTelemetry. Accepts a string (
"verbose","info","warn","error","fatal") or the corresponding Carbonite integer value (-2through2). Independent of/log/level. If the OTel level is lower than the core level, those messages will never arrive regardless. Defaults to"warn". Reloaded at runtime if changed programmatically./observability/logs/channels/Per-channel emit level overrides. Each key is a channel name pattern (
*matches zero or more characters,?matches exactly one) and the value is a severity level using the same format as/observability/logs/emitLevel. Channel settings override the base emit level; when multiple patterns match, the first takes precedence. Defaults to empty (no overrides). Reloaded at runtime if changed programmatically./observability/logs/adoptLoggingChannelSettingsWhen
true, the OTel log listener reads all patterns from/log/channels/at startup and uses them as its initial channel filters, keeping local and remote log configuration in sync without duplication. Adoption is one-time at init; settings become independent afterward. Any explicit/observability/logs/channels/entries take precedence over adopted values. Defaults tofalse.
Logging channel names…
The following is an incomplete list of channels that can be controlled.
Do not rely on this list as these values will change over time and depend on the Kit app and extensions in-use.
Channel |
Description |
|---|---|
|
GPU performance profiling and timing data |
|
Progress tracking for long-running operations |
|
Loading and parsing lidar sensor configuration files |
|
Sensor Processing Graph pipeline execution |
|
Routing and dispatching of incoming server commands |
|
Server coordination and cluster state management |
|
Server extension startup and shutdown |
|
Server operation events published to the message queue |
|
Internal thread that assigns render jobs to workers |
|
Render quality setting changes and adjustments |
|
Creating a new USD stage at runtime via RPC |
|
Creating a sensor set via RPC |
|
Deleting a USD stage at runtime via RPC |
|
Reading scene data values via RPC |
|
Streaming scene data to clients via RPC |
|
Triggering a sensor set render via RPC |
|
Writing scene data values via RPC |
|
RTX render job scheduling and result handling on the server |
|
Server-side automated test output |
|
Thread pool that reads scene data in parallel |
|
Asset caching for textures, geometry, and materials |
|
Health and availability of cache storage nodes |
|
Action Graph node execution and event handling |
|
OmniGraph bundle data read and write operations |
|
Registration and lookup of OmniGraph execution passes |
|
Hydra scene delegate that translates USD to the renderer |
|
Iray path tracer renderer |
|
Per-task timing from the Kit application update loop |
|
Instanced geometry scattering across a scene |
|
General RTX renderer messages, including missing textures and file errors |
|
RTX Hydra render delegate synchronization with the scene |
|
Internal scene index tracking what the renderer knows about the scene |
|
Material loading, compilation, and binding (MDL, MaterialX) |
|
Rendering pipeline stages: denoising, post-processing, and color management |
|
Synthetic sensor rendering for cameras, lidar, and radar |
|
Extended Reality (XR/VR/AR) rendering output |
|
Scheduling and orchestration of sensor render requests |
|
Debug overlay showing sensor scheduling and timing information |
|
Sensor data output and annotator generation |
|
Network transport for sensor output data |
|
Scene graph memory and allocation statistics exported to OpenTelemetry |
|
Application memory usage exported to OpenTelemetry |
|
OpenTelemetry Metrics subsystem startup and configuration |
|
Renderer and GPU statistics exported as OpenTelemetry metrics |
|
OpenTelemetry subsystem startup and configuration |
|
Disk and storage usage exported to OpenTelemetry |
|
OpenTelemetry distributed tracing startup and configuration |
|
Background texture compression and asset pre-caching pipeline |
|
USD audio playback and spatialization |
|
Sensor registration and lifecycle events for multi-rate tick configurations |
|
Per-frame rendering for sensors running at different tick rates |
|
Resolving USD asset paths to files on disk, Nucleus, or other backends |
|
Geometry upload and processing on the GPU |
|
Hydra scene traversal and synchronization with the RTX renderer |
|
Streaming large geometry assets into the renderer on demand |
|
Internal database of geometry, lights, and instances visible to the renderer |
|
Low-level USD asset path resolution (distinct from |