Overview#

omni.metrics.core is an internal extension that provides a bridge for other extensions to make use of OpenTelemetry based metrics. This extension is not intended to be used directly by third party extensions. This only provides a C++ based API for use in other NVIDIA developed extensions. No Python API is provided. Other extensions that need to use metrics should use the OpenTelemetry API directly, either through C++ or Python.

Key Features#

Provides access to OpenTelemetry metrics capabilities through the carb.metrics plugin
Offers a Kit-native approach to metrics collection and reporting
Mimics the carb.stats.IStats interface for familiar integration
Supports various metric types including counters, gauges, and histograms
Configurable through environment variables and carb settings
Enables integration with industry-standard observability tools

Integration with OpenTelemetry#

The extension integrates with the OpenTelemetry framework, which is an industry-standard, vendor-neutral set of tools, APIs, and SDKs used to instrument, generate, collect, and export telemetry data. This integration allows metrics to be:

Collected in a standardized format
Exported to various backends including Prometheus, Grafana, and other observability platforms
Analyzed alongside traces and logs for comprehensive system monitoring
Correlated across distributed systems for end-to-end visibility

Usage Example#

To use metrics in your Python code, use the OpenTelemetry API directly. The example code below defines the functional portion of a Python Kit extension that records a histogram of frame render times for 40 frames, then forces the Kit app to quit. When this extension is used, it creates a meter object and a histogram instrument, then starts to record values into the histogram instrument. These will be exported by OpenTelemetry once per second.

To successfully export the data from this metric, the following environment variables should be defined for the process. These tell the OpenTelemetry SDK where exported metrics should go and how they should be exported:

OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318

In this example, the http://otel-collector:4318 endpoint should be replaced by the correct endpoint for a running instance of a data ingestion server that can receive and process OTLP (OpenTelemetry Protocol) data.

Alternatively, these environment variables can be used to write the metrics results to a local file. Note that using OTEL_EXPORTER_OTLP_METRICS_ENDPOINT will require a file URI to be used which always requires an absolute path to be provided. If the OMNI_OTEL_METRICS_FILE_PATH environment variable is used instead, a relative file path may be provided if needed.

OTEL_METRICS_EXPORTER: file

Windows:
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: file:///C:/path/to/file.txt

Linux:
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: file:////path/to/file.txt

Extension example code:

import time

import carb
import omni.ext
import omni.kit.app
from opentelemetry import metrics
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource


METRIC_NAME = "kit_histogram_demo_frame_time_ms"
METRIC_DESCRIPTION = "Sample frame/update duration emitted by Kit histogram demo"
METRIC_UNIT = "ms"


class HistogramMetricsExtension(omni.ext.IExt):
    def on_startup(self, ext_id):
        self._sub = None
        self._count = 0
        self._last_record_time = 0.0
        self._provider = self._setup_metrics()
        self._histogram = self._create_histogram(ext_id)

        carb.log_info(f"[HISTOGRAM_DEMO] Python startup: {ext_id}")

        app = omni.kit.app.get_app()
        self._sub = app.get_update_event_stream().create_subscription_to_pop(
            self._on_update,
            name="com.example.histogram.metrics.update",
        )

    def on_shutdown(self):
        if self._provider:
            self._provider.force_flush()
            self._provider.shutdown()

        carb.log_info(f"[HISTOGRAM_DEMO] Python shutdown after {self._count} histogram samples")
        self._sub = None
        self._histogram = None
        self._provider = None

    def _setup_metrics(self):
        exporter = OTLPMetricExporter()
        reader = PeriodicExportingMetricReader(exporter, export_interval_millis=1000)
        provider = MeterProvider(
            metric_readers=[reader],
            resource=Resource.create({"service.name": "kit-histogram-demo"}),
        )

        metrics.set_meter_provider(provider)
        carb.log_info("[HISTOGRAM_DEMO] configured OpenTelemetry metrics provider")
        return provider

    def _create_histogram(self, ext_id):
        meter = metrics.get_meter(ext_id, "0.1.0")
        carb.log_info(f"[HISTOGRAM_DEMO] created OpenTelemetry histogram {METRIC_NAME}")
        return meter.create_histogram(
            METRIC_NAME,
            description=METRIC_DESCRIPTION,
            unit=METRIC_UNIT,
        )

    def _on_update(self, _event):
        now = time.perf_counter()
        if now - self._last_record_time < 0.25:
            return

        self._count += 1
        elapsed_ms = (now - self._last_record_time) * 1000
        self._last_record_time = now
        self._histogram.record(elapsed_ms, {"extension": "com.example.histogram.metrics"})
        carb.log_info(f"[HISTOGRAM_DEMO] recorded histogram sample #{self._count}: {elapsed_ms:.3f} {METRIC_UNIT}")

        if self._count >= 40:
            self._provider.force_flush()
            carb.log_info("[HISTOGRAM_DEMO] sample target reached; requesting app quit")
            omni.kit.app.get_app().post_quit()