OpenTelemetry Based Observability Support#

Overview#

The Omniverse ecosystem will be adopting OpenTelemetry as a standard for observability functionality. This includes logging, metrics, traces, and telemetry events. Crash reporting is also included in the realm of observability, but is not directly supported by OpenTelemetry. To more widely support OpenTelemetry based observability, the omni.telemetry.core library has been created as a platform and SDK independent helper library that can be used in Omniverse projects to allow for a consistent feature set and configuration across all Omniverse apps regardless of their deployment environment.

For detailed information about using the metrics API, see Metrics.

The main goals of this library are:

  • To provide a helper library that handles all of the OpenTelemetry configuration and startup for logging, metrics, and traces. Telemetry events will be handled as log OpenTelemetry events.

  • To provide support for observability communications through the OpenTelemetry Protocol (OTLP) for logs, metrics, and traces. Each of these signals will be supported over HTTP and gRPC at a minimum. Local file and stdout/stderr output will also be supported for debugging purposes but is not intended for production use.

  • To provide a Carbonite plugin that wraps the omni.telemetry.core library and provides access to its functionality for use in Carbonite or Kit based apps and services.

  • To provide the core library in a Carbonite-minimized manner so that other non-Carbonite, non-Kit Omniverse apps can also consume it and get the same functionality as any Carbonite or Kit based app would get. Some Carbonite public headers will still be needed in order to build against the core library, but they will be packaged with it as needed.

  • To respect all of the OpenTelemetry configuration environment variables (ie: OTEL_SERVICE_NAME, etc ) that apply to the feature sets we are using and offering. The environment variables will always take configuration precedence over other configuration methods.

  • To provide Carbonite/Kit settings that can be used to populate the OpenTelemetry environment variables if not defined.

  • To merge existing early OpenTelemetry related functionality into the core library.

  • To convert the existing legacy telemetry events system to support OTLP event/log signals. The previous behavior should still be present, just disabled by default.

  • To provide a configurable way to launch an optional OpenTelemetry collector instance for each Kit process. This will be disabled by default with the expectation that the app is configured to directly transmit to an OTLP endpoint. The collector can be helpful to speed up and batch transmission of signals in high traffic cases.

Library Initialization and Shutdown#

The omni.observability.core library provides a straightforward API for initializing and shutting down OpenTelemetry functionality in your application. The library handles all the complex OpenTelemetry SDK configuration internally, allowing you to focus on your application logic while ensuring proper observability setup.

Key Components#

Core Functions and Structures#

  • omni::observability::acquireIObservabilityProvider(): C API function that returns the main IObservabilityProvider interface. This is a singleton object that manages the OpenTelemetry system.

  • omni::observability::IObservabilityProvider: The main interface for controlling the observability system. Key methods include:

    • initialize(const InitializeDesc& desc): Initializes OpenTelemetry with the provided configuration

    • finish(): Shuts down OpenTelemetry and flushes all queues

    • addResource(name, value): Adds resource attributes to be included with all telemetry data

    • registerLogger(logger): Registers a debug logger for internal library messages

  • omni::observability::InitializeDesc: Configuration structure used to initialize the observability system. Contains optional parameters for session ID, settings provider, and debug logger.

  • omni::observability::State: RAII wrapper class that automatically manages initialization and shutdown. Recommended for most applications as it ensures proper cleanup.

  • omni::observability::OtelSettings: Abstract base class for providing configuration settings as an alternative to environment variables. Implement this interface to integrate with your application’s configuration system.

Initialization Approaches#

The library supports two main approaches for initialization:

  1. Automatic RAII Management (Recommended): Use the omni::observability::State wrapper class which handles initialization in its constructor and shutdown in its destructor.

  2. Manual Management: Directly use the IObservabilityProvider interface for more control over the lifecycle, but requires careful handling of initialize() and finish() calls.

Basic Usage Example#

Here’s a simple example showing how to initialize observability in your application:

#include <omni/observability/IObservabilityProvider.h>

int main()
{
    // Basic initialization with default settings
    // OpenTelemetry will be configured based on environment variables
    omni::observability::State observabilityState;

    // Your application code here
    // OpenTelemetry is now active and will export telemetry data
    // based on the configured exporters

    // Automatic shutdown when observabilityState goes out of scope
    return 0;
}

Advanced Usage Example#

For more control over the initialization process:

#include <omni/observability/IObservabilityProvider.h>
#include <omni/observability/OtelSettings.h>

// Custom settings implementation (optional)
class MyAppSettings : public omni::observability::OtelSettings
{
    // Implement getString(), getInt(), getBool(), etc.
    // to integrate with your app's configuration system
};

int main()
{
    // Create initialization descriptor with custom configuration
    omni::observability::InitializeDesc desc;

    // Set session ID for correlating telemetry data
    std::string sessionId = "my-app-session-12345";
    desc.sessionId = sessionId;

    // Optional: provide custom settings implementation
    MyAppSettings customSettings;
    desc.settings = &customSettings;

    // Optional: provide custom logger for internal debug messages
    // desc.logger = &myCustomLogger;

    // Add custom resource attributes that will be included with all telemetry
    omni::observability::ResourcePair resources[] =
    {
        { "service.name", "my-application" },
        { "service.version", "1.0.0" },
        { "deployment.environment", "production" },
    };
    desc.resources = resources;

    // Initialize observability with custom configuration
    omni::observability::State observabilityState(desc);

    // Your application code here

    // Automatic shutdown when observabilityState goes out of scope
    return 0;
}

Manual Lifecycle Management#

For applications that need more control over the observability lifecycle:

#include <omni/observability/IObservabilityProvider.h>

int main()
{
    // Acquire the provider interface
    auto* provider = omni::observability::acquireIObservabilityProvider();

    // Initialize with default settings
    omni::observability::InitializeDesc desc;
    std::string sessionId = "my-session";
    desc.sessionId = sessionId;

    provider->initialize(desc);

    // Your application code here

    // Important: Must call finish() to properly shutdown and flush data
    provider->finish();

    return 0;
}

Minimum Configuration Requirements#

To create a functional observability-enabled application, you need at minimum:

Environment Variables (choose one approach per signal type):#

For File-based Output (Development/Testing): The file exporter is intended for use as a debugging tool, not for production use. The output that is written to the specified file is provided in an unstructured format. The output is similar to JSON, but is not compliant at the moment. The output is intended to be more human readable than JSON output. Newer output formats will be added in future versions.

# Enable traces output to file
export OTEL_TRACES_EXPORTER=file
export OMNI_OTEL_TRACES_FILE_PATH=/tmp/traces.log

# Enable metrics output to file
export OTEL_METRICS_EXPORTER=file
export OMNI_OTEL_METRICS_FILE_PATH=/tmp/metrics.log

# Enable logs output to file
export OTEL_LOGS_EXPORTER=file
export OMNI_OTEL_LOGS_FILE_PATH=/tmp/logs.log

# Set service name and resource attributes
export OTEL_SERVICE_NAME=my-application
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0,deployment.environment=development"

For OTLP Output (Production):

# Enable OTLP exporters
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp

# Configure OTLP endpoints
export OTEL_EXPORTER_OTLP_ENDPOINT=http://your-collector:4318
# Or use signal-specific endpoints:
# export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://your-collector:4318/v1/traces
# export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://your-collector:4318/v1/metrics
# export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://your-collector:4318/v1/logs

# Set service name and resource attributes
export OTEL_SERVICE_NAME=my-application
export OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0,deployment.environment=production"

# Optional: Set protocol (defaults to HTTP)
export OTEL_EXPORTER_OTLP_PROTOCOL=http

Alternative: Carbonite Settings#

Instead of environment variables, you can use Carbonite settings by implementing the OtelSettings interface or configuring the settings in your application’s configuration files:

/observability/serviceName = "my-application"
/observability/resourceAttributes = "service.version=1.0.0,deployment.environment=development"
/observability/traces/exporter = "file"
/observability/traces/file/path = "/tmp/traces.json"
/observability/metrics/exporter = "file"
/observability/metrics/file/path = "/tmp/metrics.json"
/observability/logs/exporter = "file"
/observability/logs/file/path = "/tmp/logs.json"

Important Notes#

  1. Reference Counting: The IObservabilityProvider uses reference counting. Each call to initialize() must be paired with a call to finish(). The State wrapper handles this automatically.

  2. Singleton Behavior: The observability system is designed as a singleton. Multiple State objects or initialize() calls will increment the reference count but won’t change the configuration.

  3. Thread Safety: All initialization and shutdown operations are internally serialized and thread-safe.

  4. Configuration Precedence: Environment variables always take precedence over host app settings provided through the provided OtelSettings interface.

  5. Shutdown Flushing: The library will attempt to flush all pending telemetry data during shutdown, respecting the configured flush timeout values.

Debug Logging and Troubleshooting#

The omni.observability.core library provides debugging functionality through the omni::observability::Logger abstract class. This system allows host applications to receive and process internal diagnostic messages from the library, making it easier to troubleshoot configuration issues and monitor the library’s behavior.

Overview#

The debug logging system serves several purposes:

  • Diagnostic Information: Provides insights into the library’s internal operations, including initialization, configuration parsing, and exporter setup.

  • Error Reporting: Reports configuration errors, connection issues, and other problems that may affect observability data transmission.

  • Development Support: Assists developers in understanding how the library processes their configuration and integrates with OpenTelemetry components.

Logger Abstract Class#

The omni::observability::Logger class is an abstract base class that host applications must inherit from to receive debug messages. Key characteristics:

  • Reference Counted: Inherits from RefCountedObject for automatic lifetime management

  • Thread Safe: Can receive messages from multiple threads safely

  • Recursion Protected: Prevents infinite recursion if logger implementations emit their own observability data

Core Method#

virtual void handleMsg(const LogMetadata& metadata, carb::cpp::string_view messageText) = 0;

This method receives internal log messages from the library, along with rich metadata about the message context. The metadata includes the code location where the message was emitted (filename, function name, line number), the log channel name that the message was emitted on, the severity level of the log message, the platform specific identifier of the process and thread that emitted the message, and the timestamp of when the message was emitted. Only log messages that are emitted at or higher than the current log severity level will be passed to this function. Lower severity messages will be filtered out. The default log level is ‘warn’.

The implementation of this logger object may choose to redirect the given log message to its own logging system. If that logging system is hooked up to OpenTelemetry log signals, the emitted messages will also be visible along with the rest of the log data emitted to the configured OpenTelemetry log exporter.

LogMetadata Structure#

Each debug message includes comprehensive metadata:

class LogMetadata
{
public:
    const char* channelName;            // Log channel (e.g., "omni.observability.setup")
    const char* filename;               // Source file that generated the message
    const char* function;               // Function that generated the message
    LogLevel level;                     // Message severity level
    int32_t lineNumber;                 // Source line number
    ThreadId threadId;                  // OS thread ID
    ProcessId processId;                // OS process ID
    std::chrono::nanoseconds timestamp; // Nanoseconds since midnight January 1, 1970
};

Log Levels#

The library supports four log levels for internal messages:

  • kLogLevelVerbose (-2): Detailed debugging information, potentially high volume.

  • kLogLevelInfo (-1): General diagnostic messages, warnings, and errors.

  • kLogLevelWarn (0): Warning and error messages only.

  • kLogLevelError (1): Error messages only.

  • kLogLevelNone (2): No output. This is not valid as a log level to log to, but can be set as the current log level to effectively disable the internal logging.

Log Channels#

Messages are categorized into specific channels:

  • omni.observability.setup: General library initialization and configuration

  • omni.observability.settings: Settings parsing and validation

  • omni.observability.metrics.setup: Metrics exporter configuration

  • omni.observability.traces.setup: Traces exporter configuration

  • omni.observability.logs.setup: Logs exporter configuration

Usage Examples#

Basic Logger Implementation#

#include <omni/observability/ObservabilityDebug.h>
#include <omni/observability/IObservabilityProvider.h>
#include <iostream>

class MyAppLogger : public omni::observability::Logger
{
public:
    void handleMsg(const omni::observability::LogMetadata& metadata,
                   carb::cpp::string_view messageText) override
    {
        // Filter by log level if desired
        if (metadata.level < omni::observability::kLogLevelWarn)
            return;

        // Format and output the message
        std::cout << "[" << omni::observability::getLogLevelName(metadata.level) << "] "
                  << metadata.channelName << ": " << messageText << std::endl;
    }
};

int main()
{
    // Create and register the logger
    MyAppLogger logger;

    omni::observability::InitializeDesc desc;
    desc.logger = &logger;  // Logger provided during initialization

    omni::observability::State observabilityState(desc);

    // Logger will now receive internal debug messages.
    // Your application code here.

    return 0;
}

Advanced Logger with Integration#

// This class represents an interface to the host app's internal logging system.
class MyAppLoggingSystem;

class IntegratedLogger : public omni::observability::Logger
{
private:
    MyAppLoggingSystem* m_appLogger;

public:
    IntegratedLogger(MyAppLoggingSystem* appLogger) : m_appLogger(appLogger) {}

    void handleMsg(const omni::observability::LogMetadata& metadata,
                   carb::cpp::string_view messageText) override
    {
        // Convert observability log level to app's log level.
        MyAppLogLevel appLevel = convertLogLevel(metadata.level);

        // Include rich context information.
        std::string enrichedMessage = fmt::format(
            "[{}:{}] Thread {} - {}",
            metadata.function,
            metadata.lineNumber,
            metadata.threadId,
            messageText);

        // Emit through app's logging system
        m_appLogger->log(appLevel, metadata.channelName, enrichedMessage);
    }

private:
    MyAppLogLevel convertLogLevel(omni::observability::LogLevel level)
    {
        switch (level)
        {
            case omni::observability::kLogLevelVerbose: return MyAppLogLevel::Debug;
            case omni::observability::kLogLevelInfo:    return MyAppLogLevel::Info;
            case omni::observability::kLogLevelWarn:    return MyAppLogLevel::Warning;
            case omni::observability::kLogLevelError:   return MyAppLogLevel::Error;
            case omni::observability::kLogLevelNone:    return MyAppLogLevel::None;
            default:                                    return MyAppLogLevel::Info;
        }
    }
};

Runtime Logger Management#

int main()
{
    omni::observability::State observabilityState;
    auto* provider = observabilityState.getObservabilityProvider();

    // Register additional loggers at runtime
    MyAppLogger debugLogger;
    auto loggerId = provider->registerLogger(&debugLogger);

    // Control debug output level
    provider->setLogLevel(omni::observability::kLogLevelInfo);

    // Your application code here

    // Unregister when no longer needed
    provider->unregisterLogger(loggerId);

    return 0;
}

Controlling Log Output#

Applications can control the verbosity of debug messages. Note that this only affects the log level of the internal debug loggers for the OpenTelemetry SDK and the omni.observability.core library. This does not in any way affect the logging level of the host app/service or which log levels will be passed through to the OpenTelemetry SDK by the host app or transmitted by it.

Setting Log Levels#

auto* provider = observabilityState.getObservabilityProvider();

// Only show warnings and errors
provider->setLogLevel(omni::observability::kLogLevelWarn);

// Show all messages (verbose)
provider->setLogLevel(omni::observability::kLogLevelVerbose);

// Query current level
auto currentLevel = provider->getLogLevel();

Default Behavior#

When no custom logger is registered:

  • Internal log messages are written directly to stderr.

  • All message that are at or higher than the current log level are output unconditionally. The default log level is ‘warn’.

SilentLogger Helper Class#

The omni::observability::SilentLogger class provides a convenient way to suppress all internal debug output:

#include <omni/observability/ObservabilityDebug.h>

int main()
{
    // Create a silent logger to suppress all debug output
    omni::observability::SilentLogger silentLogger;

    omni::observability::InitializeDesc desc;
    desc.logger = &silentLogger;

    omni::observability::State observabilityState(desc);

    // No debug messages will be output to stderr
    // Your application code here

    return 0;
}

Use Cases for SilentLogger#

  • Production Deployments: Suppress debug output in production environments

  • Performance Testing: Eliminate debug message overhead during performance measurements

  • Clean Output: Prevent debug messages from interfering with application output

  • Custom Handling: Use as a base when you want to selectively handle only certain message types

Important Notes#

  1. Separate from OTLP Logs: Debug logging is completely separate from OpenTelemetry log signals sent via OTLP. These are internal diagnostic messages about the library itself. The host app’s logger implementation may choose to redirect the internal log messages via OTLP, but the core library itself will not do that on its own.

  2. Thread Safety: Logger implementations must be thread-safe as they can receive messages from multiple threads simultaneously.

  3. Recursion Prevention: If your logger implementation emits observability data that goes back through the library, recursive debug messages are automatically suppressed to prevent infinite loops.

  4. Lifetime Management: Logger objects use reference counting and must remain valid while registered. The library holds references to registered loggers. The host app may release its local reference to the logger object once it has been registered, but the object itself must persist as long as it is registered. A local variable instance of this object must not be used unless it is unregistered before the end of the defining scope.

  5. Performance Considerations: Debug logging can impact performance in high-throughput scenarios. Consider using appropriate log levels or the SilentLogger in performance-critical deployments.