Configuring OpenTelemetry#

Overview#

The omni.observability.core library supports configuration through both OpenTelemetry standard environment variables and Omniverse-specific options. The library respects the standard OpenTelemetry configuration precedence where environment variables take precedence over Carbonite settings. If an environment variable is not set, the corresponding Carbonite setting (if available) will be used as a fallback.

Each configuration option below is listed with the environment variable name first, followed by the corresponding Carbonite setting path in parentheses when available. For example: ENVIRONMENT_VARIABLE (/carbonite/setting/path).

Standard OpenTelemetry Environment Variables#

These are the standard OpenTelemetry environment variables that are supported by the core library. These all begin with OTEL_. Each includes its corresponding Carbonite setting path when available.

  • OTEL_SDK_DISABLED (/observability/disabled): Globally disables the OpenTelemetry SDK when set to “true” (case-insensitive). When disabled, all observability functionality is turned off and all operations are treated as no-ops. Defaults to “false” when not set.

  • OTEL_LOG_LEVEL (/observability/otelSdkLogLevel): Controls the internal logging level of the OpenTelemetry SDK. Valid values include “debug”, “info”, “warn”, “error”, and “none”. This is used for debugging OpenTelemetry SDK issues and does not affect the emitting or transmission of log signals. Defaults to “info” when internal logging is enabled via OMNI_OTEL_ENABLE_INTERNAL_LOGGING, otherwise defaults to “none”.

  • OTEL_SERVICE_NAME (/observability/serviceName): Sets the service name reported in all telemetry signals. This is used to identify the service that emitted each signal in observability backends. A service name is not required but having one present does make it a lot easier to correlate various signals and track down issues. Defaults to an empty string (or “<no_service_name_defined>” in debug builds).

  • OTEL_PROPAGATORS (/observability/propagators): Configures trace context propagators for distributed tracing. Accepts a comma-separated list of propagator names. Determines how trace context is propagated across service boundaries. Defaults to “tracecontext”. This setting may include the propagators supported directly by OpenTelemetry including “tracecontext”, “baggage”, “b3”, “b3multi”, “jaeger”, and “none”. Other propagators may also be supported but would depend upon what the data endpoint also supports.

  • OTEL_RESOURCE_ATTRIBUTES (/observability/resourceAttributes): Configures resource attributes to be included in all telemetry signals (traces, metrics, and logs). Accepts a comma-separated list of key=value pairs in a simplified W3C baggage format. The format supports percent encoding, whitespace trimming, and quoted values. Resource attributes provide context about the service, environment, and deployment that generated the telemetry. Example: “service.version=1.0.0,deployment.environment=production”. Keys set from the environment variable will override keys set from the corresponding Carbonite setting. Note that any code points in any of either the key or value strings that are outside of the 7-bit ASCII range must be percent encoded. Some other special characters such as a comma, semicolon, and percent symbol must also be percent encoded if they are intended to be part of a key or value string.

  • OTEL_EXPORTER_OTLP_PROTOCOL: Sets the default OTLP protocol for all signal types (traces, metrics, logs). Valid values are “grpc”, “http/protobuf”, and “http/json”. This is used as a fallback when signal-specific protocol variables are not set. No direct Carbonite setting equivalent. This defaults to “http/protobuf” if not specified.

  • OTEL_EXPORTER_OTLP_ENDPOINT: Sets the default OTLP endpoint URL for all signal types. Used as a fallback when signal-specific endpoint variables are not set. Must be a valid URL. There is no direct Carbonite setting equivalent to this. The signal-specific Carbonite settings should be used instead if environment variables are not being used for configuration. The default endpoint depends on the protocol and the OMNI_OTEL_ALLOW_DEFAULT_ENDPOINT setting. If default endpoints are disabled (the default), no endpoint is provided. If enabled, defaults to http://localhost:4318 for HTTP protocols or http://localhost:4317 for gRPC protocol.

Traces Configuration#

  • OTEL_TRACES_EXPORTER (/observability/traces/exporter): Configures the traces exporter. Valid values are “otlp”, “file”, “console”, or empty to disable tracing. Determines how trace data is exported from the application. Defaults to “otlp”.

  • OTEL_EXPORTER_OTLP_TRACES_PROTOCOL (/observability/traces/otlp/protocol): Sets the OTLP protocol specifically for traces. Valid values are “grpc”, “http/protobuf”, and “http/json”. Takes precedence over the general OTEL_EXPORTER_OTLP_PROTOCOL for trace exports. Defaults to “http/protobuf”.

  • OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (/observability/traces/otlp/endpoint): Sets the OTLP endpoint URL specifically for traces. Takes precedence over the general OTEL_EXPORTER_OTLP_ENDPOINT for trace exports. Must be a valid URL. The default endpoint depends on the protocol and the OMNI_OTEL_ALLOW_DEFAULT_ENDPOINT setting. If default endpoints are disabled (the default), no endpoint is provided. If enabled, defaults to http://localhost:4318 for HTTP protocols or http://localhost:4317 for gRPC protocol.

  • OTEL_BSP_MAX_QUEUE_SIZE (/observability/traces/maxQueueSize): Sets the maximum queue size for the traces batch processor. Controls how many spans can be queued before dropping. Must be a positive integer. Default is 2048.

  • OTEL_BSP_MAX_EXPORT_BATCH_SIZE (/observability/traces/maxExportBatchSize): Sets the maximum export batch size for traces. Controls how many spans are exported in a single batch. Must be smaller than or equal to max queue size. Default is 512.

  • OTEL_BSP_SCHEDULE_DELAY (/observability/traces/scheduleDelay): Sets the schedule delay for the traces batch processor in milliseconds. Controls how frequently batches are exported. Default is 5000 (5 seconds).

Metrics Configuration#

  • OTEL_METRICS_EXPORTER (/observability/metrics/exporter): Configures the metrics exporter. Valid values are “otlp”, “file”, “console”, or empty to disable metrics. Determines how metric data is exported from the application. Defaults to “otlp”.

  • OTEL_EXPORTER_OTLP_METRICS_PROTOCOL (/observability/metrics/otlp/protocol): Sets the OTLP protocol specifically for metrics. Valid values are “grpc”, “http/protobuf”, “http/json”. Takes precedence over the general OTEL_EXPORTER_OTLP_PROTOCOL for metric exports. Defaults to “http/protobuf”.

  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT (/observability/metrics/otlp/endpoint): Sets the OTLP endpoint URL specifically for metrics. Takes precedence over the general OTEL_EXPORTER_OTLP_ENDPOINT for metric exports. Must be a valid URL. The default endpoint depends on the protocol and the OMNI_OTEL_ALLOW_DEFAULT_ENDPOINT setting. If default endpoints are disabled (the default), no endpoint is provided. If enabled, defaults to http://localhost:4318 for HTTP protocols or http://localhost:4317 for gRPC protocol.

  • OTEL_METRIC_EXPORT_INTERVAL (/observability/metrics/exportInterval): Sets the metrics export interval in milliseconds. Controls how frequently metrics are exported. This must be a positive integer. This must be greater than the OTEL_METRIC_EXPORT_TIMEOUT setting’s value, including its default value if that setting is not explicitly provided. If this value is less than or equal to the timeout value, both settings will be ignored and their defaults will be used instead. This defaults to 60000 (60 seconds).

  • OTEL_METRIC_EXPORT_TIMEOUT (/observability/metrics/exportTimeout): Sets the metrics export timeout in milliseconds. Controls the maximum time to wait for a metrics export to complete. This must be a positive integer. This must be less than the OTEL_METRIC_EXPORT_INTERVAL setting’s value, including its default value if that setting is not explicitly provided. If this value is greater than or equal to the interval value, both settings will be ignored and their defaults will be used instead. This defaults to 30000 (30 seconds).

  • OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE (/observability/metrics/temporalityPreference): Sets the global preferred aggregation temporality for metrics. This preferred value affects all instrument types that have not had specific temporalities set with OMNI_OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PER_INSTRUMENT. Valid values are “delta”, “cumulative”, or “lowMemory”. The aggregation temporality determines how metrics data is accumulated and reported at each export interval. Delta temporality means metrics represent only the data for that interval (counters reset each interval). Cumulative temporality means metrics represent data for the entire lifetime of the instrument. LowMemory temporality uses delta for counter, gauge, and histogram instruments, while using cumulative for upDownCounter and observable instruments. Note that with delta temporality, upDownCounter and observableUpDownCounter instruments still use cumulative aggregation, while with lowMemory, only synchronous instruments (counter, gauge, histogram) use delta. The default aggregation temporality for all instrument types is “cumulative”. The aggregation temporalities set with this configuration affect all instruments of each type; there is no way to control aggregation temporality on a per-instance basis.

Logs Configuration#

  • OTEL_LOGS_EXPORTER (/observability/logs/exporter): Configures the logs exporter. Valid values are “otlp”, “file”, “console”, or empty to disable logging through OpenTelemetry. Determines how log data is exported from the application. Defaults to “otlp”.

  • OTEL_EXPORTER_OTLP_LOGS_PROTOCOL (/observability/logs/otlp/protocol): Sets the OTLP protocol specifically for logs. Valid values are “grpc”, “http/protobuf”, and “http/json”. Takes precedence over the general OTEL_EXPORTER_OTLP_PROTOCOL for log exports. Defaults to “http/protobuf”.

  • OTEL_EXPORTER_OTLP_LOGS_ENDPOINT (/observability/logs/otlp/endpoint): Sets the OTLP endpoint URL specifically for logs. Takes precedence over the general OTEL_EXPORTER_OTLP_ENDPOINT for log exports. Must be a valid URL. The default endpoint depends on the protocol and the OMNI_OTEL_ALLOW_DEFAULT_ENDPOINT setting. If default endpoints are disabled (the default), no endpoint is provided. If enabled, defaults to http://localhost:4318 for HTTP protocols or http://localhost:4317 for gRPC protocol.

  • OTEL_BLRP_MAX_QUEUE_SIZE (/observability/logs/maxQueueSize): Sets the maximum queue size for the logs batch processor. Controls how many log records can be queued before dropping. Must be a positive integer. Default is 2048.

  • OTEL_BLRP_MAX_EXPORT_BATCH_SIZE (/observability/logs/maxExportBatchSize): Sets the maximum export batch size for logs. Controls how many log records are exported in a single batch. Must be smaller than or equal to max queue size. Default is 512.

  • OTEL_BLRP_SCHEDULE_DELAY (/observability/logs/scheduleDelay): Sets the schedule delay for the logs batch processor in milliseconds. Controls how frequently batches are exported. Default is 1000 (1 second).

Omniverse Specific Environment Variables#

These are Omniverse-specific environment variables that begin with OMNI_OTEL_. These extend the standard OpenTelemetry configuration with additional functionality specific to Omniverse applications.

File Output Configuration#

  • OMNI_OTEL_DEBUG_FILE_PATH: Sets a default debug file path for all telemetry outputs when using file exporters. Used as a fallback when signal-specific file paths are not set. Only relevant when exporters are set to “file”. No direct Carbonite setting equivalent. The signal specific file path settings should be used instead if the environment variable configuration is not being used.

  • OMNI_OTEL_TRACES_FILE_PATH (/observability/traces/file/path): Sets the file path for traces output when using the file exporter. Only relevant when OTEL_TRACES_EXPORTER is set to “file”. Falls back to OMNI_OTEL_DEBUG_FILE_PATH if not set.

  • OMNI_OTEL_TRACES_FILE_APPEND (/observability/traces/file/append): Controls whether the traces file should be appended to or truncated. Set to “true” (case-insensitive) to append, “false” or unset to truncate. Only relevant when using file exporter for traces. Defaults to “false”.

  • OMNI_OTEL_METRICS_FILE_PATH (/observability/metrics/file/path): Sets the file path for metrics output when using the file exporter. Only relevant when OTEL_METRICS_EXPORTER is set to “file”. Falls back to OMNI_OTEL_DEBUG_FILE_PATH if not set.

  • OMNI_OTEL_METRICS_FILE_APPEND (/observability/metrics/file/append): Controls whether the metrics file should be appended to or truncated. Set to “true” (case-insensitive) to append, “false” or unset to truncate. Only relevant when using file exporter for metrics. Defaults to “false”.

  • OMNI_OTEL_LOGS_FILE_PATH (/observability/logs/file/path): Sets the file path for logs output when using the file exporter. Only relevant when OTEL_LOGS_EXPORTER is set to “file”. Falls back to OMNI_OTEL_DEBUG_FILE_PATH if not set.

  • OMNI_OTEL_LOGS_FILE_APPEND (/observability/logs/file/append): Controls whether the logs file should be appended to or truncated. Set to “true” (case-insensitive) to append, “false” or unset to truncate. Only relevant when using file exporter for logs. Defaults to “false”.

Export Behavior Configuration#

  • OMNI_OTEL_SYNCHRONOUS_EXPORT (/observability/synchronous): Forces synchronous export for all telemetry data (traces, metrics, logs). Set to “true” (case-insensitive) to export synchronously, which can impact performance but ensures data is written immediately. Defaults to “false” for asynchronous export.

  • OMNI_OTEL_LOGS_SYNCHRONOUS_EXPORT (/observability/logs/synchronous): Forces synchronous export specifically for logs only. Set to “true” (case-insensitive) to export logs synchronously. Takes precedence over the global synchronous export setting for logs. Defaults to “false”.

Metrics Behavior Configuration#

  • OMNI_OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PER_INSTRUMENT (/observability/metrics/temporalityPerInstrument): Allows the aggregation temporality to be explicitly set on a per-instrument-type basis. The value must be a comma, semicolon, colon, or bar (|) separated list of key=value pairs where the key is an instrument type name and the value is the aggregation temporality to use for that instrument type. Both the instrument name and temporality value are case-insensitive. Wildcard characters are not supported. Valid instrument names are “counter”, “histogram”, “upDownCounter”, “gauge”, “observableCounter”, “observableUpDownCounter”, and “observableGauge”. Valid temporality values are “delta”, “cumulative”, or “lowMemory”. The aggregation temporality values have the same behavior as described in OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE. For lowMemory temporality, counter, histogram, and gauge instruments use delta aggregation, while upDownCounter and all observable instruments use cumulative aggregation. Example: “histogram=delta,counter=cumulative” sets histogram instruments to delta and counter instruments to cumulative. The aggregation temporalities set with this configuration affect all instruments of the specified type; there is no way to control aggregation temporality on a per-instance basis. Defaults to using the global preference for all instrument types.

SDK Behavior Configuration#

  • OMNI_OTEL_ENABLE_INTERNAL_LOGGING (/observability/enableInternalLogging): Enables internal logging from the OpenTelemetry SDK and the omni.observability.core library. Set to “true” (case-insensitive) to enable internal logging, which will redirect OTel SDK log messages to the ‘otel.sdk’ log channel using a custom log handler. Defaults to “false”. Note that when disabled (the default), the behavior is not compliant with the OpenTelemetry configuration specification, which requires internal SDK logging to be enabled at the info level by default. The non-compliant default avoids unnecessary or undesirable logging being emitted.

  • OMNI_OTEL_ALLOW_DEFAULT_ENDPOINT (/observability/allowDefaultEndpoint): Controls whether default endpoints should be provided for each telemetry signal type when specific endpoints are not configured. Set to “true” (case-insensitive) to enable default endpoints, which will be http://localhost:4318 for HTTP protocols (http/protobuf or http/json) or http://localhost:4317 for gRPC protocol. Defaults to “false”. Note that when disabled (the default), the behavior is not compliant with the OpenTelemetry configuration specification, which requires default endpoints to be provided for all signals. The non-compliant default allows individual signal types to be disabled by default, saving on processing time for connections that would likely fail anyway.

Shutdown Configuration#

  • OMNI_OTEL_TRACES_SHUTDOWN_FLUSH_TIMEOUT (/observability/traces/shutdownFlushTimeout): Sets the shutdown flush timeout for traces in milliseconds. Controls how long to wait for traces to flush during application shutdown. Longer timeouts increase reliability but delay shutdown. Default is 20000 (20 seconds).

  • OMNI_OTEL_METRICS_SHUTDOWN_FLUSH_TIMEOUT (/observability/metrics/shutdownFlushTimeout): Sets the shutdown flush timeout for metrics in milliseconds. Controls how long to wait for metrics to flush during application shutdown. Longer timeouts increase reliability but delay shutdown. Default is 20000 (20 seconds).

  • OMNI_OTEL_LOGS_SHUTDOWN_FLUSH_TIMEOUT (/observability/logs/shutdownFlushTimeout): Sets the shutdown flush timeout for logs in milliseconds. Controls how long to wait for logs to flush during application shutdown. Longer timeouts increase reliability but delay shutdown. Default is 20000 (20 seconds).

Distributed Tracing Configuration#

  • OMNI_OTEL_TRACES_EXTERNAL_TRACE_ID (/observability/traces/externalTraceId): Sets an external traceparent ID for distributed tracing. Specifies an external traceparent ID that should be used as the starting trace context for this process. This allows distributed tracing across multiple processes by propagating the trace context. The format follows the W3C Trace Context specification: “version-traceId-parentId-traceFlags” (e.g., “00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01”). When set, all spans created in this process will be part of the specified trace. This is useful for connecting traces across process boundaries or when launching child processes that need to participate in the same distributed trace.

Additional Carbonite Specific Settings#

omni.observability-otel.plugin provides additional Carbonite-specific configuration options. These settings are only available through the Carbonite settings system (not as environment variables) and provide fine-grained control over how Carbonite log messages are filtered and exported through OpenTelemetry.

Logs#

The following settings control the behavior of the OpenTelemetry log listener that integrates with the Carbonite logging system:

  • /observability/logs/emitLevel: Sets the minimum log severity level for messages to be emitted via OpenTelemetry. This setting establishes a global threshold that applies to all log channels unless overridden by channel-specific settings. Valid values are either integer severity levels or string severity names. String values can be “verbose”, “info”, “warn”, “error”, or “fatal”, which correspond to Carbonite’s five severity levels. Integer values use Carbonite’s severity scale where “verbose” is -2, “info” is -1, “warn” is 0, “error” is 1, and “fatal” is 2. This setting can be higher or lower than the core Carbonite logging system’s level (configured via /log/level). If the OTel emit level is lower than the core logging level, messages below the core logging level will never reach the OTel log listener. If the OTel emit level is higher than the core logging level, messages will be emitted by the core logging system but filtered out by the OTel log listener. This, for example, allows you to send verbose messages to local log files while only transmitting error-level messages to remote observability backends. This setting defaults to “warn” (level 0). This setting will be reloaded if it is changed programmatically at runtime.

  • /observability/logs/channels/: Specifies emit level overrides for specific log channels. Each key under the /observability/logs/channels/ settings branch represents a channel name pattern, and the value is the emit level for matching channels. The pattern supports wildcards where * matches zero or more characters and ? matches exactly one character. Valid values for the emit level are the same as for /observability/logs/emitLevel (integer or string severity name). Channel-specific settings override the base emit level for matching channels. If multiple patterns match a given channel name, the first matching pattern takes precedence. This allows fine-grained control over which channels emit which severity levels. For example, you might configure "/observability/logs/channels/omni.kit.*" = "error" to only emit error and fatal messages from Kit channels, while "/observability/logs/channels/omni.telemetry.*" = "info" emits info and above from telemetry channels. This defaults to an empty settings branch (no channel-specific overrides). This setting will be reloaded if it is changed programmatically at runtime.

  • /observability/logs/adoptLoggingChannelSettings: Boolean setting that controls whether the OTel log listener should automatically adopt channel settings from the main Carbonite logging system. When set to true, the OTel log listener reads all channel patterns and levels from the /log/channels/ settings branch at startup and uses them as the initial channel filters. This means any channel-specific log levels configured for the core logging system (via settings like /log/channels/omni.kit.app = “verbose”) will also apply to OTel log export. The adoption happens only at initialization time, so after startup the OTel channel settings become independent. Any channel-specific settings explicitly defined in /observability/logs/channels/ will override the adopted settings even if this option is true. This setting is useful for maintaining consistency between local log files and remote log collection without duplicating configuration. This defaults to false.