Omniverse Telemetry

Overview

When developing an app, data about user activity, performance, errors, hardware configurations and many other things can be useful to analyze. This data can be gathered through telemetry events. Telemetry in Omniverse starts with structured logging. This provides a way for an application to emit messages with structured data to a local log file at a minimal performance cost. Once the structured log events have been emitted to the local log file, a log consumer tool can be used to inspect individual events and send selected messages for consenting users to a collection servers for analysis. This helps future design efforts to be better informed by real-world usage. Events that aren’t meant to be sent to collection servers simply remain in the local log file(s).

A fundamental tenant of the telemetry system is ensuring that the user’s privacy is respected. Diagnostic data required to develop a high-quality product is anonymized. Telemetry data which may contain personally identifiable information (PII) is not emitted unless the user has provided consent. By default, it is assumed the user does not consent to providing PII. Furthermore, the user is able to toggle between providing or denying consent at any time. Relaunching individual apps may be required for new consent settings to take effect.

Quick Start

Sending telemetry events starts with emitting a structured log event from your app. This is done using the IStructuredLog interface and its related tools and macros. To get started with IStructuredLog, the Omniverse Telemetry Walkthrough will guide you through the full process of adding structured log events to an Omniverse app.

If you want a sample integration to look at, example.structuredlog (source/examples/example.structuredlog) uses IStructuredLog from both C++ and python code. This example provides a schema for a simple event, the code generation for that schema and code to send the event in both C++ and python.

Architecture

Omniverse has a built-in structured log messaging system called IStructuredLog. Any app or component based on Carbonite can make use of this system. The structured log system has three parts to it:

• The messaging core plugin that provides IStructuredLog (omni.structuredlog.plugin).

• Messaging code generated from a JSON schema.

• The message transmission system.

The messaging core plugin implements the IStructuredLog interface and all of its related interfaces. This plugin is implicitly loaded by the process when the Omni core is started up (through OMNI_CORE_INIT()). Once the core has initialized, any telemetry events can be emitted from C++ simply by using the helper macros defined in a schema’s generated header file.

The generated messaging code is integrated directly into an Omniverse app, while the messaging core plugin is shipped alongside an Omniverse app. The message transmission system is a separate component and can be integrated wherever is logical for a given project. Omniverse Kit uses a telemetry extension (omni.kit.telemetry) to start up the telemetry transmitter.

Specifics of NVIDIA’s implementation of the Omniverse telemetry pipeline can be read at Omniverse Telemetry Implementation Details. For projects needing a separate telemetry endpoint, there are several options available other than reimplementing omni.structuredlog.plugin; these can be seen here: Alternative Endpoints and Usages.

Structured Log Schemas

Each Omniverse component that wants to send telemetry events to the data servers or even write structured log messages to the local log files must describe each of its events with a JSON schema. The same JSON schema file will be used to produce the generated telemetry source code and to ‘install’ in the message transmission system so it can validate messages that need to be sent to the data servers. This ensures that events with PII are only sent to the collection servers when consent has been given. A guide on how to write these schemas, as well as documentation on our Omniverse-specific features, can be seen in Structured Log Message Schemas.

For C++ apps, the omni.structuredlog tool is used to generate a C++ header from a JSON schema. This generated header provides an interface that can be used to easily emit structured log events. Once generated, this header may be shared between multiple modules and source files. The schema defined by the generated header file will be automatically registered when the plugin or extension is loaded.

Python apps aren’t required to generate any code; they are able to pass their schema into the omni.structured python bindings and this will register it. omni.structuredlog can be used to validate your schema, then generate a python module that registers your schema and provides helper functions to send each event.

For mixed C++ and python apps, omni.structuredlog can also generate python bindings for the generated C++ code. This may be more convenient for some use cases than registering the schema in python. See Pure Python Projects for more information on this.

The tool is simple to integrate into a build system; an integration for premake is provided (this is demonstrated in the walkthrough Pure Python Projects). The tool creates a consistent set of macros to emit structured log events which will have minimal performance impact when structured logging is disabled (parameters will not be evaluated when structured logging is disabled). This way, the app itself does not need to worry about embedding the JSON schema itself as either code or data to be processed on module startup and the same schema used to generate the code can also be used to validate its messages later.

For more on using this tool and integrating the generated code, see Code Generation and Integration below.

Once integrated into a module, the schema represented by the generated code can be registered with the structured log core and new event messages can be sent using the various helper macros in the generated code. Once sent, the events and their information will be stored in the structure log core’s queue to be processed at some point in the near future. Once an event has been processed, it is written to the log file for either the schema or the app (depending on the event’s flags).

Message Transmission System

Once an event message has been written to a local log file, the structured log core’s job is done. From this point on, it is up to another messaging system to consume the messages in the log and do something with them. In Omniverse apps, the job of sending telemetry events to the data servers is handled by the transmitter app. While it is running it will find and process all the logs from all running Omniverse apps. For every message that is read from a log file, a schema will try to be found to validate it. If the schema exists and the message’s data payload matches the schema, that message will be sent to the data servers. If the message does not match any schema, the message will simply be rejected and remain in the local log file, possibly to be consumed by another messaging system.

Other non-telemetry messaging systems may also be running locally. These will consume the local log files in the same way except instead of simply sending validated event messages to the data servers, they would take other actions on them. The set of possible actions and other consumer systems is currently undefined.

Code Generation and Integration

Once a schema has been created for a new set of events, the structured log code generator tool omni.structuredlog can be used to validate the schema and generate the integration code. For a C++ integration, this tool should be integrated into the build system; see Omniverse Telemetry Walkthrough for a guide on how to do this. For a pure-python integration, it is recommended to generate a python module as a starting point.

Once the schema’s code has been generated and integrated into a module, app, or component, the schema should automatically register with the structured logging core. The C++ headers do this as part of the initialization of Omni Core for the component. The generated python modules register their schemas as part of their module initialization (note that this requires omni.core to be imported first). Note that a single schema may be registered multiple times without issue. If the same schema is registered again, the call will just be ignored and the same result as the original registration will be returned. This is done intentionally to allow for the possibility of multiple modules within an app using the same schema.

Once your schema has been registered with the structured log core, events may be sent. The C++ code does this using the set of macros at the top of the generated header; macros are used so that parameters won’t be evaluated on disabled telemetry events. The python bindings for the C++ header expose one class that offers a *_sendEvent() function for each structured log event. The generated python module also offers a *_send_event() function for each structured log event.

For more detailed information on how this tool works and how it should be integrated with a project, please see Using the ‘omni.structuredlog’ Tool.