When developing an app, data about user activity, performance, errors, hardware configurations and many other things can be useful to analyze. This data can be gathered through telemetry events. Telemetry in Omniverse starts with structured logging. This provides a way for an application to emit messages with structured data to a local log file at a minimal performance cost. Once the structured log events have been emitted to the local log file, a log consumer tool can be used to inspect individual events and send selected messages for consenting users to a collection servers for analysis. This helps future design efforts to be better informed by real-world usage. Events that aren’t meant to be sent to collection servers simply remain in the local log file(s).
A fundamental tenant of the telemetry system is ensuring that the user’s privacy is respected. Diagnostic data required to develop a high-quality product is anonymized. Telemetry data which may contain personally identifiable information (PII) is not emitted unless the user has provided consent. By default, it is assumed the user does not consent to providing PII. Furthermore, the user is able to toggle between providing or denying consent at any time. Relaunching individual apps may be required for new consent settings to take effect.
Sending telemetry events starts with emitting a structured log event from your app. This is done using the IStructuredLog interface and its related tools and macros. To get started with IStructuredLog, the Omniverse Telemetry Walkthrough will guide you through the full process of adding structured log events to an Omniverse app.
If you want a sample integration to look at, example.structuredlog (source/examples/example.structuredlog) uses IStructuredLog from both C++ and python code. This example provides a schema for a simple event, the code generation for that schema and code to send the event in both C++ and python.
Omniverse has a built-in structured log messaging system called
Any app or component based on Carbonite can make use of this system.
The structured log system has three parts to it:
The messaging core plugin that provides
Messaging code generated from a JSON schema.
The message transmission system.
The messaging core plugin implements the
IStructuredLog interface and
all of its related interfaces. This plugin is implicitly loaded by the process
when the Omni core is started up (through OMNI_CORE_INIT()). Once the core
has initialized, any telemetry events can be emitted from C++ simply by using the
helper macros defined in a schema’s generated header file.
The generated messaging code is integrated directly into an Omniverse app,
while the messaging core plugin is shipped alongside an Omniverse app.
The message transmission system is a separate component and can be integrated
wherever is logical for a given project. Omniverse Kit uses a telemetry extension
omni.kit.telemetry) to start up the telemetry transmitter.
Specifics of NVIDIA’s implementation of the Omniverse telemetry pipeline can
be read at Omniverse Telemetry Implementation Details.
For projects needing a separate telemetry endpoint, there are several options
available other than reimplementing
omni.structuredlog.plugin; these can be
seen here: Alternative Endpoints and Usages.
Structured Log Schemas
Each Omniverse component that wants to send telemetry events to the data servers or even write structured log messages to the local log files must describe each of its events with a JSON schema. The same JSON schema file will be used to produce the generated telemetry source code and to ‘install’ in the message transmission system so it can validate messages that need to be sent to the data servers. This ensures that events with PII are only sent to the collection servers when consent has been given. A guide on how to write these schemas, as well as documentation on our Omniverse-specific features, can be seen in Structured Log Message Schemas.
For C++ apps, the
omni.structuredlog tool is used to generate a C++ header
from a JSON schema.
This generated header provides an interface that can be used to easily emit structured
Once generated, this header may be shared between multiple modules and source files.
The schema defined by the generated header file will be automatically registered when
the plugin or extension is loaded.
Python apps aren’t required to generate any code; they are able to pass their
schema into the
omni.structured python bindings and this will register it.
omni.structuredlog can be used to validate your schema, then generate a python
module that registers your schema and provides helper functions to send each event.
For mixed C++ and python apps,
omni.structuredlog can also generate python
bindings for the generated C++ code.
This may be more convenient for some use cases than registering the schema in
See Pure Python Projects for more information on this.
The tool is simple to integrate into a build system; an integration for premake is provided (this is demonstrated in the walkthrough Pure Python Projects). The tool creates a consistent set of macros to emit structured log events which will have minimal performance impact when structured logging is disabled (parameters will not be evaluated when structured logging is disabled). This way, the app itself does not need to worry about embedding the JSON schema itself as either code or data to be processed on module startup and the same schema used to generate the code can also be used to validate its messages later.
For more on using this tool and integrating the generated code, see Code Generation and Integration below.
Once integrated into a module, the schema represented by the generated code can be registered with the structured log core and new event messages can be sent using the various helper macros in the generated code. Once sent, the events and their information will be stored in the structure log core’s queue to be processed at some point in the near future. Once an event has been processed, it is written to the log file for either the schema or the app (depending on the event’s flags).
Message Transmission System
Once an event message has been written to a local log file, the structured log core’s job is done. From this point on,
it is up to another messaging system to consume the messages in the log and do something with them. In Omniverse
apps, the job of sending telemetry events to the data servers is handled by the transmitter app. While it is running
it will find and process all the logs from all running Omniverse apps. For every message that is read from a log
file, a schema will try to be found to validate it. If the schema exists and the message’s
payload matches the schema, that message will be sent to the data servers. If the message does not match any
schema, the message will simply be rejected and remain in the local log file, possibly to be consumed by another
Other non-telemetry messaging systems may also be running locally. These will consume the local log files in the same way except instead of simply sending validated event messages to the data servers, they would take other actions on them. The set of possible actions and other consumer systems is currently undefined.
Code Generation and Integration
Once a schema has been created for a new set of events, the structured log code
omni.structuredlog can be used to validate the schema and
generate the integration code.
For a C++ integration, this tool should be integrated into the build system;
see Omniverse Telemetry Walkthrough for a guide on how to do this.
For a pure-python integration, it is recommended to generate a python module as
a starting point.
Once the schema’s code has been generated and integrated into a module, app, or
component, the schema should automatically register with the structured logging
The C++ headers do this as part of the initialization of Omni Core for the
The generated python modules register their schemas as part of their module
initialization (note that this requires
omni.core to be imported first).
Note that a single schema may be registered multiple times without issue.
If the same schema is registered again, the call will just be ignored and the
same result as the original registration will be returned.
This is done intentionally to allow for the possibility of multiple modules
within an app using the same schema.
Once your schema has been registered with the structured log core, events may be
The C++ code does this using the set of macros at the top of the generated header;
macros are used so that parameters won’t be evaluated on disabled telemetry events.
The python bindings for the C++ header expose one class that offers a
function for each structured log event.
The generated python module also offers a
*_send_event() function for each
structured log event.
For more detailed information on how this tool works and how it should be integrated with a project, please see Using the ‘omni.structuredlog’ Tool.