Using the ‘omni.structuredlog’ Tool

Overview

In order to use structured logging through the omni.structuredlog.plugin plugin most easily, a schema should be passed through the omni.structuredlog tool. This tool generates C++, Python bindings (through pybind11), or pure Python code to help with controlling and emitting events for a given schema. The main intention of the omni.structuredlog.plugin plugin is to provide machine-digestible data events that strictly adhere to a known schema. The design of the tool and plugin is such that each module, plugin, extension, app, etc can have one or more schemas related to it that are used to emit events. This allows for the separation of release cycles between products and extensions since there is not just one overall schema for the entire Omniverse ecosystem.

The omni.structuredlog tool is intended to be used as part of the build toolchain for a product app, plugin, extension, etc. The tool can generate multiple outputs from a single schema input. The outputs that are provided depend on the needs of the project. The tool can produce the following types of outputs:

• A C++ header file that implements a helper class for a single schema. This helper class provides an easy way to programmatically enable or disable individual events or the full schema, and helper functions to emit events as needed. Each generated header file will also include a macro to help simplify emitting each type of event for the schema. Using this generated helper class is typically the most efficient and fastest way to emit events from C++ code. See below for how to integrate this header into an app.

• A pybind11 Python bindings C++ header. This header provides helper functions that implement bindings for each of the events in a single schema. This will also include helper functions to add bindings for any enums, structs, or special values that may be required to emit events in the schema. See below for how to integrate this header into a pybind11 module.

• A pure Python module that implements helpers for use with the omni.structuredlog.plugin plugin. Using this module does not require creating a C++ bindings module or C++ header for the schema. This output is most suitable for Omniverse Kit extensions that only have Python components. See below for how to integrate this module into a Python project.

• A JSON version of the schema. This will comply with the JSON schema specification and will describe the schema in a way that can be used with a JSON validator. Events emitted with any of the above generated code can be validated against this schema to ensure they have the correct contents and structure. All events in these schemas are assumed to be strict and not allow additional or optional fields. See below for an explanation of when and where this JSON schema is intended to be used.

Using The Tool in an Omniverse Project

The omni.structuredlog tool is located in the Carbonite SDK packages. It should be referenced from that location for every project that needs to use it. Note that the project does not necessarily need to be a Carbonite based app in order to use Carbonite structured logging however. See below for an explanation of using this tool and the omni.structuredlog.plugin plugin in ‘standalone’ mode.

To include the tool in a premake5 based build project, a couple additions are required in the lua scripts:

• Add the --fail-on-write and --skip-structuredlog-format command line options to the top level repo.toml file. These options are mainly used in CI setups to allow the tool to fail a build if a generated file changes on the CI agent during the build or to skip code formatting if desired. The --fail-on-write option should be used if the project commits the generated files to the repo. If the generated files are always created as needed during the build, this option can be omitted. To register these options with the repo_build tool, simply add these lines to the [repo_build] section of the config file:

  [[repo_build.argument]]
  name = "--fail-on-write"
  help = """
      When enabled, any code generation tool will fail if it needs to write any changes to disk.  This is intended to
      be used on CI systems to ensure that all code changes have been included in any given MR.  If a build step does
      fail due to this option, a local build should be performed first before committing changes to an MR branch."""
  kwargs.required = false
  kwargs.nargs = 0
  extra_premake_args = ["--fail-on-write=1"]
  platforms = ["*"] # All platforms.
  
  [[repo_build.argument]]
  name = "--skip-structuredlog-format"
  help = """
      When enabled, the code formatting step of the 'omni.structuredlog' tool will be skipped for all schema code
      generation.  Note that if the destination file is committed to the git repo, this will likely result in
      changes being detected on the file if it was previously formatted.  This option is intended to be used
      to work around problems with the code formatter in situations where git is not available (ie: inside a
      docker container)."""
  kwargs.required = false
  kwargs.nargs = 0
  extra_premake_args = ["--skip-structuredlog-format=1"]
  platforms = ["*"] # All platforms.
  

  Note that if these options are not needed in the project builds locally or in CI, this step can be skipped.

• Register the tool with your project. Depending on the project’s setup, this can be done either at the top level premake5 script or closer to the actual usage location. For example, for Omniverse Kit extensions, the tool should be registered in the premake5.lua script for the extension itself. To register this call, the following two calls are needed in the Lua script:

  dofile('tools/omni.structuredlog/omni.structuredlog.lua')
  setup_omni_structuredlog("./")
  

  The parameter to the dofile() call should be the location of the omni.structuredlog.lua script that is located in the Carbonite SDK package. This location can vary from one project to another, but is typically found at _build/target-deps/<carb_sdk_package_name>/tools/omni.structuredlog/omni.structuredlog.lua where <carb_sdk_package_name> is either carb_sdk or carb_sdk_plugins.

  The parameter to the setup_omni_structuredlog() call should be the location of the Carbonite SDK where the tool is located. This location can vary from one project to another, but is typically found at _build/target-deps/<carb_sdk_package_name>/ where <carb_sdk_package_name> is either carb_sdk or carb_sdk_plugins.

Once these are added to the project, calls to the omni_structuredlog_schema() function can be added to projects as needed. See Omniverse Telemetry Walkthrough for more information on adding a schema helper generation step to a project.

Using the Tool Manually

The omni.structuredlog tool can also be used manually on the command line to generate helper code as needed. This method is discouraged in Omniverse Kit based projects however since the integration already exists in the premake5 scripts. This manual method of calling into the tool can however be used in projects that are not based on Omniverse Kit or Carbonite, or projects that use a build system other than premake5.

To use the tool manually, there are two shell scripts present in the tool’s folder in the Carbonite SDK package. These are located in the tools/omni.structuredlog/ folder in the package. There is a Windows batch script called structuredlog.bat and a Unix shell script (using Bash) called structuredlog.sh. These can be used to directly invoke the Python tool structuredlog.py. To get the most up-to-date usage information for the tool’s arguments, simply run the appropriate shell script for the platform with the --help option. Note that these shell scripts expect the packman tool to be present in order to run the tool through Python.

Arguments to omni_structuredlog_schema()

The tool’s integration in the premake5 based build system in Omniverse Kit and Carbonite based apps provides a function called omni_structuredlog_schema() that can be used within a project block to trigger a prebuild step to generate schema helper code as needed. This is the preferred method of invoking the tool where possible. A single call to omni_structuredlog_schema() in a project can be used to generate outputs for one or more schemas. The parameter to the function is either a single object or a table of objects. Each object specifies a single schema and its generated outputs and other options. Each object that is passed to the function can contain the following members:

• schema: [required] A string value providing the path to the schema file to generate code for. This can be either a relative or absolute path to the given schema file. This can be either a JSON schema file or a .schema file. See Structured Log Message Schemas for more information on the .schema file format and JSON schema files.

• cpp_output: [optional] A string value providing the path to generate the C++ helper header file to. This can be either a relative or absolute path to the given output file. The file name should end in ‘.gen.h’ or ‘.gen.hpp’ to better indicate that it is a generated source file. Note that this value can be used with the py_module value as well if needed, however at least one of the two output values must be present.

• pybind_output: [optional] A string value providing the path to generate the C++ pybind11 helper functions header file to. This can be either a relative or absolute path to the given output file. The file name should end in ‘.python.h’ or ‘.python.hpp’ to better indicate that is a generated source file. Note that if this value is provided, the cpp_output value is also required.

• py_module: [optional] A string value providing the path to generate the pure Python module file to. This can be either a relative or absolute path to the given output file. The file name should end in ‘.py’ and must not contain any periods (‘.’) except for the extension (Python interprets periods in module names as needing a folder separation in the file system). Note that this option can be used with the cpp_output value as well if needed, however at least one of the two output values must be present.

• namespace: [optional] A string value indicating the C++ namespace name to generate all of the C++ header files under. This must be specified as a fully qualified C++ namespace name such as omni::structuredlog. Multiple nested namespaces may be given if needed. Each namespace component must be separated by two colons (‘::’). If this value is not provided, the default namespace for all generated headers is omni::structuredlog

• skip-format: [optional] A boolean value indicating whether code formatting should be applied to the generated code before completing the task. Code formatting is applied by default and makes the generated code much more readable. This option may be used if there are issues with the code formatting setup in a project or if the formatting is not required. Note that if code formatting is disabled through this option, the generated code may be modified unexpectedly with some inputs. This option should only be used when required, and in general should only be used when the generated code is not committed to source control.

• baked: [optional] A boolean value indicating whether the input schema file is already JSON schema compliant. This defaults to disabled. This should be used if a JSON file is used as the input and it is fully compliant with the JSON schema standard. If this is disabled, the compliant JSON schema version of the input file will be output according to the bake_to value. The default behavior is to assume that the input is not a full JSON schema.

• bake_to: [optional] A string value providing the path to generate the JSON schema compliant output file to. This can be either a relative or absolute path to the given output file. The file name should end in ‘.json’. This output file can be used in a JSON validator to verify a given event is correctly formatted.

Notes on Schema Generation Projects

• Schemas should be generated in their own projects that other projects depend on. This helps to work around a bug in premake5 that causes prebuild steps to be skipped under make for Utility type projects. Due to this bug workaround, a call to omni_structuredlog_schema() in a project will change its ‘kind’ to either StaticLib or Utility depending on the platform. To avoid a dependent project’s ‘kind’ property being implicitly and unexpectedly changed, it is best to put the code generation calls in their own project.

Integrating Generated Code Into Projects

The omni.structuredlog tool can generate four types of outputs (either individually or multiple at once). One of those outputs is the baked JSON schema that represents the input schema. This file doesn’t get integrated directly into the build project anywhere, but is integrated into another repo later. The generated C++ header files get integrated into build projects to provide functionality needed for the given schema. The pure Python module can be imported into a Python extension or project along with some Carbonite bindings modules.

The generated files can be used in the following ways:

• C++ helper header file: This header file can be included in any Carbonite based project to assist in emitting events that conform to the schema it was generated for. Almost all uses of this header will purely only interact directly with the helper macros defined at the top of the generated header. The defined helper class is used internally in those macros. The other main use for the C++ header is to allow events or the full schema to be enabled or disabled as needed. When a schema or event is disabled, any attempt to emit the disabled event is simply silently ignored.

  When the generated header file is included in a compiled C++ module, the schema itself is automatically registered with the omni.structuredlog.plugin plugin when the module initializes. There are no additional steps required to get the schema or the generated helper class to be initialized. This header file may be included from multiple C++ compile units within the module as needed. The schema will only be registered once on startup.

• C++ Python bindings header file: This header file can be included in any Carbonite Python bindings module to expose helper functions for the schema to Python scripts. Generating this header also requires that the C++ helper header file be generated. Once created, this bindings helper header file can be included in a C++ source file of the bindings module. All that is required after that is to call each of the helper functions in the generated header from within a PYBIND11_MODULE() block and pass in the module object (often simply called m). Each of these helper functions will define bindings for any functions, classes, enums, and structs that are related to the schema.

  Note that each helper function in the generated header is intentionally left separate. This is so that additional custom bindings can also be added to each defined binding object as needed. Each helper function will return a pybind11 object that defines the bindings for the function, class, enum, or struct. This can then be used to add new custom bindings to the object as needed. Most bindings modules will not have to do this however. In most cases it will be sufficient to simply call each of the functions in the generated header.

• Pure Python module file: This Python module file can be imported into a Python script, extension or module to provide the functionality of the schema it was generated from. This Python module can be imported as needed, but also requires several Carbonite bindings modules and plugins in order to work properly. The import line for it depends on how the module is named and where it is located relative to the script importing it, so that is left as an exercise for the implementor. Currently these generated Python modules only provide helper functions to emit the events on the schema. They do not also allow the schema or its events to be enabled or disabled.

  When the generated Python module is imported successfully, it will register its schema when it initializes. Doing this unfortunately requires that the full JSON schema be part of the Python module itself (as data). The drawback to using this kind of integration is that the full schema will be exposed as plain text in the generated module. For many applications, this may not be an issue however.

• generated JSON schema: This JSON file is generated from the original .schema file and contains a full description of the schema itself and all its events in great detail. This JSON schema is suitable for use in a JSON validator to verify that an event conforms to the expected formatting and structure. This is not required by the build of any given project, but can be used by the Omniverse telemetry transmitter app.

  The telemetry transmitter app downloads a package of approved schemas when it starts up. In order for any given event to be sent to the transmitter’s destination (configurable as needed), it must first be validated against one of the approved schemas. The approved schemas package for NVIDIA’s telemetry endpoint is currently managed internally. Please see Telemetry Transmitter Options for more information on how to configure the telemetry transmitter to fit other needs.

Standalone Mode For omni.structuredlog.plugin

The omni.structuredlog.plugin plugin can also operate in a ‘standalone’ mode. In this mode it does not require any of the other Carbonite plugins or the Carbonite framework. This mode is currently only tested and verified working in C++ projects however (in theory the omni.structuredlog.plugin Python bindings module should also work in this mode but is untested).

When running in standalone mode, the omni.structuredlog.plugin module from any build can be used as it is. The only differences in use are:

• include/omni/structuredlog/StructuredLogStandalone.h should be used to import and initialize the library instead of the Carbonite framework. Please read the documentation in this header before using the initialization helper class in it.

• Schemas that are part of the module that calls omni::structuredlog::StructuredLogStandalone::init() will be automatically registered during the call. Other modules will need to manually register their schemas.

• The C++ header files that are generated by the omni.structuredlog tool are expected to still be used in the usual way (ie: as described above in Carbonite or Kit based apps).

• Additional structured logging interfaces may be acquired using the omniGetStructuredLogWithoutAcquire() function defined in include/omni/structuredlog/StructuredLogStandalone.h. The returned object will always be a pointer to an omni::structuredlog::IStructuredLog interface, but that can be cast to the other interface types using omni::core::ObjectPtr::as<>(). However, note that either the include/omni/structuredlog/StructuredLogStandalone.h header must always be included before other structured log interface headers or the macro STRUCTUREDLOG_STANDALONE_MODE must be set to 1 before including any other interface headers.