Omniverse Telemetry Walkthrough

Adding Structured Logging

This guide covers the process of adding a new structured log schema to an Omniverse application. The end result of this walkthrough should be that your app can emit structured log events for your schema to the local log files. This guide does not cover the process for having your schema approved by legal or having the events sent to the collection server.

Adding Structuredlog to repo_man

This is not required when using structuredlog through premake. This is useful if you want to run the omni.structuredlog tool directly.

Add the omni.structuredlog from the carb_sdk package to repo.toml to have the tool show up as an option when you launch repo_man. This is recommended over launching structuredlog.sh or structuredlog.bat directly because repo_man is aware of your repo structure.

Note that the following code assumes that you pulled the carb_sdk+plugins package with a link path to _build/target-deps/carb_sdk_plugins.

[repo]
extra_tool_paths = [
    "${root}/_build/target-deps/carb_sdk_plugins/tools/omni.structuredlog/",
]

Enable Structured Logging

To use structured logging, you will need to enable the setting: /structuredLog/enable. This setting by default is disabled to prevent applications from unexpectedly sending telemetry data, since some events are sent automatically. For example, a com.nvidia.carbonite.processlifetime.event is sent on framework startup and shutdown.

Add this to your application’s configuration file to enable telemetry:

[structuredLog]
    "enable" = true

In an Omniverse Kit-based app, this step is already done for you by default. However, one additional step is required in Kit-based apps: enable the omni.kit.telemetry extension. When this extension is enabled, it will automatically enable telemetry and structured logging, emit events for Kit startup (ie: time taken until the UI and renderer is ready for the user), emit events for system hardware information, and launch the omni.telemetry.transmitter app. All that remains in this situation is to integrate new schemas to emit any new events that are needed to help analyze the usage and performance of any given component of the Kit- based app (ie: an extension).

Note that integrating a new schema into a given Kit extension or updating to a new version of that schema does not require a new release of the full Omniverse app, just the extension that is affected. The Omniverse structured logging and telemetry system was specifically designed around decoupling each plugin’s or extension’s use of structured logging from the rest of the app.

The last step in a Kit-based app is to create and integrate the new schema that will be used to emit new events. This process is described below.

Before Creating a Schema

Before a new schema is created for your project, it is a good idea to talk to the telemetry team to discuss or come up with a plan for the data you need collected and how you would need it analyzed.

Often times these discussions help:

  • To streamline the data that needs to be produced.

  • Find potential holes in the data that would be needed for analysis.

  • Check whether certain proposed field names would cause conflicts with other existing fields in other schemas.

  • Figure out which pieces of data can and cannot be collected (ie: for privacy reasons).

  • Save development iteration time.

Creating a Schema

The first step to hooking up structured logging is creating a schema for your events. The schema will be used to generate code to send your messages. See Structured Log Message Schemas for a guide on creating your schema. The example schema from example.structuredlog is a good starting point for your new schema. These schemas can either be written in JSON or as a Python dictionary. Python dictionary format is very similar to JSON but it has several benefits:

  • Comments are supported. Being able to comment on lines in the schema can be helpful in explaining to other readers and developers why or how something was done the way it was. In contrast, JSON does not allow any comments at all.

  • Strings can be line wrapped. This is especially useful for the “description” fields for each of an event’s properties since it will be more easily readable. JSON requires that all string values occupy a single line regardless of content or length.

  • Trailing commas are accepted. Trailing commas on the last item in an object or array is a common practice and habit to help minimize code changes when adding new properties or object entries. JSON parsers will generate an error if a trailing comma is found.

Note that you can only include Python literals in this format; you cannot execute code–this is a purely data based format.

{
    "name": "example.structuredlog",
    "version": "1.1",
    "namespace": "com.nvidia.carbonite.example.structuredlog",
    "description": "Example schema to demonstrate how to use IStructuredLog."
                   " This sends a dummy startup time event.",
    "flags": [ "fSchemaFlagAnonymizeEvents" ],
    "events": {
        "startup": {
            "privacy": {
                "category": "performance",
                "description": "example event, so categorization is arbitrary"
            },
            "description": "Marks when an app was started and how long startup took."
                           " This data is gathered to demonstrate the telemetry system.",
            "flags": [],
            "properties": {
                "startupTime": {
                    "type": "uint64",
                    "description": "time to startup in nanoseconds"
                },
                "registerTime": {
                    "type": "uint64",
                    "description": "time to register this schema in nanoseconds"
                },
                "exampleString": {
                    "type": "string",
                    "description": "an example string parameter"
                },
                "example.namespace": {
                    "type": "string",
                    "description": "a test value."
                },
                "Resources.list": {
                    "type": "object",
                    "description": "test object",
                    "properties": {
                        "app.namespace": {
                            "type": "string",
                            "description": "namespace value"
                        },
                        "app.name": {
                            "type": "string",
                            "description": "app name"
                        },
                        "app.instance.id": {
                            "type": "string",
                            "description": "instance ID"
                        },
                        "app.version": {
                            "type": "string",
                            "description": "app version"
                        }
                    }
                }
            }
        },
        "standardStreamOut": {
            "privacy": {
                "category": "performance",
                "description": "example event, so categorization is arbitrary"
            },
            "description": "Demonstrates how to output an event just to stdout.",
            "flags": [ "fEventFlagOutputToStdout", "fEventFlagSkipLog" ],
            "properties": {
                "exampleString": {
                    "type": "string",
                    "description": "an arbitrary test string."
                }
            }
        },
        # Multiple events can be added to the events dictionary.
        # Comments and trailing commas are fine in this format.
    }
}

Note that if your data is intended to be used for telemetry, you should avoid creating properties with type object. Hierarchical data is not efficient to manage in a database, so telemetry data should be kept flat if possible.

You can validate that your structured logging schema satisfies the requirements of the structured logging system by running the code generator without any output file specified.

./tools/omni.structuredlog/structuredlog.sh --baked my_schema.json
./tools/omni.structuredlog/structuredlog.sh my_schema.schema

The structured logging system has been designed following Carbonite and Kit’s “ravioli” model. The schemas are no exception to this. The Omniverse ecosystem as a whole is intended to use many separate schemas, not just one schema per app or component. Any component of an Omniverse app may use one or more schemas as needed. These components include plugins, Kit extensions, apps, Python tools, etc. All schemas for all components are collected and registered in the ‘telemetry-schemas’ git repo.

Within one component, a schema may be either kept private or shared with other projects. This is a consideration on a case by case basis depending on the needs of the project and schema. For example, if multiple Kit extensions need to emit the same telemetry event, there may be a benefit to having that schema’s generated code be shared through public headers, libraries, or extensions. Some examples of these different schema sharing models may be:

  • In Kit the omni.kit.collaboration.telemetry extension is intended to provide multiple related extensions (ie: omni.kit.collaboration.*) access to a common event schema. In this particular case the schema itself and its generated code are private to the omni.kit.collaboration.telemetry extension, but that extension provides a set of public helper functions to emit its events.

  • In Carbonite, the include/omni/structuredlog/StructuredLog.ProcessLifetime.json schema is shared publicly along with its generated C++ header file so that other projects may also emit its events as needed. This schema is used internally by the omni.structuredlog.plugin plugin, but may also be used elsewhere without issue or concern.

  • In the omni.structuredlog.plugin plugin, the structuredlog.log_consumer.schema schema is private to the plugin. The schema file itself and all of its generated code are only accessible and known to that single plugin. In this case there was no need to expose it for other components to use.

Pure Python Projects

Structured logging can be integrated into Python projects without needing to add a build step to the project. This is done with the omni.structuredlog Python bindings modules. Running a Python app with this generated module will require at least the carb Python bindings module from the Carbonite package as well. These bindings modules can be found in at the following locations in the carb_sdk+plugins packages:

  • carb bindings module: _build/$platform/$config/bindings-python/carb/_carb.*.*.

  • omni.structuredlog bindings module: _build/$platform/$config/bindings-python/omni/structuredlog/_structuredlog.*.*.

Please make sure to choose the appropriate version for the version of Python that your app needs to run against. For all Kit based apps and extensions, these bindings modules will already be present for Python 3.10 in the build.

You can generate a Python module that performs the event registration for you and adds wrapper functions to more easily send telemetry events. This code generator will also validate the telemetry schema, so you are recommended to use this. The tool to handle this code generation is located in the Carbonite SDK packages (ie: either carb_sdk or carb_sdk+plugins). This shell code from the root of the Carbonite repo or package will generate a Python module from a JSON schema.

./tools/omni.structuredlog/structuredlog.sh my_schema.json --py=generated_python_module.py

As of Carbonite version 157.0, this code generation step for pure Python modules can also be integrated into an app’s or extension’s premake scripts. This is done using the py_module member in a table passed to the omni_structured_log{} helper function. Please see Arguments to omni_structuredlog_schema() for more information on how to use this tool in premake scripts.

An example of this being used in an app’s build can be found in Carbonite’s example.structuredlog.schemas project:

    project "example.structuredlog.schemas"
        location (workspaceDir.."/%{prj.name}")
        omni_structuredlog_schema {
            {
                schema = "source/examples/example.structuredlog/example.structuredlog.schema",
                cpp_output = "source/examples/example.structuredlog/example.structuredlog.gen.h",
                pybind_output = "source/examples/example.structuredlog/bindings-python/example.structuredlog.python.h",
                py_module = "source/examples/example.structuredlog/pure_python_generated_structuredlog.py",
                bake_to = "source/examples/example.structuredlog/example.structuredlog.json",
                namespace = "example::structuredlog",
            },
            {
                schema = "source/examples/example.structuredlog.dynamic/example.structuredlog.standalone.schema",
                cpp_output = "source/examples/example.structuredlog.dynamic/example.structuredlog.standalone.gen.h",
                bake_to = "source/examples/example.structuredlog.dynamic/example.structuredlog.standalone.json",
                namespace = "example::structuredlog",
            }
        }

In this example, the example.structuredlog.schema schema is used to generate a C++ telemetry header, a C++ Python bindings header, a pure Python module, and the ‘baked’ JSON schema from a single schema.

Once you’ve imported this generated Python module, all that’s needed to send a structured log messages is to call one of the helpers. Note that if you pass invalid parameters, the structured log message will not be sent and an error will be logged.

    example.structuredlog_generated.startup_send_event(
        registerTime=(end - start) * 1000,
        startupTime=0,
        exampleString="python example string (from a generated helper)",
    )

Inside the Generated Python Module

The internals of the generated Python module are straightforward. If you want to use the omni.structuredlog API directly, then you will need to write code similar to this.

The first thing done is to import the omni.structuredlog bindings.

import omni.structuredlog
import pure_python_generated_structuredlog

After the bindings have been imported, your schema is registered with the structured logging system directly from the JSON schema. The returned object is a dictionary of the events that were registered; the keys in the dictionary are the short names of the events. Note that this call may raise an exception if your schema is invalid.

    try:
        events = omni.structuredlog.register_schema(schema)
    except Exception as e:
        omni.log.error("failed to register the schema: " + str(type(e)) + " " + str(e))
        return False

With the schema registered, we can then send structured log events. Note that these calls may throw if the dictionary structure is invalid. The generated code wraps each send call with a helper function to simplify usage.

    try:
        omni.structuredlog.send_event(
            events["startup"],
            {
                "registerTime": (end - start) * 1000,
                "startupTime": 0,
                "exampleString": "python example string (from bindings)",
            },
        )
    except Exception as e:
        omni.log.error("exception thrown: " + str(type(e)) + " " + str(e))
        return False

Generating C++ Code

For C++ apps, you will need to generate a C++ header that gives you an API for your schema. The generated code will allow you to send strongly typed structured log events with minimal performance overhead.

The first step in generating a header is to include the structured log build script in your premake5.lua build script:

dofile('tools/omni.structuredlog/omni.structuredlog.lua')
setup_omni_structuredlog("./")

Also include passthroughs for the various options for the omni.structuredlog tool in your repo.toml script:

[[repo_build.argument]]
name = "--fail-on-write"
help = """
    When enabled, any code generation tool will fail if it needs to write any changes to disk.  This is intended to
    be used on CI systems to ensure that all code changes have been included in any given MR.  If a build step does
    fail due to this option, a local build should be performed first before committing changes to an MR branch."""
kwargs.required = false
kwargs.nargs = 0
extra_premake_args = ["--fail-on-write=1"]
platforms = ["*"] # All platforms.

[[repo_build.argument]]
name = "--skip-structuredlog-format"
help = """
    When enabled, the code formatting step of the 'omni.structuredlog' tool will be skipped for all schema code
    generation.  Note that if the destination file is committed to the git repo, this will likely result in
    changes being detected on the file if it was previously formatted.  This option is intended to be used
    to work around problems with the code formatter in situations where git is not available (ie: inside a
    docker container)."""
kwargs.required = false
kwargs.nargs = 0
extra_premake_args = ["--skip-structuredlog-format=1"]
platforms = ["*"] # All platforms.

You can generate code for your schema using the omni_structuredlog_schema function. The following parameters are supported:

  • The schema parameter is the schema you created.

  • The cpp_output parameter is the output path for the C++ header that will be generated from your schema.

  • The pybind_output parameter is the output path for a Python bindings header that can be optionally generated.

  • The namespace argument specifies the C++ namespace for the generated code. This defaults to omni::telemetry, but can be set to any valid C++ namespace name. Multiple nested namespaces may be used by separating them with ::.

  • The bake_to parameter is optional; if this is specified, it will be the path to write the generated JSON schema to for your simplified schema. If you are using a JSON schema, you should instead add baked = true to the arguments list.

See Structured Log Message Schemas for an explanation on the simplified and full JSON schemas.

    project "example.structuredlog.schemas"
        location (workspaceDir.."/%{prj.name}")
        omni_structuredlog_schema {
            {
                schema = "source/examples/example.structuredlog/example.structuredlog.schema",
                cpp_output = "source/examples/example.structuredlog/example.structuredlog.gen.h",
                pybind_output = "source/examples/example.structuredlog/bindings-python/example.structuredlog.python.h",
                py_module = "source/examples/example.structuredlog/pure_python_generated_structuredlog.py",
                bake_to = "source/examples/example.structuredlog/example.structuredlog.json",
                namespace = "example::structuredlog",
            },
            {
                schema = "source/examples/example.structuredlog.dynamic/example.structuredlog.standalone.schema",
                cpp_output = "source/examples/example.structuredlog.dynamic/example.structuredlog.standalone.gen.h",
                bake_to = "source/examples/example.structuredlog.dynamic/example.structuredlog.standalone.json",
                namespace = "example::structuredlog",
            }
        }
    project "example.structuredlog"
        dependson { "example.structuredlog.bindings.python" }

If more than one schema needs to be built for a given project, they may all be specified in a single call to omni_structuredlog_schema by providing an array of argument objects. Any number of schemas may be added in this manner. Having multiple calls to omni_structuredlog_schema in a single project will work on Linux, but due to some bugs in premake will cause only the first schema to build under Visual Studio.

For example, multiple schemas can be added to the build like this:

        omni_structuredlog_schema {
            {
                schema = "source/plugins/omni.structuredlog/structuredlog.log_consumer.schema",
                bake_to = "source/plugins/omni.structuredlog/structuredlog.log_consumer.json",
                cpp_output = "source/plugins/omni.structuredlog/structuredlog.log_consumer.gen.h",
            },
            {
                -- note that this build step intentionally uses a JSON schema directly so that the
                -- 'pre-baked' path of the `omni.strcturedlog` tool is getting consistently tested
                -- in a development environment.
                schema = "include/omni/structuredlog/StructuredLog.ProcessLifetime.json",
                cpp_output = "include/omni/structuredlog/StructuredLog.ProcessLifetime.gen.h",
                pybind_output = "source/bindings/python/omni.processlifetime/StructuredLog.ProcessLifetime.bindings.python.h",
                baked = true
            },
            {
                schema = "include/omni/structuredlog/StructuredLog.Cloud.schema",
                cpp_output = "include/omni/structuredlog/StructuredLog.Cloud.gen.h",
                pybind_output = "include/omni/structuredlog/StructuredLog.Cloud.python.h",
                bake_to = "include/omni/structuredlog/StructuredLog.Cloud.json",
                namespace = "omni::telemetry",
            },
        }

A generated C++ header can either be generated in its own project with a dependency listed in other projects (to ensure build order), or omni_structuredlog_schema() can be called from within the same project that depends on it. If more than one other project depends on the same generated header, bindings, or script, it is often best to generate them from the schema in a separate project then use dependson in the project to ensure the build order. If only a single project uses the generated code, the omni_structuredlog_schema() call is often better placed within the dependent project itself. In both cases the build order will be correct since the code generation step will run as a pre-build step for the project.

Using the Generated Code in C++ Apps

All that is needed for your structured logging schema to be registered is for you to include the generated header in at least one source file. Once omni.core initializes, your schema will automatically be registered and you’ll be able to send structured log events.

For most use cases, you only need to look at the set of macros at the top of the generated header file. Each of these generated macros will emit a single structured log event.

    OMNI_EXAMPLE_STRUCTUREDLOG_1_1_STARTUP(
        startupTime, 0, "example string", "example namespace",
        example::structuredlog::Schema_example_structuredlog_1_1::Struct_startup_Resources_list(
            "app namespace", "app name", "12345", "1.0.0"));

Building the Python Bindings

In addition to the C++ generated header, you can also generate a header that will wrap the C++ API with pybind11, to expose it to Python. This functionality exists because it was the original way of accessing structured logging from Python. This method of adding Python support has a few benefits: the schema won’t be directly included as text in your code, and the performance cost of sending events should be lower.

To add these bindings, you must create a cpp file for these bindings. Unless you have an unusual use case, you can copy the example binding code and change the names.

// Copyright (c) 2018-2021, NVIDIA CORPORATION. All rights reserved.
//
// NVIDIA CORPORATION and its licensors retain all intellectual property
// and proprietary rights in and to this software, related documentation
// and any modifications thereto. Any use, reproduction, disclosure or
// distribution of this software and related documentation without an express
// license agreement from NVIDIA CORPORATION is strictly prohibited.
//
#include "example.structuredlog.python.h"

CARB_BINDINGS("example.structuredlog")

namespace
{

PYBIND11_MODULE(structuredlog, m)
{
    example::structuredlog::definePythonModule_example_structuredlog(m);
}

} // namespace

The Python bindings just need to have a dependson{} call that points to your schema build target. This is done in example.structuredlog using the define_multi_binding_python() function in Carbonite.

    define_multi_bindings_python {
        project_name = "example.structuredlog.python",
        name = "structuredlog",
        namespace = "example",
        folder = "source/examples/example.structuredlog/bindings-python/",
        dependson = { "example.structuredlog.schemas" }
    }

Projects that use define_bindings_python() can create a build target like this:

project "example.structuredlog.bindings.python"
    dependson { "example.structuredlog.structuredlog_schema" }
    define_bindings_python {
        name = "structuredlog",
        namespace = "example"
    }
    files {"source/examples/example.structuredlog/bindings-python/*.*"}

Using the Generated Code Python Bindings

Using the Python bindings is simple. Once you’ve imported your bindings, you just need to create the helper class for your schema.

    inst = example.structuredlog.Schema_example_structuredlog_1_1()

Once you have the class, you can call any of the *_sendEvent() functions to send structured logging events.

    list = example.structuredlog.Struct_startup_Resources_list()
    list.app_namespace = "python app namespace"
    list.app_name = "python app name"
    list.app_instance_id = "2345"
    list.app_version = "1.0.1"
    inst.startup_sendEvent(0, int((end - start) * 1000), "python example string", "python example namespace", list)

Testing Your Structured Log Event Integration

Once the generated C++ or Python code has been integrated into a project, it should be tested. The easiest way to do this is to run the app or component with the new integration and execute past the point where a new event message would have been sent. Once that messaging code has run, at least one new message from the event should arrive in the log file. By default, the log file can be found in the $HOME/.nvidia-omniverse/logs/ folder. On Linux, $HOME will be the user’s home folder. On Windows, this will be the user’s profile folder (ie: $USERPROFILE).

Locate the log folder and search for a message in the log files with the new event’s identifier. The event identifier can be found in the event’s schema by combining the event prefix for the schema and the name of the given event being searched for. If the message can be found in the log file and all of the properties of it are correct, the telemetry integration testing can be started. If the message cannot be found or its content is not correct, simply go back to the code and figure out why it’s not being output correctly by stepping through the call in the debugger.

Testing Your Telemetry Integration

Once the new message has been confirmed to be going to the log file correctly, the job of the app’s integration of structured logging is complete. After this point, it is the job of the telemetry transmitter app omni.telemetry.transmitter to get the messages from the log file(s) and send them up to the appropriate data endpoint. To do this, a few steps need to be taken to get the telemetry transmitter app running. The way it is configured and run depends on how it has been integrated into the host app.

Note that launching the transmitter is already handled in Omniverse Kit by the omni.kit.telemetry extension. However, if a custom configuration is needed to test transmission (ie: run in ‘test’ mode, deliver to a non-default endpoint, or inject test schemas), it will still need to be launched manually. If the omni.kit.telemetry extension tries to launch the transmitter and sees that a copy of the transmitter is already running locally, the new transmitter will simply exit immediately. To a degree it is also possible to reconfigure the transmitter that is launched by omni.kit.telemetry. Any /telemetry/* settings that are passed to the Kit process (either on command line or in a config file) will be automatically passed on to the transmitter process when it is launched. The most common setting to pass in this case would be /telemetry/mode=test.

The most common way of integrating the telemetry transmitter app would be through the Kit SDK. This has an omni.kit.telemetry extension that handles launching the transmitter app on its startup. If an integration with Kit is not being used, it is the host app’s responsibility to appropriately launch the telemetry transmitter. There is a set of helper functions for doing this in include/omni/structuredlog/Telemetry.h. The main function to do this is omni::telemetry::launchTransmitter().

When the telemetry transmitter app will be sending messages to the default Omniverse data endpoints (test or production), authentication may be required. In order to accomplish this, the transmitter must retrieve the current user’s authorization token from the Omniverse Launcher app. If the Launcher app is not running, the transmitter app will go into an idle state until the Launcher becomes available. If the default Omniverse data endpoints are not used, the host app that launches the transmitter is either responsible for getting an appropriate authorization token to the transmitter, or to disable authentication using the /telemetry/authenticate setting.

Before the transmitter can send any new test events to the data endpoint, it must first be given the schema that will validate those events. This schema file is generated by the omni.structuredlog tool using the bake_to option when a ‘.schema’ file is used. Only JSON schemas are accepted by the transmitter tool. By default, the transmitter will download all approved schemas from the schema URL. To test an unapproved schema, the new schema(s) must be injected into the transmitter. This can be done with a debug build of the tool (_build/<platform>/debug/omni.telemetry.transmitter). The /telemetry/schemaFile or /telemetry/schemasDirectory options can be used to specify which set of local JSON schema file(s) get injected into this run of the transmitter app.

Note that running the transmitter for testing purposes often requires that it be launched manually. When doing this, it is easiest to also use the /telemetry/stayAlive setting so that it doesn’t quit on its own once it sees that no Omniverse apps are connected to it. When this setting is used, it is the local user’s responsibility to terminate the transmitter process when it is no longer needed. There is no way to cause it to exit gracefully on its own in ‘stay alive’ mode.

Once the transmitter has been configured correctly, it should start consuming new events that appear in the log file(s). When injecting test schemas, these will by default be sent to the Omniverse staging data table. These events can later be analyzed to make sure they will be useful in investigating the necessary trends from the data.

Registering Your New Schema

Once end-to-end testing has been done on all events in a new schema and all data in the new events are deemed to be useful for analysis, the new schema must be registered in order for data collection to start from internal or public users. This registration requires the help fo the telemetry team.

The registration occurs in the telemetry-schemas git repo. This repo has three submodules for each of the three telemetry modes - test, dev, and prod. Each submodule contains the schemas that have been approved for that particular telemetry mode. Further instructions for registering your new schema are in the ‘readme’ doc on that git repo.

The basic steps in this process are:

  • Add your new JSON schema to the appropriate submodules of the telemetry-schemas repo. This can be done with the collect.{sh|bat} tool in the repo. Note that only JSON files are accepted in here. The JSON version of a .schema file can be generated by the omni.structuredlog tool as needed (outlined above or in more detail in the Using the ‘omni.structuredlog’ Tool documentation). The collect.{sh|bat} tool is documented in the telemetry-schemas repo itself and the latest command line arguments for it can be found by simply running it with no arguments or with --help.

  • Create MR(s) for the new schemas against the specific submodules they will belong to.

  • Review the new schema(s) with the telemetry team and make changes as needed. Note that in almost all cases, any changes to the schema need to come from the original source using the omni.structuredlog tool, not by simply modifying the JSON file in the submodules in the telemetry-schemas repo. If the registered JSON schema does not exactly match the one that was generated and used in the original project (with the minor exception of a couple flags), messages generated against that schema will not be validated properly and therefore not transmitted.

  • Merge once approved.

Once merged, the telemetry team will finish the process by publishing the new schemas to all the known schema packages and open endpoints. This process typically takes a few minutes.

Future Enhancements to Your Schema

Once a schema has been approved, registered, and published, that version is set in stone forever and may not be changed. This does not however mean the schema cannot be modified at all. Fields or events may be added, removed, or to some degree modified in a new version of the schema. This new version would then need to go through the same review and approval process that the original schema went through.

The reasons for this are:

  • A full version history of each schema must be maintained so that older apps out in the wild still have functioning telemetry. If a schema were simply modified for a new app, plugin, or extension build, it could easily invalidate events produced by older builds.

  • The open endpoint support requires that a cumulative schema be registered with the backend data collectors. This cumulative schema is very strict and also maintains a full history of previously registered versions. It also requires that any new version that is registered be fully backward compatible with all older versions.

Due to the requirements of the open-endpoint support, there are also some limitations on what can be modified in a new version of a schema. These are the general guidelines of what can and cannot be done in a new schema version:

  • New fields may be added as long as their names and data types do not conflict with existing field names in other schemas.

  • Existing fields may be removed from events.

  • A field’s type can never be changed. This would cause the cumulative schema registration to fail because it is no longer backward compatible. A new field with a different name and type must be added instead.

  • A field’s name may be changed as long as it does not conflict with other existing field names in other schemas (including older versions of any schema). Renaming a field is essentially the same as removing the old field and adding a new one.

  • An event may be renamed, or removed if it is no longer needed.

  • A schema may be revoked in its entirety if no longer needed or problematic. Note that revoking a schema is not the same as removing or deleting it. The revoked schema must still remain in the telemetry-schemas repo but will not be included in the published schemas packages. A schema or any version of it may never be deleted.