Overview#

The omni.kit.livestream.app extension enables streaming of the entire application framebuffer in Omniverse Kit applications. It captures rendered frames and streams them to remote clients using the streaming implementations provided by the omni.kit.livestream.webrtc extension.

Purpose#

This extension bridges the Omniverse Kit rendering pipeline with remote streaming clients by:

  1. Capturing the application’s framebuffer after rendering

  2. Managing shared frame buffers for efficient streaming

  3. Creating and configuring streaming servers

  4. Supporting both primary and spectator streaming modes

  5. Handling window resizing and frame synchronization

Architecture#

The omni.kit.livestream.app extension consists of these key components:

  • Extension: Main entry point that initializes streaming and manages event subscriptions

  • SharedFrameBuffer: Handles the capture and sharing of rendered frames for streaming

  • Settings Management: Configures streaming parameters through Omniverse Kit’s settings system

The extension integrates into the Omniverse Kit rendering pipeline by subscribing to key events:

  • PreUpdate: For input handling and window resizing

  • PostEndRenderPass: For synchronization with the rendering pipeline

  • PostPresentBuffer: For capturing frames after they’re presented

Key Features#

  • Application-Wide Streaming: Captures and streams the entire application window

  • Multiple Stream Support: Configurable primary stream with optional spectator streams

  • Dynamic Configuration: Customizable stream parameters through settings

  • Efficient Frame Capture: Direct CUDA buffer access for high-performance encoding

  • Synchronization: Proper synchronization with the rendering pipeline

  • Port Management: Automatic port allocation with collision detection and resolution

Configuration Settings#

The extension provides the following configurable settings:

Primary Stream Settings#

[settings.exts."omni.kit.livestream.app".primaryStream]
allowDynamicResize = false  # Allow dynamic resizing of the video stream? May be detrimental to overall streaming performance. Not supported if spectator streams are used.
enableAudioCapture = false  # Enable auto captured audio for this stream? This will capture and stream all audio played by the audio driver, not just the application audio.
enableEventTracing = true   # Enable event trace logging for this stream? These can be used to diagnose issues with the stream, but can grow quite large and use disk space.
enableOpenTelemetry = false # Enable/emit open telemetry for this stream? Set OMNI_OTEL_METRICS_FILE_PATH env var or observability/metrics/file/path setting to output path.
publicIp = ""               # Fixed ip used to transport streaming media. Can help for connecting when server is behind a NAT.
signalPort = 49100          # TCP port used for negotiating a connection. Must be unique for each stream, if not specified an attempt will be made to use an unoccupied port.
streamPort = 47998          # UDP port used to transport streaming media. Must be unique for each stream, if not specified an attempt will be made to use an unoccupied port.
streamType = "webrtc"       # Type of streaming implementation to create. webrtc=standard WebRTC protocol, native=proprietary. Must be defined for the stream to be created.
targetFps = 60              # The target frames per second of the stream. May be adjusted in response to quality of service (QoS) feedback to improve streaming performance.

Spectator Stream Settings#

Spectator streams are configured as an array in TOML. Each [[...]] entry creates one spectator stream. The available settings are the same as for the primary stream, except that allowDynamicResize is always forced to false (spectator streams cannot resize).

[[settings.exts."omni.kit.livestream.app".spectatorStream]]
streamType = "webrtc"       # Must be defined for the stream to be created.
signalPort = 49200          # If not specified, an unoccupied port will be used.
streamPort = 48000          # If not specified, an unoccupied port will be used.

Usage#

Creating a Primary Stream#

The extension automatically creates a primary stream at startup if the streamType setting is defined. You can modify these settings in your application’s .kit file:

[settings.exts."omni.kit.livestream.app".primaryStream]
allowDynamicResize = true
streamPort = 47999
streamType = "native"

using arguments when running your application:

--/exts/omni.kit.livestream.app/primaryStream/allowDynamicResize=true
--/exts/omni.kit.livestream.app/primaryStream/streamPort=47999
--/exts/omni.kit.livestream.app/primaryStream/streamType="native"

or programmatically:

carb::settings::ISettings* settings = carb::getCachedInterface<carb::settings::ISettings>();
settings->set<bool>("exts/omni.kit.livestream.app/primaryStream/allowDynamicResize", true);
settings->setString("exts/omni.kit.livestream.app/primaryStream/streamType", "native");
settings->set<int32_t>("exts/omni.kit.livestream.app/primaryStream/streamPort", 47999);

Creating Spectator Streams#

You can create multiple spectator streams using the TOML array-of-tables syntax. Each [[...]] entry defines a separate spectator stream:

# First spectator stream
[[settings.exts."omni.kit.livestream.app".spectatorStream]]
streamType = "webrtc"
signalPort = 49200
streamPort = 48000

# Second spectator stream
[[settings.exts."omni.kit.livestream.app".spectatorStream]]
streamType = "native"
signalPort = 49300
streamPort = 48002

At runtime, the array entries are indexed starting at 0, so they can also be configured using command-line arguments:

--/exts/omni.kit.livestream.app/spectatorStream/0/streamType="webrtc"
--/exts/omni.kit.livestream.app/spectatorStream/0/signalPort=49200
--/exts/omni.kit.livestream.app/spectatorStream/0/streamPort=48000
--/exts/omni.kit.livestream.app/spectatorStream/1/streamType="native"
--/exts/omni.kit.livestream.app/spectatorStream/1/signalPort=49300
--/exts/omni.kit.livestream.app/spectatorStream/1/streamPort=48002

or programmatically:

// Set up the first spectator stream
settings->setString("exts/omni.kit.livestream.app/spectatorStream/0/streamType", "webrtc");
settings->set<int32_t>("exts/omni.kit.livestream.app/spectatorStream/0/signalPort", 49200);
settings->set<int32_t>("exts/omni.kit.livestream.app/spectatorStream/0/streamPort", 48000);

// Set up the second spectator stream
settings->setString("exts/omni.kit.livestream.app/spectatorStream/1/streamType", "native");
settings->set<int32_t>("exts/omni.kit.livestream.app/spectatorStream/1/signalPort", 49300);
settings->set<int32_t>("exts/omni.kit.livestream.app/spectatorStream/1/streamPort", 48002);

Implementation Details#

Frame Capture Process#

The frame capture and streaming process follows these steps:

  1. During onPostEndRenderPass, the extension prepares/cycles shared frame buffers and initiates copying of the framebuffer

  2. During onPostPresentBuffer, it waits for the copy of the framebuffer to complete, synchronizing with the GPU to ensure frame integrity

  3. Completed frames are streamed to all active servers

Stream Server Management#

The extension manages streaming servers as follows:

  1. By default, it creates the primary stream server if configured correctly

  2. It dynamically creates or updates spectator stream servers based on settings

  3. Each server gets its own unique ports to avoid conflicts

  4. All servers receive the same frame data but can have different clients

Window Resizing#

Window resizing is handled through the resizeAppWindow method, which:

  1. Queries the primary stream for the desired stream resolution

  2. Resizes the application window to match the desired stream resolution

  3. Shared frame buffers are recreated internally if needed to accommodate the new size

Integration with Other Extensions#

The omni.kit.livestream.app extension:

  • Depends on omni.kit.livestream.webrtc for the streaming implementations

  • Depends on omni.kit.renderer.core for frame capture and rendering pipeline integration

  • Works with omni.kit.livestream.messaging for structured message exchange

  • Works with omni.services.livestream.session for session management

Performance Considerations#

For optimal streaming performance:

  • Consider the impact of the target frame rate on CPU and GPU utilization

  • Be cautious with dynamic resizing as it may impact performance

  • When using spectator streams, ensure each has unique port assignments

Version History#

The extension follows semantic versioning and maintains a detailed changelog. This extension was branched from omni.kit.livestream.core in version 8.0.0.

For detailed change history, refer to the CHANGELOG.md file.