Overview#

The omni.kit.livestream.aov extension enables streaming of specific render product Arbitrary Output Variables (AOVs) from Omniverse Kit to remote clients. This extension provides a specialized streaming solution for individual viewport rendering outputs, allowing developers to stream specific render passes (such as LdrColor, depth, normals, etc.) rather than the entire application framebuffer.

Purpose#

This extension bridges the Omniverse Kit hydra rendering pipeline with remote clients by:

  1. Intercepting render product AOVs from the hydra rendering system

  2. Creating dedicated streaming servers for each AOV spectator stream

  3. Providing efficient rendering texture copying and format conversion

  4. Managing multiple concurrent streams for different viewports and AOV types

  5. Enabling fine-grained control over what render data gets streamed

Architecture#

The omni.kit.livestream.aov extension consists of several key components:

  • Extension: Main extension class that handles frame events and manages shared streams

  • SharedStream: Manages individual AOV streams, including buffer management and server creation

  • Format Conversion: Efficient texture format conversion (RGBA to BGRA)

  • Settings Management: Configures streaming parameters through Omniverse Kit’s settings system

The extension leverages the hydra rendering pipeline by subscribing to frame recording and rendering events, allowing it to capture AOV data at the optimal point in the rendering cycle.

Key Features#

  • AOV-Specific Streaming: Stream individual render product outputs (LdrColor, depth, normals, etc.)

  • Multi-Viewport Support: Handle AOVs from multiple viewports simultaneously

  • Multiple Stream Support: Create multiple spectator streams for each AOV

  • Format Conversion: Automatic format conversion from source texture formats to streaming-compatible formats

  • Dynamic Configuration: Customizable stream parameters through settings

  • Port Management: Automatic port allocation with collision detection and resolution

Usage#

Basic Configuration#

AOV streaming is configured through Kit settings using the TOML array-of-tables syntax. The general settings path is:

exts."omni.kit.livestream.aov"."<RENDER_PRODUCT>".spectatorStream

where <RENDER_PRODUCT> is the full hydra texture path, e.g. Render.OmniverseKit.HydraTextures.omni_kit_widget_viewport_ViewportTexture_0.LdrColor.

For example, to stream the primary viewport’s LdrColor AOV:

[[settings.exts."omni.kit.livestream.aov"."Render.OmniverseKit.HydraTextures.omni_kit_widget_viewport_ViewportTexture_0.LdrColor".spectatorStream]]
streamType = "webrtc"

Available Settings#

Each spectator stream entry supports the following settings:

  • streamType: Type of streaming protocol ("webrtc" or "native"). Must be defined for the stream to be created.

  • signalPort: TCP port for connection negotiation. If not specified, an unoccupied port will be used.

  • streamPort: UDP port for media transport. If not specified, an unoccupied port will be used.

  • publicIp: Public IP address for external clients (optional, helps when behind a NAT).

  • allowDynamicResize: Allow dynamic resizing of the video stream (default: true for AOV streams, since viewports can be resized by the user).

  • targetFps: Target frames per second.

Multi-Stream Configuration#

Multiple spectator streams can be created for the same AOV by adding multiple [[...]] entries:

# First stream
[[settings.exts."omni.kit.livestream.aov"."Render.OmniverseKit.HydraTextures.omni_kit_widget_viewport_ViewportTexture_0.LdrColor".spectatorStream]]
streamType = "webrtc"
signalPort = 49101

# Second stream
[[settings.exts."omni.kit.livestream.aov"."Render.OmniverseKit.HydraTextures.omni_kit_widget_viewport_ViewportTexture_0.LdrColor".spectatorStream]]
streamType = "webrtc"
signalPort = 49102

At runtime, array entries are indexed starting at 0, so they can also be configured using command-line arguments:

--/exts/omni.kit.livestream.aov/Render.OmniverseKit.HydraTextures.omni_kit_widget_viewport_ViewportTexture_0.LdrColor/spectatorStream/0/streamType="webrtc"
--/exts/omni.kit.livestream.aov/Render.OmniverseKit.HydraTextures.omni_kit_widget_viewport_ViewportTexture_0.LdrColor/spectatorStream/0/signalPort=49101

Multi-Viewport Support#

Different viewports can be streamed independently:

# Primary viewport
[[settings.exts."omni.kit.livestream.aov"."Render.OmniverseKit.HydraTextures.omni_kit_widget_viewport_ViewportTexture_0.LdrColor".spectatorStream]]
streamType = "webrtc"

# Secondary viewport
[[settings.exts."omni.kit.livestream.aov"."Render.OmniverseKit.HydraTextures.omni_kit_widget_viewport_ViewportTexture_1.LdrColor".spectatorStream]]
streamType = "webrtc"

Implementation Details#

Render Pipeline Integration#

The extension integrates with the hydra rendering pipeline through event subscriptions:

  1. Frame Recording Events: Captures AOV data when rendering is complete

  2. Frame Rendering Events: Streams buffered data to clients

  3. Render Product Processing: Iterates through all AOVs in each render product

CUDA Buffer Management#

For optimal performance, the extension:

  • Creates CUDA-accessible buffers for each stream

  • Performs direct texture-to-buffer copying using the render graph

  • Handles format conversion (e.g. RGBA to BGRA, Vulkan only for now)

  • Manages buffer lifecycle to prevent memory leaks

Automatic Resource Management#

The extension automatically:

  • Creates streams only when configured through settings

  • Handles port conflicts by auto-assigning available ports

  • Manages GPU resources across different devices

Integration with Other Extensions#

The omni.kit.livestream.aov extension:

  • Depends on omni.kit.livestream.webrtc for streaming server implementations

  • Depends on omni.kit.renderer.core for GPU foundation and render graph access

  • Depends on omni.hydra.scene_api for render product processing

  • Works with omni.kit.livestream.core through the server factory infrastructure

  • Complements omni.kit.livestream.app by providing AOV-specific streaming

Limitations#

  • No Input Processing: AOV spectator streams are view-only and cannot process client input

  • Format Dependencies: Some format conversions require specific renderers (e.g. Vulkan for certain blit operations)

  • GPU Memory: Each stream maintains its own CUDA buffers, increasing memory usage

Performance Considerations#

For optimal AOV streaming performance:

  • Limit the number of concurrent streams based on available GPU memory

  • Consider the bandwidth implications of streaming high-resolution AOVs

  • Be aware that each AOV stream adds rendering overhead for buffer copying

Version History#

The extension follows semantic versioning and maintains a detailed changelog. Major updates typically align with new Kit versions, while minor updates include feature enhancements and bug fixes.

For detailed change history, refer to the CHANGELOG.md file.