Overview#

The omni.kit.livestream.core extension serves as the foundation for the Omniverse Kit livestreaming capabilities. It provides a server factory infrastructure that enables the creation and management of different types of streaming servers within the Omniverse ecosystem.

Purpose#

This extension acts as a central hub for streaming functionality, allowing other extensions to register and create various types of streaming servers (such as WebRTC or proprietary native implementations) through a consistent interface. The core extension itself does not implement any specific streaming protocol but instead provides the architectural framework for implementing and managing streaming solutions.

Architecture#

The omni.kit.livestream.core extension is built around a server factory pattern, consisting of:

  • IServerFactory Interface: Provides methods to register, deregister, and create streaming server instances based on type.

  • ServerFactory Implementation: Implements the factory interface to manage registration of different server types and create instances on demand.

  • IServer Interface: Defines the common interface that all streaming server implementations must follow, enabling consistent handling regardless of the underlying streaming technology.

Key Features#

  • Server Factory Pattern: Enables extensions to register and create different types of streaming servers.

  • Protocol Agnostic: Works with various streaming protocols (WebRTC, native, etc.) through a unified interface.

  • Performance Optimization: Includes default settings tuned for optimal streaming performance.

  • Extensible Design: Allows new streaming implementations to be added without modifying the core infrastructure.

Usage#

Creating a Streaming Server#

To create a streaming server in C++:

  1. Acquire the server factory interface:

auto serverFactory = carb::getCachedInterface<omni::kit::livestream::IServerFactory>();

  1. Create a server of the desired type:

// Create a WebRTC server
auto server = serverFactory->createServer("webrtc");

// Configure and start the server
omni::kit::livestream::IServer::Config config;
config.width = 1920;
config.height = 1080;
config.targetFPS = 60;
server->start(config);

Implementing a Custom Streaming Server#

To implement a custom streaming server:

  1. Create a class that implements the IServer interface.

  2. Register a factory function with the server factory:

serverFactory->registerFactoryFunction("my_stream_type", MyServer::create);

  1. Users can then create instances of your server using the factory:

auto server = serverFactory->createServer("my_stream_type");

Python Bindings#

The extension provides Python bindings that allow extension developers to create and control streaming servers directly from Python, without needing to compile C++ against the IServer/IServerFactory interfaces.

Quick Start#

from omni.kit.livestream.core import Server, ServerConfig, VideoCodec

# Create and start a server (requires the corresponding protocol extension to be enabled)
server = Server("rtsp")
server.start(ServerConfig(width=1920, height=1080, video_pre_encode=VideoCodec.H264))

# In your render/capture loop:
server.stream_video_cuda_buffer(cuda_ptr, pitch, width, height)

# Or stream pre-encoded video:
server.stream_video_pre_encoded(encoded_bytes, len(encoded_bytes), width, height)

# When done:
server.stop()
server.close()

Streaming with Per-Frame Metadata#

Applications can embed arbitrary per-frame metadata into the video stream. For RTSP servers streaming pre-encoded H.264/H.265, the metadata is injected as a standard SEI User Data Unregistered NAL unit.

from omni.kit.livestream.core import Server, ServerConfig, VideoCodec

server = Server("rtsp")
server.start(ServerConfig(video_pre_encode=VideoCodec.H265, stream_port=8554))

# Define a 16-byte UUID (producer and consumer must agree on this).
MY_UUID = b"NVDS_CUSTOMMETA\x00"

# Stream with metadata.
server.stream_video_pre_encoded_with_metadata(
    encoded_bytes, len(encoded_bytes), 1920, 1080,
    start_time_ns=0, ended_time_ns=0,
    metadata=b'{"sim_time": 0.0, "frame_num": 1}',
    metadata_uuid=MY_UUID,
)

The metadata_uuid must be exactly 16 bytes (a ValueError is raised otherwise). Both metadata and metadata_uuid are optional — passing None for either disables metadata injection for that frame.

Context Manager#

with Server("webrtc") as server:
    server.start(ServerConfig(width=1920, height=1080))
    # ... stream frames ...
    server.stop()

Available Types#

  • Server - Creates and controls a streaming server instance.

  • ServerConfig - Dataclass with all configuration fields and sensible defaults.

  • VideoCodec - Enum for pre-encoded video codecs (NONE, CUSTOM, H264, H265, AV1).

  • LivestreamError - Exception raised when a C API call fails.

Server Types#

The server type passed to Server() must correspond to an enabled extension:

  • "webrtc" - Requires omni.kit.livestream.webrtc

  • "native" - Requires omni.kit.livestream.webrtc

  • "rtsp" - Requires omni.kit.livestream.rtsp

Integration with Other Extensions#

The omni.kit.livestream.core extension integrates with:

  • omni.kit.livestream.webrtc: Provides WebRTC and native streaming implementations.

  • omni.kit.livestream.rtsp: Provides RTSP streaming implementation.

  • omni.kit.livestream.app: Handles streaming the entire application framebuffer.

  • omni.kit.livestream.aov: Handles streaming specific render product AOVs.

  • omni.kit.livestream.messaging: Enables structured messaging between server and clients.

  • omni.services.livestream.session: Manages livestream sessions and provides control endpoints.

Performance Considerations#

The extension includes default settings optimized for streaming performance, such as:

  • Consistent frame rates with configurable targeting

  • Optimized run loop and rendering configurations

  • Synchronization settings to ensure smooth video delivery

These settings can be adjusted through the Omniverse Kit settings system according to specific application requirements.

Version History#

The extension follows semantic versioning and maintains a detailed changelog. Major updates typically align with new Kit versions, while minor updates include feature enhancements and bug fixes.

For detailed change history, refer to the CHANGELOG.md file.