Overview#
The omni.kit.livestream.webrtc extension provides implementations of the streaming server interface (IServer) for both WebRTC and native protocols. It enables streaming video frames from CUDA buffers to remote clients with support for bidirectional messaging and input handling.
Purpose#
This extension bridges the Omniverse Kit rendering pipeline with remote clients by:
Registering WebRTC and native streaming implementations with the
omni.kit.livestream.coreserver factoryProviding efficient CUDA-based video frame encoding and streaming
Handling client input and message events
Managing streaming sessions and connection state
Architecture#
The extension consists of several key components:
Server: Implements the
IServerinterface for both WebRTC and native protocolsInputHandler: Processes and translates input events from the client
MessageHandler: Handles custom messages between server and client
SessionHandler: Manages streaming sessions and STUN credentials
These components work together to provide a complete streaming solution built on NVIDIA’s StreamSDK technology.
Key Features#
Multiple Protocol Support: Implements both WebRTC and native streaming protocols
CUDA Integration: Direct streaming from CUDA buffers for high-performance encoding
Bidirectional Communication: Support for sending and receiving messages between server and client
Input Handling: Processing of keyboard, mouse, and gamepad input from clients
Dynamic Resizing: Ability to resize streams both before and after client connections
Session Management: Control over streaming sessions and STUN credentials
Usage#
Creating and Using a WebRTC Server#
// Get the server factory interface
auto serverFactory = carb::getCachedInterface<omni::kit::livestream::IServerFactory>();
// Create a WebRTC server
auto server = serverFactory->createServer("webrtc");
// Configure the server
omni::kit::livestream::IServer::Config config;
config.width = 1920;
config.height = 1080;
config.targetFPS = 60;
config.signalPort = 49100;
config.streamPort = 47999;
config.publicIp = ""; // Leave empty to use ICE for NAT traversal
config.allowDynamicResize = false;
// Start the server
server->start(config);
// Stream a frame from a CUDA buffer
server->streamVideoCudaBuffer(cudaBuffer, pitchBytes, width, height);
// Send a message to the client
server->sendMessageToClient("Hello from server");
// Stop the server when done
server->stop();
Creating a Native Protocol Server#
// Create a native protocol server
auto nativeServer = serverFactory->createServer("native");
// Configure and use the server the same way as WebRTC
// ...
Implementation Details#
Server Factory Registration#
During extension startup, the server implementations are registered with the server factory:
// Register the WebRTC server implementation
serverFactory->registerFactoryFunction(Server::kStreamTypeWebRTC, Server::createWebRTC);
// Register the native server implementation
serverFactory->registerFactoryFunction(Server::kStreamTypeNative, Server::createNative);
Stream Types#
The server manages three types of streams:
Video Stream: For streaming video frames from CUDA buffers
Audio Stream: For streaming audio data (when enabled)
Input Stream: For receiving input events from the client
Resizing Capabilities#
The server supports two types of resizing:
Static Resizing: Before client connection (via
resizeVideoStreamStatic)Dynamic Resizing: After client connection (via
resizeVideoStreamDynamic)
Client Connection Events#
The server dispatches events when clients connect or disconnect:
Client Connected: Dispatched when a client connects to the server
Client Disconnected: Dispatched when a client disconnects from the server
Integration with Other Extensions#
The omni.kit.livestream.webrtc extension:
Depends on
omni.kit.livestream.corefor the server factory infrastructureProvides streaming implementation for
omni.kit.livestream.appandomni.kit.livestream.aovWorks with
omni.kit.livestream.messagingfor structured message exchangeWorks with
omni.services.livestream.sessionfor session management
Performance Considerations#
For optimal streaming performance:
Use CUDA buffers directly when possible to avoid unnecessary copies
Consider the trade-offs between resolution, frame rate, and bandwidth
Be aware that dynamic resizing can impact performance
Use the native protocol when lower latency is more important than web compatibility (requires a native streaming client)
Version History#
The extension follows semantic versioning and maintains a detailed changelog. Major updates typically align with new Kit versions, while minor updates include feature enhancements and bug fixes.
For detailed change history, refer to the CHANGELOG.md file.