Omniverse Spatial Scene Integration and Messaging#

Note

Applies to: Spatial Extensions, Kit 109.0.3+, CloudXR 6

This scene integration reference covers loading USD content into your Kit XR application and using the Kit message bus for bidirectional communication between the server and XR clients. The message bus is the universal integration point: messages arrive at the same place regardless of which CloudXR runtime (Native or WebRTC) or which client (Apple, Meta, or browser) sent them.

The Omniverse Spatial Kit Message Bus#

Both CloudXR runtimes deliver opaque data channel messages to the same Kit message bus:

flowchart LR Apple["Apple Vision Pro\n(CloudXRKit)"] -- "Opaque Data Channel\n(Native)" --> NativeRT["CloudXR 6 Native"] Meta["Meta Quest 2/3/3S\n(CloudXR.js)"] -- "Opaque Data Channel\n(WebRTC)" --> WebRTCRT["CloudXR 6 WebRTC"] NativeRT --> MsgBus["Kit Message Bus"] WebRTCRT --> MsgBus MsgBus --> Handlers["Kit Extension\nMessage Handlers"] Handlers -- "Response" --> MsgBus MsgBus -- "Outbound" --> NativeRT MsgBus -- "Outbound" --> WebRTCRT

This means:

  • A Kit extension listening on the message bus responds to messages from any client

  • The JSON payload format is identical regardless of transport

  • You write your server-side logic one time and it works with all clients

Message Format#

Important

Always use the "payload" field (not "message") when sending JSON over the CloudXR opaque data channel. The "payload" path flattens fields directly into event.payload on the server, which is what the USD Viewer built-in handlers expect. The "message" field is a legacy fallback that serializes to a string and requires manual parsing.

All messages sent over the CloudXR opaque data channel use the same JSON structure:

{
  "type": "someEventName",
  "payload": {
    "key": "value"
  }
}

  • type: A string identifying the operation. The CloudXR plugin reads this field to determine which event to dispatch on the server. This field is required; messages without it are rejected.

  • payload: A JSON object with operation-specific data. The CloudXR plugin flattens each child field directly into event.payload, so server-side handlers receive event.payload["key"] etc.

How CloudXR Delivers Messages to the Server#

The OpenXR plugin (omni.kit.xr.system.openxr) supports two payload delivery modes depending on which fields it finds in the incoming JSON:

  1. "payload" field (preferred): The plugin iterates over the children of the "payload" object and copies each one directly into event.payload. This means a handler receives event.payload["url"], event.payload["paths"], etc. exactly as expected. This is the mode used by the USD Viewer messaging extension: no bridge or wrapper extension is needed.

  2. "message" field (fallback): If there is no "payload" field but a "message" field is present, the plugin serializes its value to a string and places it in event.payload["message"]. Extensions that use this path (such as omni.xr.visibility) parse the JSON string themselves.

  3. Neither: If the message contains neither "payload" nor "message", the plugin logs an error and drops the message.

The full flow for the recommended "payload" path:

  1. Client sends {"type":"openStageRequest","payload":{"url":"..."}}

  2. CloudXR plugin reads "type" and dispatches the event as openStageRequest

  3. CloudXR plugin reads "payload" and flattens its children into event.payload

  4. The USD Viewer handler _on_open_stage fires and reads event.payload["url"] to load the stage

USD Viewer Built-In Message Handlers#

If you created your application using the USD Viewer template, the setup extension already includes handlers for common operations. These handlers listen on the Kit message bus and respond to incoming messages from any client.

Important

The message handlers listed below are provided by the USD Viewer template’s built-in setup extension. If you built your application from a different template (such as Kit Base Editor or USD Composer), no message handling extension is included by default – incoming data channel messages will arrive on the Kit message bus but nothing will act on them. You will need to create your own extension that subscribes to the bus and handles the messages your client sends. See Writing Custom Message Handlers below for an example.

Available Message Handlers#

Event Type

Direction

Description

openStageRequest

Client -> Server

Load a USD asset

openedStageResult

Server -> Client

Confirm asset load success/failure

loadingStateQuery

Client -> Server

Poll whether Kit is ready

loadingStateResponse

Server -> Client

Report loading state

getChildrenRequest

Client -> Server

Get child prims of a path

getChildrenResponse

Server -> Client

Return child prim list

selectPrimsRequest

Client -> Server

Change prim selection

stageSelectionChanged

Server -> Client

Notify client of selection changes

makePrimsPickable

Client -> Server

Make prims selectable in viewport

resetStage

Client -> Server

Reset stage to initial state

updateProgressAmount

Server -> Client

Report loading progress

updateProgressActivity

Server -> Client

Report loading activity

Example Payloads#

Load a USD asset:

{
  "type": "openStageRequest",
  "payload": {
    "url": "${omni.usd_viewer.samples}/samples_data/stage01.usd"
  }
}

Reset the stage:

{
  "type": "resetStage",
  "payload": {}
}

Select specific prims:

{
  "type": "selectPrimsRequest",
  "payload": {
    "paths": ["/World/Geometry/Cube"]
  }
}

Query stage children:

{
  "type": "getChildrenRequest",
  "payload": {
    "prim_path": "/World",
    "filters": ["USDGeom"]
  }
}

Sending Messages from the Apple Generic Viewer#

The cloudxr-apple-generic-viewer includes a ServerActionsView with “Action 1” and “Action 2” buttons. By default, these send plain text strings. To trigger the USD Viewer’s built-in handlers, modify them to send the correct JSON payloads.

Original Code#

In CloudXRViewer/Common/ServerActionsView.swift, the default buttons send plain strings around line 122:

Button("Action 1") {
    sendMessage(message: "Action 1")
}

Modified Code: Trigger USD Viewer Handlers#

First, add this helper function to the top of the ServerActionsView struct in ServerActionsView.swift (before any button code). It uses JSONSerialization to build clean JSON with no stray whitespace. The "type" field tells CloudXR which event to dispatch; all other fields become the payload:

/// Build a JSON message for the CloudXR opaque data channel.
/// CloudXR reads "type" for the event name and flattens "payload"
/// children into event.payload on the server.
private func buildMessage(type: String, payload: [String: Any] = [:]) -> String? {
    let message: [String: Any] = [
        "type": type,
        "payload": payload
    ]
    guard let data = try? JSONSerialization.data(withJSONObject: message, options: []),
          let jsonString = String(data: data, encoding: .utf8) else {
        return nil
    }
    return jsonString
}

Then replace the button actions:

Button("Load Stage 1") {
    if let json = buildMessage(type: "openStageRequest", payload: [
        "url": "${omni.usd_viewer.samples}/samples_data/stage01.usd"
    ]) {
        sendMessage(message: json)
    }
}

Button("Load Stage 2") {
    if let json = buildMessage(type: "openStageRequest", payload: [
        "url": "${omni.usd_viewer.samples}/samples_data/stage02.usd"
    ]) {
        sendMessage(message: json)
    }
}

The sendMessage function already sends data over the CloudXR opaque data channel using MessageChannel.sendServerMessage(). No changes to the transport layer are needed: only the payload changes.

For more advanced patterns, including type-safe structured messaging, variant switching, camera changes, gesture-driven interaction, and portal windows, see the Apple Code Snippets Cookbook.

Remember to select a data channel before using the UI in the sample client.

Receiving Responses#

The ServerActionsView already listens for incoming messages through channel.receivedMessageStream. Responses from Kit (such as openedStageResult or loadingStateResponse) arrive on this same stream as JSON strings.

Sending Messages from the CloudXR.js React Sample#

The CloudXR.js React sample includes a 3D in-VR control panel with generic Action 1 and Action 2 buttons. By default, these send generic action messages ({ type: 'action', message: { action: 'action_1' } }). To trigger the USD Viewer’s built-in handlers instead, you only need to change two things: the button labels and the message payloads. The transport layer (sendMessage) already works and does not need any changes. The end result is two buttons: Load Stage 1 and Load Stage 2, each loading a different USD asset, matching the Apple client sample above.

The two files involved are:

  • src/CloudXRUI.tsx – The 3D button layout and labels

  • src/App.tsx – The button click handlers and the sendMessage helper

Step 1: Change the Button Labels (src/CloudXRUI.tsx)#

Find the two action buttons in the JSX. They are inside the {/* Action buttons row */} container. Change the button text:

First button – update the label from “Action 1” to “Load Stage 1”:

// Before:
<Button {...xrButton('action1', onAction1)} ...>
  <Text fontSize={48} color="black" fontWeight="medium">
    Action 1
  </Text>
</Button>

// After:
<Button {...xrButton('action1', onLoadStage1)} ...>
  <Text fontSize={48} color="black" fontWeight="medium">
    Load Stage 1
  </Text>
</Button>

Second button – update the label from “Action 2” to “Load Stage 2”:

// Before:
<Button {...xrButton('action2', onAction2)} ...>
  <Text fontSize={48} color="black" fontWeight="medium">
    Action 2
  </Text>
</Button>

// After:
<Button {...xrButton('action2', onLoadStage2)} ...>
  <Text fontSize={48} color="black" fontWeight="medium">
    Load Stage 2
  </Text>
</Button>

Then update the matching prop names in the CloudXRUIProps interface and the function’s destructured parameters:

interface CloudXRUIProps {
  onLoadStage1?: () => void;   // was onAction1
  onLoadStage2?: () => void;   // was onAction2
  // ...keep the rest unchanged...
}

Step 2: Change the Button Handlers (src/App.tsx)#

Inside src/App.tsx, look for the handleAction1 and handleAction2 functions. They are inside the App() function body, right after the sendMessage definition. Replace them in place:

  // ---- Find these (inside App(), after sendMessage): ----

  const handleAction1 = async () => {
    console.log('Action 1 pressed');
    const success = await sendMessage({ type: 'action', message: { action: 'action_1' } });
    if (!success) {
      console.error('Failed to send Action 1 message');
    }
  };

  const handleAction2 = async () => {
    console.log('Action 2 pressed');
    const success = await sendMessage({ type: 'action', message: { action: 'action_2' } });
    if (!success) {
      console.error('Failed to send Action 2 message');
    }
  };

  // ---- Replace with: ----

  const handleLoadStage1 = async () => {
    const success = await sendMessage({
      type: "openStageRequest",
      payload: {
        url: "${omni.usd_viewer.samples}/samples_data/stage01.usd"
      }
    });
    if (!success) {
      console.error('Failed to send Load Stage 1 message');
    }
  };

  const handleLoadStage2 = async () => {
    const success = await sendMessage({
      type: "openStageRequest",
      payload: {
        url: "${omni.usd_viewer.samples}/samples_data/stage02.usd"
      }
    });
    if (!success) {
      console.error('Failed to send Load Stage 2 message');
    }
  };

Important

These handlers must stay inside the App() function body. They call sendMessage, which is a closure over the component’s cloudXRSession state. Placing them outside App() (at module scope) will cause a ReferenceError.

Finally, find the <CloudXR3DUI ... /> JSX and update the two prop names to match:

// Before:
<CloudXR3DUI
  onAction1={handleAction1}
  onAction2={handleAction2}
  ...
/>

// After:
<CloudXR3DUI
  onLoadStage1={handleLoadStage1}
  onLoadStage2={handleLoadStage2}
  ...
/>

Everything else – the sendMessage helper, the <CloudXRComponent>, the message receive loop, and the handleDisconnect handler – stays unchanged.

How sendMessage Works#

For reference, the existing sendMessage function in App.tsx handles all the transport details. It tries the MessageChannel API first and falls back to the legacy sendServerMessage API, returning a boolean indicating success:

const sendMessage = async (message: any) => {
  if (!cloudXRSession) {
    console.error('CloudXR session not available');
    return false;
  }

  const channels = cloudXRSession.availableMessageChannels;
  if (channels.length > 0) {
    const channel = channels[0];
    try {
      const encoder = new TextEncoder();
      const data = encoder.encode(JSON.stringify(message));
      const success = channel.sendServerMessage(data);
      return success;
    } catch (error) {
      console.error('Error sending via MessageChannel:', error);
      return false;
    }
  }

  // Fallback to legacy API
  try {
    cloudXRSession.sendServerMessage(message);
    return true;
  } catch (error) {
    console.error('Error sending via legacy API:', error);
    return false;
  }
};

This is the CloudXR.js equivalent of the Apple client’s MessageChannel.sendServerMessage(). The JSON structure is identical across both clients: the server cannot distinguish which platform sent the message.

Receiving Responses#

The React sample already sets up a message receive loop in App.tsx that polls for MessageChannel availability and reads incoming messages. Responses from Kit (such as openedStageResult or loadingStateResponse) arrive as JSON strings. To act on them, add handling logic inside the existing parse block:

const message = JSON.parse(messageText);
if (message.type === "openedStageResult") {
  console.log("Stage loaded:", message.url);
} else if (message.type === "loadingStateResponse") {
  console.log("Loading state:", message.payload);
}

You do not need to modify sendMessage itself: it works with any JSON payload. Just pass the correct type and payload fields for the Kit handler you want to trigger (see USD Viewer Built-In Message Handlers above).

Note

The default action messages use the "message" field rather than "payload". This means the server receives event.payload["message"] as a serialized JSON string. If you are targeting the USD Viewer built-in handlers, use the "payload" field instead so the plugin flattens the values directly into event.payload.

Writing Custom Message Handlers#

You can add your own message handlers in a Kit extension. This works with any client.

Example: Custom Action Handler#

# In your extension.py

import omni.ext
import omni.kit.app
import carb.events
import json


class MyMessageHandlerExtension(omni.ext.IExt):
    def on_startup(self, ext_id):
        # Subscribe to the message bus
        bus = omni.kit.app.get_app().get_message_bus_event_stream()

        self._sub = bus.create_subscription_to_pop_by_type(
            carb.events.type_from_string("myCustomAction"),
            self._on_custom_action
        )

    def _on_custom_action(self, event):
        payload = event.payload
        print(f"Received custom action: {payload}")

        # Perform your logic here
        # e.g., change material, toggle visibility, animate, etc.

    def on_shutdown(self):
        self._sub = None

Then from any client, send:

{
  "type": "myCustomAction",
  "payload": {
    "action": "toggleVisibility",
    "target": "/World/Geometry/Cube"
  }
}

Loading USD Scenes in Omniverse Spatial#

There are three ways to load USD content into a running Omniverse Spatial application. Choose the method that matches your workflow:

Method 1: Command-Line Argument at Launch#

Pass the --/app/auto_load_usd argument when launching Kit to load a scene immediately:

./repo.sh launch -- --/app/auto_load_usd='/path/to/your/scene.usd'

The ${omni.usd_viewer.samples} token can be used to reference bundled sample scenes:

./repo.sh launch -- --/app/auto_load_usd='${omni.usd_viewer.samples}/samples_data/stage01.usd'

Method 2: Client-Driven Loading via Opaque Data Channel#

Send an openStageRequest message from any connected client (Apple or Meta) to load a scene at runtime:

{
  "type": "openStageRequest",
  "payload": {
    "url": "/path/to/your/scene.usd"
  }
}

The USD Viewer setup extension handles this message and responds with openedStageResult when loading completes. See USD Viewer Built-In Message Handlers above for the full list of supported messages.

Method 3: Programmatic Loading from a Kit Extension#

Load USD content directly from Python in your Kit extension code:

import omni.usd

# Open a stage (replaces the current scene)
omni.usd.get_context().open_stage("path/to/scene.usd")

# Add a reference to the current stage (non-destructive)
stage = omni.usd.get_context().get_stage()
prim = stage.DefinePrim("/World/MyAsset")
prim.GetReferences().AddReference("path/to/asset.usd")

Advanced: XRUsdLayer for Real-Time USD Integration#

For developers building custom XR tools and interactions, the Kit XR extensions provide XRUsdLayer: a specialized system for managing XR UI elements in the USD stage with minimal latency. XRUsdLayer bypasses normal USD composition for real-time updates and is the foundation for controller models, selection beams, teleport arcs, and 3D UI elements.

For full documentation on XRUsdLayer, see the Scene Integration development guide.

See Also#