CloudXR Cookbook: Common Feature Snippets#

Note

Applies to: Spatial Extensions, Kit 109.0.3+, CloudXR 6

This cookbook provides ready-to-use code snippets for common features you may want to add to your CloudXR client application. Each snippet builds on top of the Generic Viewer and the CloudXRKit SDK.

Note

All snippets assume you have a connected CloudXRSession with at least one available MessageChannel. The Generic Viewer’s ServerActionsView.swift demonstrates the basics of channel selection and raw string messaging; the snippets below show how to build structured, typed interactions on top of that foundation.

Snippet Overview#

1. Structured Messaging

Type-safe JSON messages to the server.

#sending-structured-json-messages-to-the-server

2. Variant Switching

Switch colors, styles, and materials.

#switching-a-usd-variant-color-style-material

3. Camera Switching

Switch between scene cameras.

#switching-the-active-camera

4. Rotation Gesture

Rotate the scene with hand gestures.

#rotating-the-streamed-scene-with-a-gesture-visionos

5. Scale Gesture

Pinch to scale the scene.

#scaling-the-streamed-scene-with-a-pinch-gesture-visionos

6. Lighting Slider

Control lighting with SwiftUI slider.

#controlling-scene-lighting-with-a-slider

7. Visibility Toggle

Show/hide objects using variants.

#toggling-object-visibility

8. Opacity Animation

Smooth fade in/out transitions.

#animating-entity-opacity-fade-inout

9. Server Notifications

Listen for server command results.

#listening-for-server-completion-notifications

10. Portal Window

Display content in a visionOS portal.

#creating-a-portal-window-visionos

Sending Structured JSON Messages to the Server#

Send typed, structured JSON messages to your Omniverse application instead of raw strings.

Prerequisites

A connected CloudXRSession with an available MessageChannel.

ServerMessage protocol and send helper#

import Foundation
import CloudXRKit

// A protocol for structured messages that encode to JSON with a "Type" key.
protocol ServerMessage: Encodable {
    associatedtype Payload: Encodable
    var type: String { get }
    var message: [String: Payload] { get }
}

// Encode any ServerMessage to Data for sending over a MessageChannel.
private let jsonEncoder = JSONEncoder()

func encodeMessage<T: ServerMessage>(_ msg: T) -> Data {
    try! jsonEncoder.encode(msg)
}

// Send a structured message over the first available channel.
func sendToServer<T: ServerMessage>(_ msg: T, session: Session) -> Bool {
    let data = encodeMessage(msg)
    guard let channelInfo = session.availableMessageChannels.first,
          let channel = session.getMessageChannel(channelInfo) else {
        return false
    }
    return channel.sendServerMessage(data)
}

How It Works#

Define a ServerMessage struct for each command your Omniverse app expects. The type field maps to the "Type" key in the JSON payload that your server-side extension dispatches on. Calling sendToServer encodes the struct to JSON and sends it over the first available opaque data channel.

Client Side

The ServerMessage protocol ensures type safety at compile time. Each message type is a separate struct, so it is not possible to accidentally send malformed JSON.

Server Side

Your Omniverse extension should register a listener on the opaque data channel and dispatch based on the "Type" field in the received JSON.

Switching a USD Variant (Color, Style, Material)#

Tell the server to switch a USD variant set – the most common configurator interaction.

Prerequisites

Snippet 1 (structured messaging), an Omniverse scene with variant sets configured.

Variant switching message and type-safe enum#

// Define the message the server expects for variant switching.
struct SetVariantMessage: ServerMessage {
    let type = "setVariantSelection"
    let message: [String: String]

    init(variantSetName: String, variantName: String) {
        message = [
            "variantSetName": variantSetName,
            "variantName": variantName
        ]
    }
}

// Define your variant options as a Swift enum for type safety.
enum MaterialColor: String, CaseIterable, Identifiable {
    case red = "Red"
    case blue = "Blue"
    case black = "Black"
    case white = "White"

    var id: String { rawValue }
}

// Usage: switch the "color" variant set to "Red"
let msg = SetVariantMessage(variantSetName: "color", variantName: MaterialColor.red.rawValue)
sendToServer(msg, session: session)

How It Works#

The SetVariantMessage struct produces JSON like:

Wire format#

{"type": "setVariantSelection", "variantSetName": "color", "variantName": "Red"}

On the server, your Omniverse extension receives this and calls the USD variant-switching API for the specified prim. The enum gives you compile-time safety and makes it easy to build picker UI with ForEach(MaterialColor.allCases).

Server Side

Your Omniverse extension should handle messages of type "setVariantSelection" and apply the variant using prim.GetVariantSets().SetSelection(variantSetName, variantName).

Switching the Active Camera#

Switch between predefined camera positions in your Omniverse scene.

Prerequisites

Snippet 1 (structured messaging), camera prims set up in your USD scene.

Camera switching message#

struct SetActiveCameraMessage: ServerMessage {
    let type = "setActiveCamera"
    let message: [String: String]

    /// cameraPath: the full USD prim path to the camera, e.g.
    /// "/World/Cameras/Front" or "/World/Cameras/TopDown"
    init(cameraPath: String) {
        message = ["cameraPath": cameraPath]
    }
}

// Usage: switch to the front camera
let cameraPrefix = "/World/Cameras/"
let msg = SetActiveCameraMessage(cameraPath: "\(cameraPrefix)Front")
sendToServer(msg, session: session)

How It Works#

The message tells your Omniverse application to switch its active viewport camera to the specified prim path. Camera paths are fully qualified USD prim paths. You can build a camera picker UI by defining an array of camera names and constructing the full path from a shared prefix.

Server Side

Your Omniverse extension should handle "setActiveCamera" messages and set the active camera on the viewport using the provided cameraPath.

Rotating the Streamed Scene with a Gesture (visionOS)#

Allow the user to rotate the streamed CloudXR content around the Y-axis using visionOS hand gestures.

Prerequisites

A CloudXRSessionComponent entity in your RealityView.

Y-axis rotation gesture for visionOS#

import SwiftUI
import RealityKit

// State to track rotation across gesture updates
@Observable
class GestureState {
    var lastRotation: Float = 0
    var totalRotation: Float = 0
}

// Build a RotateGesture3D that rotates the session entity around the Y-axis.
@MainActor
func makeRotationGesture(sessionEntity: Entity, state: GestureState) -> some Gesture {
    RotateGesture3D(constrainedToAxis: .z, minimumAngleDelta: .degrees(2))
        .onChanged { value in
            // RotateGesture3D has limited range; multiply by a factor for usability
            let rotationFactor: Float = 3.0
            let radians = Float(value.rotation.angle.radians) * -sign(value.rotation.axis.z)
            let amplified = radians * rotationFactor

            // Compute the delta since last update
            let delta = amplified - state.lastRotation
            state.lastRotation = amplified

            // Apply cumulative Y-axis rotation to the session entity
            state.totalRotation += delta
            sessionEntity.setOrientation(
                simd_quatf(angle: state.totalRotation, axis: simd_float3(0, 1, 0)),
                relativeTo: nil
            )
        }
        .onEnded { _ in
            state.lastRotation = 0
        }
}

How It Works#

RotateGesture3D on visionOS tracks a two-handed twist gesture. Its raw rotation range is small, so a 3x multiplier is applied for usability. The gesture provides cumulative angles, so we compute the delta between updates and accumulate it into a total rotation. The session entity (which parents all streamed content) is rotated around the Y-axis, causing the entire remote scene to visually rotate in place. Resetting lastRotation on gesture end ensures clean deltas for the next gesture.

Scaling the Streamed Scene with a Pinch Gesture (visionOS)#

Allow the user to scale the streamed content using a visionOS pinch (magnify) gesture.

Prerequisites

A CloudXRSessionComponent entity in your RealityView.

Pinch-to-scale gesture with clamping and snap-to-1x#

import SwiftUI
import RealityKit

// Track the scale at the start of each gesture
@Observable
class ScaleState {
    var baseScale: Float = 1.0
}

// Clamp and snap scale values for a good user experience.
func correctedScale(factor: Float, baseScale: Float) -> Float {
    let raw = factor * baseScale
    if raw > 5.0 { return 5.0 }         // Maximum scale
    if raw < 0.2 { return 0.2 }         // Minimum scale
    if raw > 0.9 && raw < 1.1 { return 1.0 } // Snap to 1.0 near identity
    return raw
}

@MainActor
func makeScaleGesture(sessionEntity: Entity, state: ScaleState) -> some Gesture {
    MagnifyGesture(minimumScaleDelta: 0.075)
        .onChanged { value in
            let scale = correctedScale(factor: Float(value.magnification), baseScale: state.baseScale)
            sessionEntity.scale = .one * scale
        }
        .onEnded { value in
            let scale = correctedScale(factor: Float(value.magnification), baseScale: state.baseScale)
            sessionEntity.scale = .one * scale
            state.baseScale = scale
        }
}

How It Works#

MagnifyGesture reports a magnification factor relative to the gesture start (1.0 = no change). We multiply it by the baseScale captured at the previous gesture’s end to get cumulative scaling. The correctedScale function clamps to a 0.2–5.0 range and snaps to exactly 1.0 when the user is close to the original size, preventing the frustrating “almost but not quite at 1x” problem. On gesture end, we store the final scale as the new baseline.

Controlling Scene Lighting with a Slider#

Send a lighting intensity value from a SwiftUI slider to the Omniverse scene.

Prerequisites

Snippet 1 (structured messaging), a server extension that handles lighting commands.

Lighting intensity slider with live updates#

import SwiftUI
import CloudXRKit

// Message to set scene light intensity on the server.
struct SetLightIntensityMessage: ServerMessage {
    let type = "setLightSlider"
    let message: [String: Float]

    init(intensity: Float, range: ClosedRange<Float> = 0.0...2.0) {
        let clamped = min(max(intensity, range.lowerBound), range.upperBound)
        message = ["intensity": clamped]
    }
}

// A SwiftUI view with a lighting slider.
struct LightingSliderView: View {
    @State private var intensity: Float = 1.0
    let session: Session
    let range: ClosedRange<Float> = 0.0...2.0

    var body: some View {
        VStack {
            Text("Lighting")
            Slider(value: $intensity, in: range)
                .onChange(of: intensity) {
                    let msg = SetLightIntensityMessage(intensity: intensity, range: range)
                    sendToServer(msg, session: session)
                }
        }
        .padding()
    }
}

How It Works#

The slider’s value is bound to intensity. On each change, a SetLightIntensityMessage is sent to the server with the clamped intensity value. The server-side extension applies this to the scene’s light source(s). The range can be adjusted to match your scene’s lighting setup.

Server Side

Your Omniverse extension should handle "setLightSlider" messages and set the intensity attribute on the relevant UsdLux light prim(s).

Toggling Object Visibility#

Show or hide objects in the Omniverse scene by switching a visibility variant.

Prerequisites

Snippet 1 (structured messaging), a visibility variant set on the target prim.

Visibility toggle using variant switching#

// Reuses the same SetVariantMessage from Snippet 2.
enum ObjectVisibility: String {
    case visible = "Visible"
    case hidden = "Hidden"
}

func setObjectVisibility(_ visibility: ObjectVisibility, session: Session) {
    let msg = SetVariantMessage(
        variantSetName: "Visibility",
        variantName: visibility.rawValue
    )
    sendToServer(msg, session: session)
}

// Usage in a SwiftUI Toggle
struct VisibilityToggleView: View {
    @State private var isVisible = true
    let session: Session

    var body: some View {
        Toggle("Show Object", isOn: $isVisible)
            .onChange(of: isVisible) {
                setObjectVisibility(isVisible ? .visible : .hidden, session: session)
            }
    }
}

How It Works#

This reuses the variant-switching pattern from Snippet 2 with a "Visibility" variant set. The SwiftUI Toggle provides a clean on/off control. When toggled, it sends either "Visible" or "Hidden" as the variant name. This is a simple pattern you can replicate for any boolean feature in your scene.

Server Side

Configure a variant set named "Visibility" on the target prim with "Visible" and "Hidden" variants that control the prim’s visibility attribute.

Animating Entity Opacity (Fade In/Out)#

Smoothly animate the opacity of any RealityKit entity – useful for transitions, loading states, or visual feedback.

Prerequisites

RealityKit, Combine. No CloudXR dependency – this is a pure RealityKit utility.

Full implementation – Entity opacity extension

Entity+Opacity.swift – reusable opacity animation#

import RealityKit
import Combine

// File-level storage for animation completion subscriptions.
private var opacitySubscriptions: Set<AnyCancellable> = .init()

extension Entity {
    /// The opacity of this entity and its descendants.
    /// Automatically adds an OpacityComponent if one doesn't exist.
    var opacity: Float {
        get { components[OpacityComponent.self]?.opacity ?? 1 }
        set {
            if components.has(OpacityComponent.self) {
                components[OpacityComponent.self]?.opacity = newValue
            } else {
                components[OpacityComponent.self] = OpacityComponent(opacity: newValue)
            }
        }
    }

    /// Animate opacity from one value to another.
    /// - Parameters:
    ///   - target: The target opacity (0.0 to 1.0).
    ///   - from: The starting opacity. Defaults to the entity's current opacity.
    ///   - duration: Animation duration in seconds.
    ///   - delay: Delay before the animation starts.
    ///   - completion: Called when the animation finishes.
    func animateOpacity(
        to target: Float,
        from: Float? = nil,
        duration: TimeInterval = 0.5,
        delay: TimeInterval = 0,
        completion: (() -> Void)? = nil
    ) {
        let startValue = from ?? self.opacity

        if !components.has(OpacityComponent.self) {
            components[OpacityComponent.self] = OpacityComponent(opacity: startValue)
        }

        let animation = FromToByAnimation(
            name: "opacityFade",
            from: startValue,
            to: target,
            duration: duration,
            timing: .linear,
            isAdditive: false,
            bindTarget: .opacity,
            delay: delay
        )

        do {
            let resource = try AnimationResource.generate(with: animation)
            let controller = playAnimation(resource)

            if let completion {
                scene?.publisher(for: AnimationEvents.PlaybackCompleted.self)
                    .filter { $0.playbackController == controller }
                    .sink { _ in completion() }
                    .store(in: &opacitySubscriptions)
            }
        } catch {
            assertionFailure("Could not generate opacity animation: \(error.localizedDescription)")
        }
    }
}

Usage#

// Fade out over 0.25 seconds
entity.animateOpacity(to: 0.0, duration: 0.25)

// Fade in over 0.5 seconds with a completion handler
entity.animateOpacity(to: 1.0, duration: 0.5) {
    print("Fade-in complete")
}

How It Works#

This extension uses RealityKit’s FromToByAnimation bound to the .opacity target. It ensures an OpacityComponent exists on the entity before animating. The animation is linear and affects the entity and all its descendants. An optional completion handler is connected through Combine by subscribing to AnimationEvents.PlaybackCompleted and filtering to the specific playback controller.

Listening for Server Completion Notifications#

Detect when the server finishes processing a command (e.g., a variant switch) so you can update your UI.

Prerequisites

A connected MessageChannel, JSON messages from the server.

Async listener for server messages#

import CloudXRKit
import Foundation

/// Start listening for messages on a channel and call the handler for each parsed message.
func listenForServerMessages(
    channel: MessageChannel,
    onMessage: @escaping ([String: Any]) -> Void
) -> Task<Void, Never> {
    Task {
        for await messageData in channel.receivedMessageStream {
            if let json = try? JSONSerialization.jsonObject(with: messageData) as? [String: Any] {
                onMessage(json)
            }
        }
    }
}

// Example: detect when a variant switch completes
func handleServerMessage(_ json: [String: Any]) {
    guard let type = json["Type"] as? String else { return }

    switch type {
    case "switchVariantComplete":
        // The server finished applying a variant change.
        if let variantName = json["variantSetName"] as? String {
            print("Variant switch completed: \(variantName)")
            // Update your UI state here, e.g. re-enable buttons or dismiss a spinner.
        }
    default:
        break
    }
}

Usage#

// Start listening when the channel becomes available
let listenerTask = listenForServerMessages(channel: channel, onMessage: handleServerMessage)

// Cancel when done:
listenerTask.cancel()

How It Works#

The receivedMessageStream on a MessageChannel is an AsyncSequence that yields Data for each incoming server message. We parse each message as JSON and dispatch based on the "Type" field. For variant switches, the server sends a "switchVariantComplete" message when it finishes applying the change, which you can use to update loading indicators or re-enable UI controls.

Server Side

After processing a command, your Omniverse extension should send a completion notification back through the same opaque data channel with a "Type" field your client can match on.

Creating a Portal Window (visionOS)#

Display the streamed CloudXR content inside a portal – a window into another world that exists within the user’s physical space.

Prerequisites

RealityKit, visionOS, a CloudXRSessionComponent entity.

Usage#

Setting up a portal in your RealityView#

// 1. Create a world entity and mark it as a portal world
let portalWorld = Entity()
portalWorld.components[WorldComponent.self] = .init()

// 2. Move your session entity (CloudXR content) into the portal world
portalWorld.addChild(sessionEntity)

// 3. Create the portal
let (container, portal) = makePortal(world: portalWorld)

// 4. Add both the world and the portal container to your scene
sceneEntity.addChild(portalWorld)
sceneEntity.addChild(container)

How It Works#

A visionOS portal uses three key components:

WorldComponent: Defines a separate “world” – content parented under this entity is only visible through a portal.
PortalMaterial: Applied to a plane mesh, this material acts as a window into the target world.
PortalComponent: Links the portal plane to the world entity, creating the see-through effect.

The drag bar above the portal uses HoverEffectComponent for visual feedback and InputTargetComponent with collision shapes to support drag gestures for repositioning.

End-to-End: Hand Gesture Triggers a Server Action#

This pattern connects a visionOS hand gesture on the Apple Vision Pro to a server-side Kit action via the CloudXR opaque data channel. The gesture (a tap) sends a selectPrimsRequest message to the Kit server, which selects the specified prim in the USD stage.

Prerequisites

This snippet assumes you have a working CloudXRKit session and a MessageChannel reference. See Snippet 1: Sending Structured JSON Messages for the messaging setup.

Tap gesture sends selectPrimsRequest to Kit server#

import SwiftUI
import RealityKit

struct InteractiveImmersiveView: View {
    let channel: MessageChannel

    var body: some View {
        RealityView { content in
            // Your session entity setup
        }
        .gesture(
            SpatialTapGesture()
                .targetedToAnyEntity()
                .onEnded { value in
                    let tappedEntity = value.entity
                    let primPath = tappedEntity.name

                    let message: [String: Any] = [
                        "type": "selectPrimsRequest",
                        "payload": ["paths": [primPath]]
                    ]

                    if let data = try? JSONSerialization.data(withJSONObject: message),
                       let json = String(data: data, encoding: .utf8) {
                        channel.sendServerMessage(json)
                    }
                }
        )
    }
}

How It Works#

SpatialTapGesture detects when the user taps an entity using visionOS hand tracking
The tapped entity’s name is used as the USD prim path (this works when entity names match prim paths in the streamed scene)
A selectPrimsRequest JSON payload is constructed using the "payload" field format (preferred over "message")
The message is sent over the CloudXR opaque data channel to the Kit server
The USD Viewer’s built-in selectPrimsRequest handler receives the message and updates the prim selection on the stage

This pattern generalizes to any gesture-to-server-action flow: replace SpatialTapGesture with RotateGesture3D, MagnifyGesture, or DragGesture, and replace selectPrimsRequest with any custom event type your Kit extension handles.