Scene Integration#

Note

Applies to: Spatial Extensions, Kit 109.0.3+, CloudXR 6

This scene integration reference covers integrating XR with USD scenes using XRUsdLayer for efficient real-time updates.

XRUsdLayer Overview#

XRUsdLayer is a specialized system for managing XR UI elements in the USD stage with minimal latency.

Why XRUsdLayer?#

Traditional USD composition is too slow for real-time XR updates. XRUsdLayer solves this by:

Using USD-RT: Bypasses USD composition for fast updates
Late-Stage Updates: Updates controller positions after USD composition
Managed Objects: Special handling for beams, arcs, controller models
Coordinate Conversion: Automatic transformation between XR and USD spaces

When to Use XRUsdLayer#

Use XRUsdLayer for:

Controller models and attachments
Selection beams
Teleport arcs
Dynamic XR UI elements
Objects that update every frame

Use Regular USD for:

Static scene content
Objects that rarely change
Traditional 3D assets

Creating XRUsdLayer#

import omni.kit.xr.core

xr_core = omni.kit.xr.core.XRCore.get_singleton()

# Create XR UI layer
xr_usd_layer = xr_core.create_xr_usd_layer(
    usd_path="/_xr/my_ui",           # USD path for XR elements
    meters_per_unit=0.01,             # 1 cm units (common for XR)
    up_axis='y',                      # Y-up coordinate system
    ui_layer_name='my_xr_ui',         # Optional: UI layer name
    component_layer_name='my_xr_comp' # Optional: Component layer name
)

# Check if valid
if xr_usd_layer.is_valid():
    print(f"Created layer at: {xr_usd_layer.get_top_level_prim_path()}")

Layer Properties#

# Get layer information
layer_name = xr_usd_layer.get_layer_name()
ui_layer_name = xr_usd_layer.get_ui_layer_name()
top_level_path = xr_usd_layer.get_top_level_prim_path()

# Get USD layers
sdf_layer = xr_usd_layer.get_layer()
ui_layer = xr_usd_layer.get_ui_layer()
component_layer = xr_usd_layer.get_component_layer()

# Get edit context for manual USD operations
with xr_usd_layer.get_edit_context():
    # USD operations here will edit this layer
    pass

# Get coordinate system
coord_system = xr_usd_layer.get_coordinate_system()
print(f"Meters per unit: {coord_system.meters_per_unit}")
print(f"Up axis: {coord_system.up_axis}")

Managed Objects#

XRUsdLayer manages several types of objects optimized for XR.

Assets#

Load and position 3D models.

# Add controller model
controller_path = xr_usd_layer.add_asset(
    path="/_xr/my_ui/left_controller",
    group="controllers",
    file_path="/path/to/controller.usd",
    layer_identifier="",              # Optional: specific USD layer
    transform=None,                   # Initial transform (or None)
    transform_type=omni.kit.xr.core.XRTransformType.local,
    pickable=False,                   # Can be selected by raycast
    visible=True
)

print(f"Added asset at: {controller_path}")

Parameters:

path: USD path for the asset
group: Logical grouping for bulk operations
file_path: Path to USD file to load
transform: Initial 4x4 transform matrix
transform_type: local, stage, or scope
pickable: Whether raycast can hit this object
visible: Initial visibility

Selection Beams#

Create ray visualizations for pointing/selection.

# Add selection beam
beam_path = xr_usd_layer.add_beam(
    path="/_xr/my_ui/left_beam",
    group="tools",
    material_reference="/World/Materials/BeamMaterial",
    transform=None,
    transform_type=omni.kit.xr.core.XRTransformType.local,
    max_length=1000.0,  # 10 meters (in cm)
    length=-1.0,        # -1 = use full max_length
    tube_radius=0.5,    # 0.5cm radius
    visible=True
)

# Update beam length dynamically
xr_usd_layer.set_length(beam_path, 500.0)  # 5 meters

# Update beam radius
xr_usd_layer.set_radius(beam_path, 0.3)

# Get beam properties
length = xr_usd_layer.get_length(beam_path)
radius = xr_usd_layer.get_radius(beam_path)
max_length = xr_usd_layer.get_max_length(beam_path)

Teleport Arcs#

Create curved arcs for locomotion visualization.

# Add teleport arc
arc_path = xr_usd_layer.add_teleport_arc(
    path="/_xr/my_ui/teleport_arc",
    group="tools",
    material_reference="/World/Materials/ArcMaterial",
    transform=None,
    transform_type=omni.kit.xr.core.XRTransformType.local,
    max_height=300.0,    # 3 meters (in cm)
    tube_radius=2.0,     # 2cm radius
    num_segments=120,    # Smoothness (more = smoother)
    visible=True
)

# Update arc height
xr_usd_layer.set_max_height(arc_path, 200.0)  # 2 meters

# Get arc properties
max_height = xr_usd_layer.get_max_height(arc_path)

Transforms#

Create empty transform nodes.

# Add transform (like a group/pivot)
transform_path = xr_usd_layer.add_transform(
    path="/_xr/my_ui/pivot",
    group="ui",
    transform=None,
    transform_type=omni.kit.xr.core.XRTransformType.local,
    visible=True
)

# Child objects can be added under this transform
child_path = xr_usd_layer.add_asset(
    path=f"{transform_path}/child_object",
    group="ui",
    file_path="/path/to/asset.usd"
)

Links#

Parent objects to existing USD prims.

# Link to existing USD prim
link_path = xr_usd_layer.add_link(
    path="/_xr/my_ui/linked_object",
    group="ui",
    link_path="/World/MyExistingPrim",
    transform=None,
    transform_type=omni.kit.xr.core.XRTransformType.local,
    visible=True
)

Links to Input Devices#

Parent objects to controllers or HMD.

from pxr import Gf

# Ensure device prim exists
device_path = xr_usd_layer.ensure_device_prim_path("/user/hand/left")

# Link object to controller
linked_path = xr_usd_layer.add_link_to_input_device(
    path=f"{device_path}/attached_object",
    group="controllers",
    input_device_name="/user/hand/left",
    pose_name="grip",
    transform=Gf.Matrix4d().SetTranslate(Gf.Vec3d(0, 5, 0)),  # 5cm offset
    transform_type=omni.kit.xr.core.XRTransformType.local,
    visible=True
)

# Object now follows controller automatically!

Complete Example:

def setup_controller_model():
    """Set up controller model with selection beam."""
    device_path = xr_usd_layer.ensure_device_prim_path("/user/hand/left")
    
    # Add controller model
    controller_model = xr_usd_layer.add_asset(
        path=f"{device_path}/model",
        group="controllers",
        file_path="/assets/controllers/generic_controller.usd",
        visible=True
    )
    
    # Link to controller's grip pose
    xr_usd_layer.add_link_to_input_device(
        path=controller_model,
        group="controllers",
        input_device_name="/user/hand/left",
        pose_name="grip"
    )
    
    # Add selection beam linked to aim pose
    beam = xr_usd_layer.add_beam(
        path=f"{device_path}/beam",
        group="tools",
        material_reference="/World/Materials/SelectionBeam",
        max_length=1000.0,
        tube_radius=0.5
    )
    
    xr_usd_layer.add_link_to_input_device(
        path=beam,
        group="tools",
        input_device_name="/user/hand/left",
        pose_name="aim"
    )

References#

Add USD references to other prims.

# Add reference to existing prim
ref_path = xr_usd_layer.add_reference(
    path="/_xr/my_ui/referenced_object",
    group="ui",
    reference_path="/World/Templates/ButtonTemplate",
    transform=None,
    transform_type=omni.kit.xr.core.XRTransformType.local,
    pickable=True,
    visible=True
)

Reorient#

Automatically reorient objects based on device orientation.

# Create reorient node (e.g., for billboarding UI)
reorient_path = xr_usd_layer.add_reorient(
    path="/_xr/my_ui/billboard",
    group="ui",
    alignment=omni.kit.xr.core.XROrientationAlignment.device_up_right,
    input_device="/user/head",
    pose_name=""
)

# Add UI as child - it will automatically face the user
ui_element = xr_usd_layer.add_asset(
    path=f"{reorient_path}/ui_panel",
    group="ui",
    file_path="/assets/ui/panel.usd"
)

Alignment Options:

world - World-aligned
device - Follow device orientation
device_no_roll - Follow device, no roll
device_up_right - Face device, keep up vector
anchor - Follow anchor
anchor_no_roll - Follow anchor, no roll
anchor_up_right - Face anchor, keep up vector

Transform Management#

Setting Transforms#

from pxr import Gf

# Create transform matrix
transform = Gf.Matrix4d()
transform.SetTranslate(Gf.Vec3d(100, 150, -200))  # Position in cm
transform.SetRotateOnly(Gf.Rotation(Gf.Vec3d(0, 1, 0), 45))  # Rotate 45° around Y

# Set transform
xr_usd_layer.set_transform(
    path="/_xr/my_ui/my_object",
    transform=transform,
    transform_type=omni.kit.xr.core.XRTransformType.local,
    use_usd=False  # Use USD-RT for fast updates
)

# Set position only (simpler)
xr_usd_layer.set_position(
    path="/_xr/my_ui/my_object",
    position=Gf.Vec3d(100, 150, -200),
    transform_type=omni.kit.xr.core.XRTransformType.local,
    use_usd=False
)

Getting Transforms#

# Get transform
transform = xr_usd_layer.get_transform(
    path="/_xr/my_ui/my_object",
    transform_type=omni.kit.xr.core.XRTransformType.local,
    use_usd=False
)

# Get position only
position = xr_usd_layer.get_position(
    path="/_xr/my_ui/my_object",
    transform_type=omni.kit.xr.core.XRTransformType.local
)

print(f"Position: {position}")

Transform Types#

XRTransformType.local: Relative to parent
XRTransformType.stage: World space in USD stage
XRTransformType.scope: Intermediate scope transform

Committing Link Transforms#

When an object is linked to a device, you can “commit” its current offset:

# Object is linked to controller
# User manually repositions it
# Commit new offset to layer

xr_usd_layer.commit_link_transform(
    path="/_xr/my_ui/controller/attached_object",
    layer=""  # Use default layer
)

Visibility Control#

# Show object
xr_usd_layer.show("/_xr/my_ui/my_object")

# Hide object
xr_usd_layer.hide("/_xr/my_ui/my_object")

# Check visibility
is_visible = xr_usd_layer.is_visible("/_xr/my_ui/my_object")

# Hide entire group
xr_usd_layer.hide()  # Hides all objects in layer

Pickability (Raycasting)#

Control which objects can be hit by raycasts.

# Make object pickable
xr_usd_layer.set_pickable("/_xr/my_ui/button", True)

# Make object non-pickable
xr_usd_layer.set_pickable("/_xr/my_ui/background", False)

# Check if pickable
is_pickable = xr_usd_layer.get_pickable("/_xr/my_ui/button")

Target Info#

Get information about raycast hits on managed objects.

# After raycast hit
target_info = xr_usd_layer.get_target_info("/_xr/my_ui/hit_object")

if target_info.valid:
    position = target_info.position
    normal = target_info.normal
    instance_id = target_info.instance_id
    
    target_path = target_info.get_target_usd_path()
    model_path = target_info.get_target_enclosing_model_usd_path()
    
    print(f"Hit at: {position}")
    print(f"Target: {target_path}")

Grabbability#

Mark objects as grabbable for grab tools.

# Make object grabbable
xr_usd_layer.set_grabbable("/_xr/my_ui/movable_object", True)

# Check if grabbable
is_grabbable = xr_usd_layer.is_grabbable("/_xr/my_ui/movable_object")

Group Management#

Groups allow bulk operations on related objects.

# Add objects to group "controllers"
xr_usd_layer.add_asset(
    path="/_xr/ui/left_ctrl",
    group="controllers",
    file_path="/assets/controller.usd"
)

xr_usd_layer.add_asset(
    path="/_xr/ui/right_ctrl",
    group="controllers",
    file_path="/assets/controller.usd"
)

# Remove entire group
xr_usd_layer.remove_group("controllers")

Removing Objects#

# Remove single object
xr_usd_layer.remove("/_xr/my_ui/my_object")

# Remove group
xr_usd_layer.remove_group("tools")

# Clear entire layer
xr_usd_layer.clear()

# Check if managed
is_managed = xr_usd_layer.is_managed_prim("/_xr/my_ui/my_object")

Coordinate Conversion#

Convert vectors between USD stage and XR session coordinates.

from pxr import Gf

# Stage to session (XR space)
stage_vec = Gf.Vec3d(100, 0, 0)  # 100cm in USD
session_vec = xr_usd_layer.convert_vector_from_stage_to_session(stage_vec)

# Session to stage
session_vec = Gf.Vec3d(1.0, 0, 0)  # 1m in XR
stage_vec = xr_usd_layer.convert_vector_from_session_to_stage(session_vec)

# Normal vectors (for lighting, physics)
stage_normal = Gf.Vec3d(0, 1, 0)
session_normal = xr_usd_layer.convert_normal_vector_from_stage_to_session(stage_normal)

Asset Loading#

Load assets from XR asset packages.

# Load asset by name (resolved from asset packages)
asset_path = xr_usd_layer.load_asset("generic_controller")
print(f"Loaded asset: {asset_path}")

# Use with add_asset
xr_usd_layer.add_asset(
    path="/_xr/ui/controller",
    group="controllers",
    file_path=asset_path
)

Metadata#

Store and retrieve custom metadata on the layer.

# Set metadata
xr_usd_layer.set_meta_data("tool_mode", "select")
xr_usd_layer.set_meta_data("ui_state", "menu_open")

# Get metadata
tool_mode = xr_usd_layer.get_meta_data("tool_mode")
print(f"Tool mode: {tool_mode}")

# Clear metadata by prefix
xr_usd_layer.clear_meta_data("ui_")  # Clears all ui_* keys

Complete Example: Interactive XR Menu#

import omni.kit.xr.core
from pxr import Gf

class XRMenuSystem:
    """Interactive 3D menu in XR."""
    
    def __init__(self, xr_core):
        self.xr_core = xr_core
        self.xr_usd_layer = None
        self.menu_visible = False
    
    def create(self):
        """Create XR menu."""
        # Create USD layer
        self.xr_usd_layer = self.xr_core.create_xr_usd_layer(
            usd_path="/_xr/menu_system",
            meters_per_unit=0.01,
            up_axis='y'
        )
        
        # Create menu root (billboard facing user)
        menu_root = self.xr_usd_layer.add_reorient(
            path="/_xr/menu_system/menu_root",
            group="menu",
            alignment=omni.kit.xr.core.XROrientationAlignment.device_up_right,
            input_device="/user/head",
            visible=False  # Start hidden
        )
        
        # Add menu background
        background = self.xr_usd_layer.add_asset(
            path=f"{menu_root}/background",
            group="menu",
            file_path="/assets/ui/menu_background.usd",
            pickable=True
        )
        
        # Add menu buttons
        button_spacing = 15.0  # 15cm
        for i, button_name in enumerate(["Select", "Move", "Delete"]):
            button_path = self.xr_usd_layer.add_asset(
                path=f"{menu_root}/button_{i}",
                group="menu",
                file_path="/assets/ui/button.usd",
                transform=Gf.Matrix4d().SetTranslate(
                    Gf.Vec3d(0, -i * button_spacing, 0)
                ),
                pickable=True
            )
            
            # Store metadata
            self.xr_usd_layer.set_meta_data(
                f"button_{i}_action",
                button_name.lower()
            )
        
        return menu_root
    
    def show(self, position=None):
        """Show menu at position."""
        if not self.xr_usd_layer:
            return
        
        if position:
            self.xr_usd_layer.set_position(
                "/_xr/menu_system/menu_root",
                position
            )
        
        self.xr_usd_layer.show("/_xr/menu_system/menu_root")
        self.menu_visible = True
    
    def hide(self):
        """Hide menu."""
        if self.xr_usd_layer:
            self.xr_usd_layer.hide("/_xr/menu_system/menu_root")
            self.menu_visible = False
    
    def handle_selection(self, hit_path):
        """Handle button selection."""
        # Check if it's a button
        if "button_" in hit_path:
            # Extract button index
            button_idx = hit_path.split("button_")[1].split("/")[0]
            
            # Get button action from metadata
            action = self.xr_usd_layer.get_meta_data(
                f"button_{button_idx}_action"
            )
            
            print(f"Button pressed: {action}")
            return action
        
        return None
    
    def cleanup(self):
        """Clean up menu."""
        if self.xr_usd_layer:
            self.xr_usd_layer.clear()
            self.xr_usd_layer.invalidate()

# Usage
menu_system = XRMenuSystem(xr_core)
menu_system.create()

# Show menu in front of user
menu_system.show(position=Gf.Vec3d(0, 150, -100))  # 1.5m up, 1m forward

# On raycast hit
def on_menu_hit(hit_path):
    action = menu_system.handle_selection(hit_path)
    if action:
        print(f"Execute action: {action}")

# Hide menu
menu_system.hide()

Best Practices#

Performance#

Use Groups: Organize objects for bulk operations
Minimize Transforms: Batch transform updates when possible
Use USD-RT: Set use_usd=False for real-time updates
Limit Managed Objects: Only manage objects that need fast updates

Organization#

Consistent Naming: Use clear, hierarchical paths
Logical Groups: Group related objects (tools, controllers, ui)
Metadata for State: Store component state in metadata

Coordinate Systems#

Be Consistent: Stick to one unit system (usually cm for XR)
Use Conversions: Use built-in conversion functions
Test Scaling: Verify object sizes in XR

Next Steps#

Custom Tools – Build tools using XRUsdLayer
Event System – React to XR events
Input & Interaction – Handle user input

Key Takeaways

XRUsdLayer provides fast USD integration for XR
Managed objects include assets, beams, arcs, links
Links to input devices enable automatic following
Transform management supports multiple coordinate systems
Groups enable bulk operations
Use USD-RT for real-time performance