Input & Interaction#

Note

Applies to: Spatial Extensions, Kit 109.0.3+, CloudXR 6

This input and interaction reference covers working with XR input devices including controllers, HMDs, and handling user interactions.

Input Device Fundamentals#

XRCore manages all input devices connected to the XR system. Devices include controllers, head-mounted displays (HMDs), and eye trackers.

Device Names (OpenXR Standard)#

Standard device names follow the OpenXR path convention:

  • /user/head - HMD/Head tracking

  • /user/hand/left - Left controller

  • /user/hand/right - Right controller

  • /user/eye/left - Left eye tracking

  • /user/eye/right - Right eye tracking

Getting Input Devices#

import omni.kit.xr.core

xr_core = omni.kit.xr.core.XRCore.get_singleton()

# Get specific device
left_hand = xr_core.get_input_device("/user/hand/left")
right_hand = xr_core.get_input_device("/user/hand/right")
head = xr_core.get_input_device("/user/head")

# Get all devices
all_devices = xr_core.get_all_input_devices()
for device in all_devices:
    name = str(device.get_name())
    device_type = str(device.get_type())
    print(f"Device: {name}, Type: {device_type}")

# Check if device exists
if xr_core.has_input_device("/user/hand/left"):
    print("Left controller is connected")

Device Properties#

device = xr_core.get_input_device("/user/hand/left")

# Get device name (XRToken)
name_token = device.get_name()
name_str = str(name_token)  # Convert to string

# Get device type
type_token = device.get_type()
type_str = str(type_token)  # e.g., "controller", "hmd"

# Check if device is active
if device.is_active():
    print("Device is active")

Working with Poses#

Poses represent the position and orientation of devices in 3D space. Each device can have multiple poses.

Pose Types#

1. Raw Pose#

Unfiltered data directly from XR runtime.

raw_pose = device.get_raw_pose()

Use Case: When you need unfiltered, lowest latency data

2. Pose (Filtered)#

Filtered real-world coordinates.

pose = device.get_pose()  # Default pose (usually "aim" for controllers)

Use Case: General-purpose pose with noise filtering

3. Virtual World Pose#

Pose transformed into USD stage coordinates.

virtual_pose = device.get_virtual_world_pose()

Use Case: Positioning objects in the USD scene

Named Poses#

Controllers support multiple named poses:

  • grip: Physical grip position (where hand holds controller)

  • aim: Pointing position (typically extends from controller tip)

# Get grip pose (where hand holds)
grip_pose = device.get_pose("grip")

# Get aim pose (for pointing/raycasting)
aim_pose = device.get_pose("aim")

# List all available poses
pose_names = device.get_pose_names()
for pose_name in pose_names:
    print(f"Available pose: {str(pose_name)}")

Extracting Position and Orientation#

Poses are returned as Gf.Matrix4d (4x4 transformation matrices).

from pxr import Gf

# Get pose
pose = device.get_pose("aim")

# Extract position (translation)
position = pose.ExtractTranslation()  # Returns Gf.Vec3d
print(f"Position: x={position[0]}, y={position[1]}, z={position[2]}")

# Extract orientation (rotation as quaternion)
rotation = pose.ExtractRotation()  # Returns Gf.Quatd
print(f"Rotation: {rotation}")

# Extract as Euler angles (requires conversion)
rotation_mat = rotation.GetRotation()
# Or use pose.ExtractRotationMatrix()

# Get forward direction (for raycasting)
forward = pose.TransformDir(Gf.Vec3d(0, 0, -1))  # -Z is forward

Complete Pose Example#

def print_device_pose(device_name):
    """Print detailed pose information for a device."""
    device = xr_core.get_input_device(device_name)
    
    if not device or not device.is_active():
        print(f"Device {device_name} not active")
        return
    
    # Get different pose types
    raw_pose = device.get_raw_pose()
    filtered_pose = device.get_pose()
    virtual_pose = device.get_virtual_world_pose()
    
    # Extract position from virtual pose
    position = virtual_pose.ExtractTranslation()
    rotation = virtual_pose.ExtractRotation()
    
    print(f"Device: {device_name}")
    print(f"  Position: ({position[0]:.3f}, {position[1]:.3f}, {position[2]:.3f})")
    print(f"  Rotation: {rotation}")
    
    # Get forward direction
    forward = virtual_pose.TransformDir(Gf.Vec3d(0, 0, -1))
    print(f"  Forward: ({forward[0]:.3f}, {forward[1]:.3f}, {forward[2]:.3f})")

# Use it
print_device_pose("/user/hand/left")

Button and Gesture Input#

Input Names and Gestures#

Each input device has buttons and sensors that can be queried.

Common Button Names:

  • trigger - Index finger trigger

  • squeeze - Grip/squeeze button

  • menu - Menu/system button

  • thumbstick - Analog thumbstick

  • trackpad - Touch trackpad

  • a, b, x, y - Face buttons

Gesture Types:

  • click - Button pressed (0.0 or 1.0)

  • touch - Button touched (0.0 or 1.0)

  • value - Analog value (0.0 to 1.0, e.g., trigger pressure)

  • x - Thumbstick/trackpad X axis (-1.0 to 1.0)

  • y - Thumbstick/trackpad Y axis (-1.0 to 1.0)

Reading Button States#

device = xr_core.get_input_device("/user/hand/left")

# Read trigger value (0.0 to 1.0)
trigger_value = device.get_input_gesture_value("trigger", "value")
print(f"Trigger: {trigger_value}")

# Check if button is clicked (1.0 = pressed, 0.0 = not pressed)
menu_clicked = device.get_input_gesture_value("menu", "click")
if menu_clicked > 0.5:
    print("Menu button pressed")

# Read thumbstick axes
thumbstick_x = device.get_input_gesture_value("thumbstick", "x")
thumbstick_y = device.get_input_gesture_value("thumbstick", "y")
print(f"Thumbstick: ({thumbstick_x:.2f}, {thumbstick_y:.2f})")

# Check if input is touched
trigger_touched = device.get_input_gesture_value("trigger", "touch")

Discovering Available Inputs#

device = xr_core.get_input_device("/user/hand/left")

# Get all input names for this device
input_names = device.get_input_names()
print("Available inputs:")
for input_name in input_names:
    print(f"  - {str(input_name)}")
    
    # Get gestures for this input
    gestures = device.get_input_gesture_names(input_name)
    for gesture in gestures:
        print(f"    - {str(gesture)}")

# Check if specific input exists
if device.has_input("trigger"):
    print("Device has trigger")

if device.has_input_gesture("trigger", "value"):
    print("Trigger supports analog value")

Simulated Inputs#

XRCore automatically creates simulated inputs based on physical inputs:

D-Pad from Thumbstick:

  • dpad_up, dpad_down, dpad_left, dpad_right

# These are simulated from thumbstick
dpad_up = device.get_input_gesture_value("dpad_up", "click")

# Check what input it's based on
base_input = device.get_input_base("dpad_up")
print(f"dpad_up is based on: {str(base_input)}")  # "thumbstick"

# Check for overlapping inputs (to avoid binding conflicts)
overlapping = device.get_overlapping_inputs("dpad_up")
for overlap in overlapping:
    print(f"Overlaps with: {str(overlap)}")  # "thumbstick"

Event-Based Input#

Instead of polling button states every frame, use event generators for reactive input handling.

Binding Event Generators#

import carb.events

device = xr_core.get_input_device("/user/hand/left")

# Bind event generator for trigger
event_generator = device.bind_event_generator(
    "trigger",           # Input name
    "my_trigger_event",  # Event name prefix
    ["press", "release", "update"],  # Event types to generate
    {"press": "Fire weapon"}  # Optional tooltips (displayed in VR)
)

# Subscribe to press event
message_bus = xr_core.get_message_bus()

def on_trigger_press(event):
    print("Trigger pressed!")

trigger_press_sub = message_bus.create_subscription_to_pop_by_type(
    carb.events.type_from_string("my_trigger_event.press"),
    on_trigger_press
)

Available Event Types#

  • press - Button pressed (transition from 0 to 1)

  • release - Button released (transition from 1 to 0)

  • update - Sent every frame while pressed, includes state data

  • touch - Button touched

  • lift - Button no longer touched

  • suspend - Focus taken by another event generator

  • resume - Focus returned

  • state - Sent every frame with complete button state

Extracting Event Data#

Use XRInputDeviceGeneratorEvent wrapper to extract data from events:

from omni.kit.xr.core import XRInputDeviceGeneratorEvent

def on_trigger_update(event):
    # Wrap event for easy data access
    input_event = XRInputDeviceGeneratorEvent(event)
    
    # Get gesture values
    trigger_value = input_event.get_gesture_value("value")
    is_touched = input_event.get_gesture_value("touch")
    is_clicked = input_event.get_gesture_value("click")
    
    print(f"Trigger value: {trigger_value:.2f}")
    
    # Get device that generated event
    device = input_event.get_device()
    device_name = str(device.get_name())
    
    # Get input name
    input_name = str(input_event.get_input_name())

# Subscribe to update events
message_bus.create_subscription_to_pop_by_type(
    carb.events.type_from_string("my_trigger_event.update"),
    on_trigger_update
)

Complete Event Example#

class ControllerInputManager:
    def __init__(self, xr_core):
        self.xr_core = xr_core
        self.subscriptions = []
        
    def setup(self):
        left_hand = self.xr_core.get_input_device("/user/hand/left")
        
        if left_hand:
            # Bind trigger events
            left_hand.bind_event_generator(
                "trigger", "left_trigger", 
                ["press", "release"],
                {"press": "Select", "release": "Deselect"}
            )
            
            # Bind menu events
            left_hand.bind_event_generator(
                "menu", "left_menu",
                ["press"],
                {"press": "Open Menu"}
            )
            
            # Bind thumbstick for continuous updates
            left_hand.bind_event_generator(
                "thumbstick", "left_thumbstick",
                ["state"]  # Continuous updates
            )
        
        # Subscribe to events
        message_bus = self.xr_core.get_message_bus()
        
        self.subscriptions.append(
            message_bus.create_subscription_to_pop_by_type(
                carb.events.type_from_string("left_trigger.press"),
                self._on_trigger_press
            )
        )
        
        self.subscriptions.append(
            message_bus.create_subscription_to_pop_by_type(
                carb.events.type_from_string("left_thumbstick.state"),
                self._on_thumbstick_update
            )
        )
    
    def _on_trigger_press(self, event):
        print("Left trigger pressed - perform selection")
        # Implement selection logic here
    
    def _on_thumbstick_update(self, event):
        input_event = XRInputDeviceGeneratorEvent(event)
        x = input_event.get_gesture_value("x")
        y = input_event.get_gesture_value("y")
        
        if abs(x) > 0.1 or abs(y) > 0.1:
            print(f"Thumbstick: ({x:.2f}, {y:.2f})")
            # Implement navigation logic here
    
    def cleanup(self):
        self.subscriptions = []

# Usage
manager = ControllerInputManager(xr_core)
manager.setup()

Action Maps#

Action maps provide controller-agnostic input binding, allowing the same code to work across different controller types.

Getting Action Map#

action_map = xr_core.get_action_map()

if action_map:
    # Action map is available
    # This provides higher-level abstractions
    pass

Using Action Maps#

Action maps abstract device-specific inputs into semantic actions:

# Instead of binding to "trigger" specifically, bind to "select" action
# The action map handles mapping "trigger" on controllers,
# mouse click on desktop, etc.

Note: Action maps are configured through XR system settings and profile configurations. Refer to Settings Reference for details.

Raycasting for Selection#

Cast rays from controllers to select objects in the scene.

Basic Raycasting#

from omni.kit.xr.core import XRRay, XRRayQueryResult

def cast_selection_ray():
    left_hand = xr_core.get_input_device("/user/hand/left")
    pose = left_hand.get_pose("aim")
    
    # Get ray origin and direction
    origin = pose.ExtractTranslation()
    forward = pose.TransformDir(Gf.Vec3d(0, 0, -1))
    
    # Create ray
    ray = XRRay(
        origin,           # Ray origin
        forward,          # Ray direction (normalized)
        0.0,              # Min distance (meters)
        1000.0            # Max distance (meters)
    )
    
    # Submit raycast query
    xr_core.submit_raycast_query(ray, on_raycast_result)

def on_raycast_result(ray, result):
    """Callback when raycast completes (may be several frames later)."""
    if result.valid:
        hit_path = result.get_target_usd_path()
        hit_pos = result.hit_position
        hit_normal = result.normal
        hit_distance = result.hit_t
        
        print(f"Hit: {hit_path}")
        print(f"Position: {hit_pos}")
        print(f"Distance: {hit_distance}m")
        
        # Get model root (useful for selection)
        model_path = result.get_target_enclosing_model_usd_path()
        print(f"Model: {model_path}")

Async Raycasting#

Use async/await for cleaner code:

import asyncio
from omni.kit.xr.core import XRRay

async def async_raycast_example():
    left_hand = xr_core.get_input_device("/user/hand/left")
    pose = left_hand.get_pose("aim")
    
    origin = pose.ExtractTranslation()
    forward = pose.TransformDir(Gf.Vec3d(0, 0, -1))
    
    ray = XRRay(origin, forward, 0.0, 1000.0)
    
    # Await result
    result = await xr_core.execute_raycast_query_async(ray)
    
    if result.valid:
        print(f"Hit: {result.get_target_usd_path()}")
        return result.get_target_usd_path()
    else:
        print("No hit")
        return None

# Run async
asyncio.ensure_future(async_raycast_example())

Multi-Raycasting#

Cast multiple rays simultaneously:

def multi_raycast_example():
    # Create multiple rays
    rays = []
    for device_name in ["/user/hand/left", "/user/hand/right"]:
        device = xr_core.get_input_device(device_name)
        pose = device.get_pose("aim")
        
        origin = pose.ExtractTranslation()
        forward = pose.TransformDir(Gf.Vec3d(0, 0, -1))
        
        rays.append(XRRay(origin, forward, 0.0, 1000.0))
    
    # Submit all at once
    xr_core.submit_multi_raycast_query(rays, on_multi_raycast_result)

def on_multi_raycast_result(rays, results):
    """Callback with all results."""
    for i, result in enumerate(results):
        if result.valid:
            print(f"Ray {i} hit: {result.get_target_usd_path()}")

Controlling Pickable Objects#

Limit raycasting to specific objects:

# Make object pickable
xr_core.set_pickable_path("/World/Objects/MyObject", True)

# Make object non-pickable
xr_core.set_pickable_path("/World/Objects/Background", False)

# Reset to default
xr_core.unset_pickable_path("/World/Objects/MyObject")

Input Smoothing#

Enable input smoothing for steadier poses:

import carb.settings

settings = carb.settings.get_settings()

# Enable smoothing
settings.set("/persistent/xr/profile/vr/inputSmoothing/enabled", True)

# Set smoothing factor (0.0 = no smoothing, 1.0 = max smoothing)
settings.set("/persistent/xr/profile/vr/inputSmoothing/factor", 0.3)

Trade-off: Smoothing reduces jitter but adds latency.

Dominant Hand Configuration#

# Set dominant hand for controller layout
settings.set("/xr/persistent/tools/dominantHand", "right")  # or "left"

Next Steps#

Key Takeaways

  • Input devices follow OpenXR naming conventions

  • Three pose types: raw, filtered, virtual

  • Event generators provide reactive input handling

  • Raycasting enables object selection

  • Action maps provide controller-agnostic input