1. OmniGraph: Input Devices

1.1. Learning Objectives

This tutorial introduces how to use OmniGraph to connect up input devices such as a gamepad controller or a keyboard for interacting with a simulation in Omniverse Isaac Sim. After this tutorial, you will know how to map inputs from a gamepad to operate a mobile robot in simulation, and receive inputs from keyboard to manipulate objects on stage.

1.2. Getting Started

Prerequisites

  • Review the GUI Tutorial series, especially OmniGraph, prior to beginning this tutorial.

  • The gamepad tutorial requires an XBox controller to run. Other gamepads may function; however their bindings may be different than this tutorial.

1.3. Gamepad Inputs

1.3.1. Running the Example

The NVIDIA Kaya robot is a three-wheeled holonomic drive robot designed to be as accessible as possible. It is made of 3D-printed parts and hobbyist components, and features a RealSense depth camera for mapping, localization, and object recognition. An on-board IMU and position feedback from the wheel motors provide accurate odometry information.

The Kaya’s sensor array and omni-directional motion can be simulated in Omniverse Isaac Sim. Let’s learn how:

  1. Begin by connecting the gamepad controller to the PC running Omniverse Isaac Sim.

  2. Next, open the Kaya Gamepad example: go to the top Menu Bar and Click Isaac Examples > Input Devices > Kaya Gamepad.

  3. Press the LOAD button to load a Kaya robot into the scene.

  4. Press Play to start controlling the Kaya via the gamepad.

1.3.2. OmniGraph Layout

To see the layout of the graph:

  1. Make sure the example is loaded and that there is a Kaya on the stage, open visual programming window by Window > Visual Scripting > Action Graph.

  2. Click on the Edit Graph icon inside the Action Graph tab.

  3. In the pop window, select /World/ActionGraph. There may be multiple action graphs for a given stage, they will all be listed here.

  4. The graph seeing in the image below will open. To examine the properties of each node, simply select the node on the graph display, the corresponding properties will display in the Property tab on the bottom right. The nodes are also listed on the stage tree. You can select the nodes there too.

../_images/isaac_tutorial_advanced_inputs_gamepad_graph.png

For each joystick axis or button on the gamepad, there is a dedicated node. All the mapping and hardware connection are done behind the scenes, and the output of the node is merely the digital readout of the selected control. Those then are processed and fed into the holonomic controller that calculates the rotational speed of the wheels, which then is fed into the articulation controller that is assigned to the robot. OmniGraph tutorial under the GUI Tutorials section has more details regarding the controllers.

1.4. Keyboard Inputs

1.4.1. Running the Example

The OmniGraph Keyboard example is designed to show how to receive keyboard inputs and connect them up to modifying specific attributes of objects on stage.

To run the example:

  1. Open the example by going to the top Menu Bar and Click Isaac Examples > Input Devices > Omniverse Keyboard.

  2. Press Play. Now you can grow the size of the cube on stage by pressing A and shrink it by pressing D on the keyboard.

1.4.2. OmniGraph Layout

To see the layout of the graph:

  1. Make sure the example is loaded and that there is a cube on stage, open visual programming window by Window > Visual Scripting > Action Graph.

  2. Click on the Edit Graph icon inside the Action Graph tab.

  3. In the pop window, select /World/ActionGraph. There may be multiple action graphs for a given stage, they will all be listed here.

  4. The graph seeing in the image below will open. To examine the properties of each node, simply select the node on the graph display, the corresponding properties will display in the Property tab on the bottom right. The nodes are also listed on the stage tree. You can select the nodes there too. Note for each key on the keyboard, there is a dedicated node.

../_images/isaac_tutorial_advanced_inputs_keyboard_graph.png

1.5. Summary

This tutorial covered the following topics:

  1. Using OmniGraph to receive inputs from external devices such as gamepad and keyboard.

  2. How to read and write object’s USD attributes in OmniGraph

1.5.1. Further Reading