Joystick Manipulation (Manip)

Here we will cover how to go about using the Joystic Manipulation extension (or Manip for short). If you have not already taken a look at the Kaya Sample Application its suggested to do so now. The Kaya sample we provide uses this extension to allow you to control the Kaya robot with a game pad.

If not already enabled, enable this extension from the Windows -> Extensions menu at the top of the Omniverse Isaac Sim window. Search for the omni.isaac.manip extension and ensure it is enabled.

Python API

Like most extensions, the Manip extension has a python API, allowing you to bind controls to joints and drives via script. Most of the functionality of the extension is handled exclusively in python, with only a few exceptions.

Acquiring the binding interface for Manip is done exactly like any other extension in Omniverse Isaac Sim: you simply call acquire_manip_interface. In a custom extension this should be done within your “on_startup” method.

1
2
3
from omni.isaac.manip import _manip

manip_interface = _manip.acquire_manip_interface()

The gamepad bindings operate using a simple event system, meaning you will need to define a function to serve as the gamepad event handler that will be called whenever a gamepad event is triggered. The event function must have the following signature

1
2
3
4
5
6
7
def func_name(axis: GamePadAxis, signal: float):
    # do stuff

# OR

def func_name(self, axis: GamePadAxis, signal: float):
    # do stuff

And you can bind this function to the gamepad by calling

1
manip_interface.bind_gamepad(func_name)

The bound function will only be called when the state of the controller changes. The axis is an enumeration value that indicates which axis is receiving signal. Every event will only have a single axis, even if the input to the controller is across multiple axes and in that case, the event function will be called multiple times. The signal is a float on [0,1] and can be interpreted as “how much” the state of the controller has changed.

A good example for event handling in the joystick manipulation extension can be found in our Kaya preview
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
.
.
.
self._joystick_deadzone = 0.2
self._gains = (4, 4, 0.5)
self._vel_target = np.zeros(3)
.
.
.
def _on_event_fn(self, axis, signal):
    if abs(signal) < self._joystick_deadzone:
        signal = 0

    if axis == GamePadAxis.eLeftStickY:
        self._vel_target[0] = signal * self._gains[0]
    elif axis == GamePadAxis.eLeftStickX:
        self._vel_target[1] = -signal * self._gains[1]
    elif axis == GamePadAxis.eRightStickX:
        self._vel_target[2] = -signal * self._gains[2]
    else:
        pass
    self.kaya.move(self._vel_target)

Notice that we are binding the joystick output directly to the target velocity we want. If the user quickly moves the joystick to one side, the velocity target will change dramatically, but if it’s held there, the target velocity will change very little, if at all.

Here is a good experiment to run in order to get a handle on exactly what the bindings are doing. Open the live script editor by selecting Window -> Script Editor from the menu bar at the top of the Omniverse Isaac Sim window. You should see two boxes appear in the script editor tab. Paste the following into the bottom box (you might need to use a keyboard shortcut to paste to that box)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
from omni.isaac.manip import _manip, GamePadAxis

manip_interface = _manip.acquire_manip_interface()

def myfunc(axis: GamePadAxis, data: float):
    print("called!  Axis is ", axis, " signal is", data)
    if axis == GamePadAxis.eLeftStickX:
        print("****in LX")
    elif axis == GamePadAxis.eLeftStickY:
        print("****in LY")
    if axis == GamePadAxis.eRightStickX:
        print("****in RX")
    elif axis == GamePadAxis.eRightStickY:
        print("****in RY")

manip_interface.bind_gamepad(myfunc)

In the Script Editor click the “Run” button or press Ctrl + Enter to execute the script. Then press the play button. As you manipulate your controller you will see the corresponding data print to the terminal from which you are running Omniverse Isaac Sim.

As an added bonus, you can modify that script in the editor window and execute it, and the print out to the terminal will change without needing to pause and replay the scene.