4. Hello Robot

4.1. Learning Objectives

This tutorial details how to add and move a mobile robot in Omniverse Isaac Sim in an extension application. After this tutorial, you will understand how to add a robot to the simulation and apply actions to its wheels using python.

10-15 Minute Tutorial

4.2. Getting Started

Prerequisites

  • Please review Hello World prior to beginning this tutorial.

Begin with the source code of the Awesome Example developed in the previous tutorial, Hello World.

4.3. Adding a Robot

Let’s begin by adding a NVIDIA Jetbot to the scene. We can access the library of Omniverse Isaac Sim robots, sensors, and environments located on a Omniverse Nucleus Server using python as well as navigating through it using the Content window. Note: the server connected to is the one that will be shown. Make sure you went through Isaac Sim First Run before proceeding.

../_images/isaac_sim_nucleus_navigate.gif
  • Add the assets by simply dragging them to the stage window or the viewport.

  • Now lets try to do the same thing through python in our Awesome Example

  • Create a new stage File > new > Don’t Save

  • Open the exts/omni.isaac.examples/omni/isaac/examples/user_examples/hello_world.py file by clicking on the Open Source Code button in the Awesome Example window.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    from omni.isaac.examples.base_sample import BaseSample
    from omni.isaac.core.utils.nucleus import find_nucleus_server
    from omni.isaac.core.utils.stage import add_reference_to_stage
    from omni.isaac.core.robots import Robot
    import carb
    
    class HelloWorld(BaseSample):
        def __init__(self) -> None:
            super().__init__()
            return
    
        def setup_scene(self):
            world = self.get_world()
            world.scene.add_default_ground_plane()
            # Use the find_nucleus_server instead of changing it every time
            # you configure a new server with /Isaac folder in it
            result, nucleus_server = find_nucleus_server()
            if result is False:
                # Use carb to log warnings, errors and infos in your application (shown on terminal)
                carb.log_error("Could not find nucleus server with /Isaac folder")
            asset_path = nucleus_server + "/Isaac/Robots/Jetbot/jetbot.usd"
            # This will create a new XFormPrim and point it to the usd file as a reference
            # Similar to how pointers work in memory
            add_reference_to_stage(usd_path=asset_path, prim_path="/World/Fancy_Robot")
            # Wrap the jetbot prim root under a Robot class and add it to the Scene
            # to use high level api to set/ get attributes as well as initializing
            # physics handles needed..etc.
            # Note: this call doesn't create the Jetbot in the stage window, it was already
            # created with the add_reference_to_stage
            jetbot_robot = world.scene.add(Robot(prim_path="/World/Fancy_Robot", name="fancy_robot"))
            # self.log_info is a convenience function to print info on the example window instead of
            # the terminal for debugging.
            # Note: before a reset is called, we can't access information related to an Articulation
            # because physics handles are not intialized yet. setup_post_load is called after
            # the first reset so we can do so there
            self.log_info("Num of degrees of freedom before first reset: " + str(jetbot_robot.num_dof)) # prints None
            return
    
        async def setup_post_load(self):
            self._world = self.get_world()
            self._jetbot = self._world.scene.get_object("fancy_robot")
            # Print info about the jetbot after the first reset is called
            self.log_info("Num of degrees of freedom after first reset: " + str(self._jetbot.num_dof)) # prints 2
            self.log_info("Joint Positions after first reset: " + str(self._jetbot.get_joint_positions()))
            return
    

Press the PLAY button to start simulating the robot. Although it is simulating, it is not moving. The next section walks through how to make the robot move.

4.4. Move the Robot

In Omniverse Isaac Sim, Robots are constructed of physically accurate articulated joints. Applying actions to these articulations make them move.

Let’s apply random velocities to the Jetbot’s articulation controller to get it moving.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
from omni.isaac.examples.base_sample import BaseSample
from omni.isaac.core.utils.nucleus import find_nucleus_server
from omni.isaac.core.utils.stage import add_reference_to_stage
from omni.isaac.core.robots import Robot
from omni.isaac.core.utils.types import ArticulationAction
import carb
import numpy as np


class HelloWorld(BaseSample):
    def __init__(self) -> None:
        super().__init__()
        return

    def setup_scene(self):
        world = self.get_world()
        world.scene.add_default_ground_plane()
        result, nucleus_server = find_nucleus_server()
        if result is False:
            carb.log_error("Could not find nucleus server with /Isaac folder")
        asset_path = nucleus_server + "/Isaac/Robots/Jetbot/jetbot.usd"
        add_reference_to_stage(usd_path=asset_path, prim_path="/World/Fancy_Robot")
        jetbot_robot = world.scene.add(Robot(prim_path="/World/Fancy_Robot", name="fancy_robot"))
        return

    async def setup_post_load(self):
        self._world = self.get_world()
        self._jetbot = self._world.scene.get_object("fancy_robot")
        # This is an implicit PD controller of the jetbot/ articulation
        # setting PD gains, applying actions, switching control modes..etc.
        # can be done through this controller.
        # Note: should be only called after the first reset happens to the world
        self._jetbot_articulation_controller = self._jetbot.get_articulation_controller()
        # Adding a physics callback to send the actions to apply actions with every
        # physics step executed.
        self._world.add_physics_callback("sending_actions", callback_fn=self.send_robot_actions)
        return


    def send_robot_actions(self, step_size):
        # Every articulation controller has apply_action method
        # which takes in ArticulationAction with joint_positions, joint_efforts and joint_velocities
        # as optional args. It accepts numpy arrays of floats OR lists of floats and None
        # None means that nothing is applied to this dof index in this step
        # ALTERNATIVELY, same method is called from self._jetbot.apply_action(...)
        self._jetbot_articulation_controller.apply_action(ArticulationAction(joint_positions=None,
                                                                            joint_efforts=None,
                                                                            joint_velocities=5 * np.random.rand(2,)))
        return

Press the PLAY button to start simulating the robot. Note: pressing STOP then PLAY in this workflow might not reset the world properly. To reset press the RESET button instead.

Extra Practice This example applies random velocities to the Jetbot’s articulation controller. Try the following exercises:

  • Make the Jetbot move backwards.

  • Make the Jetbot turn right.

  • Make the Jetbot stop after 5 seconds.

4.5. Using the Jetbot Class

Omniverse Isaac Sim also has Robot specific extensions that provide further customized functions and access to other controllers and tasks (more on this later on). Let’s re-write the previous code using the Jetbot Class to make it simpler.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
from omni.isaac.examples.base_sample import BaseSample
from omni.isaac.jetbot import Jetbot
from omni.isaac.core.utils.types import ArticulationAction
import numpy as np


class HelloWorld(BaseSample):
    def __init__(self) -> None:
        super().__init__()
        return

    def setup_scene(self):
        world = self.get_world()
        world.scene.add_default_ground_plane()
        # This takes care of adding a reference to the appropriate asset
        # in case that the prim path doesn't exist in the stage yet
        jetbot_robot = world.scene.add(Jetbot(prim_path="/World/Fancy_Robot", name="fancy_robot"))
        return

    async def setup_post_load(self):
        self._world = self.get_world()
        self._jetbot = self._world.scene.get_object("fancy_robot")
        self._world.add_physics_callback("sending_actions", callback_fn=self.send_robot_actions)
        return


    def send_robot_actions(self, step_size):
        self._jetbot.apply_wheel_actions(ArticulationAction(joint_positions=None,
                                                            joint_efforts=None,
                                                            joint_velocities=5 * np.random.rand(2,)))
        return
../_images/isaac_sim_hello_robot.gif

4.6. Summary

This tutorial covered the following topics:

  1. Adding Omniverse Isaac Sim library components from a Nucleus Server

  2. Adding a Robot to the World

  3. Using a Joint-Level controller to articulate a Robot

  4. Using Robot Classes

  5. Using Robot specific extensions

4.6.1. Next Steps

Continue on to the next tutorial in our Essential Tutorials series, Adding A Controller, to learn how to add a controller to Jetbot.

4.6.2. Further Learning

Nucleus Server

Robot Specific Extensions

  • Omniverse Isaac Sim provides several robot extensions such as omni.isaac.franka, omni.isaac.universal_robots and much more. Checkout the standalone examples located at standalone_examples/api/omni.isaac.franka and standalone_examples/api/omni.isaac.universal_robots/ if you are interested to know more.