Running Isaac Sim For the First Time

Now that you have installed Isaac Sim 2020.1 either locally on your workstation or in the containerized / remote-streaming configuration, let’s start with a tour of fundamental features. Our tour will focus on NVIDIA’s Kaya robot, designed as a showcase example for NVIDIA’s Isaac Robot Engine and Jetson Nano compute platform. You can build your own Kaya using a 3D printer and off-the-shelf parts, but before you go there you might to check it out in simulation. This tour is written as if you are running the simulation locally on a workstation, but applies equally if you are running the remote client and container instead.

Kaya Tour Begins Here

When you start Isaac Sim , you are prompted to log in to an Omniverse Nucleus server through a dialog box, as shown in the image below. The dialog box in the image shows the server is ‘localhost’, which applies if you are connecting to a Nucleus server installed on your local workstation alongside Isaac Sim. If instead you wish to connect to a Nucleus server on a remote machine, type in the IP address and port of the Nucleus server you configured during the installation. The Nucleus server holds all of the example assets for this tour, as well as other starter content that ships with Isaac Sim 2020.1, so if you haven’t yet installed Nucleus, you must do so before proceeding with the tour.


Now that you are logged into the Nucleus server, you should see several folders appear in the Content panel at the bottom of your Isaac Sim application window, as shown in the image below. Click on the folder labeled ‘Isaac’ in the tree view index on the left hand side of the Content panel, and you should then see a set of folders appear in the right-hand side of the panel, as shown in the image.


Next, look for the menu labeled ‘Isaac Robotics’ in the menu bar at the top of the Isaac Sim application window. If there is no menu item labeled ‘Isaac Robotics’, you can enable it by choosing menu: Window->Extension Manager to open the panel as shown below. Scroll down and find ‘omni.isaac.samples’. Place a check mark next to it as illustrated and close the Extension Manager panel.


Important: If you do not already have a game controller connected to your PC, now would be a good time to do that.

Once you find the Isaac Robotics item in the menu bar, select Samples->Kaya Joystick as shown in the image below, and you should notice that a dialog box with two buttons and a checkbox will appear as illustrated here. You can dock this new Kaya Joystick panel by dragging it onto one of the other panels in the application window.


On the Kaya Joystick panel, press the top button to load the Kaya sample scene, and then press the second button to connect the game-pad to your Kaya model. Then as shown in the video below, press the ‘Play’ button located on the vertical bar to the left of the Viewport to start the simulation loop. The Kaya should now respond to the joysticks on your game controller. When you are finished test driving the model robot, press the ‘Stop’ button to return Kaya to its starting pose.

Kaya Takes the Stage

Using terminology from Universal Scene Description, the system Omniverse uses to composite assets for simulation, in Isaac Sim the action you see in the Viewport takes place on a ‘Stage’. In the image below of the Isaac Sim application window, as well as in most similar images in our Kaya tour, the Stage panel appears to the right of the Viewport panel in the Isaac Sim GUI. In the image you can see the top-level view of the scene composited on the Stage, comprising objects for the World, physics, kaya and background. In this context you might hear Kaya and other objects on the Stage referred to as ‘USD assets’ or ‘USD models’. When you press the ‘Play’ button, the simulation clock starts running and action begins on the Stage as seen in the Viewport.

Below the Stage panel in the image below, notice the panels labeled ‘Details’ and ‘PhysX Properties’. Together with the Viewport, the Stage and the Content panels, the Details and PhysX Properties panels are fundamental to Isaac Sim, and you should become familiar with them as you begin to develop your own simulations. We will return to explore details of the USD model of Kaya and the Stage later in the tour.


Play with Lighting

There are multiple lights and types of lights at use in the Kaya scene. The main light switches to turn off all types of lights are found on the “Real Time Mode Settings” below. You can bring up this panel by choosing Menu:Rendering->Real Time Mode Settings. To turn off ‘Direct Lights’, such as ‘Distant’, ‘Sphere’ or ‘Dome’ lights that are placed on the Stage, use the checkbox in the ‘Direct Light’ section of the panel.


If you turn off Direct Lights using the checkbox, you should see Kaya in ‘ambient light’, also known as “Indirect Diffuse Light”, as in the image below.


If you set the ‘Intensity’ of Indirect Diffuse Light to zero, you should now see an image like the one below. Notice that the white lines on the floor are ‘emissive’ materials, so they glow in the dark when all lights are switched off. The yellow outline of the Kaya is visible because the Kaya model is selected on the Stage.


Next let’s create a new Direct light and attach it to the Kaya itself. Start by choosing Menu->Create->Light->Sphere Light. With the new light selected in the Stage panel, adjust the Scale of the light by typing new values in the Scale fields of the Details panel as shown in the upper right of the GUI in the image below. At this point the new Sphere Light is not attached to the Kaya, but sits in some position near it as shown.


To attach the light to the Kaya, use your mouse to select the Sphere Light object in the Stage panel, and drag/drop it onto the object named “base_link” in the hierarchy of Kaya. The base_link is the physical rigid body corresponding to the Kaya chassis. These appear in the Stage panel in the image below. Note that the light has been moved to in front and above the Kaya.


If all went well with the procedure above, you should see results as in the video below. Press ‘Play’ to start the simulation and drive the Kaya around the environment, and you should see the light moving with it.

Attach a Camera

If you are following this guide step-by-step, up until now you have been using the default ‘Perspective’ view in the Viewport. Let’s create a camera and attach it to the Kaya, so that the camera moves with the robot. Start by using the mouse and keyboard to move the Perspective camera to a desired location. In the image below the viewpoint has been adjusted to just behind and above the robot, looking in the direction the robot would travel ‘straight ahead’.


Now using your mouse, click on the camera icon in the upper-left of the Viewport, where it should say ‘Perspective’ currently. Select Camera–>Create Camera from View. This will create a new object on the stage called ‘Camera’ by default, you can rename it if you choose. Although you have created the camera, it is not yet the active camera in the Viewport, so your view is being generated by the Perspective camera currently.


Now follow the video below to select your Camera on the Stage panel, and drag-drop it onto the Kaya->base_link entry further down on the Stage. This will attach the camera to the physical rigid body that represents the chassis of the robot in the simulation. Next, click on the camera selector in the Viewport, the icon resembling a camera in the upper-left, and choose the new Camera for the Viewport. Finally, hit the Play button and drive the Kaya around the environment. You should see the perspective of a third-person follow-cam. The bumpy wheels cause a noticeable vibration in the camera, this is not an error, it is a plausible physical result of the design of the robot.

If you would like to see the new Camera itself, change back to Perspective view and move back, as shown below.


The camera you created can be configured as to focal length and distance, aperture and other properties, as below. You can read more about about them in the camera docs here.


Importing or Building Robot Models

This section will take a first look at how to import a robot model defined in URDF, as well as building a simulated robot from an engineering design model (CAD). Thorough treatment of these topics requires familiarity with several important systems in Isaac Sim, such as scene management, scripting, and physics. This section will introduce the important concepts, and additional details are available in the documentation for specific systems and pre-built models. Please note that if you find the GUI display panels for the robot layout on the Stage to be cumbersome or confusing, these are limitations of the early-access release. The Isaac Sim team is improving the robotics-specific GUI tools for an upcoming release.

Import Kaya from URDF to USD

From the Gazebo tutorial:: “The Unified Robotic Description Format (URDF) is an XML file format used in ROS to describe all elements of a robot. To use a URDF file in Gazebo, some additional simulation-specific tags must be added to work properly with Gazebo.”

Because many users of Isaac Sim already work with ROS and the Gazebo simulator with some particular robot, Isaac Sim offers a pathway to import URDF files in order to pull existing robot models into the simulation. But as the tutorial quote above indicates, URDF itself is not a complete specification suitable for simulation, and even Gazebo cannot simulate a URDF-defined robot properly unless additional Gazebo-specific data is supplied.

Although URDF importer support is an important feature for Isaac Sim, note that URDF is not a native representation for robots in Isaac Sim or in the underlying Omniverse engine. Isaac Sim and Omniverse represent all objects, including machines and environments, using the Universal Scene Description (USD) system. The URDF format is generally useful only within ROS, it is not easily portable, and in general is not a great choice to define simulation models.

Earlier in this tour you created a Kaya model by clicking a button labeled ‘Load Kaya’ on the control panel for the Kaya Joystick sample. Behind the scenes, what happens here is that the button invokes Python code that loads the Kaya model from a USD file that was prepared earlier.

    self._stage = self._usd_context.get_stage()
    print("loading environment")
    asset_path = "omni:/Isaac/Robots/Kaya"
    kaya_usd = asset_path + "/kaya.usd"
    speed_gain = 10.0


    self.kaya = Kaya(stage=self._stage, dc=self._dc, usd_path=kaya_usd, prim_path="/kaya", speed_gain=speed_gain)
    if self._create_background_chk.value:
            offset=Gf.Vec3d(0, 0, -9),

The model loaded by the script can also be loaded directly in the GUI from the Content browser panel, the USD files are found in the folder /Isaac/Robots/Kaya.

This next section of the tour will explore importing the Kaya model from URDF and converting to USD, which you might wish to do with another robot model when you begin using Isaac Sim in your work.

Follow the video below by choosing Menu: Isaac Robotics –>URDF –> Kaya to bring up a panel for loading the URDF model and converting it to USD. The top button loads the robot itself, the second button creates the environment, initializes world physics and configures the simulation.

Next, let’s take a closer look at the USD generated by the URDF importer. Take some time here to explore the components of the model displayed in the tree view in the Stage panel. Notice in the image below that the Console panel is visible, and a record of the importing process appears there.


Also note the purple outlines on the shapes of the robot, these are physics collision shapes. In order to see them on your screen, enable the PhysX Debug panel by choosing Menu: Physics–>PhysX Debug Window, then dock the window and choose ‘Selected’ for the value of “Show Collision Shapes”, and make sure ‘Kaya’ is selected in the Stage. See image below.


Examining the top-level structure of the Kaya as seen in the Stage panel reveals that all of the ‘XForm’ type items under Kaya are PhysX rigid bodies. This can be confirmed by inspecting the PhysX Properties panel for each XForm child of the Kaya object, here illustrated with the base_link XForm selected. When an XForm is a rigid body, the PhysX Properties panel will display the annotations for Physics Body and PhysX Rigid Body as shown. An important point here is that the robot is laid out as a set of rigid bodies: the base link (or chassis), the three axle assemblies, and thirty rollers. The visual meshes of the rollers are actually attached to the axles, while the roller objects themselves are not visible except through debug rendering modes. This should become clear in the images that follow.


In the image below most of the visual meshes of the Kaya have been hidden by clicking on the ‘eye’ icons in the Stage for all components of the Kaya, except for two of the wheels and one of the axles. This was done to illustrate how the collision shapes for the rigid bodies comprising the robot relate to the visual appearance. In particular, notice that the collision shape of each wheel is made of a dense collection of purple spheres, one for each ‘roller’ in the holonomic drive wheels.


In the next view, only a single roller is ‘selected’, so the purple collision sphere of the single selected roller appears below the axle.


Below is the same view as above, except that the visual wheels and axle have been hidden, and the visual meshes of the chassis are visible for reference.


Kaya comprises a relatively complex arrangement of rigid bodies due to the design of the holonomic drive wheels, and inspection of the components on the USD stage shows that the structure is not entirely intuitive. Each part labeled “roller_x_y_Z” that lives at the top level of the Kaya hierarchy defines a rigid body that implements a roller, but note that the visual mesh for each roller is not included in the roller part. The visual mesh hierarchy for each wheel descends from a part labeled ‘axle_#’, which are found at the same level in the hierarchy as the rollers. See detail below, which shows the definition of a roller (“roller_2_1_4”) and the details of axle_0. There the objects labeled “dae” are the visual meshes imported from the URDF file in Digital Asset Exchange format (the Collada mesh description system). Also notice the objects “roller_x_y_z_joint”, which store the joint information that connect the roller rigid bodies to the axle/wheel hub rigid bodies.


The image below presents visualization detail of an axle joint. The green wire-frame disk represents the rotational freedom of a revolute joint whose rotation axis is aligned to the symmetry axis of the green axle mesh at the center of the wheel. This joint connects the wheel / hub assembly rigid body to the rigid body of the chassis.


Compare the next two images below to the image above. You will notice a joint visualization for one of the rollers, whose visual mesh is hidden from view in the first image below, and shown un-hidden in the second, for the purpose of illustration.


In each of these images, take note that the PhysX Properties panel is displayed on the lower-right of the application window, below the Stage panel. When an object is selected in the stage, for example the roller joint in the image above, the details of the physics configuration appear on the PhysX Properties panel. All of the physical properties of the components of the Kaya model were set by the URDF importer. The importer will set some of these values to a reasonable default configuration if the URDF does not specify sufficient detail. Take time to explore the many components of the model by clicking through them on the stage, and observe the PhysX Properties for the various rigid bodies, collision shapes and joints.


Build Kaya from CAD Model