Running Isaac Sim For the First Time

Now that you have installed Omniverse Isaac Sim either locally on your workstation or in the containerized / remote-streaming configuration, let’s start with a tour of fundamental features. Our first sample will focus on NVIDIA JetBot, designed as a showcase example for NVIDIA Jetson Nano compute platform. Before you go and get yourself a JetBot, you might to check it out in simulation. This tour is written as if you are running the simulation locally on a workstation, but applies equally if you are running the remote client and container instead.

JetBot Sample Application

Getting Started

To run the samples, you will want to log in to an Omniverse Nucleus server. Depending on which version you have, this is done either through a dialog box automatically displayed in the center of the screen, or as a pop up option on the lower left corner of the screen under content tag, as shown in the image below. The dialog box in the image shows the server is ‘localhost’, which applies if you are connecting to a Nucleus server installed on your local workstation alongside Omniverse Isaac Sim. If instead you wish to connect to a Nucleus server on a remote machine, type in the IP address and port of the Nucleus server you configured during the installation. The Nucleus server holds all of the example assets for this tour, as well as other starter content that ships with Isaac Sim, so if you haven’t yet installed Omniverse Nucleus, you must do so before proceeding with the tour.

../_images/isaac_jetbot_1.png
../_images/isaac_jetbot_1a.png

Now that you are logged into the Nucleus server, you should see several folders appear in the Content panel at the bottom of your Omniverse Isaac Sim application window, as shown in the image below. Click on the folder labeled ‘Isaac’ in the tree view index on the left hand side of the Content panel, and you should then see a set of folders appear in the right-hand side of the panel, as shown in the image.

../_images/isaac_jetbot_2.png

Run the Sample

Next, look for the menu labeled ‘Isaac Robotics’ in the menu bar at the top of the Omniverse Isaac Sim application window. If there is no menu item labeled ‘Isaac Robotics’, you can enable it by choosing menu: Window->Extension Manager to open the panel as shown below. Scroll down and find ‘omni.isaac.samples’. Place a check mark next to it as illustrated and close the Extension Manager panel.

../_images/isaac_jetbot_extensions.jpg

Once you find the Isaac Robotics item in the menu bar, select Samples->JetBot Sample as shown in the image below, and you should notice that a dialog box with two buttons and a block of text pops up. You can dock this new JetBot Sample panel by dragging it onto one of the other panels in the application window.

../_images/isaac_jetbot_load.png

On the JetBot Sample panel, press the top button to load the JetBot sample scene. Then as shown in the video below, press the ‘Play’ button located on the vertical bar to the left of the Viewport to start the simulation loop. The JetBot should now respond to the w, s, a, and d keys on your keyboard. Press the ‘Reset’ button on the panel to return the robot to the origin of the scene. When you are finished test driving the model robot, press ‘Stop’ button to return JetBot to its starting pose.

Layout Explained

First let’s get familiar with the layout of Omniverse Isaac Sim GUI, so that you can locate the various menus and tabs that we will be mentioning throughout the tutorials.

Using terminology from Universal Scene Description, the system Omniverse uses to composite assets for simulation, in Isaac Sim the action you see in the Viewport takes place on a ‘Stage’. In the image below of Isaac Sim application window, the Stage panel appears to the right of the Viewport pan Isaac Sim GUI. In the image you can see the top-level view of the scene composited on the Stage, comprising objects for the World, physics, jetbot, and background. In this context you might hear JetBot and other objects on the Stage referred to as ‘USD assets’ or ‘USD models’. When you press the ‘Play’ button, the simulation clock starts running and action begins on the Stage as seen in the Viewport.

Below the Stage panel in the image below, notice the panels labeled ‘Details’ and ‘PhysX Properties’. Together with the Viewport, the Stage and the Content panels, the Details and PhysX Properties panels are fundamental to Isaac Sim, and you should become familiar with them as you begin to develop your own simulations.

../_images/isaac_jetbot_stage.png

Play with Lighting

There are multiple lights and types of lights at use in the JetBot scene. The main light switches to turn off all types of lights are found on the “Real Time Mode Settings” below. You can bring up this panel by choosing Menu:Rendering->Real Time Mode Settings.

../_images/isaac_jetbot_render-panel.jpg

To turn off ‘Direct Lights’, such as ‘Distant’, ‘Sphere’ or ‘Dome’ lights that are placed on the Stage, use the checkbox in the ‘Direct Light’ section of the panel. If you turn off Direct Lights using the checkbox, you should see JetBot in ‘ambient light’, also known as “Indirect Diffuse Light”.

If you set the ‘Intensity’ of Indirect Diffuse Light to zero, you should now see an image like the one below. Notice that the white lines on the floor are ‘emissive’ materials, so they glow in the dark when all lights are switched off. The yellow outline of JetBot is visible because the JetBot model is selected on the Stage.

../_images/isaac_jetbot_no-light.png

Next let’s create a new Direct light and attach it to JetBot itself. Start by choosing in the Menu Create->Light->Sphere Light. With the new light selected in the Stage panel, adjust the Scale of the light by typing new values in the Scale fields of the Details panel.

At this point the new Sphere Light is not attached to the JetBot. To attach the light to the JetBot, use your mouse to select the Sphere Light object in the Stage panel, and drag/drop it onto the object named “chassis” in the hierarchy of JetBot. The chassis is the physical rigid body corresponding to the JetBot body. The light now should now be attached to the JetBot. Press ‘Play’ to start the simulation and drive the JetBot around the environment, and you should see the light moving with it.

Play with the View

If you are following this guide step-by-step, up until now you have been using the default ‘Perspective’ view in the Viewport. We can change the view by going to the upper left corner of the viewport, click on “Perspective”. The dropdown menu shows that our default options other than “Perspective”, are “Top”, “Front”, and “Right”. Click on any of them to see the corresponding view.

../_images/isaac_jetbot_viewport.png

There is also an option “Camera”. Once we open that up, we can see that our other option is the “jetbot camera”. This camera is part of JetBot asset, created when it was first converted from CAD to the USD format. It corresponds to the onboard camera sensor. We can change the viewport to show the view from this camera by selecting “jetbot camera”. We can see from the JetBot’s image that the camera is at the front of the robot and pointing slightly down. You may even see a grey camera sitting on top of the JetBot when you first loaded the robot. Hence, the new view from the onboard camera should be a few white lines on a blue background, because we would be looking directly at the ground. If you press “Play” and move the jetbot around, you can see the image changes corresponding to the robot’s movement.

If you want to get rid of the grey camera. Click on the grey camera in the viewport, it should highlight the correponding Xform on the Stage tree on the left. In this case, it should be “jetbot->Chassis->rgb_camera->jetbot_camera”. Click on the eye icon in the middle column to hide it from view.

Attach a Camera

Now let’s create a camera ourselves and attach it to the JetBot.

  • Start by using the mouse and keyboard to move the Perspective camera to a desired location. Let’s put the viewpoint just behind and above the robot, looking in the direction the robot would travel ‘straight ahead’.

  • Use your mouse, click on the camera icon in the upper-left of the Viewport, where it should say ‘Perspective’ currently. Select Camera–>Create Camera from View. This will create a new object on the stage called ‘Camera’ by default, you can rename it if you choose.

  • Then select your Camera on the Stage panel, and drag-drop it onto the jetbot -> chassis. This will attach the camera to the physical rigid body that represents the chassis of the robot in the simulation.

  • To see the view from the new camera, click on the camera selector in the Viewport, the icon resembling a camera in the upper-left, and choose the new camera for the Viewport.

  • Finally, hit the Play button and drive the JetBot around the environment. You should see the perspective of a third-person follow-cam. The bumpy wheels cause a noticeable vibration in the camera, this is not an error, it is a plausible physical result of the design of the robot.

If you would like to see the new Camera itself, change back to Perspective view and move back, as shown below.

../_images/isaac_jetbot_camera-perspective.png

The camera you created can be configured as to focal length and distance, aperture and other properties, as below. You can read more about about them in the camera docs here.

../_images/isaac_jetbot_camera-properties.jpg

Import JetBot from CAD to USD

The Omniverse Isaac Sim package comes with a STEP version of the JetBot. To load it, go to Window->Isaac->Step Importer, select the Built-in Step Files, pick the JetBot. You can make changes to its tesselation quality and material shader properties. For more details on that, check The Step Importer Guide.

Once all import changes are made to satisfaction, click on Finish Import, and select Save Flattened. This will save the entire jetbot as a single USD, making it easier to make further changes to its structure.

Creating the Camera Sensor

Let’s create the camera sensor now. First go to Create->Camera, and a camera will be created on /Root/Camera. To facilitate its positioning, place it inside the lens’ prim by dragging the camera into the lens’ mesh on the Stage window.

Note

To find which mesh is the Lens, you can click on the lens mesh in the viewport.

Notice on the details pane the Extent property, which shows the distance to the bounding box on every direction from the mesh center.since the lens forward axis is Y, the distance to the lens top is near 0.260cm in the Y direction.

../_images/isaac_jetbot_first_extent.png

After dragging the camera inside the lens mesh, change its local transform to (0, 0.260, 0) so it sits right on the edge of the lens. To adjust the camera orientation, rotate it on the X axis by 90 degrees, and on the Z axis by 180 degrees.

Now, move the camera out of the lens mesh by dragging it back to /Root, and the camera path now should be /Root/Camera, but this will not be the final path, as there are more steps before we get the final asset.

Simplifying the Model

Having every individual components is good for setting up the robot, but once everything is settled, it makes sense to simplify the stucture to be computationally more efficient. To do so, we use the mesh merging tool to consolidate meshes together as a single mesh.

This results in less transforms being reported at every timestep, reducing the computation time. The merge process will retain all material assignment so the robot will look the same.

We will create the following structure for the JetBot:

  • Root
    • Chassis
      • Body

      • Left Antenna

      • Right Antenna

      • Caster Ball Case (back and front)

      • Caster Ball (back and front)

      • Camera

    • Left Wheel

    • Right Wheel

The choice of breaking the chassis in multiple smaller components are due to two reasons: For the Camera component, it is due to its adjustable nature, so we can configure the position the camera is pointing to. For the other components, it is to achieve a middle ground between performance and precision on the collision shapes. Each subcomponent will be set up as a convex hull, so breaking it down by the most prominent features (such as the antennas), allows us to perform a significant performance increase over selecting Physx convex decomposition for the whole body.

To create that structure, First drag /Root/chassis/electronics into /Root/chassis/body, and then move the /Root/chassis/body/caster_ball_assembly and /Root/chassis/body/caster_ball_assembly_1_ to /Root.

Now to merge all body components on a single mesh open menu Window->Isaac->Mesh Merge Tool, select /Root/chassis/body and click Merge Selected Prim. This will create a single mesh with GeomSubsets for each material it uses in /Merged/body. Delete all contents inside /Root/chassis, and reset its orientation to (0,0,0), then drag /Merged/body to /Root/chassis.

Repeat this process for the wheels, antennas and the camera shroud, but don’t delete the camera base prim. Instead, delete all its contents, and move the merged mesh into it, along with /Root/Camera. Move /Root/camera_base/ into /Root/chassis. You can now adjust the camera orientation by changing the rotation angle of the Y axis in the transform /Root/chassis/rgb_camera.

../_images/isaac_jetbot_first_run_camera_angle.png

Move the caster case and ball meshes into the /Root/chassis, delete empty Xform Prims, and your asset will be ready for rigging physics.

To keep meshes organized, you may opt to create another base transform to keep all the chassis components inside. In the asset provided, we created the transform /Root/chassis/geometry for that purpose.

Setting up Physics

First, the robot will be set up as an articulated chain. Click on /Root, then open the Physx Properties tab, and add Articulation API under the Add Prim Component dropdown. Similarly, add PhysicsAPI and massAPI:mass to /Root/chassis, /Root/left_wheel and /Root/right_wheel. For each component inside Root/chassis and also for the wheels, add CollisionAPI. This will flag the simulation to treat those prims as physics-enabled rigid bodies, with a given mass. Set the Root/chassis mass to 1.0, and the wheels to 0.200.

../_images/isaac_jetbot_first_run_physics_api.png

For the Caster Balls and wheels, change the Collision Approximation to Bounding Sphere. This will ensure a single contact point will be cast with the ground during simulation, providing additional stability and performance.

Now let’s set up the joints between the chassis and the wheels.

First, set the root of the articulation to the chassis. Select /Root/chassis, and click on Physics->Add->Joint->To world space. The joint will be created on /Root/chassis/revolute_joint. Rename it to root_joint by double-clicking on the joint, editing the name and then click OK. On the Physx Properties tab, remove the RevoluteJoint component, and add ArticulationJoint. Scroll down and set the articulation type to articulatedRoot.

To connect the wheel to the chassis, select the /Root/chassis/left_wheel, and then also select /Root/chassis by pressing ctrl and clicking on it, and click on Physics->Add->Joint->between selected. Rename the joint to left_wheel_joint, and on the Physx Properties tab, add the ArticulatedJoint and Drive components. Change the joint localPos1 to (0, 0, 0) and localRot1 to (1, 0, 0, 0) to align it with the wheel axis. Change the joint drive type to velocity, and set the damping to 1e6.

Likewise, do the same for the right wheel, with the exception that localRot1 must be set to (0, 0, 0, 1).

We could add spherical joints for the caster balls, but having them with a low friction to the ground is an equivalent, faster simulation. To add a low friction material to the casters, click on Physics->Add->Physics material, rename it to caster_material, and set the dynamic and static friction to 0.00 on the details tab. Then, click on the caster balls and assign the caster ball material in the details tab under Physics materials section. Do the same for the left and right wheels for a wheel material with friction 1.0, and physics should be all set.

Note

You can enable debug visualization of the collision meshes by clicking on the Show/Hide button (eye icon on the viewport)->Show by type->Physics Mesh->Selected or All

../_images/isaac_jetbot_see_collision.png

Testing JetBot

Finally, let’s make the final settings for the JetBot Asset. Ensure that /Root is set as default Prim. You can set it as the default prim by right-clicking on it in the stage editor tab, and selecting set as Default Prim.

then. click on Physics->Add->Physics Scene, and Physics->Add->Ground Plane, Drag the /Root/PhysicsScene and to /World. This is important because Everything that is under the default prim gets loaded on authored stages when the USD is referenced. Keeping the physics scene out of the default prim allows us to locally test the asset, while not interferring with it when loaded by other stages. Finally, copy Root/staticPlaneActor to /World and set its position to (0, 0, -3.360).

Now you can press the Play button and check if jetbot remains assembled. If it does, select the wheel joints and give them some target value to check if jetbot moves.

Note

If changing the joint target by the UI, remember to un-select the joint prim before stopping the simulation, as it may update the joint pose with incorrect transforms.

Training Your JetBot

Take a look at the Training Your JetBot in NVIDIA Isaac Sim blog post on how to train you JetBot in Omniverse Isaac Sim. Please also apply the JetBot Isaac Sim Patch.