Isaac Sim Reference Architecture

This document presents an overview of the end-to-end workflows possible with Omniverse Isaac Sim and highlights the crucial building blocks needed to accomplish them. This is depicted through a Reference Architecture. Almost all use-cases involve a few common initial steps, which are highlighted in the diagrams below. Methods to interact with simulation and build applications on top of Isaac Sim are also summarized

The Reference Architecture for Isaac Sim includes the following components:

Geometry Authoring

Importing Assets

Scene Setup

Interaction with Digital Twin

Use Case (End Goal)

Geometry Authoring

The simulation environment (scene) is composed of various components including robots, static and dynamic objects. The mechanical and geometrical design for these components is usually done with CAD software like Solidworks, Pro-E, Catia, AutoCad, or Creo. Parts and components of varying complexity can be designed and assembled.

Developers can also leverage existing 3D asset libraries, which provide a vast collection of existing 3D assets. Omniverse and Isaac Sim leverage a file format called Universal Scene Description (OpenUSD). All assets need to be converted to OpenUSD before they can be used with Isaac Sim, and the default unit for Isaac Sim is meters.

NVIDIA provides a vast collection of OpenUSD ‘SimReady’ assets. SimReady, or simulation-ready, assets are physically accurate 3D objects that encompass accurate physical properties, behavior, and connected data streams to represent the real world in simulated digital worlds. Developers can use these building blocks to construct scenes and generate data as per their requirement. The Warehouse asset collection comprises over 800 3D assets of commonly available tools, equipment and items in a warehouse including forklifts, pallets, racks, and shelves.

Importing Assets

We provide extensions that enable importing CAD (Computer Aided Design) files into Isaac Sim which handles conversion to OpenUSD. Extensions are core building blocks that interact with and add or extend the functionality of Isaac Sim and are described in more detail later.

Importing and Creating Environments

The asset importer can be leveraged for importing OBJ, FBX and glTF formats. The CAD converter extension supports a variety of popular CAD files from applications including Catia, Solidworks, AutoCad, Creo, and many others. This enables users to easily convert and import their environments into Isaac Sim

OpenUSD Connections and Data Exchange, formerly Omniverse Connect, is a collection of importers, exporters, converters, and USD file format plug-ins that enable various 3D applications, products, and file formats to exchange data using OpenUSD. Some CAD applications have connectors with Omniverse, this allows them to bring over more relevant and contextual information when they are converted to USD. For example, PTC Creo, Autodesk Revit, or Autodesk Alias have their corresponding connectors. The files generated from this CAD converter will have all the visual meshes represented in OpenUSD. Adding physics before simulating will be an important step for most use cases and is explained in more detail in this document

Importing Robots

Isaac Sim provides advanced options for importing robots.

A common file format representation in robotics amongst the ROS community is URDF. In this format, along with the visual meshes and the prim hierarchies (child-parent relationships) more information about the collision meshes, joints, and sensors are encoded.

The URDF Importer extension can be used and accessed via the GUI, as well as Python scripting. All the collision meshes and joint information will be included in the USD of the robot. Similarly, the Onshape Importer and MJCF Importer can also be used. You will have to add the joint drives in this case and may have to tune them, the Gain Tuner Extension allows you to visualize them.

Isaac Sim comes with a variety of robots which can be found on the Robot Assets page

Scene Setup

The scene can be set up once all the necessary assets are converted to OpenUSD and imported into Isaac Sim

Adding Physics

After importing the required assets into Isaac Sim, we may need to add appropriate Physics to them. Some asset importers like the URDF and Onshape Importer carry over most Physics parameters and configurations, for the rest of the imported assets adding physics before proceeding would be necessary. The NVIDIA Omniverse™ Physics simulation extension is powered by the NVIDIA PhysX SDK. It supports Rigid Body Simulation, Character Control, Deformable Body Simulation, Particle Simulation, and Articulations. The important steps for adding Physics to your scene are:

Creating the physics scene

Assigning collision settings

Adding joints and drives.

Creating the physics scene

The first step is to create a Physics Scene and ensure that the default parameters for it are acceptable. For example, verify the direction and magnitude of gravity in the scene. If the imported scene does not contain a ground plane, make sure to add one before proceeding. It will prevent any physics-enabled objects from falling below it. Unless you are simulating hundreds of rigid bodies and robots, it is more efficient to use the CPU solver instead of the GPU solver. Refer to the Environment Setup tutorial

Assigning collision settings

Collision enables rigid bodies to interact with each other in an environment. The geometry of the object can be approximated by convex hull, convex decomposition, bounding sphere, bounding box, and SDF collision meshes. Each of them approximates the geometry using different methods and may be better suited for specific use cases. PhysX supports exact representations for Cube, Capsule, and Sphere shapes. Cones and Cylinders are supported through the custom geometry flag and are particularly useful when setting collision approximations for wheels of robots. Rigid-body physics materials provide friction, restitution (a.k.a. ‘bounciness’), and material density properties

Adding joints and drives

After adding the appropriate collision meshes to prims in the scene, we need to ensure that they interact correctly with one another. We can do this by defining appropriate joints between prims connected to each other. Joints give you the ability to connect physics objects by defining how the objects may move relative to each other. There are various joints types to select from including revolute, prismatic, spherical, fixed, etc.

Adding Sensors

Isaac Sim sensor simulation extensions can simulate ground truth perception and physics-based sensors, and has a library of realistic sensor models. You can simulate camera, lidar, radar, and physics-based sensors. It is possible to use camera calibration parameters obtained from OpenCV or ROS by converting them to Isaac Sim units, refer to the Camera Sensors and Render Extensions page. RTX Lidar and Radar sensors are simulated at render time on the GPU with RTX hardware. Examples of Lidar sensors supported via the use of a config file are: Hesai, Ouster, SICK, Velodyne. Refer to the Omniverse Lidar Extension documentation. A variety of physics-based sensors like contact sensors, IMUs sensor, force sensor, effort sensor, and proximity sensor are also included. These sensors can be added at the appropriate locations in the stage hierarchy (for example, a camera or lidar might be added near the front/top of the robot). The Sensor Assets highlights all the available physical sensor assets available with Isaac Sim

Import/Create Materials

Materials in Isaac Sim are supported using NVIDIA Material Definition Language (MDL) , a shading language designed for defining and describing the appearance of materials in computer graphics. It allows artists and developers to create highly realistic materials by specifying their physical properties, surface characteristics, and how they interact with light. Omniverse comes with several template materials, including a physically based glass; several general purpose multi-lobed materials useful for dielectric and non-dielectric materials, skin, hair, liquids and other materials requiring subsurface scattering or transmissive effects; and USD’s UsdPreviewSurface.

Interaction with Digital Twin

Once the assets have been imported and the scene has been set up, there are various ways to interact with the simulated environment, which are summarized below.

GUI

The GUI provides intuitive controls for scene management, object manipulation, and real-time monitoring, providing a streamlined interface for developing and testing robotic systems. Pre-packaged examples, robots and environments can easily be accessed and added to the scene via the GUI. Create tools make it easy to assemble, illuminate, simulate, and render scenes large and small, therefore making it the ideal place to build your virtual worlds, assemble robots, and examine physics. Refer to the Build Your First Virtual World for getting started with the GUI tutorials.

Standalone Python

Isaac Sim provides a built-in Python Environment that packages can use, like a system-level Python install. This is the recommended environment for running Python Scripts with Isaac Sim. All Isaac Sim libraries and dependencies can be imported and accessed through this Python environment. It also allows users to script and run their entire worflkow headlessly. For using libraries and tools which are not a part of Isaac Sim, ensure that they work with Python 3.10 first. A collection of standalone python examples is provided with Isaac Sim and serves as a good starting point to understand the overall steps involved. Jupyter notebook and Visual Studio Code support is also available. Workflows from the GUI can be completely scripted in Python and can be run in headless mode too.

Extensions

Extensions enable developers to add functionality and integrate other tools for Isaac Sim. They are individually built application modules. All the tools used in Isaac Sim are built as extensions. Various extensions enable easier interaction with sensors, robots and prims in the scene. The ROS Bridge (1 and 2) extension can be used to connect your ROS packages and code to Isaac Sim. Developers can write their own extensions in C++, Python or a combination of both.

OmniGraph

OmniGraph is the visual scripting language for Omniverse. It is not a single type of graph, but a composition of many different graph systems under a single framework. Many Isaac Sim extensions provide nodes for building graphs for common use cases. Core, sensor, and ROS extensions are a few examples that contain such OmniGraph Nodes.

Consumption of Digitial Twin (Use Case)

Synthetic Data Generation (SDG)

Developers can generate physically accurate synthetic data that can enhance the training and performance of AI perception networks used for robotics using Omniverse Replicator. Replicator is a collection of extensions, Python APIs, workflows, and tools that enable synthetic data generation tasks.

Once the scene has been set up, Replicator can be used to modify and randomize various features like position, rotation, lighting, size, and textures of assets in the scene. A wide range of annotations are supported including 2D bounding boxes, 3D bounding boxes, semantic and instance segmentation masks, normals, depth, pointclouds, and more, with data being written in common formats like COCO and KITTI formats. Custom annotators and writers can also be implemented for advanced use cases like pose estimation. This enables developers to seamlessly integrate the generated data with their training pipelines

To get started, developers can leverage the Python API provided by Omniverse Replicator for generating synthetic data. The same scripts can be used to generate data headlessly in the cloud through the Isaac Sim docker container (instructions Container Installation) on a developer’s preferred CSP (AWS, Alibaba, Azure, GCP) with the Cloud Deployment guide. Replicator YAML can be used for low-code situations where scripts can easily be edited by non-technical experts. They offer a high level of portability and care suitable for cloud use cases.

Software in the Loop (SIL)

Once the simulation scenario has been set up, developers can tune and test various aspects of their robotics software stack. The insights gained after varying parameters and configurations of the software stack and the simulated robot enable an easier and accurate transition to the physical real-world robot. A few common use cases are highlighted below:

AI Model Evaluation

Evaluating models is easy in simulation because of direct access to ground truth which could be through Physics, the state of a robot, or reading of a sensor. These can then be compared with the model predictions to obtain the evaluation metrics.

For example, in computer vision tasks, the rendered image from the simulated camera can be passed through the model for obtaining predictions. This can then be compared with the ground truth (available directly from simulation or via Replicator) to obtain evaluation metrics. This can also be done for other sensors like Lidars and can be easily extended to multimodal applications

Perception

Isaac Perceptor is a reference workflow of NVIDIA-accelerated libraries and AI models that helps you quickly build robust autonomous mobile robots (AMRs) to perceive, localize, and operate in unstructured environments like warehouses or factories. It works with inputs from simulated environments in Isaac Sim

Manipulation

Isaac Manipulator can be leveraged for manipulation tasks and verified in simulation. It is a collection of GPU-accelerated packages for perception driven manipulation, providing capabilities such as object detection and pose estimation. Time optimal collision-free motion can be generated with cuMotion. Nvblox can be used for local 3D reconstruction and obstacle detection. MoveIt is also supported via the ROS Bridge in Isaac Sim.

Reinforcement Learning

Isaac Lab is a unied and modular framework for robot learning that aims to simplify common workows in robotics research (such as RL, learning from demonstrations, and motion planning). It is built upon Omniverse Isaac Sim.

Hardware in the Loop (HIL)

Hardware in the loop testing and evaluation can be done with Isaac Sim. The target deployment device will receive all the data from the simulated robot and sensors which can be fed to the needed software stacks/algorithms. ROS 2 can be leveraged as the middleware which handles sending and receiving all the data from the simulation computer to the target device. For example, a simulated camera from Isaac Sim will stream over all the image data via the ROS 2 bridge from Isaac Sim to an NVIDIA Jetson Orin, the embedded device on the robot which will run the computer vision application. This is particularly useful when selecting the target deployment device to verify it can run the software stack needed before physically setting up the robot.

CI/CD

OSMO

OSMO is a cloud-native workow orchestration platform that lets you easily scale your workloads across distributed environments — from on-premises to private and public cloud. You can now apply for early access.

Sizing Calculator

On a single OVX server, developers can expect ~5FPS with a sensor setup consisting of 4 stereo cameras, 4 monocular cameras and 5 Lidars. This can be used as an approximate sizing guide to determine the compute needed based on the sensor and robot setup.