Robot Engine Bridge

Omniverse Isaac Sim integrates with NVIDIA Isaac SDK™ via the Robot Engine Bridge (REB) extension. The extension enables message communication between the two platforms over TCP. These messages include simulated sensor data, drive commands, ground truth state of simulated assets and scenario management.

The extension can be enabled by going to Window -> Extensions and enabling the omni.isaac.robot_engine_bridge extension.

The Robot Engine Bridge contains a sim side Isaac SDK application that sends and receives messages, these messages are converted to/from native SDK Cap’n Proto messages for use in the simulator.

To start the simulator side SDK application go to the Robot Engine Bridge UI window and click Create Application.

Main Robot Engine Bridge Window

Once Pressed the specified application in the Application Json Path field will be launched. Custom application can be specified by modifying this path. The application can be stopped by pressing the Destroy Application button.

The default isaacsim application contains a interface node, an input isaac::alice::TcpSubscriber component on port 55001 and a output isaac::alice::TcpPublisher component on port 55000

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
  {
  "name": "isaacsim",
  "modules": [],
  "graph": {
      "nodes": [
      {
          "name": "interface",
          "components": [
          {
              "name": "ledger",
              "type": "isaac::alice::MessageLedger"
          },
          {
              "name": "input",
              "type": "isaac::alice::TcpSubscriber"
          },
          {
              "name": "output",
              "type": "isaac::alice::TcpPublisher"
          },
          {
              "name": "time_sync",
              "type": "isaac::alice::TimeSynchronizer"
          }
          ]
      }
      ],
      "edges": []
  },
  "config": {
      "interface": {
      "input": {
          "port": 55001,
          "host": "localhost"
      },
      "output": {
          "port": 55000
      }
      }
  }
  }

The json path can also be provided at runtime by using the --carb/isaac/robot_engine_bridge/json="path/to/app.json" commandline argument

Note

  • The set of Isaac SDK components currently supported is limited to: isaac::alice::MessageLedger, isaac::alice::TcpSubscriber, isaac::alice::TcpPublisher and isaac::alice::TimeSynchronizer and at least one of each should be present to work correctly. Multiple subscribers and publishers can be created and used by changing the default inputComponent and outputComponent fields in the components below.

  • The Robot Engine Bridge extension does not support NVIDIA Ampere graphics cards such as the RTX 30 series since it is not recommended to be used with Isaac SDK.

Robot Engine Bridge Components

All of the properties below can be edited in the “Raw USD Properties” section of the properties tab. Relationships can also be directly modified there.

The following properties are shared between all components

Name

Type

Default

Description

nodeName

string

interface

Name of the node in the application to publish/subscribe to

enabled

bool

True

Enable or disable this component from sending/receiving data

timeOffset

double

0

Time offset applied to acquisition time when sending to SDK

Scene Loader

The Scene loader component is a special component that allows the user to load USD files once the application has been created and play is pressed to start simulation. It is controlled via the collapsible UI in the main Robot Engine Extension UI.

Robot Engine Bridge Scene Loader

Name

Type

Default

Description

requestChannel

string

scenario_control

Channel name that receives json message with information on what usd stage to load

inputComponent

string

input

The name of the component on which commands are received

outputComponent

string

output

Then name of the component on which reply is sent

replyChannel

string

scenario_reply

Channel name to send reply on once scene is loaded

cameraControl

string

camera_switch

Channel name that receives a camera name to switch the viewport to

Differential Base

This component is used to control a differential base

Name

Type

Default

Description

accelerationSmoothing

float

1.0

A smoothing factor for the estimated acceleration

chassisPrim

relationship

USD Prim Path for Chassis Prim

inputChannel

string

base_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

leftWheelJointName

string

Name of left wheel joint

maxSpeed

float2

(1.5, 1.0)

The maximal allowed linear and angular speed in m/s and rad/s

maxTimeWithoutCommand

float

0.2

The maximum allowed time duration which the robot will continue with the last sent speed command in the absence of speed commands

outputChannel

string

output

The name of the channel on which state is published

outputComponent

string

base_state

The name of the component on which state is published

rightWheelJointName

string

Name of right wheel joint

robotFront

float3

(1,0,0)

Vector representing the robots forward direction

wheelBase

float

0.5

Distance between wheel and center of chassis in meters

wheelRadius

float

0.1

Radius of driven wheels in meters

Holonomic Base

This component is used to control a holonomic base with three wheels such as Kaya

Name

Type

Default

Description

accelerationSmoothing

float

1.0

A smoothing factor for the estimated acceleration

articulationPrim

relationship

USD Prim Path for Articulatuion

inputChannel

string

base_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

maxSpeed

float2

(1.5, 1.0)

The maximal allowed linear and angular speed in m/s and rad/s

maxTimeWithoutCommand

float

0.2

The maximum allowed time duration which the robot will continue with the last sent speed command in the absence of speed commands

outputChannel

string

output

The name of the channel on which state is published

outputComponent

string

base_state

The name of the component on which state is published

robotFront

float3

(1,0,0)

Vector representing the robots forward direction

wheel1JointName

string

Name of first wheel joint

wheel2JointName

string

Name of second wheel joint

wheel3JointName

string

Name of third wheel joint

wheelBase

float

0.125

Distance from wheel to chassis center in meters

wheelRadius

float

0.04

Radius of the wheel in meters

Vehicle Control

This component is used to control a PxVehicle with acceleration and steering commands

Name

Type

Default

Description

inputChannel

string

vehicle_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

outputChannel

string

vehicle_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

vehiclePrim

relationship

USD Prim path to the PxVehicle

usePID

bool

false

Enable PID controller for acceleration

historyLength

int

100

Length of buffer used to smooth acceleration data

controllerPIDValues

float3

(1,1,1)

PID controller values

Joint Control

This component publishes and subscribes the joint positions of an articulation

Name

Type

Default

Description

articulationPrimRel

relationship

USD Prim path to the articulation being controlled

inputChannel

string

joint_position

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

outputChannel

string

joint_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

Scissor Lift

This component controls a vertical translational joint on an articulation

Name

Type

Default

Description

articulationPrim

relationship

USD Prim path to the articulation being controlled

inputChannel

string

lift_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

liftJointName

string

lift_joint

Name of the translational joint on the articulation being controlled

liftSpeed

float

0.02

Speed at which the joint raises and lowers in m/s

outputChannel

string

lift_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

Surface Gripper

This component simulates a suction cup gripper

Name

Type

Default

Description

inputChannel

string

io_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

gripperEntity

string

gripper

name returned by the component

outputChannel

string

io_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

parentPrim

relationship

Rigid body prim representing the suction cup gripper

gripThreshold

float

1

forceLimit

float

1e10

Max force before joint breaks

torqueLimit

float

1e10

Max torque before joint breaks

bendAngle

float

0

Max angle the joint can bend before hitting a limit

stiffness

float

1e10

Stiffness of the joint

damping

float

1e3

damping parameter for the joint

offsetPosition

float3

(0,0,0)

offset from the parent prim where the joint is created

offsetRotation

float4

(1,0,0,0)

quaternion representing rotation offset of the joint from the parent prim

Two Finger Gripper

This component simulates a gripper with two finger joints

Name

Type

Default

Description

articulationPrim

relationship

The articulation that contains the two finger joints

inputChannel

string

io_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

gripperEntity

string

gripper

name returned by the component

outputChannel

string

io_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

closedDistance

float

0

Distance between fingers when closed

openDistance

float

0.04

Distance between fingers when open

leftFingerJoint

string

left_finger

The name of the left finger joint

rightFingerJoint

string

right_finger

The name of the right finger joint

speed

float

0

Zero speed will instantly close/open based on joint parameters

Rigid Body Sink

This component publishes the rigid body poses for a list of primitives

Name

Type

Default

Description

rigidBodyPrims

relationship

list of relationship targets to rigid bodies that need to be published

outputChannel

string

bodies

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

Teleport

This component controls the pose for a list of primitives

Name

Type

Default

Description

teleportPrims

relationship

list of relationship targets to rigid bodies that need to be teleported

inputChannel

string

teleport

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

Scenario From message

This component can create new prims, teleport them and publish their rigid body pose

Name

Type

Default

Description

inputChannel

string

scenario_actors

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

teleportInputComponent

string

input

name returned by the component

teleportInputChannel

string

teleport

Name of the node in the application to publish/subscribe to

rigidBodySinkOutputChannel

string

bodies

The name of the channel on which state is published

rigidBodySinkOutputComponent

string

output

The name of the component on which state is published

Lidar

This component publishes lidar data

Name

Type

Default

Description

lidarPrim

relationship

USD Prim path to lidar sensor

outputChannel

string

rangescan

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

Camera

This component publishes RGB, Depth and Segmentation data

Name

Type

Default

Description

rgbOutputChannel

string

color

The name of the channel on which color state is published

rgbOutputComponent

string

output

The name of the component on which color state is published

depthOutputChannel

string

depth

The name of the channel on which depth state is published

depthOutputComponent

string

output

The name of the component on which depth state is published

segmentationOutputChannel

string

segmentation

The name of the channel on which segmentation state is published

segmentationOutputComponent

string

output

The name of the component on which segmentation state is published

boundingBox2DOutputChannel

string

bbox

The name of the channel on which 2D bbox state is published

boundingBox2DOutputComponent

string

output

The name of the component on which 2D bbox state is published

boundingBox3DOutputChannel

string

bbox3d

The name of the channel on which 3D bbox state is published

boundingBox3DOutputComponent

string

output

The name of the component on which 3D bbox state is published

boundingBox3DClassList

string

The list of class names to filter

rgbEnabled

bool

true

true if rgb data is published

depthEnabled

bool

false

true if depth data is published

segmentationEnabled

bool

false

true if segmentation data is published

boundingBox2dEnabled

bool

false

true if 2D bbox data is published

boundingBox3dEnabled

bool

false

true if 3D bbox data is published

cameraPrim

relationship

USD Prim path to the active camera in the viewport

resolution

int2

(1280, 720)

Resolution of the viewport

Contact Monitor

This component publishes a message if a specified object collides with another.

Name

Type

Default

Description

targetPrim

relationship

USD Prim path to rigid body to report contacts for

ignoredPrims

relationship

USD Prims paths that will be ignored when reporting contacts

outputChannel

string

collision

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

forceThreshold

float

1000

Contact force threshold above which contacts are reported

Occupancy Grid Map

This component publishes occupancy grid map

Name

Type

Default

Description

parentPrim

relationship

If specified, used to compute mapping location

outputChannel

string

occupancy_map

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

offset

float3

(0,0,0)

Mapping location offset, if parentPrim is not specified this is the offset from global origin

cellSize

float

0.1

Size of each pixel in meters

degreesPerRay

float

5

Number of degrees per ray when doing a 360 sweep

surfaceOffset

float

0.02

Offset from surface in meters to compute next sweep

occupancyThreshold

float

1

Probability used to determine occupancy

maxRays

float

1000000

Max number of rays to compute before terminating

mapSize

int2

(256,256)

Map size in pixels, the mapping area is computed using cell size

occupiedValue

float

1.0

Value returned for a region that is occupied

unoccupiedValue

float

0.0

Value returned for a region that is unoccupied

unknownValue

float

0.5

Value returned for a region that could not be mapped from the start location

debugDraw

bool

false

Enable Debug Drawing

Ultrasonic

This component publishes ultrasonic data

Name

Type

Default

Description

ultrasonicPrim

relationship

USD Prim path to ultrasonic sensor array

outputChannel

string

uss_envelopes

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

Polyline Visualizer

This component visualizes polyline

Name

Type

Default

Description

inputChannel

string

sight_plan

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

width

float

1.0

Width of polyline

color

color4f

(1, 1, 1, 1)

Color of polyline

offset

float3

(0,0,0)

Translation offset for polyline

parentPrim

relationship

If specified will be used as origin transform for polyline

Sample Applications

Below are some sample Isaac SDK applications leveraging Omniverse Isaac Sim. For more details, please refer to Isaac SDK Documentation.

Warehouse Navigation with Carter

Carter robot navigating in the Warehouse simulation environment. For more details, please refer to Isaac SDK Carter Navigation sample.

Cart Delivery

The robot delivering a cart in the Warehouse simulation environment. For more details, please refer to Isaac SDK Cart Delivery sample.

Shuffle Box with Simulator

The UR10 robot arm with suction gripper picks and drops a pink box in the simulation environment. For more details, please refer to Isaac SDK Shuffle Box sample.

Simple Joint Control

The UR10 robot arm is manipulated in the simulation environment via joint commands from Isaac SDK. For more details, please refer to Isaac SDK Joint Control sample.