Robot Engine Bridge

Omniverse Isaac Sim integrates with Isaac SDK via the Robot Engine Bridge (REB) extension. The extension enables message communication between the two platforms over TCP. These messages include simulated sensor data, drive commands, ground truth state of simulated assets and scenario management.

The extension can be enabled by going to Window->Extension Manager and enabling the omni.isaac.robot_engine_bridge extension.

The Robot Engine Bridge contains a sim side Isaac SDK application that sends and receives messages, these messages are converted to/from native SDK Cap’n Proto messages for use in the simulator.

To start the simulator side SDK application go to the Robot Engine Bridge UI window and click Create Application.

Main Robot Engine Bridge Window

Once Pressed the specified application in the Application Json Path field will be launched. Custom application can be specified by modifying this path. The application can be stopped by pressing the Destroy Application button.

The default isaacsim application contains a interface node, an input isaac::alice::TcpSubscriber component on port 55001 and a output isaac::alice::TcpPublisher component on port 55000

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
  {
  "name": "isaacsim",
  "modules": [],
  "graph": {
      "nodes": [
      {
          "name": "interface",
          "components": [
          {
              "name": "ledger",
              "type": "isaac::alice::MessageLedger"
          },
          {
              "name": "input",
              "type": "isaac::alice::TcpSubscriber"
          },
          {
              "name": "output",
              "type": "isaac::alice::TcpPublisher"
          },
          {
              "name": "time_sync",
              "type": "isaac::alice::TimeSynchronizer"
          }
          ]
      }
      ],
      "edges": []
  },
  "config": {
      "interface": {
      "input": {
          "port": 55001,
          "host": "localhost"
      },
      "output": {
          "port": 55000
      }
      }
  }
  }

The json path can also be provided at runtime by using the --carb/isaac/robot_engine_bridge/json="path/to/app.json" commandline argument

Note

the set of Isaac SDK components currently supported is limited to: isaac::alice::MessageLedger, isaac::alice::TcpSubscriber, isaac::alice::TcpPublisher and isaac::alice::TimeSynchronizer and at least one of each should be present to work correctly. Multiple subscribers and publishers can be created and used by changing the default inputComponent and outputComponent fields in the components below.

Robot Engine Bridge Components

For any of the components below with a type of relationship use the Relationship Editor UI UI to modify

Scene Loader

The Scene loader component is a special component that allows the user to load USD files once the application has been created and play is pressed to start simulation. It is controlled via the collapsible UI in the main Robot Engine Extension UI.

Robot Engine Bridge Scene Loader

Name

Type

Default

Description

inputChannel

string

base_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

outputComponent

string

output

Then name of the component on which reply is sent

replyChannel

string

scenario_reply

Channel name to send reply on once scene is loaded

Differential Base

This component is used to control a differential base

Name

Type

Default

Description

accelerationSmoothing

float

1.0

A smoothing factor for the estimated acceleration

brakeTorque

float

100

brakeTorque applied when braking is requested

chassisPrim

relationship

USD Prim Path for Chassis Prim

inputChannel

string

base_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

leftWheelJointName

string

Name of left wheel joint

maxMotorTorque

float

10

The maximal motorTorque apply to the driving wheels

maxSpeed

float2

(1.5, 1.0)

The maximal allowed linear and angular speed

maxTimeWithoutCommand

float

0.2

The maximum allowed time duration which the robot will continue with the last sent speed command in the absence of speed commands

nodeName

string

interface

Name of the node in the application to publish/subscribe to

outputChannelName

string

output

The name of the channel on which state is published

outputComponentName

string

base_state

The name of the component on which state is published

proportionalGain

float

100

Proportional controller gain

rightWheelJointName

string

Name of right wheel joint

robotFrontDirection

float3

(1,0,0)

Vector representing the robots forward direction

useProportionalGainDriver

bool

true

Whether to use a proportional driver (true) or always apply mMaxMotorTorque (false)

Joint Control

This component publishes and subscribes the joint positions of an articulation

Name

Type

Default

Description

articulationPrimRel | relationship | | USD Prim path to the articulation being controlled

inputChannel

string

joint_position

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

nodeName

string

interface

Name of the node in the application to publish/subscribe to

outputChannel

string

joint_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

Scissor Lift

This component controls a vertical translational joint on an articulation

Name

Type

Default

Description

articulationPrimRel

relationship

USD Prim path to the articulation being controlled

inputChannel

string

lift_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

liftJointName

string

lift_joint

Name of the translational joint on the articulation being controlled

nodeName

string

interface

Name of the node in the application to publish/subscribe to

outputChannel

string

lift_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

Surface Gripper

This component simulates a suction cup gripper

Name

Type

Default

Description

inputChannel

string

io_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

gripperEntity

string

gripper

name returned by the component

nodeName

string

interface

Name of the node in the application to publish/subscribe to

outputChannel

string

io_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

parentPrim

relationship

Rigid body prim representing the suction cup gripper

gripThreshold

float

1

forceLimit

float

1e10

Max force before joint breaks

TorqueLimit

float

1e10

Max torque before joint breaks

BendAngle

float

0

Max angle the joint can bend before hitting a limit

stiffness

float

1e10

Stiffness of the joint

damping

float

1e3

damping parameter for the joint

offsetPosition

float3

(0,0,0)

offset from the parent prim where the joint is created

offsetRotation

float4

(1,0,0,0)

quaternion representing rotation offset of the joint from the parent prim

Two Finger Gripper

This component simulates a gripper with two finger joints

Name

Type

Default

Description

articulationPrim

relationship

The articulation that contains the two finger joints

inputChannel

string

io_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

gripperEntity

string

gripper

name returned by the component

nodeName

string

interface

Name of the node in the application to publish/subscribe to

outputChannel

string

io_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

closedDistance

float

0

Distance between fingers when closed

openDistance

float

0.04

Distance between fingers when open

Rigid Body Sink

This component publishes the rigid body poses for a list of primitives

Name

Type

Default

Description

rigidBodyPrims

relationship

list of relationship targets to rigid bodies that need to be published

nodeName

string

interface

Name of the node in the application to publish/subscribe to

outputChannel

string

bodies

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

Teleport

This component controls the pose for a list of primitives

Name

Type

Default

Description

teleportPrims

relationship

list of relationship targets to rigid bodies that need to be teleported

inputChannel

string

teleport

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

nodeName

string

interface

Name of the node in the application to publish/subscribe to

Scenario From message

This component can create new prims, teleport them and publish their rigid body pose

Name

Type

Default

Description

inputChannel

string

scenario_actors

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

nodeName | string | interface | Name of the node in the application to publish/subscribe to

teleportInputComponent | string | input | name returned by the component

teleportInputChannel

string

teleport

Name of the node in the application to publish/subscribe to

rigidBodySinkOutputChannel

string

bodies

The name of the channel on which state is published

rigidBodySinkOutputComponent

string

output

The name of the component on which state is published

Lidar

This component publishes lidar data

Name

Type

Default

Description

lidarPrim

relationship

USD Prim path to lidar sensor

nodeName

string

interface

Name of the node in the application to publish/subscribe to

outputChannel

string

rangescan

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

Camera

This component publishes RGB, Depth and Segmentation data

Name

Type

Default

Description

rgbOutputChannel

string

color

The name of the channel on which color state is published

rgbOutputComponent

string

output

The name of the component on which color state is published

depthOutputChannel

string

depth

The name of the channel on which depth state is published

depthOutputComponent

string

output

The name of the component on which depth state is published

segmentationOutputChannel

string

segmentation

The name of the channel on which segmentation state is published

segmentationOutputComponent

string

output

The name of the component on which segmentation state is published

bbox2dOutputChannel

string

bbox

The name of the channel on which bbox state is published

bbox2dOutputComponent

string

output

The name of the component on which bbox state is published

rgbEnabled

bool

true

true if rgb data is published

depthEnabled

bool

false

true if depth data is published

segmentationEnabled

bool

false

true if segmentation data is published

boundingBox2dEnabled

bool

false

true if bbox data is published

Contact Monitor

This component publishes a message if a specified object collides with another.

Name

Type

Default

Description

targetPrim

relationship

USD Prim path to rigid body to report contacts for

ignoredPrims

relationship

USD Prims paths that will be ignored when reporting contacts

nodeName

string

interface

Name of the node in the application to publish/subscribe to

outputChannel

string

collision

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

forceThreshold

float

1000 | Contact force threshold above which contacts are reported

Sample Applications

Below are some sample Isaac SDK applications leveraging Omniverse Isaac Sim. For more details, please refer to Isaac SDK documentation.

Warehouse Navigation with Carter

Carter robot navigating in the Warehouse simulation environment. For more details, please refer to Isaac SDK Carter Navigation sample.

Cart Delivery

The robot delivering a cart in the Warehouse simulation environment. For more details, please refer to Isaac SDK Cart Delivery sample.

Shuffle Box with Simulator

The UR10 robot arm with suction gripper picks and drops a pink box in the simulation environment. For more details, please refer to Isaac SDK Shuffle Box sample.

Simple Joint Control

The UR10 robot arm is manipulated in the simulation environment via joint commands from Isaac SDK. For more details, please refer to Isaac SDK Joint Control sample.

Relationship Editor UI

This utility extension allows you to edit USD references on the various components in the Robot Engine Bridge, ROS Bridge and Domain Randomization

The Relationship Editor UI can be found at Window->Isaac->Relationship Editor

If you click on a prim that contains any USD references the Relationship Editor will automatically populate with those references. Multiple references per prim and multiple targets per reference are supported. Note that even if you can specify multiple targets for a reference, that component might only use the first one.

In this example we have a stage with a Lidar sensor at World/Lidar and a Lidar Robot Engine Bridge Component at World/REB_Lidar

Editing Lidar Reference

When World/REB_Lidar is selected you will see the Relationship Editor UI will automatically show lidarPrim with a single default entry that you can modify.

As you enter the new path the label to the right of the text box will change from Invalid to Valid

Editing Lidar Reference Invalid
Editing Lidar Reference Valid

Once Add is pressed you can modify or remove that reference, or add additional targets to that reference if needed.

Editing Lidar Reference Add, Modify, Remove