# Robot Engine / Isaac SDK Bridge¶

The Robot Engine / Isaac SDK Bridge Extension is used to connect Omniverse Isaac Sim with NVIDIA Isaac SDK™. Isaac SDK is a developer toolbox for accelerating the development and deployment of AI-powered robots. This extension enables message communication between the two platforms over TCP. These messages include simulated sensor data, drive commands, ground truth state of simulated assets and scenario management.

This extension is enabled by default. If it is ever disabled, it can be re-enabled from the Extension Manager by searching for omni.isaac.robot_engine_bridge.

## API Documentation¶

See the API Documentation for complete usage information.

## User Interface¶

The Robot Engine Bridge contains an IsaacSim-side Isaac SDK application that sends and receives messages, these messages are converted to/from native SDK Cap’n Proto messages for use in the simulator.

To start the IsaacSim-side IsaacSDK application:

• Open the Robot Engine Bridge UI window from the Isaac Utils -> Robot Engine Bridge menu item.

• Click Create Application to start the default Isaac Sim application.

Once Pressed the specified application in the Application Json Path field will be launched. Custom application can be specified by modifying this path. The application can be stopped by pressing the Destroy Application button.

The default isaacsim application contains a interface node, an input isaac::alice::TcpSubscriber component on port 55001 and a output isaac::alice::TcpPublisher component on port 55000

Expand to see the full app config File
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41  { "name": "isaacsim", "modules": [], "graph": { "nodes": [ { "name": "interface", "components": [ { "name": "ledger", "type": "isaac::alice::MessageLedger" }, { "name": "input", "type": "isaac::alice::TcpSubscriber" }, { "name": "output", "type": "isaac::alice::TcpPublisher" }, { "name": "time_sync", "type": "isaac::alice::TimeSynchronizer" } ] } ], "edges": [] }, "config": { "interface": { "input": { "port": 55001, "host": "localhost" }, "output": { "port": 55000 } } } } 

The json path can also be provided at runtime by using the --carb/isaac/robot_engine_bridge/json="path/to/app.json" commandline argument

Note

• The set of Isaac SDK components currently supported is limited to: isaac::alice::MessageLedger, isaac::alice::TcpSubscriber, isaac::alice::TcpPublisher and isaac::alice::TimeSynchronizer and at least one of each should be present to work correctly. Multiple subscribers and publishers can be created and used by changing the default inputComponent and outputComponent fields in the components below.

## Robot Engine Bridge Components¶

All of the properties below can be edited in the “Raw USD Properties” section of the properties tab. Relationships can also be directly modified there.

The following properties are shared between all components

Expand to See Properties

Name

Type

Default

Description

nodeName

string

interface

Name of the node in the application to publish/subscribe to

enabled

bool

True

Enable or disable this component from sending/receiving data

timeOffset

double

0

Time offset applied to acquisition time when sending to SDK

The Scene loader component is a special component that allows the user to load USD files once the application has been created and play is pressed to start simulation. It is controlled via the collapsible UI in the main Robot Engine Extension UI.

Expand to See Properties

Name

Type

Default

Description

requestChannel

string

scenario_control

Channel name that receives json message with information on what usd stage to load

inputComponent

string

input

The name of the component on which commands are received

outputComponent

string

output

Then name of the component on which reply is sent

string

cameraControl

string

camera_switch

Channel name that receives a camera name to switch the viewport to

### Differential Base¶

This component is used to control a differential base

Expand to See Properties

Name

Type

Default

Description

accelerationSmoothing

float

1.0

A smoothing factor for the estimated acceleration

chassisPrim

relationship

USD Prim Path for Chassis Prim

inputChannel

string

base_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

leftWheelJointName

string

Name of left wheel joint

maxSpeed

float2

(1.5, 1.0)

The maximal allowed linear and angular speed in m/s and rad/s

maxTimeWithoutCommand

float

0.2

The maximum allowed time duration which the robot will continue with the last sent speed command in the absence of speed commands

outputChannel

string

output

The name of the channel on which state is published

outputComponent

string

base_state

The name of the component on which state is published

rightWheelJointName

string

Name of right wheel joint

robotFront

float3

(1,0,0)

Vector representing the robots forward direction

wheelBase

float

0.5

Distance between wheel and center of chassis in meters

float

0.1

Radius of driven wheels in meters

### Holonomic Base¶

This component is used to control a holonomic base with three wheels such as Kaya

Expand to See Properties

Name

Type

Default

Description

accelerationSmoothing

float

1.0

A smoothing factor for the estimated acceleration

articulationPrim

relationship

USD Prim Path for Articulation

inputChannel

string

base_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

maxSpeed

float2

(1.5, 1.0)

The maximal allowed linear and angular speed in m/s and rad/s

maxTimeWithoutCommand

float

0.2

The maximum allowed time duration which the robot will continue with the last sent speed command in the absence of speed commands

outputChannel

string

output

The name of the channel on which state is published

outputComponent

string

base_state

The name of the component on which state is published

robotFront

float3

(1,0,0)

Vector representing the robots forward direction

wheel1JointName

string

Name of first wheel joint

wheel2JointName

string

Name of second wheel joint

wheel3JointName

string

Name of third wheel joint

wheelBase

float

0.125

Distance from wheel to chassis center in meters

float

0.04

Radius of the wheel in meters

### Vehicle Control¶

This component is used to control a PxVehicle with acceleration and steering commands

Expand to See Properties

Name

Type

Default

Description

inputChannel

string

vehicle_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

outputChannel

string

vehicle_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

vehiclePrim

relationship

USD Prim path to the PxVehicle

usePID

bool

false

Enable PID controller for acceleration

historyLength

int

100

Length of buffer used to smooth acceleration data

controllerPIDValues

float3

(1,1,1)

PID controller values

### Joint Control¶

This component publishes and subscribes the joint positions of an articulation

Expand to See Properties

Name

Type

Default

Description

articulationPrim

relationship

USD Prim path to the articulation being controlled

inputChannel

string

joint_position

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

outputChannel

string

joint_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

### Scissor Lift¶

This component controls a vertical translational joint on an articulation

Expand to See Properties

Name

Type

Default

Description

articulationPrim

relationship

USD Prim path to the articulation being controlled

inputChannel

string

lift_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

liftJointName

string

lift_joint

Name of the translational joint on the articulation being controlled

liftSpeed

float

0.02

Speed at which the joint raises and lowers in m/s

outputChannel

string

lift_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

### Surface Gripper¶

This component simulates a suction cup gripper

Expand to See Properties

Name

Type

Default

Description

inputChannel

string

io_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

gripperEntity

string

gripper

name returned by the component

outputChannel

string

io_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

parentPrim

relationship

Rigid body prim representing the suction cup gripper

gripThreshold

float

1

forceLimit

float

1e10

Max force before joint breaks

torqueLimit

float

1e10

Max torque before joint breaks

bendAngle

float

0

Max angle the joint can bend before hitting a limit

stiffness

float

1e10

Stiffness of the joint

damping

float

1e3

damping parameter for the joint

offsetPosition

float3

(0,0,0)

offset from the parent prim where the joint is created

offsetRotation

float4

(1,0,0,0)

quaternion representing rotation offset of the joint from the parent prim

### Two Finger Gripper¶

This component simulates a gripper with two finger joints

Expand to See Properties

Name

Type

Default

Description

articulationPrim

relationship

The articulation that contains the two finger joints

inputChannel

string

io_command

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

gripperEntity

string

gripper

name returned by the component

outputChannel

string

io_state

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

closedDistance

float

0

Distance between fingers when closed

openDistance

float

0.04

Distance between fingers when open

leftFingerJoint

string

left_finger

The name of the left finger joint

rightFingerJoint

string

right_finger

The name of the right finger joint

speed

float

0

Zero speed will instantly close/open based on joint parameters

### Rigid Body Sink¶

This component publishes the rigid body poses for a list of primitives

Expand to See Properties

Name

Type

Default

Description

rigidBodyPrims

relationship

list of relationship targets to rigid bodies that need to be published

outputChannel

string

bodies

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

### Teleport¶

This component controls the pose for a list of primitives

Expand to See Properties

Name

Type

Default

Description

teleportPrims

relationship

list of relationship targets to rigid bodies that need to be teleported

inputChannel

string

teleport

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

### Scenario From message¶

This component can create new prims, teleport them and publish their rigid body pose

Expand to See Properties

Name

Type

Default

Description

inputChannel

string

scenario_actors

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

teleportInputComponent

string

input

name returned by the component

teleportInputChannel

string

teleport

Name of the node in the application to publish/subscribe to

rigidBodySinkOutputChannel

string

bodies

The name of the channel on which state is published

rigidBodySinkOutputComponent

string

output

The name of the component on which state is published

### Camera¶

This component publishes RGB, Depth and Segmentation data

Expand to See Properties

Name

Type

Default

Description

rgbOutputChannel

string

color

The name of the channel on which color state is published

rgbOutputComponent

string

output

The name of the component on which color state is published

depthOutputChannel

string

depth

The name of the channel on which depth state is published

depthOutputComponent

string

output

The name of the component on which depth state is published

segmentationOutputChannel

string

segmentation

The name of the channel on which segmentation state is published

segmentationOutputComponent

string

output

The name of the component on which segmentation state is published

boundingBox2DOutputChannel

string

bbox

The name of the channel on which 2D bbox state is published

boundingBox2DOutputComponent

string

output

The name of the component on which 2D bbox state is published

boundingBox3DOutputChannel

string

bbox3d

The name of the channel on which 3D bbox state is published

boundingBox3DOutputComponent

string

output

The name of the component on which 3D bbox state is published

boundingBox3DClassList

string

The list of class names to filter

rgbEnabled

bool

true

true if rgb data is published

depthEnabled

bool

false

true if depth data is published

segmentationEnabled

bool

false

true if segmentation data is published

boundingBox2dEnabled

bool

false

true if 2D bbox data is published

boundingBox3dEnabled

bool

false

true if 3D bbox data is published

cameraPrim

relationship

USD Prim path to the active camera in the viewport

resolution

int2

(1280, 720)

Resolution of the viewport

### Lidar¶

This component publishes lidar data

Expand to See Properties

Name

Type

Default

Description

lidarPrim

relationship

USD Prim path to lidar sensor

outputChannel

string

rangescan

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

### Contact Monitor¶

This component publishes a message if a specified object collides with another.

Expand to See Properties

Name

Type

Default

Description

targetPrim

relationship

USD Prim path to rigid body to report contacts for

ignoredPrims

relationship

USD Prims paths that will be ignored when reporting contacts

outputChannel

string

collision

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

forceThreshold

float

100000

Contact force threshold above which contacts are reported

### Occupancy Grid Map¶

This component publishes occupancy grid map

Expand to See Properties

Name

Type

Default

Description

parentPrim

relationship

If specified, used to compute mapping location

outputChannel

string

occupancy_map

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

offset

float3

(0,0,0)

Mapping location offset, if parentPrim is not specified this is the offset from global origin

cellSize

float

0.1

Size of each pixel in meters

degreesPerRay

float

5

Number of degrees per ray when doing a 360 sweep

surfaceOffset

float

0.02

Offset from surface in meters to compute next sweep

occupancyThreshold

float

1

Probability used to determine occupancy

maxRays

float

1000000

Max number of rays to compute before terminating

mapSize

int2

(256,256)

Map size in pixels, the mapping area is computed using cell size

occupiedValue

float

1.0

Value returned for a region that is occupied

unoccupiedValue

float

0.0

Value returned for a region that is unoccupied

unknownValue

float

0.5

Value returned for a region that could not be mapped from the start location

debugDraw

bool

false

Enable Debug Drawing

### Ultrasonic¶

This component publishes ultrasonic data

Expand to See Properties

Name

Type

Default

Description

ultrasonicPrim

relationship

USD Prim path to ultrasonic sensor array

outputChannel

string

uss_envelopes

The name of the channel on which state is published

outputComponent

string

output

The name of the component on which state is published

### Polyline Visualizer¶

This component visualizes polyline

Expand to See Properties

Name

Type

Default

Description

inputChannel

string

sight_plan

The name of the channel on which commands are received

inputComponent

string

input

The name of the component on which commands are received

width

float

1.0

Width of polyline

color

color4f

(1, 1, 1, 1)

Color of polyline

offset

float3

(0,0,0)

Translation offset for polyline

parentPrim

relationship

If specified will be used as origin transform for polyline

### Sample Applications¶

Below are some sample Isaac SDK applications leveraging Omniverse Isaac Sim. For more details, please refer to Isaac SDK Documentation.

Carter robot navigating in the Warehouse simulation environment. For more details, please refer to Isaac SDK Carter Navigation sample.

### Cart Delivery¶

The robot delivering a cart in the Warehouse simulation environment. For more details, please refer to Isaac SDK Cart Delivery sample.

### Shuffle Box with Simulator¶

The UR10 robot arm with suction gripper picks and drops a pink box in the simulation environment. For more details, please refer to Isaac SDK Shuffle Box sample.

### Simple Joint Control¶

The UR10 robot arm is manipulated in the simulation environment via joint commands from Isaac SDK. For more details, please refer to Isaac SDK Joint Control sample.

## Code Snippets¶

The following python-only examples show how to interface with the Isaac SDK when running in pure python mode. Make sure that the steps in the Setup section have been completed.

This sample loads any usd stage and starts the Omniverse Isaac Sim side sdk application to publish and receive TCP messages.

./python.sh standalone_examples/api/omni.isaac.robot_engine_bridge/load_stage.py --usd_path /Samples/Isaac_SDK/Scenario/carter_warehouse_with_forklifts.usd


### Pose Estimation¶

This sample demonstrates how to do two pass rendering for a pose estimation data generation scenario. In the first pass, the object of interest, a cube, is rendered occluded in the environment. In the second the object is rendered unoccluded with the environment hidden.

The textures in the environment, camera pose and object pose are all randomized. While running the current FPS is output to the screen.

./python.sh standalone_examples/api/omni.isaac.robot_engine_bridge/pose_estimation.py


### Publishing custom messages¶

This sample demonstrates how to build a custom Composite proto message and publish on a given port. This sample can be used as a template for publishing any data with a custom structure using one of the provided Isaac SDK proto messages. All proto messages supported by Isaac SDK are available via the pyalice python interface

./python.sh standalone_examples/api/omni.isaac.robot_engine_bridge/custom_message.py


## Tutorials & Examples¶

The following example showcases how to best use this extension:

Examples

• Isaac SDK Vehicle Control Example: Isaac Examples > Isaac SDK > Isaac SDK Vehicle Control