Replicator Isaac Extension [omni.replicator.isaac]

This extension builds upon omni.replicator to provide additional functionalities relevant to robotics.

Reinforcement Learning Domain Randomization

The following methods provide randomization functionalities of various parameters pertaining to omni.isaac.core.prims.RigidPrimView, omni.isaac.core.articulations.ArticulationView, and omni.isaac.core.SimulationContext. These methods are designed to perform randomizations in simulations where update to USD is disabled for faster simulation speed. These methods directly set randomized values to Physx as opposed to existing methods in omni.replicator.core, such as omni.replicator.core.modify.pose or omni.replicator.core.physics.physics_material, which utilizes USD APIs to set values and hence cannot be used when update to USD is disabled. Therefore, the following methods provided in omni.replicator.isaac are particularly useful for domain randomization in reinforcement learning (RL) as RL usecases benefit immensely from disabling update to USD to achieve faster simulation speed.

The following is a simple demo that showcases the workflow and the various features of omni.replicator.isaac for domain randomization. This demo script can be launched as a standalone example, which can be found by going to Isaac Sim’s root directory and go to standalone_examples/api/omni.replicator.isaac/randomization_demo.py.

The omni.replicator.isaac extension for domain randomization functions by constructing an OmniGraph action graph, which consists of nodes that generate random values, regulate frequency intervals of various randomizaiton properties, and write the random values to Physx. This action graph gets executed according to the way in which the triggers are set up. Note that it is necessesary to register the views to be randomized before constructing this action graph.

The first step is to create an entry point of the action graph using with dr.trigger.on_rl_frame(num_envs=num_envs):. It is worth noting that all views to be used with this extension must have the same number of encapsulated prims, equaling num_envs that is passed as an argument to the on_rl_frame trigger.

After creating this entry point, there are two types of gates that determine when the nodes can write to Physx: on_interval(interval) and on_env_reset(); these gates work in conjunction with omni.replicator.isaac.physics_view.step_randomization(reset_inds). There exists an internal step counter for every environment in the views. Everytime the step_randomization method is called, it resets the counter to zero for every environment listed in the reset_inds argument while incrementing the counter for every other environment. The on_interval(interval) gate then ensures the nodes to write to Physx whenever the counter for an environment is a multiple of the interval argument. The on_env_reset() gate writes to Physx whenever an environment’s counter gets reset by the reset_inds argument passed in the step_randomization method.

Within each gate, the following methods can be used to randomize view properties: randomize_simulation_context, randomize_rigid_prim_view and randomize_articulation_view. For each of these methods, there exists an operation argument, which can be set to direct, additive, or scaling. direct sets the random values directly to Physx, while the behaviour of additive and scaling depends on the gate it is controlled by. Under on_env_reset(), additive adds the random values to the default values of a given attribute of the view, and scaling multiplies the random values to the default values. Under on_interval(interval), additive and scaling adds or multiplies the random values to the last values set during on_env_reset(). In essence, on_env_reset() randomization should be treated as correlated noise that lasts until the next reset, while on_interval(interval) randomization should be treated as uncorrelated noise.

After setting up this action graph, it is necessesary to run omni.replicator.core.orchestrator.run().

from omni.isaac.kit import SimulationApp
simulation_app = SimulationApp({"headless": False})

import numpy as np
from omni.isaac.core import World
from omni.isaac.core.prims.rigid_prim_view import RigidPrimView
from omni.isaac.core.articulations import ArticulationView
from omni.isaac.core.utils.prims import get_prim_at_path, define_prim
from omni.isaac.core.utils.stage import get_current_stage, add_reference_to_stage
from omni.isaac.nucleus import get_assets_root_path
from omni.isaac.core.objects import DynamicSphere
from omni.isaac.cloner import GridCloner

# create the world
world = World(stage_units_in_meters=1.0, physics_prim_path="/physicsScene", backend="numpy")
world.scene.add_default_ground_plane()

# set up grid cloner
cloner = GridCloner(spacing=1.5)
cloner.define_base_env("/World/envs")
define_prim("/World/envs/env_0")

# set up the first environment
DynamicSphere(prim_path="/World/envs/env_0/object", radius=0.1, position=np.array([0.75, 0.0, 0.2]))
add_reference_to_stage(
    usd_path=get_assets_root_path()+ "/Isaac/Robots/Franka/franka.usd",
    prim_path="/World/envs/env_0/franka",
)

# clone environments
num_envs = 4
prim_paths = cloner.generate_paths("/World/envs/env", num_envs)
env_pos = cloner.clone(source_prim_path="/World/envs/env_0", prim_paths=prim_paths)

# creates the views and set up world
object_view = RigidPrimView(prim_paths_expr="/World/envs/*/object", name="object_view")
franka_view = ArticulationView(prim_paths_expr="/World/envs/*/franka", name="franka_view")
world.scene.add(object_view)
world.scene.add(franka_view)
world.reset()

num_dof = franka_view.num_dof

# set up randomization with omni.replicator.isaac, imported as dr
import omni.replicator.isaac as dr
import omni.replicator.core as rep

dr.physics_view.register_simulation_context(world)
dr.physics_view.register_rigid_prim_view(object_view)
dr.physics_view.register_articulation_view(franka_view)

with dr.trigger.on_rl_frame(num_envs=num_envs):
    with dr.gate.on_interval(interval=20):
        dr.physics_view.randomize_simulation_context(
            operation="scaling",
            gravity=rep.distribution.uniform((1, 1, 0.0), (1, 1, 2.0)),
        )
    with dr.gate.on_interval(interval=50):
        dr.physics_view.randomize_rigid_prim_view(
            view_name=object_view.name,
            operation="direct",
            force=rep.distribution.uniform((0, 0, 2.5), (0, 0, 5.0)),
        )
    with dr.gate.on_interval(interval=10):
        dr.physics_view.randomize_articulation_view(
            view_name=franka_view.name,
            operation="direct",
            joint_velocities=rep.distribution.uniform(tuple([-2]*num_dof), tuple([2]*num_dof)),
        )
    with dr.gate.on_env_reset():
        dr.physics_view.randomize_rigid_prim_view(
            view_name=object_view.name,
            operation="additive",
            position=rep.distribution.normal((0.0, 0.0, 0.0), (0.2, 0.2, 0.0)),
            velocity=[0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
        )
        dr.physics_view.randomize_articulation_view(
            view_name=franka_view.name,
            operation="additive",
            joint_positions=rep.distribution.uniform(tuple([-0.5]*num_dof), tuple([0.5]*num_dof)),
            position=rep.distribution.normal((0.0, 0.0, 0.0), (0.2, 0.2, 0.0)),
        )
rep.orchestrator.run()


frame_idx = 0
while simulation_app.is_running():
    if world.is_playing():
        reset_inds = list()
        if frame_idx % 200 == 0:
            # triggers reset every 200 steps
            reset_inds = np.arange(num_envs)
        dr.physics_view.step_randomization(reset_inds)
        world.step(render=True)
        frame_idx += 1

Pytorch Online Writer and Listener

The PytorchWriter and PytorchListener are APIs for using omni.replicator’s writer API to retrieve various data such as RGB from the specified cameras (supports multiple cameras) and provides them to the user in both default format (eg. PNG for RGB data) and batched pytorch tensors. The PytorchListener provides an API to directly retrieve data sent to the PytorchWriter without the need to access the stored by omni.replicator’s BackendDispatch.

Omnigraph Nodes

OmniGraph Nodes

OgnSampleInSphere

Assignes uniformly sampled location in a sphere.

Inputs

prims (target): prims to randomize. Default to [].
execIn (execution): exec. Default to 0.
radius (float): sphere radius. Default to 1.0.

Outputs

execOut (execution): exec.

Random3f

This node outputs the poses of assets with semantic labels

Inputs

minimum (float[3]): minimum range for randomization.
maximum (float[3]): minimum range for randomization.

Outputs

output (float[3]): Random float 3.

OgnSampleOnSphere

Assignes uniformly sampled location on a sphere.

Inputs

prims (target): prims to randomize. Default to [].
execIn (execution): exec. Default to 0.
radius (float): sphere radius. Default to 1.0.

Outputs

execOut (execution): exec.

OgnWritePhysicsSimulationContext

This node writes physics attributes to TensorAPI views

Inputs

prims (string): Name of registered view to randomize.
attribute (string): Name of attribute that is to be written.
operation (string): Type of randomization operation to be applied.
distribution (string): Type of distribution used to sample values.
dist_param_1 (float[]): Distribution parameter 1.
dist_param_2 (float[]): Distribution parameter 2.
num_buckets (int): Number of buckets to randomize from.
values (float[]): Values to be assigned to the physics attribute.
indices (int[]): Indices of the environments to assign the physics attribute.
on_reset (bool): indicates whether an on_reset context triggered the execution.
execIn (execution): exec.

Outputs

execOut (execution): exec.

Pose

This node outputs the poses of assets with semantic labels

Inputs

exec (execution): Trigger.
includeOccludedPrims (bool): Set to True if poses (and if enabled, centers) of fully occluded/out-of-frame semantic entities should be output.
getCenters (bool): Set to True if producing center coordinates of every semantic entity projected in the image space.
cameraRotation (float[]): Rotation of the desired camera frame from the default camera frame, as XYZ Euler angles.
imageWidth (uint): Width of the viewport.
imageHeight (uint): Height of the viewport.
cameraViewTransform (matrixd[4]): Camera view matrix.
cameraProjection (matrixd[4]): Camera projection matrix.
sdIMNumSemantics (uint): Number of semantic entities in the semantic arrays.
sdIMNumSemanticTokens (uint): Number of semantics token including the semantic entity path, the semantic entity types and if the number of semantic types is greater than one.
sdIMInstanceSemanticMap (uchar[]): Raw array of uint16_t of size sdIMNumInstances*sdIMMaxSemanticHierarchyDepth containing the mapping from the instances index to their inherited semantic entities.
sdIMSemanticTokenMap (token[]): Semantic array of token of size numSemantics * numSemanticTypes containing the mapping from the semantic entities to the semantic entity path and semantic types.
sdIMMinSemanticIndex (uint): Semantic id of the first instance in the instance arrays.
sdIMMaxSemanticHierarchyDepth (uint): Maximal number of semantic entities inherited by an instance.
sdIMSemanticWorldTransform (float[]): Semantic array of 4x4 float matrices containing the transform from local to world space for every semantic entity.
data (uchar[]): Buffer array data. Default to [].
bufferSize (uint): Size (in bytes) of the buffer (0 if the input is a texture).
semanticTypes (token[]): Semantic Types to consider. Default to [‘class’].

Outputs

exec (execution): Trigger.
data (uchar[]): Semantic array of 4x4 float matrices containing the transform from local to view space for every semantic entity. Additionally, an optional semantic array of float[2] vectors containing the center coordinates of every semantic entity projected in the image space.
idToLabels (string): Mapping from id to semantic labels.
primPaths (token[]): Prim paths corresponding to each pose.
bufferSize (uint): Size (in bytes) of the buffer (0 if the input is a texture).
height (uint): Shape of the data.
width (uint): Shape of the data.

OgnOnRLFrame

[‘Triggered every frame in an Rl setting’]

Inputs

run (bool): Run.
num_envs (int): number of RL environments.

Outputs

execOut (execution): Output Execution.
resetInds (int[]): indices of environments to be reset.
frameNum (int[]): frame number for every environment.

OgnIntervalFiltering

Outputs indices if their frame count is a multiple of the interval

Inputs

interval (int): randomization will take place once every interval frames.
frameCounts (int[]): the current frame number for every environment.
indices (int[]): a list of indices to use in case of ignoring interval.
ignoreInterval (bool): if true, will just pass indices to downstream node. Default to False.
execIn (execution): exec.

Outputs

execOut (execution): exec.
indices (int[]): the indices that are at the interval and need to be randomized.
on_reset (bool): indicates whether an on_reset context triggered the execution.

OgnWritePhysicsArticulationView

This node writes physics attributes to TensorAPI views

Inputs

prims (string): Name of registered view to randomize.
attribute (string): Name of attribute that is to be written.
operation (string): Type of randomization operation to be applied.
distribution (string): Type of distribution used to sample values.
dist_param_1 (float[]): Distribution parameter 1.
dist_param_2 (float[]): Distribution parameter 2.
num_buckets (int): Number of buckets to randomize from.
values (float[]): Values to be assigned to the physics attribute.
indices (int[]): Indices of the environments to assign the physics attribute.
on_reset (bool): indicates whether an on_reset context triggered the execution.
execIn (execution): exec.

Outputs

execOut (execution): exec.

OgnSampleBetweenSpheres

Assignes uniformly sampled between two spheres

Inputs

prims (target): prims to randomize. Default to [].
execIn (execution): exec. Default to 0.
radius1 (float): inner sphere radius. Default to 0.5.
radius2 (float): outer sphere radius. Default to 1.0.

Outputs

execOut (execution): exec.

OgnCountIndices

[‘Get length of array’]

Inputs

indices (int[]): indices to count.

Outputs

count (uint): Count.

OgnWritePhysicsRigidPrimView

This node writes physics attributes to TensorAPI views

Inputs

prims (string): Name of registered view to randomize.
attribute (string): Name of attribute that is to be written.
operation (string): Type of randomization operation to be applied.
distribution (string): Type of distribution used to sample values.
dist_param_1 (float[]): Distribution parameter 1.
dist_param_2 (float[]): Distribution parameter 2.
num_buckets (int): Number of buckets to randomize from.
values (float[]): Values to be assigned to the physics attribute.
indices (int[]): Indices of the environments to assign the physics attribute.
on_reset (bool): indicates whether an on_reset context triggered the execution.
execIn (execution): exec.

Outputs

execOut (execution): exec.

Dope

Gets poses of assets as required by ground truth file for DOPE training

Inputs

exec (execution): Trigger.
cameraModel (token): Camera model (pinhole or fisheye models).
cameraViewTransform (matrixd[4]): Camera view matrix.
cameraProjection (matrixd[4]): Camera projection matrix.
cameraFisheyeNominalWidth (int): Camera fisheye nominal width.
cameraFisheyeNominalHeight (int): Camera fisheye nominal height.
cameraFisheyeOpticalCentre (float[2]): Camera fisheye optical centre.
cameraFisheyeMaxFOV (float): Camera fisheye maximum field of view.
cameraFisheyePolynomial (float[]): Camera fisheye polynomial.
cameraNearFar (float[2]): Camera near/far clipping range.
horizontalAperture (float): Horizontal aperture of camera.
focalLength (float): Camera fisheye maximum field of view.
cameraRotation (float[3]): Camera rotation in euler angles.
width (uint): Width of the viewport.
height (uint): Height of the viewport.
boundingBox3d (uchar[]): 3d bounding box data.
occlusion (uchar[]): Occlusion data.
sdIMNumSemantics (uint): Number of semantic entities in the semantic arrays.
sdIMNumSemanticTokens (uint): Number of semantics token including the semantic entity path, the semantic entity types and if the number of semantic types is greater than one.
sdIMInstanceSemanticMap (uchar[]): Raw array of uint16_t of size sdIMNumInstances*sdIMMaxSemanticHierarchyDepth containing the mapping from the instances index to their inherited semantic entities.
sdIMSemanticTokenMap (token[]): Semantic array of token of size numSemantics * numSemanticTypes containing the mapping from the semantic entities to the semantic entity path and semantic types.
sdIMMinSemanticIndex (uint): Semantic id of the first instance in the instance arrays.
sdIMMaxSemanticHierarchyDepth (uint): Maximal number of semantic entities inherited by an instance.
semanticTypes (token[]): Semantic Types to consider. Default to [‘class’].

Outputs

exec (execution): Trigger.
data (uchar[]): Semantic array of 4x4 float matrices containing the transform from local to view space for every semantic entity. Additionally, an optional semantic array of float[2] vectors containing the center coordinates of every semantic entity projected in the image space.
idToLabels (string): Mapping from id to semantic labels.
bufferSize (uint): Size (in bytes) of the buffer (0 if the input is a texture).
height (uint): Shape of the data.
width (uint): Shape of the data.