Replicator Isaac Extension [omni.replicator.isaac]
This extension builds upon omni.replicator to provide additional functionalities relevant to robotics.
Reinforcement Learning Domain Randomization
The following methods provide randomization functionalities of various parameters pertaining to omni.isaac.core.prims.RigidPrimView, omni.isaac.core.articulations.ArticulationView, and omni.isaac.core.SimulationContext. These methods are designed to perform randomizations in simulations where update to USD is disabled for faster simulation speed. These methods directly set randomized values to Physx as opposed to existing methods in omni.replicator.core, such as omni.replicator.core.modify.pose or omni.replicator.core.physics.physics_material, which utilizes USD APIs to set values and hence cannot be used when update to USD is disabled. Therefore, the following methods provided in omni.replicator.isaac are particularly useful for domain randomization in reinforcement learning (RL) as RL usecases benefit immensely from disabling update to USD to achieve faster simulation speed.
The following is a simple demo that showcases the workflow and the various features of
omni.replicator.isaac
for domain randomization. This demo script can be launched as
a standalone example, which can be found by going to Isaac Sim’s root directory and
go to standalone_examples/api/omni.replicator.isaac/randomization_demo.py
.
The omni.replicator.isaac
extension for domain randomization functions by constructing
an OmniGraph action graph, which consists of nodes that generate random values, regulate
frequency intervals of various randomizaiton properties, and write the random values to Physx.
This action graph gets executed according to the way in which the triggers are set up. Note
that it is necessesary to register the views to be randomized before constructing this action graph.
The first step is to create an entry point of the action graph using with dr.trigger.on_rl_frame(num_envs=num_envs):
.
It is worth noting that all views to be used with this extension must have the same number of
encapsulated prims, equaling num_envs
that is passed as an argument to the on_rl_frame
trigger.
After creating this entry point, there are two types of gates that determine when the nodes can write
to Physx: on_interval(interval)
and on_env_reset()
; these gates work in conjunction with
omni.replicator.isaac.physics_view.step_randomization(reset_inds)
. There exists an internal
step counter for every environment in the views. Everytime the step_randomization
method is called,
it resets the counter to zero for every environment listed in the reset_inds
argument while
incrementing the counter for every other environment. The on_interval(interval)
gate then ensures
the nodes to write to Physx whenever the counter for an environment is a multiple of the interval
argument. The on_env_reset()
gate writes to Physx whenever an environment’s counter gets reset by
the reset_inds
argument passed in the step_randomization
method.
Within each gate, the following methods can be used to randomize view properties: randomize_simulation_context
,
randomize_rigid_prim_view
and randomize_articulation_view
. For each of these methods, there
exists an operation
argument, which can be set to direct
, additive
, or scaling
. direct
sets the random values directly to Physx, while the behaviour of additive
and scaling
depends on
the gate it is controlled by. Under on_env_reset()
, additive
adds the random values to the default
values of a given attribute of the view, and scaling
multiplies the random values to the default values.
Under on_interval(interval)
, additive
and scaling
adds or multiplies the random values to the
last values set during on_env_reset()
. In essence, on_env_reset()
randomization should be treated
as correlated noise that lasts until the next reset, while on_interval(interval)
randomization should
be treated as uncorrelated noise.
After setting up this action graph, it is necessesary to run omni.replicator.core.orchestrator.run()
.
from omni.isaac.kit import SimulationApp
simulation_app = SimulationApp({"headless": False})
import numpy as np
from omni.isaac.core import World
from omni.isaac.core.prims.rigid_prim_view import RigidPrimView
from omni.isaac.core.articulations import ArticulationView
from omni.isaac.core.utils.prims import get_prim_at_path, define_prim
from omni.isaac.core.utils.stage import get_current_stage, add_reference_to_stage
from omni.isaac.nucleus import get_assets_root_path
from omni.isaac.core.objects import DynamicSphere
from omni.isaac.cloner import GridCloner
# create the world
world = World(stage_units_in_meters=1.0, physics_prim_path="/physicsScene", backend="numpy")
world.scene.add_default_ground_plane()
# set up grid cloner
cloner = GridCloner(spacing=1.5)
cloner.define_base_env("/World/envs")
define_prim("/World/envs/env_0")
# set up the first environment
DynamicSphere(prim_path="/World/envs/env_0/object", radius=0.1, position=np.array([0.75, 0.0, 0.2]))
add_reference_to_stage(
usd_path=get_assets_root_path()+ "/Isaac/Robots/Franka/franka.usd",
prim_path="/World/envs/env_0/franka",
)
# clone environments
num_envs = 4
prim_paths = cloner.generate_paths("/World/envs/env", num_envs)
env_pos = cloner.clone(source_prim_path="/World/envs/env_0", prim_paths=prim_paths)
# creates the views and set up world
object_view = RigidPrimView(prim_paths_expr="/World/envs/*/object", name="object_view")
franka_view = ArticulationView(prim_paths_expr="/World/envs/*/franka", name="franka_view")
world.scene.add(object_view)
world.scene.add(franka_view)
world.reset()
num_dof = franka_view.num_dof
# set up randomization with omni.replicator.isaac, imported as dr
import omni.replicator.isaac as dr
import omni.replicator.core as rep
dr.physics_view.register_simulation_context(world)
dr.physics_view.register_rigid_prim_view(object_view)
dr.physics_view.register_articulation_view(franka_view)
with dr.trigger.on_rl_frame(num_envs=num_envs):
with dr.gate.on_interval(interval=20):
dr.physics_view.randomize_simulation_context(
operation="scaling",
gravity=rep.distribution.uniform((1, 1, 0.0), (1, 1, 2.0)),
)
with dr.gate.on_interval(interval=50):
dr.physics_view.randomize_rigid_prim_view(
view_name=object_view.name,
operation="direct",
force=rep.distribution.uniform((0, 0, 2.5), (0, 0, 5.0)),
)
with dr.gate.on_interval(interval=10):
dr.physics_view.randomize_articulation_view(
view_name=franka_view.name,
operation="direct",
joint_velocities=rep.distribution.uniform(tuple([-2]*num_dof), tuple([2]*num_dof)),
)
with dr.gate.on_env_reset():
dr.physics_view.randomize_rigid_prim_view(
view_name=object_view.name,
operation="additive",
position=rep.distribution.normal((0.0, 0.0, 0.0), (0.2, 0.2, 0.0)),
velocity=[0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
)
dr.physics_view.randomize_articulation_view(
view_name=franka_view.name,
operation="additive",
joint_positions=rep.distribution.uniform(tuple([-0.5]*num_dof), tuple([0.5]*num_dof)),
position=rep.distribution.normal((0.0, 0.0, 0.0), (0.2, 0.2, 0.0)),
)
rep.orchestrator.run()
frame_idx = 0
while simulation_app.is_running():
if world.is_playing():
reset_inds = list()
if frame_idx % 200 == 0:
# triggers reset every 200 steps
reset_inds = np.arange(num_envs)
dr.physics_view.step_randomization(reset_inds)
world.step(render=True)
frame_idx += 1
Pytorch Online Writer and Listener
The PytorchWriter and PytorchListener are APIs for using omni.replicator’s writer API to retrieve various data such as RGB from the specified cameras (supports multiple cameras) and provides them to the user in both default format (eg. PNG for RGB data) and batched pytorch tensors. The PytorchListener provides an API to directly retrieve data sent to the PytorchWriter without the need to access the stored by omni.replicator’s BackendDispatch.
Omnigraph Nodes
OmniGraph Nodes
OgnSampleInSphere
Assignes uniformly sampled location in a sphere.
- Inputs
prims (target): prims to randomize. Default to [].
execIn (execution): exec. Default to 0.
radius (float): sphere radius. Default to 1.0.
- Outputs
execOut (execution): exec.
Random3f
This node outputs the poses of assets with semantic labels
- Inputs
minimum (float[3]): minimum range for randomization.
maximum (float[3]): minimum range for randomization.
- Outputs
output (float[3]): Random float 3.
OgnSampleOnSphere
Assignes uniformly sampled location on a sphere.
- Inputs
prims (target): prims to randomize. Default to [].
execIn (execution): exec. Default to 0.
radius (float): sphere radius. Default to 1.0.
- Outputs
execOut (execution): exec.
OgnWritePhysicsSimulationContext
This node writes physics attributes to TensorAPI views
- Inputs
prims (string): Name of registered view to randomize.
attribute (string): Name of attribute that is to be written.
operation (string): Type of randomization operation to be applied.
distribution (string): Type of distribution used to sample values.
dist_param_1 (float[]): Distribution parameter 1.
dist_param_2 (float[]): Distribution parameter 2.
num_buckets (int): Number of buckets to randomize from.
values (float[]): Values to be assigned to the physics attribute.
indices (int[]): Indices of the environments to assign the physics attribute.
on_reset (bool): indicates whether an on_reset context triggered the execution.
execIn (execution): exec.
- Outputs
execOut (execution): exec.
Pose
This node outputs the poses of assets with semantic labels
- Inputs
exec (execution): Trigger.
includeOccludedPrims (bool): Set to True if poses (and if enabled, centers) of fully occluded/out-of-frame semantic entities should be output.
getCenters (bool): Set to True if producing center coordinates of every semantic entity projected in the image space.
cameraRotation (float[]): Rotation of the desired camera frame from the default camera frame, as XYZ Euler angles.
imageWidth (uint): Width of the viewport.
imageHeight (uint): Height of the viewport.
cameraViewTransform (matrixd[4]): Camera view matrix.
cameraProjection (matrixd[4]): Camera projection matrix.
sdIMNumSemantics (uint): Number of semantic entities in the semantic arrays.
sdIMNumSemanticTokens (uint): Number of semantics token including the semantic entity path, the semantic entity types and if the number of semantic types is greater than one.
sdIMInstanceSemanticMap (uchar[]): Raw array of uint16_t of size sdIMNumInstances*sdIMMaxSemanticHierarchyDepth containing the mapping from the instances index to their inherited semantic entities.
sdIMSemanticTokenMap (token[]): Semantic array of token of size numSemantics * numSemanticTypes containing the mapping from the semantic entities to the semantic entity path and semantic types.
sdIMMinSemanticIndex (uint): Semantic id of the first instance in the instance arrays.
sdIMMaxSemanticHierarchyDepth (uint): Maximal number of semantic entities inherited by an instance.
sdIMSemanticWorldTransform (float[]): Semantic array of 4x4 float matrices containing the transform from local to world space for every semantic entity.
data (uchar[]): Buffer array data. Default to [].
bufferSize (uint): Size (in bytes) of the buffer (0 if the input is a texture).
semanticTypes (token[]): Semantic Types to consider. Default to [‘class’].
- Outputs
exec (execution): Trigger.
data (uchar[]): Semantic array of 4x4 float matrices containing the transform from local to view space for every semantic entity. Additionally, an optional semantic array of float[2] vectors containing the center coordinates of every semantic entity projected in the image space.
idToLabels (string): Mapping from id to semantic labels.
primPaths (token[]): Prim paths corresponding to each pose.
bufferSize (uint): Size (in bytes) of the buffer (0 if the input is a texture).
height (uint): Shape of the data.
width (uint): Shape of the data.
OgnOnRLFrame
[‘Triggered every frame in an Rl setting’]
- Inputs
run (bool): Run.
num_envs (int): number of RL environments.
- Outputs
execOut (execution): Output Execution.
resetInds (int[]): indices of environments to be reset.
frameNum (int[]): frame number for every environment.
OgnIntervalFiltering
Outputs indices if their frame count is a multiple of the interval
- Inputs
interval (int): randomization will take place once every interval frames.
frameCounts (int[]): the current frame number for every environment.
indices (int[]): a list of indices to use in case of ignoring interval.
ignoreInterval (bool): if true, will just pass indices to downstream node. Default to False.
execIn (execution): exec.
- Outputs
execOut (execution): exec.
indices (int[]): the indices that are at the interval and need to be randomized.
on_reset (bool): indicates whether an on_reset context triggered the execution.
OgnWritePhysicsArticulationView
This node writes physics attributes to TensorAPI views
- Inputs
prims (string): Name of registered view to randomize.
attribute (string): Name of attribute that is to be written.
operation (string): Type of randomization operation to be applied.
distribution (string): Type of distribution used to sample values.
dist_param_1 (float[]): Distribution parameter 1.
dist_param_2 (float[]): Distribution parameter 2.
num_buckets (int): Number of buckets to randomize from.
values (float[]): Values to be assigned to the physics attribute.
indices (int[]): Indices of the environments to assign the physics attribute.
on_reset (bool): indicates whether an on_reset context triggered the execution.
execIn (execution): exec.
- Outputs
execOut (execution): exec.
OgnSampleBetweenSpheres
Assignes uniformly sampled between two spheres
- Inputs
prims (target): prims to randomize. Default to [].
execIn (execution): exec. Default to 0.
radius1 (float): inner sphere radius. Default to 0.5.
radius2 (float): outer sphere radius. Default to 1.0.
- Outputs
execOut (execution): exec.
OgnCountIndices
[‘Get length of array’]
- Inputs
indices (int[]): indices to count.
- Outputs
count (uint): Count.
OgnWritePhysicsRigidPrimView
This node writes physics attributes to TensorAPI views
- Inputs
prims (string): Name of registered view to randomize.
attribute (string): Name of attribute that is to be written.
operation (string): Type of randomization operation to be applied.
distribution (string): Type of distribution used to sample values.
dist_param_1 (float[]): Distribution parameter 1.
dist_param_2 (float[]): Distribution parameter 2.
num_buckets (int): Number of buckets to randomize from.
values (float[]): Values to be assigned to the physics attribute.
indices (int[]): Indices of the environments to assign the physics attribute.
on_reset (bool): indicates whether an on_reset context triggered the execution.
execIn (execution): exec.
- Outputs
execOut (execution): exec.
Dope
Gets poses of assets as required by ground truth file for DOPE training
- Inputs
exec (execution): Trigger.
cameraModel (token): Camera model (pinhole or fisheye models).
cameraViewTransform (matrixd[4]): Camera view matrix.
cameraProjection (matrixd[4]): Camera projection matrix.
cameraFisheyeNominalWidth (int): Camera fisheye nominal width.
cameraFisheyeNominalHeight (int): Camera fisheye nominal height.
cameraFisheyeOpticalCentre (float[2]): Camera fisheye optical centre.
cameraFisheyeMaxFOV (float): Camera fisheye maximum field of view.
cameraFisheyePolynomial (float[]): Camera fisheye polynomial.
cameraNearFar (float[2]): Camera near/far clipping range.
horizontalAperture (float): Horizontal aperture of camera.
focalLength (float): Camera fisheye maximum field of view.
cameraRotation (float[3]): Camera rotation in euler angles.
width (uint): Width of the viewport.
height (uint): Height of the viewport.
boundingBox3d (uchar[]): 3d bounding box data.
occlusion (uchar[]): Occlusion data.
sdIMNumSemantics (uint): Number of semantic entities in the semantic arrays.
sdIMNumSemanticTokens (uint): Number of semantics token including the semantic entity path, the semantic entity types and if the number of semantic types is greater than one.
sdIMInstanceSemanticMap (uchar[]): Raw array of uint16_t of size sdIMNumInstances*sdIMMaxSemanticHierarchyDepth containing the mapping from the instances index to their inherited semantic entities.
sdIMSemanticTokenMap (token[]): Semantic array of token of size numSemantics * numSemanticTypes containing the mapping from the semantic entities to the semantic entity path and semantic types.
sdIMMinSemanticIndex (uint): Semantic id of the first instance in the instance arrays.
sdIMMaxSemanticHierarchyDepth (uint): Maximal number of semantic entities inherited by an instance.
semanticTypes (token[]): Semantic Types to consider. Default to [‘class’].
- Outputs
exec (execution): Trigger.
data (uchar[]): Semantic array of 4x4 float matrices containing the transform from local to view space for every semantic entity. Additionally, an optional semantic array of float[2] vectors containing the center coordinates of every semantic entity projected in the image space.
idToLabels (string): Mapping from id to semantic labels.
bufferSize (uint): Size (in bytes) of the buffer (0 if the input is a texture).
height (uint): Shape of the data.
width (uint): Shape of the data.