5. Replicator Composer

Replicator Composer is a tool for creating parameterizable offline datasets in Isaac Sim.

This tutorial will guide you through generating datasets with Replicator Composer.

• The Replicator Composer Parameter List provides the list of input parameters.

• The Replicator Composer Manual explains how to write the input parameter file with the parameters.

5.1. Learning Objectives

This tutorial includes the following:

1. Replicator Composer Overview

2. Prepare for Generation

3. Generate Data

4. Retrieve Output

5.2. Replicator Composer Overview

Below are sample models from the provided Nucleus Server. The scenes are generated using Replicator Composer.

Below diagrams the inputs, outputs, and parameterization of Replicator Composer.

A dataset parameterization is input from a YAML file containing “key, value” pairs that set parameter names and values. To create random datasets, parameter values can be set to distributions, which are sampled from per scene.

The input parameter file can point to TXT files called asset lists, which contain lists of elements (colors, model paths, coordinates, etc.) to sample from.

To generate a virtual scene using Isaac Sim Replicator, each parameter is sampled from. Then, models, textures, and material files are pulled from the Nucleus Server. Data is captured and output to a dataset directory (Retrieve the Output).

Replicator Composer supports the following outputs, with stereo and mono for each output (see Output Data Types for more information):

• RGB, Normals

• Distance To Camera(Depth), Distance To Image Plane

• Instance Segmentation, Semantic Segmentation, Instance Id Segmentation

• Bounding Box 2d Tight, Bounding Box 2d Loose, Bounding Box 3d

• Occlusion, Motion Vectors

5.3. Prepare for Generation

Replicator Composer can be run from the Omniverse Launcher.

5.3.1. Launcher

1. Create a <workspace> folder for input and output files.

2. Follow Workstation Installation.

3. From the Isaac Sim App Selector step, click on “Open in Terminal”.

4. From the terminal, follow Generate Data to start composing.

5.4. Generate Data

Replicator Composer is a command line tool. Below are steps to generate data from sample inputs or custom inputs.

Check Command Line Reference for additional usage information.

5.4.1. Sample Input Parameterizations

Generate datasets from provided inputs. The --headless argument can be removed if running from Launcher.

1. Generate a realistic dataset in a warehouse.

$ ./python.sh tools/composer/src/main.py \
--input parameters/warehouse.yaml --output */datasets/warehouse \
--num-scenes 10 --headless --mount <workspace>

2. Generate a challenging and chaotic domain-randomized dataset inspired by the FlyingThings3D synthetic dataset.

$ ./python.sh tools/composer/src/main.py \
--input parameters/flying_things_3d.yaml --output */datasets/flying_things_3d \
--num-scenes 10 --headless --mount <workspace>

3. Generate a sequential “4D” version of the FlyingThings3D dataset with random object motion.

$ ./python.sh tools/composer/src/main.py \
--input parameters/flying_things_4d.yaml --output */datasets/flying_things_4d \
--num-scenes 3 --headless --mount <workspace>

5.4.2. Custom Input Parameterization

Generate a dataset from a custom input.

5.4.2.1. Copy Provided Input Files

In the source code for Isaac Sim, there are baked-in input files. Several are used in the sample input parameterizations.

Accessing these inputs files will help template new custom inputs.

1. Copy input files from source code. Run the cmds in the opened terminal.

$ cp -r tools/composer/parameters <workspace>
$ cp -r tools/composer/assets <workspace>

5.4.2.2. Create an Input Parameter File

An Input Parameter file is a YAML file that parameterizes the dataset.

1. Follow Replicator Composer Parameter List and Replicator Composer Manual pages to create <input_parameter_file>.

2. If needed, review and/or template from the sample parameter files in <workspace>/parameters.

3. Move <input_parameter_file> into <workspace>/parameters.

5.4.2.3. Create Asset List Files

An Asset List file is a TXT file containing a list of elements (e.g. colors, object model paths, camera coordinates etc.) that can be pointed to by parameters in the Input Parameter file.

Note, if the provided Assets Lists already suffice the custom input parameter file, new Asset Lists are not needed.

1. Follow the steps in Asset Lists.

2. If needed, review and/or template from the sample Assets Lists in <workspace>/assets.

3. Move new Asset List files into <workspace>/assets.

5.4.2.4. Generate Custom Dataset

Compose a dataset with custom inputs.

$ ./python.sh tools/composer/src/main.py \
--input */parameters/<input_parameter_file> \
--output */datasets/<dataset_name> \
--num-scenes <num_scenes> \
--headless \
--mount <workspace>

5.5. Retrieve the Output

Dataset and other files will be output to the output directory (given by the output_dir parameter, which can be set by the --output argument).

Example output file tree (with 1 data sample, bounding_box_2d_tight, rgb, instance_segmentation and semantic_segmentation annotators enabled.):

<output_dir>
├── data
│   ├── left
│   │   ├── bounding_box_2d_tight
│   │   │   ├── bounding_box_2d_tight_0.npy
│   │   │   └── bounding_box_2d_tight_labels_0.json
│   │   ├── instance_segmentation
│   │   │   ├── instance_segmentation_0.png
│   │   │   ├── instance_segmentation_mapping_0.json
│   │   │   └── instance_segmentation_semantics_mapping_0.json
│   │   ├── rgb
│   │   │   └── rgb_0.png
│   │   └── semantic_segmentation
│   │       ├── semantic_segmentation_0.png
│   │       └── semantic_segmentation_labels_0.json
│   └── right
│       ├── bounding_box_2d_tight
│       │   ├── bounding_box_2d_tight_0.npy
│       │   └── bounding_box_2d_tight_labels_0.json
│       ├── instance_segmentation
│       │   ├── instance_segmentation_0.png
│       │   ├── instance_segmentation_mapping_0.json
│       │   └── instance_segmentation_semantics_mapping_0.json
│       ├── rgb
│       │   └── rgb_0.png
│       └── semantic_segmentation
│           ├── semantic_segmentation_0.png
│           └── semantic_segmentation_labels_0.json
└── metadata.txt

5.6. Visualize the Models

Running in Visualize Models mode (enabled with the --visualize-models argument) will output visuals (rendered in Isaac Sim) of each object model defined in the input parameter file with a non-zero obj_count, instead of outputting a dataset.

The Visualize Models mode is useful for curating the models defined in a parameterization before generating a dataset.

Each visual includes four RGBs. Visuals are output to <output_dir>/visualize_models/<model_file_name>.png.

Generate visuals.

$ ./python.sh tools/composer/src/main.py \
--input parameters/warehouse.yaml --output */datasets/warehouse \
--headless --mount <workspace> --visualize-models

5.7. Command Line Reference

The following describes command-line options.

$ ./python.sh tools/composer/src/main.py -h

usage: main.py [-h] [--input INPUT] [--visualize-models] [--mount MOUNT]
                   [--headless] [--nap] [--overwrite] [--output OUTPUT]
                   [--num-scenes NUM_SCENES] [--nucleus-server NUCLEUS_SERVER]

optional arguments:
  -h, --help            show this help message and exit


  --input INPUT         Path to input parameter file.

  --mount MOUNT         Path to mount symbolized in parameter files via '*'.

  --headless            Will not launch Isaac SIM window.

  --nap                 Will nap Isaac SIM after the first scene is generated.

  --overwrite           Overwrites dataset in output directory.

  --output OUTPUT       Output directory. Overrides 'output_dir' param.

  --num-scenes NUM_SCENES, --num_scenes NUM_SCENES
                            Num scenes in dataset. Overrides 'num_scenes' param.

  --nucleus-server NUCLEUS_SERVER, --nucleus_server NUCLEUS_SERVER
                            Nucleus Server URL. Overrides 'nucleus_server' param.

---

input

The path to the input parameter file. Follows Paths rules.

---

---

mount

See 2. Mounted Paths in Paths.

---

headless

If given, Replicator Composer will run without launching the Isaac Sim client.

---

nap

If given, Replicator Composer will nap after the first scene is generated.

If set without headless and run in Launcher, allows the user to examine or debug the first scene in the Isaac Sim client.

---

overwrite

If given, generation will always start from the 0 index and overwrite any files in the output_dir.

If not given, if file is existed in the output_dir, it will keeps the old file. Otherwise, it will writes new data to the output_dir.

---

output

If given, overwrites the output_dir parameter. Follows Paths rules.

---

num-scenes

If given, overwrites the num_scenes parameter.

---

nucleus-server

If given, overwrites the nucleus_server parameter.

---

5.8. UI

Running the script from the UI can be done by enabling the Composer Window from Replicator->Composer. The UI enables loading, visualizing, and recording the scene using the editor. The aforementioned paths can be set in the Scene frame and the scene can be loaded using the Load button. The generated graph can then be run/stopped/paused/resumed/previewed using the buttons in the Control frame.