7. Replicator Composer

Replicator Composer is a tool for creating parameterizable offline datasets in Isaac Sim.

This tutorial will guide you through generating datasets with Replicator Composer.

7.1. Learning Objectives

This tutorial includes the following:

  1. Replicator Composer Overview

  2. Prepare for Generation

  3. Generate Data

  4. Retrieve Output

7.2. Replicator Composer Overview

Below are sample models from the provided Nucleus Server. The scenes are generated using Replicator Composer.

../_images/isaac_replicator_composer_samples_1.jpg ../_images/isaac_replicator_composer_samples_2.jpg ../_images/isaac_replicator_composer_samples_3.jpg ../_images/isaac_replicator_composer_samples_4.jpg ../_images/isaac_replicator_composer_samples_5.jpg ../_images/isaac_replicator_composer_samples_6.jpg

Below diagrams the inputs, outputs, and parameterization of Replicator Composer.

../_images/isaac_replicator_composer_arch.png

A dataset parameterization is input from a YAML file containing “key, value” pairs that set parameter names and values. To create random datasets, parameter values can be set to distributions, which are sampled from per scene.

The input parameter file can point to other TXT files called asset lists, which contain lists of elements (colors, model paths, coordinates, etc.) to sample from.

To generate a virtual scene using Isaac Sim Replicator, each parameter is sampled from. Then, models, textures, and material files are pulled from the Nucleus Server. Data is captured and output to a dataset directory (Retrieve the Output).

Replicator Composer supports the following outputs, with stereo and mono for each output (see Output Data Types for more information):

  • RGB

  • Depth, Disparity

  • Instance Segmentation, Semantic Segmentation

  • Bounding Box 2d Tight, Bounding Box 2d Loose, Bounding Box 3d

7.3. Prepare for Generation

Replicator Composer can be run from the Omniverse Launcher or in the Isaac Sim Docker container.

7.3.1. Launcher

  1. Create a <workspace> folder for input and output files.

  2. Follow Basic Isaac Sim Installation.

  3. From the Isaac Sim App Selector step, click on “Open in Terminal”.

  4. From the terminal, follow Generate Data to start composing.

7.3.2. Docker

  1. Create a <workspace> folder for inputs and output files.

  2. Follow Container Deployment through step 6.

  3. Run Isaac Sim container with an interactive BASH session.

    $ sudo docker run --name isaac-sim --entrypoint bash -it --gpus all -e "ACCEPT_EULA=Y" --rm --network=host \
    -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
    -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
    -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
    -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
    -v ~/docker/isaac-sim/config:/root/.nvidia-omniverse/config:rw \
    -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
    -v ~/docker/isaac-sim/documents:/root/Documents:rw \
    -v <workspace>:<workspace> \
    nvcr.io/nvidia/isaac-sim:2021.2.0
    
  1. Start Nucleus Server in the Isaac Sim container.

    $ ./start_nucleus.sh
    
  2. Inside the container, follow Generate Data to start composing.

7.4. Generate Data

Replicator Composer is a command line tool. Below are steps to generate data from sample inputs or custom inputs.

Check Command Line Reference for additional usage information.

7.4.1. Sample Input Parameterizations

Generate datasets from provided inputs. The --headless argument can be removed if running from Launcher.

  1. Generate a realistic dataset in a warehouse.

$ ./python.sh tools/composer/src/main.py \
--input parameters/warehouse.yaml --output */datasets/warehouse \
--num-scenes 10 --headless --mount <workspace>
../_images/isaac_replicator_composer_warehouse_1.jpg ../_images/isaac_replicator_composer_warehouse_2.jpg ../_images/isaac_replicator_composer_warehouse_3.jpg ../_images/isaac_replicator_composer_warehouse_4.jpg
  1. Generate a challenging and chaotic domain-randomized dataset inspired by the FlyingThings3D synthetic dataset.

$ ./python.sh tools/composer/src/main.py \
--input parameters/flying_things_3d.yaml --output */datasets/flying_things_3d \
--num-scenes 10 --headless --mount <workspace>
../_images/isaac_replicator_composer_flying_things_3d_1.jpg ../_images/isaac_replicator_composer_flying_things_3d_2.jpg ../_images/isaac_replicator_composer_flying_things_3d_3.jpg ../_images/isaac_replicator_composer_flying_things_3d_4.jpg
  1. Generate a sequential “4D” version of the FlyingThings3D dataset with random object motion.

$ ./python.sh tools/composer/src/main.py \
--input parameters/flying_things_4d.yaml --output */datasets/flying_things_4d \
--num-scenes 3 --headless --mount <workspace>
../_images/isaac_replicator_composer_flying_things_4d_1.gif ../_images/isaac_replicator_composer_flying_things_4d_2.gif ../_images/isaac_replicator_composer_flying_things_4d_3.gif ../_images/isaac_replicator_composer_flying_things_4d_4.gif

7.4.2. Custom Input Parameterization

Generate a dataset from a custom input.

7.4.2.1. Copy Provided Input Files

In the source code for Isaac Sim, there are baked-in input files. Several are used in the sample input parameterizations.
Accessing these inputs files will help template new custom inputs.
  1. Copy input files from source code.

    1. If following Launcher steps, run the cmds in the opened terminal.

    2. Otherwise, if following Docker steps, run the cmds in the opened container.

$ cp -r tools/composer/parameters <workspace>
$ cp -r tools/composer/assets <workspace>

7.4.2.2. Create an Input Parameter File

An Input Parameter file is a YAML file that parameterizes the dataset.

  1. Follow Replicator Composer Parameter List and Replicator Composer Manual pages to create <input_parameter_file>.

  2. If needed, review and/or template from the sample parameter files in <workspace>/parameters.

  3. Move <input_parameter_file> into <workspace>/parameters.

7.4.2.3. Create Asset List Files

An Asset List file is a TXT file containing a list of elements (e.g. colors, object model paths, camera coordinates etc.) that can be pointed to by parameters in the Input Parameter file.

Note, if the provided Assets Lists already suffice the custom input parameter file, new Asset Lists are not needed.

  1. Follow the steps in Asset Lists.

  2. If needed, review and/or template from the sample Assets Lists in <workspace>/assets.

  3. Move new Asset List files into <workspace>/assets.

7.4.2.4. Generate Custom Dataset

Compose a dataset with custom inputs.

$ ./python.sh tools/composer/src/main.py \
--input */parameters/<input_parameter_file> \
--output */datasets/<dataset_name> \
--num-scenes <num_scenes> \
--headless \
--mount <workspace>

7.5. Retrieve the Output

Dataset and other files will be output to the output directory (given by the output_dir parameter, which can be set by the --output argument).

Example output file tree (with 1 data sample):

<output_dir>
├── data
│   ├── left
│   │   ├── rgb
│   │   │   └── 000.png
│   │   └── disparity
│   │     ├── 000.pfm
│   │     └── visuals
│   │         └── 000.png
│   └── right
│       └── rgb
│           └── 000.png
├── log
│   ├── sampling_log.yaml
│   └── metrics.txt
└── parameters
    ├── input.yaml
    └── profiles
        └── default.yaml

7.6. Visualize the Models

Running in Visualize Models mode (enabled with the --visualize-models argument) will output visuals (rendered in Isaac Sim) of each object model defined in the input parameter file with a non-zero obj_count, instead of outputting a dataset.

The Visualize Models mode is useful for curating the models defined in a parameterization before generating a dataset.

Each visual includes four RGBs and four wireframes. Visuals are output to <output_dir>/data/visuals.

Generate visuals.

$ ./python.sh tools/composer/src/main.py \
--input parameters/warehouse.yaml --output */datasets/warehouse \
--headless --mount <workspace> --visualize-models
../_images/isaac_replicator_composer_visual_1.jpg ../_images/isaac_replicator_composer_visual_3.jpg ../_images/isaac_replicator_composer_visual_2.jpg ../_images/isaac_replicator_composer_visual_4.jpg

7.7. Command Line Reference

The following describes command-line options.

$ ./python.sh tools/composer/src/main.py -h

usage: main.py [-h] [--input INPUT] [--visualize-models] [--mount MOUNT]
                   [--headless] [--nap] [--overwrite] [--output OUTPUT]
                   [--num-scenes NUM_SCENES] [--nucleus-server NUCLEUS_SERVER]

optional arguments:
  -h, --help            show this help message and exit


  --input INPUT         Path to input parameter file.


  --visualize-models, --visualize_models
                            Output visuals of all object models defined in input
                            parameter file, instead of outputting a dataset.

  --mount MOUNT         Path to mount symbolized in parameter files via '*'.

  --headless            Will not launch Isaac SIM window.

  --nap                 Will nap Isaac SIM after the first scene is generated.

  --overwrite           Overwrites dataset in output directory.

  --output OUTPUT       Output directory. Overrides 'output_dir' param.

  --num-scenes NUM_SCENES, --num_scenes NUM_SCENES
                            Num scenes in dataset. Overrides 'num_scenes' param.

  --nucleus-server NUCLEUS_SERVER, --nucleus_server NUCLEUS_SERVER
                            Nucleus Server URL. Overrides 'nucleus_server' param.

input

The path to the input parameter file. Follows Paths rules.

visualize-models

See the Visualize the Models section.


mount

See 2. Mounted Paths in Paths.


headless

If given, Replicator Composer will run without launching the Isaac Sim client.


nap

If given, Replicator Composer will nap after the first scene is generated.
If set without headless and run in Launcher, allows the user to examine or debug the first scene in the Isaac Sim client.

overwrite

If given, generation will always start from the 0 index and overwrite any files in the output_dir.
If not given, generation will start from the last file index found in the output_dir.

output

If given, overwrites the output_dir parameter. Follows Paths rules.

num-scenes

If given, overwrites the num_scenes parameter.


nucleus-server

If given, overwrites the nucleus_server parameter.