.. _isaac_sim_app_tutorial_replicator_composer: Replicator Composer =================== Replicator Composer is a tool for creating parameterizable offline datasets in Isaac Sim. This tutorial will guide you through generating datasets with Replicator Composer. * The :ref:`isaac_sim_app_manual_replicator_composer_parameter_list` provides the list of input parameters. * The :ref:`isaac_sim_app_manual_replicator_composer` explains how to write the input parameter file with the parameters. Learning Objectives ------------------- This tutorial includes the following: 1. Replicator Composer Overview 2. Prepare for Generation 3. Generate Data 4. Retrieve Output Replicator Composer Overview ---------------------------- Below are sample models from the provided Nucleus Server. The scenes are generated using Replicator Composer. .. image:: /content/images/isaac_replicator_composer_samples_1.jpg :width: 16.3% .. image:: /content/images/isaac_replicator_composer_samples_2.jpg :width: 16.3% .. image:: /content/images/isaac_replicator_composer_samples_3.jpg :width: 16.3% .. image:: /content/images/isaac_replicator_composer_samples_4.jpg :width: 16.3% .. image:: /content/images/isaac_replicator_composer_samples_5.jpg :width: 16.3% .. image:: /content/images/isaac_replicator_composer_samples_6.jpg :width: 16.3% Below diagrams the inputs, outputs, and parameterization of Replicator Composer. .. image:: /content/images/isaac_replicator_composer_arch.png :width: 900px :align: center A dataset parameterization is input from a YAML file containing "key, value" pairs that set parameter names and values. To create random datasets, parameter values can be set to **distributions**, which are sampled from per scene. The input parameter file can point to TXT files called asset lists, which contain lists of elements (colors, model paths, coordinates, etc.) to sample from. To generate a virtual scene using Isaac Sim Replicator, each parameter is sampled from. Then, models, textures, and material files are pulled from the Nucleus Server. Data is captured and output to a dataset directory (`Retrieve the Output`_). Replicator Composer supports the following outputs, with stereo and mono for each output (see :ref:`isaac_sim_app_manual_replicator_composer_output_data_types` for more information): * RGB, Normals * Distance To Camera(Depth), Distance To Image Plane * Instance Segmentation, Semantic Segmentation, Instance Id Segmentation * Bounding Box 2d Tight, Bounding Box 2d Loose, Bounding Box 3d * Occlusion, Motion Vectors Prepare for Generation ---------------------- Replicator Composer can be run from the Omniverse Launcher. Launcher ^^^^^^^^ 1. Create a ```` folder for input and output files. 2. Follow :ref:`isaac_sim_app_install_workstation`. 3. From the :ref:`isaac_sim_setup_native_app_launcher` step, click on "Open in Terminal". 4. From the terminal, follow `Generate Data`_ to start composing. Generate Data ------------- Replicator Composer is a command line tool. Below are steps to generate data from sample inputs or custom inputs. Check `Command Line Reference`_ for additional usage information. Sample Input Parameterizations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Generate datasets from provided inputs. The ``--headless`` argument can be removed if running from Launcher. 1. Generate a realistic dataset in a warehouse. .. code-block:: console $ ./python.sh tools/composer/src/main.py \ --input parameters/warehouse.yaml --output */datasets/warehouse \ --num-scenes 10 --headless --mount .. image:: /content/images/isaac_replicator_composer_warehouse_1.jpg :width: 24.6% .. image:: /content/images/isaac_replicator_composer_warehouse_2.jpg :width: 24.6% .. image:: /content/images/isaac_replicator_composer_warehouse_3.jpg :width: 24.6% .. image:: /content/images/isaac_replicator_composer_warehouse_4.jpg :width: 24.6% 2. Generate a challenging and chaotic domain-randomized dataset inspired by the `FlyingThings3D `_ synthetic dataset. .. code-block:: console $ ./python.sh tools/composer/src/main.py \ --input parameters/flying_things_3d.yaml --output */datasets/flying_things_3d \ --num-scenes 10 --headless --mount .. image:: /content/images/isaac_replicator_composer_flying_things_3d_1.jpg :width: 24.6% .. image:: /content/images/isaac_replicator_composer_flying_things_3d_2.jpg :width: 24.6% .. image:: /content/images/isaac_replicator_composer_flying_things_3d_3.jpg :width: 24.6% .. image:: /content/images/isaac_replicator_composer_flying_things_3d_4.jpg :width: 24.6% 3. Generate a sequential "4D" version of the FlyingThings3D dataset with random object motion. .. code-block:: console $ ./python.sh tools/composer/src/main.py \ --input parameters/flying_things_4d.yaml --output */datasets/flying_things_4d \ --num-scenes 3 --headless --mount .. image:: /content/images/isaac_replicator_composer_flying_things_4d_1.gif :width: 24.6% .. image:: /content/images/isaac_replicator_composer_flying_things_4d_2.gif :width: 24.6% .. image:: /content/images/isaac_replicator_composer_flying_things_4d_3.gif :width: 24.6% .. image:: /content/images/isaac_replicator_composer_flying_things_4d_4.gif :width: 24.6% Custom Input Parameterization ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Generate a dataset from a custom input. Copy Provided Input Files ************************* | In the source code for Isaac Sim, there are baked-in input files. Several are used in the sample input parameterizations. | Accessing these inputs files will help template new custom inputs. 1. Copy input files from source code. Run the cmds in the opened terminal. .. code-block:: console $ cp -r tools/composer/parameters $ cp -r tools/composer/assets Create an Input Parameter File ****************************** An Input Parameter file is a YAML file that parameterizes the dataset. 1. Follow :ref:`isaac_sim_app_manual_replicator_composer_parameter_list` and :ref:`isaac_sim_app_manual_replicator_composer` pages to create ````. 2. If needed, review and/or template from the sample parameter files in ``/parameters``. 3. Move ```` into ``/parameters``. Create Asset List Files *********************** An Asset List file is a TXT file containing a list of elements (e.g. colors, object model paths, camera coordinates etc.) that can be pointed to by parameters in the Input Parameter file. Note, if the provided Assets Lists already suffice the custom input parameter file, new Asset Lists are not needed. 1. Follow the steps in :ref:`Asset Lists `. 2. If needed, review and/or template from the sample Assets Lists in ``/assets``. 3. Move new Asset List files into ``/assets``. Generate Custom Dataset *********************** Compose a dataset with custom inputs. .. code-block:: console $ ./python.sh tools/composer/src/main.py \ --input */parameters/ \ --output */datasets/ \ --num-scenes \ --headless \ --mount Retrieve the Output ------------------- Dataset and other files will be output to the output directory (given by the **output_dir** parameter, which can be set by the ``--output`` argument). Example output file tree (with 1 data sample, `bounding_box_2d_tight`, `rgb`, `instance_segmentation` and `semantic_segmentation` annotators enabled.): .. code:: ├── data │ ├── left │ │ ├── bounding_box_2d_tight │ │ │ ├── bounding_box_2d_tight_0.npy │ │ │ └── bounding_box_2d_tight_labels_0.json │ │ ├── instance_segmentation │ │ │ ├── instance_segmentation_0.png │ │ │ ├── instance_segmentation_mapping_0.json │ │ │ └── instance_segmentation_semantics_mapping_0.json │ │ ├── rgb │ │ │ └── rgb_0.png │ │ └── semantic_segmentation │ │ ├── semantic_segmentation_0.png │ │ └── semantic_segmentation_labels_0.json │ └── right │ ├── bounding_box_2d_tight │ │ ├── bounding_box_2d_tight_0.npy │ │ └── bounding_box_2d_tight_labels_0.json │ ├── instance_segmentation │ │ ├── instance_segmentation_0.png │ │ ├── instance_segmentation_mapping_0.json │ │ └── instance_segmentation_semantics_mapping_0.json │ ├── rgb │ │ └── rgb_0.png │ └── semantic_segmentation │ ├── semantic_segmentation_0.png │ └── semantic_segmentation_labels_0.json └── metadata.txt Visualize the Models -------------------- Running in Visualize Models mode (enabled with the ``--visualize-models`` argument) will output visuals (rendered in Isaac Sim) of each object model defined in the input parameter file with a non-zero **obj_count**, instead of outputting a dataset. The Visualize Models mode is useful for curating the models defined in a parameterization before generating a dataset. Each visual includes four RGBs. Visuals are output to ``/visualize_models/.png``. Generate visuals. .. code-block:: console $ ./python.sh tools/composer/src/main.py \ --input parameters/warehouse.yaml --output */datasets/warehouse \ --headless --mount --visualize-models .. image:: /content/images/isaac_replicator_composer_visual_1.png :width: 49.7% .. image:: /content/images/isaac_replicator_composer_visual_3.png :width: 49.7% .. image:: /content/images/isaac_replicator_composer_visual_2.png :width: 49.7% .. image:: /content/images/isaac_replicator_composer_visual_4.png :width: 49.7% .. _isaac_sim_app_tutorial_replicator_composer_command_line_reference: Command Line Reference ---------------------- The following describes command-line options. .. code-block:: console $ ./python.sh tools/composer/src/main.py -h usage: main.py [-h] [--input INPUT] [--visualize-models] [--mount MOUNT] [--headless] [--nap] [--overwrite] [--output OUTPUT] [--num-scenes NUM_SCENES] [--nucleus-server NUCLEUS_SERVER] optional arguments: -h, --help show this help message and exit --input INPUT Path to input parameter file. --mount MOUNT Path to mount symbolized in parameter files via '*'. --headless Will not launch Isaac SIM window. --nap Will nap Isaac SIM after the first scene is generated. --overwrite Overwrites dataset in output directory. --output OUTPUT Output directory. Overrides 'output_dir' param. --num-scenes NUM_SCENES, --num_scenes NUM_SCENES Num scenes in dataset. Overrides 'num_scenes' param. --nucleus-server NUCLEUS_SERVER, --nucleus_server NUCLEUS_SERVER Nucleus Server URL. Overrides 'nucleus_server' param. ---------------------------------- **input** | The path to the input parameter file. Follows :ref:`Paths ` rules. ---------------------------------- .. **visualize-models** .. See the `Visualize the Models`_ section. ---------------------------------- **mount** See **2. Mounted Paths** in :ref:`Paths `. ---------------------------------- **headless** If given, Replicator Composer will run without launching the Isaac Sim client. ---------------------------------- **nap** | If given, Replicator Composer will nap after the first scene is generated. | If set without **headless** and run in Launcher, allows the user to examine or debug the first scene in the Isaac Sim client. ---------------------------------- **overwrite** | If given, generation will always start from the 0 index and overwrite any files in the **output_dir**. | If not given, if file is existed in the **output_dir**, it will keeps the old file. Otherwise, it will writes new data to the **output_dir**. ---------------------------------- **output** | If given, overwrites the **output_dir** parameter. Follows :ref:`Paths ` rules. ---------------------------------- **num-scenes** If given, overwrites the **num_scenes** parameter. ---------------------------------- **nucleus-server** If given, overwrites the **nucleus_server** parameter. ---------------------------------- UI ---------------------- Running the script from the UI can be done by enabling the Composer Window from `Replicator->Composer`. The UI enables loading, visualizing, and recording the scene using the editor. The aforementioned paths can be set in the `Scene` frame and the scene can be loaded using the `Load` button. The generated graph can then be run/stopped/paused/resumed/previewed using the buttons in the `Control` frame. .. figure:: /content/images/isaac_replicator_composer_ui.png :align: center :alt: Composer UI