.. _isaac_sim_app_tutorial_replicator_recorder:

==========================================
Synthetic Data Recorder
==========================================

Learning Objectives
=======================

This tutorial introduces the Synthetic Data Recorder for |isaac-sim|, a GUI extension for recording synthetic data 
with the possibility of using custom `Replicator writers <../../prod_extensions/prod_extensions/ext_replicator/custom_writer.html>`_ to record the data in any custom format.

*15-20 Minute Tutorial*

Getting Started
========================

The tutorial uses the following stage as an example:

``Isaac/Samples/Replicator/Stage/full_warehouse_worker_and_anim_cameras.usd``

.. figure:: /content/images/isaac_synth-data_recorder_stage.png
    :align: center
    :alt: Synthetic Data Recorder

The example stage comes preloaded with semantic annotations, as well as multiple cameras, a few of them being animated to move around the scene when running the simulation. To create custom camera movement animations take a look at the `Camera Animation Tutorial <../../prod_extensions/prod_extensions/ext_animation-timeline.html>`_.

.. Note:: When using other scenes make sure to add `semantic annotations <../../prod_extensions/prod_extensions/ext_replicator/semantics_schema_editor.html>`_ to the scene otherwise most annotators (semantic_segmentation, 3d_bounding_box, etc.) will not be able to produce data.

Basic Usage
========================

The recorder is split into two main parts: the ``Writer`` frame - containing sensor, data, and output parameters; and the ``Control`` frame - containing the recording functionalities such as start, stop, pause, and parameters such as the number of frames to execute.

.. figure:: /content/images/isaac_synth-data_recorder_window.png
    :align: center
    :alt: Synthetic Data Recorder Window


Writer Parameters
^^^^^^^^^^^^^^^^^^^^

The **Render Products** frame creates a list of render product entries using the ``Add New Render Product`` button. By default a new entry is added to the list using the active viewport camera as its camera path (see left figure). If however cameras are selected in the stage viewer, these are added to the render products list (see right figure).
The render products list can contain the same camera path multiple times, however each time with a different resolution. All the entry values (camera path or resolution) can also be manually edited in the input fields.

.. figure:: /content/images/isaac_synth-data_recorder_rp.png
    :align: center
    :alt: Synthetic Data Recorder Render Products


The **Parameters** frame gives the possibility to choose between the default built-in Replicator writer (`BasicWriter`) or to choose a custom writer. The default writer parameters (mostly annotators) can be selected from the checkbox list. As custom writers have unknown parameters, these can be provided by the user in form of a json file containing all the required parameters. The path to the json file can be added in the ``Parameters Path`` input field.

.. figure:: /content/images/isaac_synth-data_recorder_writer_params.png
    :align: center
    :alt: Synthetic Data Recorder Parameters

The **Output** frame (left figure) contains the working directory path where the data will be saved together with the folder name used for the current recording. By selecting between ``Overwrite``, ``Increment``, or ``Timestamp`` the output folder name can be kept the same or modified to avoid overwriting previous recordings. The ``Overwrite`` option will maintain the output path and write over any existing data. The ``Increment`` option will start by appending `_0` to the folder name and increment it by 1 for each recording. The ``Timestamp`` option will append a timestamp (`_YYYY-mm-dd-HH-MM-SS`) to the folder name. The recorder can also write to S3 buckets by checking ``Use S3`` and providing the required fields and having the AWS credentials set up.

.. Note:: When writing to S3 the ``Increment`` folder naming is not supported and will default to ``Timestamp``.

The **Config** frame (right figure) can load and save the GUI writer state as a json config file. By default the extension loads the previously used configuration state.

.. figure:: /content/images/isaac_synth-data_recorder_writer_out_config.png
    :align: center
    :alt: Synthetic Data Recorder Output and Config


Control 
========================

The **Control** frame contains the recording functionalities such as Start/Stop and Pause/Resume, and parameters such as the number of frames to record, or the number of subframes to render for each recorded frame. The ``Start`` button will create a writer given the selected parameters and start the recording. The ``Stop`` button will stop the recording and clear the writer. The ``Pause`` button will pause the recording without clearing the writer, and the ``Resume`` button will resume the recording. The ``Number of frames`` input field will set the number of frames to record, after which the recorder will be stopped and the writer cleared. If the value is set to 0, the recording will run indefinitely until the ``Stop`` button is pressed. The ``RTSubframes`` field will set the number of additional subframes to render for each per frame. This can be used if randomized materials are not loaded in time or if temporal rendering artifacts (such as ghosting) are present due to objects being teleported. The ``Control Timeline`` checkbox will start/stop/pause/resume the timeline together with the recorder, after each recording the timeline is moved to timestamp `0`.

.. figure:: /content/images/isaac_synth-data_recorder_control.png
    :align: center
    :alt: Synthetic Data Recorder Output and Config
    :width: 60%

.. Note:: In the current version, Replicator and Isaac Sim are both interacting with the timeline, and in some cases they can cause unwanted behavior. Some known issues are:
    
    * when running Replicator, the timeline's ``loop`` mode is ignored
    * when using the Recorder the timeline should be controlled either through the ``Control Timeline`` flag or through the Isaac Sim GUI, not both

Custom Writer Example
========================

In order to support custom data formats custom writer can be registered and loaded from the GUI. In this example, we register a custom writer called ``MyCustomWriter`` using the ``Script Editor`` and use it with the recorder. 

.. raw:: html

    <details open>
    <summary>MyCustomWriter</summary>

.. code-block:: python
    :linenos:

    from omni.replicator.core import AnnotatorRegistry, BackendDispatch, Writer, WriterRegistry
        
    class MyCustomWriter(Writer):
        def __init__(
            self,
            output_dir,
            rgb = False,
            semantic_segmentation = False,
        ):
            self.version = "0.0.1"
            self.backend = BackendDispatch({"paths": {"out_dir": output_dir}})
            if rgb:
                self.annotators.append(AnnotatorRegistry.get_annotator("rgb"))
            if semantic_segmentation:
                self.annotators.append(AnnotatorRegistry.get_annotator("semantic_segmentation"))            
            self._frame_id = 0

        def write(self, data):
            if "rgb" in data:
                filename = f"rgb_{self._frame_id}.png"
                self.backend.write_image(filename, data["rgb"])
            if "semantic_segmentation" in data:
                filename = f"semantic_segmentation_{self._frame_id}.png"
                self.backend.write_image(filename, data["semantic_segmentation"])
            print(f"Frame {self._frame_id} written to {self.backend.output_dir}")
            self._frame_id += 1

        def on_final_frame(self):
            print(f"Final frame {self._frame_id} reached")
            self._frame_id = 0

    WriterRegistry.register(MyCustomWriter)

.. raw:: html

    <details open>
    <summary>my_params.json</summary>

.. code-block:: json
    :linenos:

    {
        "rgb": true
    }


.. figure:: /content/images/isaac_synth-data_recorder_custom_writer.png
    :align: center
    :alt: Synthetic Data Recorder Custom Writer


Replicator Randomized Cameras
==============================

In order to take advantage of Replicator randomization techniques, these can be loaded using the Script Editor before starting the recorder to run scene randomizations during recording. In this example we create a randomized camera using Replicator API. This can be attached as a render product to the recorder and for each frame the camera will be randomized with the given parameters.

.. raw:: html

    <details open>
    <summary>Randomized Camera</summary>

.. code-block:: python
    :linenos:

    import omni.replicator.core as rep

    with rep.new_layer():
        camera = rep.create.camera()
        with rep.trigger.on_frame():
            with camera:
                rep.modify.pose(
                    position=rep.distribution.uniform((-5, 5, 1), (-1, 15, 5)),
                    look_at="/Root/SM_CardBoxA_3",
                )

.. figure:: /content/images/isaac_synth-data_recorder_replicator_camera.png
    :align: center
    :alt: Synthetic Data Recorder Custom Writer


The following figure shows the steps by instructions on setting up a Replicator randomization and running it with the recorder.


.. figure:: /content/images/isaac_synth-data_recorder_replicator_camera.gif
    :align: center
    :alt: Synthetic Data Recorder Custom Writer Steps