Is this page helpful?

Randomizing appearance, placement and orientation of an existing 3D assets with a built-in writer#

Learning Objectives#

Let’s say we want to create a DNN model that is able to identify an object within a large set of environments. In order to create such a model a training dataset would be needed where the target object is placed in a variety of different ways in a variety of different environments. In some cases this target object may not be completely visible and could be occluded by other assets in the scene. In this example, we would walk through the process of synthetically creating this dataset. The intent of this tutorial is to show an end-to-end example of a synthetic data generation process where a target object is placed in randomized environments. We end this tutorial with a writer that outputs annotated data in a form that can be used to train a machine learning model.

For this example we use Toy Jensen as our target object that we place in randomized environments along with other assets. Assets placements on the scene and rotations are selected from uniform random distributions. In addition to randomizing assets in this example we also demonstrate camera placement randomization to generate images from different angles to capture different backgrounds. Finally, images and annotations are written to the disk using a predefined writer.

Using the Replicator APIs#

To run the following script follow set up instructions in Getting started with Omniverse Replicator.

Let’s import omni.replicator APIs first. We also define paths for 3D assets used in this example.

import omni.replicator.core as rep

# Define paths for Base and Toy Jensen (TJ) assets used in this example

BASE = 'omniverse://ov-content/Users/jlafleche@nvidia.com/Where_is_TJ/'

TJ = 'omniverse://ov-content/Users/ecameracci@nvidia.com/Toy_Jensen_Copy/Collected_ToyJensen/ToyJensen.usd'

In the following function we define a randomizer for Base assets. As a first step, we create a subset (size = 50) of these Base assets. We want to use different Base assets in different frames to ensure that we have enough variety of background generated for this dataset. Once this subset is selected the next step is to place these Base assets in random positions with random rotations.

def env_props(size=50):
    instances = rep.randomizer.instantiate(rep.utils.get_usd_files(BASE + 'props'), size=size, mode='point_instance')
        with instances:
            rep.modify.pose(
                position=rep.distribution.uniform((-500, 0, -500), (500, 0, 500)),
                rotation=rep.distribution.uniform((0,-180, 0), (0, 180, 0)),)

    return instances.node

Next we define a randomizer for our target object - Toy Jensen (TJ). As with the other Base assets we also randomize the position and rotation of the Toy Jensen asset. In addition, we also randomize material on the TJ’s shirt and pants to give a different look in different frames.

def toy_jensen():
    tj = rep.create.from_usd(BASE + "TJ/TJ.usd", semantics=[('class', 'toy_jensen')])

    with tj:
        with rep.get.prims("shirt|pants", prim_types=["Mesh"]):
            rep.randomizer.materials(rep.get.prims("Toy_JHH_MDL_Ver", prim_types=["Material"]))
        rep.modify.pose(
            position=rep.distribution.uniform((-500, 0, -500), (500, 0, 500)),
            rotation=rep.distribution.uniform((0,-45, 0), (0, 45, 0)),)

    return tj

In the following lines, we register the two randomizers we defined above.

rep.randomizer.register(env_props)

rep.randomizer.register(toy_jensen)

Next we set up our static elements of the scene, room and the table on which the assets are going to be placed. We also isetup camera and attach to a render product.

# Setup the static elements
env = rep.create.from_usd(BASE + 'env/room.usd')

table = rep.create.from_usd(BASE + 'env/table.usd')

# Setup camera and attach it to render product
 camera = rep.create.camera (
    focus_distance=800,
    f_stop=0.5)

render_product = rep.create.render_product(camera, resolution=(1024, 1024))

As a last step we initialize our default writer and define an output directory where we want to store our images. This writer is then attached to the render product we defined in the previous step.

# Initialize and attach writer

writer = rep.WriterRegistry.get("OmniWriter")

writer.initialize(output_dir="_output", rgb=True, bounding_box_2d_tight=True)

writer.attach([render_product])

Until now, we have defined our target and base assets, have defined randomizers, have defined our environment, have set up camera, render product and a default writer. After all the setup is complete, we are not ready to generate data. In the following lines, we trigger our registered randomizers every frame. For every frame Base and target assets are randomized along with the camera position to ensure we have images captured from different angles for our dataset.

with rep.trigger.on_frame(num_frames=10000):
    rep.randomizer.env_props(75)

    rep.randomizer.toy_jensen()

    with camera:
        rep.modify.pose(position=rep.distribution.uniform((-500, 200, 1000), (500, 500, 1500)), look_at=table)

Below is the complete code that can be copied over and tested for convenience.

import omni.replicator.core as rep

# Define paths for Base and Toy Jensen (TJ) assets used in this example

BASE = 'omniverse://ov-content/Users/jlafleche@nvidia.com/Where_is_TJ/'

TJ = 'omniverse://ov-content/Users/ecameracci@nvidia.com/Toy_Jensen_Copy/Collected_ToyJensen/ToyJensen.usd'

# Define randomizer function for Base assets. This randomization includes placement and rotation of the assets on the table.

def env_props(size=50):
    instances = rep.randomizer.instantiate(rep.utils.get_usd_files(BASE + 'props'), size=size, mode='point_instance')
        with instances:
            rep.modify.pose(
                position=rep.distribution.uniform((-500, 0, -500), (500, 0, 500)),
                rotation=rep.distribution.uniform((0,-180, 0), (0, 180, 0)),)

    return instances.node

def toy_jensen():
    tj = rep.create.from_usd(BASE + "TJ/TJ.usd", semantics=[('class', 'toy_jensen')])

    with tj:
        with rep.get.prims("shirt|pants", prim_types=["Mesh"]):
            rep.randomizer.materials(rep.get.prims("Toy_JHH_MDL_Ver", prim_types=["Material"]))
        rep.modify.pose(
            position=rep.distribution.uniform((-500, 0, -500), (500, 0, 500)),
            rotation=rep.distribution.uniform((0,-45, 0), (0, 45, 0)),)

    return tj

# Register randomization

rep.randomizer.register(env_props)

rep.randomizer.register(toy_jensen)

# Setup the static elements

# Setup the static elements
env = rep.create.from_usd(BASE + 'env/room.usd')

table = rep.create.from_usd(BASE + 'env/table.usd')

# Setup camera and attach it to render product
 camera = rep.create.camera (
    focus_distance=800,
    f_stop=0.5)

render_product = rep.create.render_product(camera, resolution=(1024, 1024))

# Initialize and attach writer
writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(output_dir="_output", rgb=True, bounding_box_2d_tight=True)
writer.attach([render_product])


with rep.trigger.on_frame(num_frames=10000):
    rep.randomizer.env_props(75)

    rep.randomizer.toy_jensen()

    with camera:
        rep.modify.pose(position=rep.distribution.uniform((-500, 200, 1000), (500, 500, 1500)), look_at=table)