Practical Guides

Note

please ensure farm agent and farm queue are installed, set up correctly and are running before continuing.

Rendering with Farm

Omniverse Create comes bundled with the ability to submit render jobs to Queue. In omniverse, setting up a cluster for distributed rendering has been integrated and is relatively easy to use.

In this example, we will walk through the needed steps to distribute your renders across your local network on spare machines. To begin, let’s create a (render) task that an Agent will pick up and process.

What you will need…. All items available for download in the Omniverse Launcher.

  • Omniverse Create

    • Or other Omniverse Kit based App with Movie Capture enabled.

  • Omniverse Farm

  • Omniverse Queue

Send a Render to the Queue

  1. In Omniverse Create open a scene with content you want to render. File > Open

  2. At the top of the screen and select Rendering > Movie Capture.

  3. Within the Movie Capture Panel select the scene you wish to render.

    Note

    It is important to be sure the scene you wish to render and the assets it contains are stored in a location that is accessible to all farm agents. Typically a shared drive or a common nucleus server address will work best.

  4. In the Settings section of the Movie Capture Panel, select localhost Queue or a discovered network queue depending queue on configuration.

  5. Optionally, add a description to the task so you can more easily identify it later. This can be useful task recognition when running many tasks or for historical context after running the queue.

    Submitting a render to the Queue from Omniverse Create
  6. Review and ensure all render settings and path locations are filled in according to your preference.

  7. Ensure the Output : Path is set to a desirable location.

  8. Submit the task to Queue by clicking the Submit to Queue button at the bottom of the Movie Capture Panel.

Once submitted, the Queue dashboard should automatically update to show that the task has been created, and that its status is set to Submitted while waiting for an Agent to pick it up.

Omniverse Task Dashboard

Add Agents to execute on the Render Task

Agents are responsible for executing on tasks in Omniverse Farm. Therefore in order to render our task, we will need at least one but can comprise of a multitude of agents.

Requirements for an Agent

  • Access to any network location requested by the Farm Queue

  • Network Access to the Farm Queue Host

  • Omniverse Agent

To begin, agent must be launched

  1. Upon starting, you will be presented with a UI allowing specification of the URL to your Queue.

  2. For convenience, a default URL is provided. This url assumes the queue and agent are both operating on the localhost. If you are not running agent on the same machine as queue, provide its hostname or IP address in the text field and click the Test connection to confirm the Agent is indeed able to reach the Queue.

  3. After clicking the Connect button, Agent will retrieve the list of pending tasks from Queue and begin executing.

  4. Reviewing the Queue’s dashboard after a few seconds should then reflect the new state of the system, where:

    • An Agent is now available, since it was launched from the Omniverse Launcher

    • The task previously submitted has transitioned to the Running state, illustrating that processing is currently underway.

    • The Agent’s status is Active, and the ID of its active task matches the one submitted earlier.

    Omniverse Task Dashboard

Once the task has been completed

  • Agent will return to an Idle state

  • Task status will change to Finished.

In the Movie Capture Panel, select the folder icon alongside the Output : Path input field will link you to the selected path allowing you to review the results of the render.

Custom Decimation Task

Omniverse farm is more than just a render queue. In fact it can process virtually anything within omniverse or without. In this example, we will define a custom job and we will use Blender to decimate the meshes.

Example: Using Blender to decimate meshes

Now that we’ve learned how to execute built-in tasks, let’s see how to define and run our own custom ones.

For the purposes of this example, we will be using Omniverse Agent to automate a mesh simplification task in Blender.

You can get Blender free from the Blender website or as an App Download in the Omniverse Launcher ../.

An illustration of typical batch operations you may find in studio or enterprise scenarios. The task will:

  1. Open an OBJ file

  2. Decimate its meshes to a given ratio

  3. Export the scene as a USD asset

While this is a simple scenario, it does demonstrates important notions:

  • No task is too small to benefit from automation.

  • Creating custom job types for Omniverse Agent is a fairly straightforward process.

  • USD is the open-source pillar of the 3D world to facilitate data exchange and interoperability. We are sure you will love working with it just as much as we do.

Blender mesh decimation script

Let’s start where most automation tasks begin: by running our task locally.

Below is our Python script which will be driving the work to do in Blender. It supports the following arguments:

  • --source: The absolute path to the OBJ file to decimate.

  • --destination: The absolute path where the resulting USD file will be saved.

  • --ratio: An optional parameter between 0.0 and 1.0, specifying the amount by which meshes should be simplified.

job.blender-decimate.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
# File: job.blender-decimate.py

import argparse
import sys
from typing import List

import bpy


class BlenderArgumentParser(argparse.ArgumentParser):
    """Argument parser for Python scripts running in Blender."""

    def _get_arguments_after_double_dash(self) -> List[str]:
        """
        Extract any arguments intended to be passed to Blender scripts.

        This uses the fact that Blender will ignore any command-line arguments provided after the `--` command. This
        effectively extracts any arguments following `--`, and forwards them to the default `argparse.parse_args()`
        implementation.

        Args:
            None

        Returns:
            List[str]: The command-line arguments provided to the Blender executable, and intended for the Python script.
        """
        try:
            double_dash_index = sys.argv.index("--")
            return sys.argv[double_dash_index + 1:]
        except ValueError:  # Case where no arguments were provided after `--`.
            return []

    def parse_args(self) -> argparse.Namespace:
        return super().parse_args(args=self._get_arguments_after_double_dash())


def main() -> None:
    """Decimation script for Blender."""

    # Parse the arguments provided to the script:
    parser = BlenderArgumentParser()
    parser.add_argument("--source", required=True, help="Path of the source file to decimate.")
    parser.add_argument("--destination", required=True, help="Path of the destination file to save.")
    parser.add_argument("--ratio", type=float, default=0.5, help="Mesh decimation ratio (between 0.0 and 1.0).")
    args = parser.parse_args()

    # Clear the Blender scene, effectively removing the default Cube, Light and Camera:
    bpy.ops.wm.read_factory_settings(use_empty=True)

    # Import the given OBJ file into the scene:
    bpy.ops.import_scene.obj(filepath=args.source)

    # Decimate the meshes of the scene using the provided decimation ratio.
    #
    # The decimation ratio represented the ratio of faces to keep from the original mesh:
    #   * At 1.0, the mesh remains unchanged.
    #   * At 0.5, edges have been collapsed in such a way that half the number of faces are retained.
    #   * At 0.0, all faces have been removed.
    for object in bpy.data.objects:
        if object.type == "MESH":
            modifier = object.modifiers.new(name="DecimateModifier", type="DECIMATE")
            modifier.ratio = args.ratio

    # Export the scene to the given USD file:
    bpy.ops.wm.usd_export(filepath=args.destination)


if __name__ == "__main__":
    main()

Testing our mesh decimation script

Following best practices, this task can be executed locally from the command-line. This makes it possible to develop, validate and debug it to allow fast iteration cycles, and also promotes reusability of components.

Before integrating our task, let’s first make sure it runs as expected in standalone mode by running it from the command-line:

1
2
3
4
5
6
7
./blender \
    --background \
    --python path/to/job.blender-decimate.py \
    -- \
    --source path/to/source.obj \
    --destination path/to/destination.usd \
    --ratio 0.5

Using the mandatory Suzanne monkey as an OBJ source file, we should see the following result, with the original model on the left and its simplified version on the right:

Mesh decimation comparison

Integrating the job with Omniverse Agent and Queue

Now that we know that our task works as expected, we will integrate it in Omniverse Agent so anyone can submit tasks and benefit from our work.

This can be done either by code, or by using a more User-friendly wizard that will assist in the creation of a task. Since we have already done a fair share of code for the day, let’s use the wizard to make the rest of the day easier:

  1. From the Omniverse Launcher, launch an instance of Agent.

  2. From the UI, under the Agent settings section, click the Configure or edit job definitions button.

Clicking the button will direct you to the Job Definition Wizard, from where you can either create new job definitions that Agents can execute, or edit any previously-saved job:

Job definition wizard

Let’s follow along the steps, and provide information about the job we want to create:

  • Job name: blender-decimate

  • Job type: Command or executable, as we will be launching the blender executable. For other use cases, we could also select Kit-based application or Kit-based service to select a Kit extension as a driver for the process.

  • Command: blender${exe_ext}, for which we can supply either the full path to Blender’s executable on the Agent, or the name of any command to execute.

  • Process arguments: The list of arguments to supply to the Blender command, one per line. These are the values common to all executions of this job.

  • Allowed arguments: The parameters to supply to the job, which might differ from one execution to another. This can include a definition for default values, in case one is not supplied.

And click Save!

Let’s now complete our project by submitting a task:

1
2
3
4
curl -X POST "http://localhost:8222/queue/management/tasks/submit" \
    --header "Accept: application/json" \
    --header "Content-Type: application/json" \
    --data '{"user":"my-user-id","task_type":"blender-decimate","task_args":{"source":"path/to/suzanne.obj","destination":"path/to/suzanne-decimated.usd","ratio":0.2},"task_comment":"My first decimation job!"}'

The only thing left to do now is to wait a few seconds for an Agent to pick up the task!

Note

Since jobs are defined in Kit files using the TOML syntax, the wizard will produce a schema similar to the following, which Agents will use to launch and process our task. Should you want to share or distribute your job to multiple machines, all you would have left to do would be to place it in the same location on other Agents of your pool.

job.blender-decimate.kit
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# File: job.blender-decimate.kit

[package]
title = "Blender decimation task"
description = "Decimate a mesh using Blender and export it as a USD asset"
version = "1.0.0"
category = "jobs"
authors = ["Omniverse Team"]
readme = "Omniverse decimation task"
keywords = ["job"]

# Schema for the Blender mesh decimation task.
[job.blender-decimate]
job_type = "base"
name = "blender-decimate"
log_to_stdout = true # Capture information from `stdout` and `stderr` for the task's logs
# Since the `command` property supports standard Kit token resolution, we're using `${exe_ext}` so the executable resolves to `blender.exe` on Windows and
# `blender` on Linux. For the environment where Agent will be running, we have the choice of either:
#   * Making sure the `blender` executable can be found in the `PATH` or `HOME` environment variable.
#   * Providing the full path to the Blender executable.
command = "blender${exe_ext}"
# The command-line arguments listed in the `args` array will be supplied to the `command` for each task of this type. This has 2 main benefits:
#    1. It prevents having to provide these arguments for each task.
#    2. Makes it possible to seamlessly roll out updates to the "job.blender-decimate.py" script on the Agent, without the need for clients to also
#       update their APIs. For example, Agents could update to a "job.blender-decimate.v2.py" and clients could still submit tasks using the same
#       `allowed_args` listed below.
args = [
    "--background",
    "--python", "path/to/job.blender-decimate.py",
    "--", # In the case of Blender, this additional `--` token is required to supply arguments to the Python script itself.
]

# The `allowed_args` are appended to the "static" `args` listed above for each task. Effectively, this means that the resulting command-line executed by
# Agents takes the form of:
#    $ <command> <args> <allowed_args>
#
# In the case of our example, this resolves to:
#    $ blender \
#       --background --python path/to/job.blender-decimate.py -- \
#       --source <source_obj> --destination <destination_usd> --ratio <ratio=0.5>
#
# Each `allowed_args` supports 2 properties:
#    * `arg`: The argument token as it will appear on the command-line.
#    * `default`: A default value for the argument, if none is supplied by clients.
#
# Note that `allowed_args` defines the explicit list of arguments which can be submitted by clients, and that any additional arbitrary arguments will be
# ignored. For security reasons, you would not want external clients to be able to supply extra arguments, which could otherwise allow arbitrary external
# code execution. In our example, this prevents Users from submitting tasks to Blender with the `--python-expr <script>` argument, which executes the
# provided text string as a Python script.
[job.blender-decimate.allowed_args]
source      = { arg = "--source",      default = "" }
destination = { arg = "--destination", default = "" }
ratio       = { arg = "--ratio",       default = "0.5" }

Where to go from here

To make this feature accessible to others, we could now bring it to the next level by:

  • Writing a script or plugin to submit tasks directly from DCC applications making it possible for artists to submit tasks easily on asset revision.

  • Alternatively, we could also submit tasks directly from our production tracking tools delivering automatic system submissions of tasks once they have been reviewed and approved.

Note

You may have noticed that it might be more performant for Agents to execute multiple tasks at the same time. Since an Agent is already launched, it might be more efficient to have it process a number of conversions at the same time, rather than only process a single file.

A few ideas come to mind on this topic:

  • Perhaps we could edit our script to support supplying the path to a source folder to find and process all OBJ files, instead of a single file.

  • Perhaps we could even provide our script with a glob wildcard pattern to find source files of multiple formats, and export them all as USD.

Conclusion

With this ability to scale, distribute and share tasks across multiple environments, there are no limits to what we could do automatically or by manual submission:

  • Render turntable-style preview of assets once they are submitted to the production tracking system.

  • Making sure all resources are checked in, and that no texture files are missing from files before .

  • Validating that asset nomenclature conforms to studio requirements.

  • Updating fluids or animation caches.

  • Training machine learning on new datasets.

  • etc.

Want to showcase your creations, or see what others from the community have created? Join the Showcase section of our forums to join the conversation!