Creating Job Definitions

About

When creating a new job to be distributed on Omniverse Farm, one of the first steps you may wish to take is creating a job definition for it.

Omniverse Farm job definitions act as the point of entry for the work to be executed, and provide information about the requirements and dependencies necessary for their operations. Using this information, the services bundled in Farm Agents are then able to select the next task it can execute when querying the Farm Queue about awaiting tasks.

In the following section, we will look at job definitions in greater details, so you will have the information you need to start creating your own distributed jobs, whether they are implemented as:

Job definition schema: System Executables

Job definitions are nothing more than KIT files you should already be familiar with if you have already created an extension for an Omniverse application. If you have not yet had the opportunity to get acquainted with the development of extensions, you may be interested in looking at some of the resources available on that topic to get started.

Let’s start with a simple example printing a mandatory “Hello Omniverse!” message, in order to provide an overview of what we will be describing in greater details:

minimal-job-definition.kit
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Standard KIT metadata about the package for the job, providing information about what the feature accomplishes so
# it can be made visible to Users in Omniverse applications:
[package]
title = "Minimal Omniverse Farm job definition"
description = "A simple job definition for an Omniverse Farm job, printing a welcoming message."
category = "jobs"
version = "1.0.0"
authors = ["Omniverse Team"]
keywords = ["job"]

# Schema for the job definition of a system command or executable:
[job.hello-omniverse]
# Type of the job. Using "base" makes it possible to run executable files:
job_type = "base"
# User-friendly display name for the job:
name = "simple-hello-omniverse-job"
# The command or application that will be executed by the job:
command = "echo"
# Arguments to supply to the command specified above:
args = ["Hello Omniverse!"]
# Capture information from `stdout` and `stderr` for the job's logs:
log_to_stdout = true

As you may have noticed, we have included some comments and annotations in the file.

As a best practice, we encourage you to provide documentation in the job definition, as it acts as the entry point for the work that will be executed. This not only makes it easier to maintain your work over time, but also makes it easier to share it with others so they can reuse the service you created, and build even larger workflows thanks to the fruit of your labor.

Why use a KIT file for job definitions?

You may be wondering why use KIT files to define jobs, if JSON or YAML could have supported similar features.

The main motivation is that KIT files offer an number of additional features offered to Omniverse applications, such as token expansion and platform-specific options. Additionally, we strongly believe that creating services for Omniverse Farms should not be a different experience from the standard development of extensions and features for Omniverse applications, allowing authors to reuse the work they have already done on the platform to scale it to multiply its efficiency.

Another driving factor for using this format is that it allows you to package your job, along with its dependencies, so its bundle can be hosted in a location accessible to Farm Agents, or shared for others to reuse thanks to the built-in capabilities of KIT files.

Example: Token expansion

As concrete examples of the benefit of using the familiar KIT file format, is that the command property of the job definition schema supports standard token resolution supported by Omniverse applications. This means we could be using the my-app${exe_ext} token so executable file extensions resolve to my-app.exe on Windows and my-app on Linux.

Example: Platform-specific configuration

Another feature of the KIT file format is that is offers flexibility in terms of platform-specific options. On top of the token expansion mentioned previously, the format offers the ability to augment or filter definitions based on a number of features available to the host, making it possible to support Linux or Windows hosts with minimal configuration.

As an illustrative example of this capability, consider the simple job definition we crafted earlier. Using the KIT feature of append elements to a list, we could easily change the greeting message depending on the host where the job will be executed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Continuation from the job definition declared earlier:
[job.hello-omniverse]
# [...]
# Arguments to supply to the `echo` command:
args = ["Hello Omniverse!"]

# Add a Linux-specific item to the list of `args` that will be supplied to the `echo` command:
[job.hello-omniverse.args."filter:platform"."linux-x86_64"]
"++" = [" from Linux"]

# Add a Windows-specific item to the list of `args` that will be supplied to the `echo` command:
[job.hello-omniverse.args."filter:platform"."windows-x86_64"]
"++" = [" from Windows"]

While a simplistic example for demonstration purposes, you could envision using this ability for any platform-specific configuration for your job, such as:

  • Declaring environment variables

  • Enabling/disabling extensions

  • Setting default task arguments

  • etc.

Note

Now that you have a better understanding of the structure of job definitions for Omniverse Farm, head over to the Omniverse Farm examples page for practical guides to using the feature for production.

Job definition schema: Omniverse Services

Now that you know how to define a simple job definition and launch a command on the system, let’s see how to launch an Omniverse application to start building larger workflows.

In this example, we will go one step beyond our earlier example and introduce a few additional properties of the job definition to let you create more complex workflows:

omniverse-application-job-definition.kit
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# Standard KIT metadata about the package for the job, providing information about what the feature accomplishes so
# it can be made visible to Users in Omniverse applications:
[package]
title = "Sample Omniverse job definition"
description = "Sample job definition showcasing how to launch Omniverse applications."
version = "1.0.0"
authors = ["Omniverse Team"]
category = "jobs"
keywords = ["job"]

# Schema for the job definition of an Omniverse application:
[job.sample-omniverse-application-job]
# Type of the job. Using "kit-service" makes it possible to execute services exposed by Omniverse applications:
job_type = "kit-service"
# User-friendly display name for the job:
name = "sample-omniverse-application-job"
# Resolve and launch the "Create" application from the Omniverse Launcher (note the 3 slashes in the URL):
command = "launcher:///create"
# List of command-line arguments to provide to the Omniverse application when launched:
args = [
    # Make sure the Omniverse application can be closed when the processing of the job has completed, and that the
    # notification asking if the USD stage from the active session should be saved prior to closing does not prevent
    # the application from shutting down:
    "--/app/file/ignoreUnsavedOnExit=true",
    # Make sure Omniverse application are active upon launch, and that notification prompt asking for User inputs
    # are not preventing the session from being interactive:
    "--/app/extensions/excluded/0='omni.kit.window.privacy",
    # Add any additional setting required by the Omniverse application, or your own extensions:
    # [...]
]
# Path of the service endpoint where to route arguments in order to start the processing of the job:
task_function = "sample-processing-extension.run"
# Flag indicating whether to execute the Omniverse application in headless mode (i.e. the equivalent of supplying it
# with the `--no-window` command-line option):
headless = true
# Capture information from `stdout` and `stderr` for the job's logs:
log_to_stdout = true

# Supply a list of folders where extensions on which the job depends can be found:
[settings.app.exts.folders]
"++" = [
    "${job}/exts-sample-omniverse-application-job",
    # ...
]

# List of extensions on which the job depends in order to execute:
[dependencies]
"omni.services.farm.agent.runner" = {}
# ...

# When running the job, enable the following extensions:
[settings.app.exts]
enabled = [
    # Extension exposing a "run" endpoint, which will receive the arguments of the task as payload, and start the
    # job process:
    "sample-processing-extension",
    # ...
]

Fundamentally, jobs implemented as Omniverse application services declare a set of extensions which should be enabled by the application, and the path to the endpoint that one of them exposes in order to fulfill the task.

The process should be familiar to you if you have already created an Omniverse extension, as it follows the typical development workflow. For clarity, a few details nonetheless about the example above, where we are:

  1. Providing configuration options to the Omniverse application, so it can launch in a state that will allow it to perform the work it will receive.

  2. Specifying the location of the extension(s) that we expect the Omniverse application to load for us.

  3. Enabling any extension we require from the Omniverse application, along with the one that will act as the entrypoint for incoming requests to kickstart the execution of the task.

This entrypoint extension is expected to expose an endpoint that the location defined by the task_function option of the schema. This endpoint, implemented using the Service stack, will be called by the Agent tasked with performing a job, and that will supply the endpoint with any information it needs in order to execute the work.

Note

For concrete examples of how arguments can be supplied to the endpoint service, head over to the Omniverse Farm examples page.

A few additional notes about the layout of this job definition for Omniverse services:

  • We used Omniverse Create from the Launcher for demonstration purposes in this sample, however you are free to use any application available on the Launcher by supplying its unique identifier to the command property of the job definition. For example, you could be using isaac_sim to target workflows based on Isaac Sim.

  • For convenience, the headless flag can be used during development as a way of inspecting the operations performed by the service, in order see the progress of the operations performed. Once deployed in a production context, running the application in headless mode make it both more performant and easier to scale, as batch workflows typically do not require a user interface to perform actions, and thus makes an entire desktop environment optional.

Schema Reference

For reference, the following is a brief list of properties available for job definitions:

Property

Type

Description

job_type

string

Type of the job, can be either base or kit-service.

name

string

User-friendly name uniquely identifying the job.

task_function

string

Module to execute when when specifying a kit-service.

command

string

Application or command to be executed by the job.

working_directory

string

Directory where the command should be executed.

success_return_codes

Array<int>

List of return codes from the command that should be considered as successful executions.

args

Array<string>

List of arguments to supply to the command, and identical to all jobs instances.

allowed_args

Dict<string,Dict>

Dictionary of arguments which may be unique to each execution of a job, including default values. Arguments can be defined as:

[job.sample-job.allowed_args]
source      = { arg = "--source",      default = "" }
destination = { arg = "--destination", default = "" }
ratio       = { arg = "--ratio",       default = "0.5" }

env

Dict<string,string>

Dictionary of environment variables to supply to the command.

extension_paths

Array<string>

List of extension paths.

log_to_stdout

boolean

Flag indicating whether to capture information from stdout and stderr in the task’s logs.

headless

boolean

Flag indicating whether the application should be run in headless mode.

active

boolean

Flag indicating whether the task is enabled.

container

string

Image location of a Docker container to execute.

capacity_requirements

Dict<string,any>

Capacity requirements to specify container resource limits, volumes or volume mounts when executing tasks in a cluster. Requirements can be defined as:

[job.sample-job.capacity_requirements.resource_limits]
cpu = 1
memory = "4096Mi"
"nvidia.com/gpu" = 1

[job.sample-job.capacity_requirements.volumes]
configMap = { name = "credentials" }
name = "creds"

[job.sample-job.capacity_requirements.volume_mounts]
mountPath = "/root/.provider/"
name = "creds"