Farm Agent

Overview

Omniverse Farm Agent is responsible for executing a set of tasks based on the capabilities of the environment on which it runs.

Once launched, Agent queries Omniverse Queue for the tasks that have been submitted, and selects one for processing. For this reason, Agents depend on a Queue for the book-keeping of tasks, and for reporting progress information in a meaningful way. The documentation for Farm Queue contains additional details on how to get started.

During the execution of a task, it regularly reports its progress back to Queue, so that the entire fleet of Agents and tasks can be managed from a single location.

Installation

Agent can be installed from the Omniverse Launcher, by navigating to the Exchange header menu, then selecting Agent from the Apps section of the left-hand sidebar.

After first installing Agent, the Install button then allows you to launch it and start processing content.

Omniverse Launcher: Agent

Interface Overview

Omniverse Agent interface

#

UI Element

Action

1

Queue address

Enter Queue URL (host) the Agent (client) should connect to in order to retrieve the list of pending tasks.

2

Connect/Disconnect

After validating that the connection is successful, clicking the Connect button will have the Agent query, pick up and start processing tasks. Alternatively, clicking the Disconnect button will release the Agent from the pool.

3

Test connection

Click this button to validate that the provided Queue URL can be reached by the Agent. This can be helpful in order to investigate and diagnose potential network or connectivity issues when running Agents and Queues on different environments.

4

Edit job definition

Click this button to open a Wizard to edit existing job definitions, or create new ones on the fly. Once saved, the Agent will be ready to accept incoming requests from Queue about executing this type of task.

5

Agent ID

Label displaying the unique identifier of the Agent. When running multiple Agents on the same environment, this makes it possible to connect/disconnect, or otherwise perform tasks on individual Agents via their own UI.

6

Agent task status

Label displaying the status of the task currently performed by the Agent.

7

Agent connection status

Label displaying the status of the connection from the Agent to the pool.

Headless Installation on Linux

To automate deployment of Farm Agents on Linux, and scale the compute capabilities to multiple machines, it may be convenient to install Agents in headless manner.

Prerequisites include Ubuntu Server 20.04 or greater, with an Internet connection in order to download the necessary additional software and packages.

Note

Other Linux distributions should also be compatible with Omniverse Farm Agent, although only Ubuntu 20.04 is officially supported for production use.

  1. Start by installing the required software dependencies:

    $ sudo apt-get install -y --no-install-recommends \
        libatomic1 \
        libxi6 \
        libxrandr2 \
        libxt6 \
        libegl1 \
        libglu1-mesa \
        libgomp1 \
        libsm6 \
        unzip

  2. Upload the farm_agent_install.sh script to the server running Farm Agent, and place in the /opt/ove folder after creating it:

    farm_agent_install.sh
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
    #!/bin/bash

#
# Note: Specific package versions can be retrieved from the Omniverse Launcher.
#

# Install the Omniverse Farm Agent package, containing the core Agent capabilities:
mkdir -p ov-farm-agent
cd ov-farm-agent
pwd
curl https://d4i3qtqj3r0z5.cloudfront.net/farm-agent-launcher%40103.1.0%2Bmaster.53.238d4340.teamcity.linux-x86_64.release.zip > farm-agent-launcher.zip
unzip farm-agent-launcher.zip
rm farm-agent-launcher.zip

# Install the Kit SDK package, containing the set of features and extensions shared by Omniverse applications:
mkdir kit
cd kit
pwd
curl https://d4i3qtqj3r0z5.cloudfront.net/kit-sdk-launcher@103.1%2Brelease.6024.1fc2e16c.tc.linux-x86_64.release.zip  > kit-sdk-launcher.zip
unzip kit-sdk-launcher.zip
rm kit-sdk-launcher.zip

# Install the Multiview Batch package:
cd ..
mkdir -p jobs/multiview-batch
cd jobs/multiview-batch
pwd
curl https://d4i3qtqj3r0z5.cloudfront.net/farm-job-multiview-batch-render%40103.1.0%2Bmaster.60.bd48c66c.teamcity.linux-x86_64.release.zip > farm-job-multiview-batch.zip
unzip farm-job-multiview-batch.zip
rm farm-job-multiview-batch.zip

# Install the "create-render" package, containing the job definition for the rendering task:
cd ..
mkdir -p jobs/create-render
cd jobs/create-render
pwd
curl https://d4i3qtqj3r0z5.cloudfront.net/farm-job-create-render@103.1.0%2Bmaster.63.581a1b80.teamcity.linux-x86_64.release.zip > farm-job-create-render.zip
unzip farm-job-create-render.zip
rm farm-job-create-render.zip

cd ../..

# Create a boilerplate launch script for the Agent:
cat << 'EOF' > agent.sh
#!/bin/bash

BASEDIR=$(dirname "$0")
exec $BASEDIR/kit/kit $BASEDIR/apps/omni.farm.agent.headless.kit \
    --ext-folder $BASEDIR/exts-farm-agent \
    --/exts/omni.services.farm.agent.operator/job_store_args/job_directories/0=$BASEDIR/jobs/* \
    --/exts/omni.services.farm.agent.controller/manager_host=http://<QUEUE IP>:<QUEUE PORT> \
    --/exts/omni.services.farm.agent.operator/manager_host=http://<QUEUE IP>:<QUEUE PORT>
EOF

chmod +x agent.sh

  3. Change the permission of the farm_agent_install.sh script in order to make it executable:

    $ chown +x farm_agent_install.sh

  4. Run the script from within the /opt/ove folder as a non-root user:

    $ ./farm_agent_install.sh

  5. Once the files are downloaded and extracted, you will have a folder named /opt/ove/ov-farm-agent.

  6. Ensure that the /opt/ove/ov-farm-agent and all the files within are owned by a non-root user.

  7. Configure the Farm Agent Controller and Operator addresses to supply the Farm Queue Server address to agent.sh:

    agent.sh
    # [...]

--/exts/omni.services.farm.agent.controller/manager_host=http://<QUEUE IP>:<QUEUE PORT>
--/exts/omni.services.farm.agent.operator/manager_host=http://<QUEUE IP>:<QUEUE PORT>

  8. Once the Controller and Operator addresses configured, launch the Agent:

    $ ./agent.sh &

Output Log

When executing the Agent, the console will display an output similar to the following to indicate it is running successfully:

[user@machine ov-farm-agent] $ ./agent.sh
[Info] [carb] Logging to file: /home/user/.nvidia-omniverse/logs/Kit/omni.farm.agent.headless/102.1/kit_20220502_132007.log
[0.346s] [ext: omni.kit.pipapi-0.0.0] startup
[0.360s] [ext: omni.services.pip_archive-0.3.0] startup
. . .
[1.464s] [ext: omni.farm.agent.headless-102.1.0] startup
[1.574s] app ready

In case of error due to an Agent is running without proper acceleration, an output similar to the following will be displayed:

[1.964s] [ext: omni.farm.agent.headless-102.1.0] startup
2022-04-29 20:57:11 [2,049ms] [Error] [omni.services.farm.facilities.agent.capacity.managers.base] Failed to load capacities for omni.services.farm.facilities.agent.capacity.GPU: NVML Shared Library Not Found
[2.075s] app ready

See the Linux Troubleshooting for any installation issues.