1. Overview & Getting Started¶
A variety of reinforcement learning tasks are provided at OmniIsaacGymEnvs. Please make sure to grab the latest from the main branch to make sure contents are in sync with the latest Isaac Sim release. If you have a previous release of OmniIsaacGymEnvs checked out, please run git pull origin main to update to the latest.
The Omniverse Isaac Gym extension provides an interface for performing reinforcement learning training and inferencing in Isaac Sim. This framework simplifies the process of connecting reinforcement learning libraries and algorithms with other components in Isaac Sim. Similar to existing frameworks and environment wrapper classes that inherit from gym.Env, the Omniverse Isaac Gym extension also provides an interface inheriting from gym.Env and implements a simple set of APIs required by most common RL libraries. This interface can be used as a bridge connecting RL libraries with physics simulation and tasks running in the Isaac Sim framework.
We can view the RL ecosystem as three main pieces: the Task, the RL policy, and the Environment wrapper that provides an interface for communication between the task and the RL policy.
The Task is where main task logic is implemented, such as computing observations and rewards. This is where we can collect states of actors in the scene and apply controls or actions to our actors. Omniverse Isaac Gym allows for tasks to be defined following the BaseTask definition in omni.isaac.core. This provides flexibility for users to re-use task implementations for both RL and non-RL use cases.
The main purpose of the Omniverse Isaac Gym extension is to provide Environment Wrapper interfaces that allow for RL policies to communicate with simulation in Isaac Sim. As a base interface, we are providing a class named VecEnvBase, a vectorized interface inheriting from gym.Env that implements common RL APIs. This class can also be easily extended towards RL libraries that require additional APIs by creating a new derived class.
Commonly used APIs provided by the base wrapper class VecEnvBase include:
render(self, mode: str = “human”): renders the current frame
close(self): closes the simulator
seed(self, seed: int = -1): sets a seed. Use -1 for a random seed.
step(self, actions: Union[np.ndarray, torch.Tensor]): triggers task pre_physics_step with actions, steps simulation and renderer, computes observations, rewards, dones, and returns state buffers
reset(self): triggers task reset(), steps simulation, and re-computes observations
1.1. Learning Objectives¶
In this tutorial, we will set up our reinforcement learning example repository: OmniIsaacGymEnvs. We will
Install OmniIsaacGymEnvs for Isaac Sim
Running inferencing and training example in OmniIsaacGymEnvs
10-15 Minute Tutorial
1.2. Getting Started¶
Refer to Default Python Environment to learn about Isaac Sim’s python environment and locate the python executable in Isaac Sim.
Refer to Basic Isaac Sim Installation for setting up the Nucleus server for access to pre-trained checkpoints.
1.3. Installing Examples Repository¶
To set up these examples, first clone the repository:
git clone https://github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs.git
We can install the examples as a python module in Isaac Sim. Locate the Isaac Sim python executable, which by default should be python.sh on Linux or python.bat on Windows, located at the root of the Isaac Sim directory. We will refer to this path as PYTHON_PATH.
To set a PYTHON_PATH variable in the terminal that links to the python executable, we can run a command that resembles the following. Make sure to update the paths to your local path.
For Linux: alias PYTHON_PATH=~/.local/share/ov/pkg/isaac_sim-*/python.sh
For Windows: doskey PYTHON_PATH=C:\Users\user\AppData\Local\ov\pkg\isaac_sim-*\python.bat $*
Install OmniIsaacGymEnvs to PYTHON_PATH by running the following from the root of OmniIsaacGymEnvs:
PYTHON_PATH -m pip install -e .
The following error may appear during the initial installation. This error is harmless and can be ignored.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
1.4. Running Examples¶
Example scripts should be launched from omniisaacgymenvs/omniisaacgymenvs.
1.4.1. Launching Training Examples¶
To train your first policy, run:
PYTHON_PATH scripts/rlgames_train.py task=Cartpole
We will see an Isaac Sim window pop up. Once Isaac Sim initialization completes (which may take a few minutes if launching for the first time), the Cartpole scene will be constructed and simulation will start running automatically. The process will terminate once training finishes.
1.4.2. Running Inference¶
To load a trained checkpoint and perform inference (no training), pass test=True as an argument, along with the checkpoint name.
PYTHON_PATH scripts/rlgames_train.py task=Cartpole test=True checkpoint=runs/Cartpole/nn/Cartpole.pth
1.4.3. Inferencing with Pre-Trained Checkpoints¶
Pre-trained checkpoints are provided for each task on the Nucleus server, under Assets/Isaac/2022.1/Isaac/Samples/OmniIsaacGymEnvs/Checkpoints.
To load a pre-trained checkpoint and run inferencing, run:
PYTHON_PATH scripts/rlgames_train.py task=Cartpole test=True checkpoint=omniverse://localhost/NVIDIA/Assets/Isaac/2022.1/Isaac/Samples/OmniIsaacGymEnvs/Checkpoints/cartpole.pth
1.5. Summary¶
This tutorial covered the following topics:
Installation of OmniIsaacGymEnvs
Running training examples in OmniIsaacGymEnvs
Running inferencing examples in OmniIsaacGymEnvs
1.5.1. Next Steps¶
Continue on to the next tutorial in our Reinforcement Learning Tutorials series, Domain Randomization for RL, to learn about domain randomization for reinforcement learning.
If you are interested in setting up a new reinforcement learning task outside of OmniIsaacGymEnvs, please see the tutorial, Custom RL Example using Stable Baselines, to learn about using the stable baselines library with Isaac Sim.