Troubleshooting & Known Issues

Troubleshooting

Setting default Nucleus Server

To set the default Nucleus server when running natively, edit the file at:

_build/linux-x86_64/release/apps/omni.isaac.sim.base.kit
[settings.isaac]
nucleus.default = "omniverse://localhost"

Change localhost to the IP address of the Nucleus server.

You could also run Isaac Sim with the flag –/isaac/nucleus/default=”omniverse://<ip_address>”.

To set the default Nucleus server when running in Docker, use the flag -e “OMNI_SERVER=<ip_address>” where <ip_address> is the IP address of the Nucleus Server.

$ sudo docker run --gpus all -e "ACCEPT_EULA=Y" -e "OMNI_SERVER=<ip_address>" --rm --network=host nvcr.io/nvidia/isaac-sim:2021.1.0

Setting default Username and Password connecting to Nucleus Server

To set the default credentials when running natively, run:

$ export OMNI_USER=<username>
$ export OMNI_PASS=<username>

To set the default credentials when running in Docker, use the flag -e “OMNI_USER=<username>” -e “OMNI_PASS=<username>” where <username> is any text (the default is dockeruser).

$ sudo docker run --gpus all -e "ACCEPT_EULA=Y" -e "OMNI_USER=<username>" -e "OMNI_PASS=<username>" --rm --network=host nvcr.io/nvidia/isaac-sim:2021.1.0

Note

It is recommended to use the same text for username and password for our current version. The current Omniverse Nucleus Server does not support password authentication.

Save Isaac Sim configs on local disk

To keep Isaac Sim configuration persistent when running in container, use the flags below when running the docker container.

-v ~/docker/isaac-sim/documents:/root/Documents:rw           #For adding new or saving files
-v ~/docker/isaac-sim/cache:/root/.cache/ov:rw               #For shader cache
-v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw   #For log files
-v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw     #For Isaac Sim configs
$ sudo docker run --gpus all -e "ACCEPT_EULA=Y" --rm --network=host -v ~/docker/isaac-sim/documents:/root/Documents:rw -v ~/docker/isaac-sim/cache:/root/.cache/ov:rw -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw nvcr.io/nvidia/isaac-sim:2021.1.0

Note

These flags will use the use Home folder to save the Isaac Sim cache, logs, config and data.

Problem connecting to Docker container

To resolve some problems connecting to a docker container, try using the –network=host flag when running the docker container.

$ sudo docker run --gpus all -e "ACCEPT_EULA=Y" --rm --network=host nvcr.io/nvidia/isaac-sim:2021.1.0

Note

This flag is needed to connect to a Nucleus server.

Access Remote Ubuntu Workstation

This section shows you how to access a remote Ubuntu workstation.

Steps

  1. If you have access to the remote workstation physically, install an SSH server to allow remote access:

    $ sudo apt update
    $ sudo apt install openssh-server
    
  2. Run the command below to get the remote workstaion IP address:

    $ ifconfig
    
  3. Run the command below to access the remote workstation:

    $ ssh <remote_workstation_username>@<remote_workstation_ip_address>
    <remote_workstation_username>@<remote_workstation_ip_address>'s password:
    
  4. Proceed to Isaac Sim Headless Container

Reading the Logs in a Container

To ensure the Omniverse Isaac Sim in a container is running, read the logs. Here’s how:

  1. If the Omniverse Isaac Sim container is on a remote machine, SSH into the docker host using a terminal. Run this command from where your pem key folder is, replace the ubuntu@ec2-52-53-177-94.us-west-1.compute.amazonaws.com with your instance or remote host IP address:

    $ ssh -i "yourkey.pem" ubuntu@ec2-52-53-177-94.us-west-1.compute.amazonaws.com
    
  2. Access the running container by running:

    $ docker exec -it <container_id_or_name> bash
    $ cd /root/.nvidia-omniverse/logs/Kit/Isaac-Sim/<version_number>
    

Getting IP Addresses of AWS EC2 Instance

To get the public and private IP addresses of an AWS EC2 instance, go to the Instances section of the EC2 Dashboard and select the instance. See the image below for an example of the Private and Public IPs:

../_images/isaac_main_aws_ip_address.png

SSH into the AWS EC2 Instance

If you need to directly access an AWS EC2 instance that was created from the deployment above, run these steps to SSH into the instance:

$ ssh -i "<ssh_key_name>.pem" ubuntu@<public_ip_address>

Restarting the Container

The steps below is to restart a headless container.

  1. SSH into the host machine running Omniverse Isaac Sim Container or AWS instance.

    $ ssh -i "<ssh_key_name>.pem" ubuntu@<public_ip_address>
    
  2. List all running containers and find the container ID running Omniverse Isaac Sim.

    $ sudo docker ps
    CONTAINER ID        IMAGE
    823686a7036d      nvcr.io/nvidia/isaac-sim...2021.1.0
    
  1. Restart the container.

    $ sudo docker restart [CONTAINER ID]
    
  2. View docker logs.

    $ sudo docker logs [CONTAINER ID]
    

Restart Omniverse Isaac Sim inside Docker

If you want to restart Omniverse Isaac Sim while keeping the docker running, you must start the docker with bash as the entrypoint, so that you can manually start or stop Omniverse Isaac Sim.

  1. Start the docker with bash, and start Omniverse Isaac Sim manually.

    $ sudo docker run -it --entrypoint bash --gpus all -e "ACCEPT_EULA=Y" --rm --network=host nvcr.io/nvidia/isaac-sim:2021.1.0
    $ runheadless
    
  2. Proceed to Omniverse Kit Remote Client to live-stream Omniverse Isaac Sim remotely.

  3. When you need to exit, in a separate terminal, start an interactive bash session inside the same container that’s running the headless server, and kill the Omniverse Isaac Sim related processes.

    $ docker exec -it <container_id> bash
    $ pkill omniverse-kit
    
  4. To restart Omniverse Isaac Sim:

    $ runheadless
    

Save Docker Image

If you made significant changes inside the docker, for example, installed ROS or other libraries, you may want to save the docker image so that you can restart the docker without having to reinstall everything.

  1. Find the container’s id and commit it.

    $ docker ps
    $ docker commit <CONTAINER ID> <new docker name>
    
  2. To reload a specific docker:

    $ docker run -it --entrypoint bash --gpus all -e "ACCEPT_EULA=Y" --rm --network=host -d <new docker name>
    

Setting up Docker

Once you have docker on Linux installed, follow the instructions at Post-installation steps for Linux to set it up so you would not need to use sudo to run a docker container.

Mount a Folder to the Container

To add data from the host machine to a container, mounting a folder is needed.

$ sudo docker run --gpus all --rm -e "ACCEPT_EULA=Y" -v ~/docker/isaac-sim/documents:/root/Documents:rw nvcr.io/nvidia/isaac-sim:2021.1.0

Note

Can now copy files to docker/isaac-sim/documents in your Home folder and it will show up in the Isaac Sim container at /root/Documents.

Installing NVIDIA GPU Drivers

It is recommended to install the Latest Long Lived Branch Version of NVIDIA GPU Drivers and using the .run installer.

To download, go to:

To install, follow these steps:

Omniverse Isaac Sim Container on NGC

To download the Omniverse Isaac Sim container go to:

Note

Please use your NVIDIA Developer Program credentials on the NVIDIA NGC website.

Creating AWS Access Key

Create an Access Key by following the instructions here:

Creating SSH Key

*On Linux*

  1. Run:

    $ mkdir ~/.ssh
    $ chmod 700 ~/.ssh
    $ ssh-keygen -t rsa
    
  2. Enter your passphrase twice.

  3. Your public key is at .ssh/id_rsa.pub in your home folder and private key at .ssh/id_rsa.

*On Windows*

  1. Download PuTTYgen.

  2. Launch PuTTYgen, and click on “Generate a public/private key pair”.

  3. Click on “Save public key” and name the file “${ssh_key_name}.pub”. This is your Public Key file.

  4. From the “Conversions” menu, select “Export OpenSSH key” and name the file “${ssh_key_name}.pem”. This is your Private Key file.

  5. Edit the properties of the “${ssh_key_name}.pem” file.

    • Go to security settings, click “Advanced”

    • Remove inheritance

    • Set current user as owner of the file and full permissions to only that user.

    • This is to prevent permission errors when trying to SSH into the instance

Creating NGC API Key

If you don’t already have an API key:

  1. Generate your NGC API Key.

  2. Go to AWS Secrets Manager section of the AWS console:

  3. Click Store a new secret.

  4. Select Other type of secrets.

  5. Set the following two sets of Secret key/value pairs

    Key

    Value

    Username

    $oauthtoken

    Password

    <Your_NGC_API Key>

  6. Give the secret a name

Known Issues

General

  1. When running samples from the python_samples via ./python.sh they might not exit/shutdown cleanly:

    [Warning] [carb.tasking.plugin] Leaking 1 carb::tasking::Counter instances!
    Shutting Down Complete
    ./python.sh: line 27:  8164 Segmentation fault      (core dumped) $python_exe $@
    There was an error running python
    
  2. If running Omniverse Isaac Sim headless connected via the remote client and you exit on shutdown the following error can occur, it can be ignored:

    [ext: omni.physx] shutdown
    Fatal Python error: Segmentation fault
    
    Thread 0x00007f46f8faa740 (most recent call first):
    File "..._build/target-deps/kit_sdk_release/_build/linux-x86_64/release/extsPhysics/omni.physx/omni/physx/scripts/extension.py", line 30 in on_shutdown
    File "..._build/target-deps/kit_sdk_release/_build/linux-x86_64/release/plugins/bindings-python/omni/ext/impl/_internal.py", line 225 in shutdown_all
    File "..._build/target-deps/kit_sdk_release/_build/linux-x86_64/release/plugins/bindings-python/omni/ext/impl/_internal.py", line 261 in shutdown_all_extensions
    File "..._build/target-deps/kit_sdk_release/_build/linux-x86_64/release", line 3 in <module>
    
  3. When running in windowed container, the following errors may be ignored and the app will continue to run after waiting for awhile:

    ERROR: Could not find a version that satisfies the requirement psutil (from versions: none)
    ERROR: No matching distribution found for psutil
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcdcc284dd8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/psutil/``
    
  4. A joint position target value less than -2 * pi or greater than 2 * pi can cause the joint to rotate indefinitely.

  5. Enabling the ROS2 extension is not persistent across restarts of Isaac Sim

Warnings

  1. Warnings similar to the following can be ignored:

    • [Warning] [omni.usd] Warning (secondary thread)

    • [Warning] [carb.tasking.plugin] Counter 0x7f25e002f8d0

    • [Warning] [rtx.neuraylib.plugin] [MDLC:COMPILER]   1.0   MDLC   comp warn

    • [Warning] [rtx.mdltranslator.plugin] Unable to resolve

    • [Warning] [omni.tagging.plugin] Failed to discover tagging service

    • [Warning] [omni.isaac.dynamic_control.plugin] DcFindArticulationDof: Function called while not simulating

    • [Warning] [omni.isaac.dynamic_control.plugin] DcSetDofProperties: Function called while not simulating

    • [Warning] [omni.client.plugin]  Tick: authentication: Could not connect to discovery service at "wss://...

  2. If Physics is not needed this message can be ignored:

    • [Warning] [omni.physx.plugin] Physics USD: Physics scene not found. A temporary default PhysicsScene prim was added automatically!

  3. If there is unwanted noise in simulated depth images, disable anti-aliasing under the Render Settings ->Ray Tracing -> Anti-Aliasing tab by setting the Algorithm to None

Errors

  1. Errors similar to the following can be ignored:

    • [Error] [omni.client.plugin]  Main: usd_plugin: Type mismatch for <...>: expected 'SdfAssetPath', got 'GfVec3f'

    • [Error] [omni.client.plugin]  Main: usd_plugin: Failed verification:

    • [Error] [omni.isaac.dynamic_control.plugin] DcFindArticulationDof: Invalid or expired articulation handle

    • [Error] [omni.isaac.dynamic_control.plugin] DcSetDofProperties: Invalid or expired dof handle

    • [Error] [omni.ui.python] Unable to cast Python instance to C++ type (compile in debug mode for details)

    • [Error] [omni.physx.plugin] Transformation change on non-root links is not supported. Link: ...

  2. If you see the following, please make sure that python processes for any previously executed training runs are killed:

    • [Error] [carb.cudainterop.plugin] CUDA error 708: cudaErrorSetOnActiveProcess - cannot set while device is active in this process)

  3. When enabling the omni.isaac.shapenet extension, these errors can be ignored:

    • ERROR: requests 2.23.0 has requirement urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1, but you'll have urllib3 ... which is incompatible.

  4. These errors can be ignored while running the running Kit Remote Client, if the client works as normal:

    ERROR [BifrostClient: Streamer] {2B436200} -  updateVideoSettingsForNVbProfile: profile 8 is not handled
    ERROR [NVST:ClientSession] {1E8D2700} -  Number of channels(2) is not valid for surround configuration
    ERROR [NVST:ClientSession] {1E8D2700} -  Either in stereo or error in receiving opus information from server
    
    ERROR [LAVCDecoder] {FC4A0700} - GERONIMO_ERROR 0xC0040006 GERONIMO_LAVCDECODER_DECODE: Packet decode failure.
    ERROR [GIOInterface] {DAFFD700} - GERONIMO_ERROR 0xC0020003 GERONIMO_IOINTERFACE_INVALID_AUDIO_FUNC: Audio Renderer is NULL.
    ERROR [NVST:ClientLibraryWrapper] {1E8D2700} -  Cannot find streamEventRaised for stream.media type 1
    ERROR [NVST:ClientLibraryWrapper] {1E8D2700} -  Cannot find streamEventRaised for stream.media type 2
    
    ERROR [NVST:RtspSessionPocoBase] {FB280700} -  perform() failed: 0
    ERROR [NVST:RtspPocoEvent] {FB280700} -  RTSP-XNvEvent Polling failed: 0, rc: 408
    
    ERROR [NVST:UdpRtpSource] {F9A7D700} -  UDP RTP Source: failed to receive data (Error: 0x80000013)
    ERROR [NVST:RtpSourceQueue] {F9A7D700} -  RtpSourceQueue: failed to read RTP packet (Result: 0X80000013)
    ERROR [BifrostClient: NvscWrapper] {1D0CF700} - Received old frame - current 7536 received 0
    ERROR [BifrostClient: Interface] {2B436200} - nvbSendInputEvent(). SessionIdentifier is ''
    
  5. Error similar to the below can be ignored when disabling and enabling again the Onshape Importer/ STEP Importer / URDF Importer Extensions:

    2021-06-01 19:16:51 [65,842ms] [Error] [carb.settings.python] AttributeError: 'NoneType' object has no attribute '...'
    
    At:
    <...>
    
  6. If you see this error “Could not find registered CUDA function ‘updateBodyExternalVelocitiesLaunch”, please add import omni.physx before all of your import torch:

    import omni.physx
    import torch
    

Hang

  1. There is a known issue where if a new stage is created or loaded while an existing stage is loading, Isaac Sim will hang

  2. Disabling the ROS2 extension will cause Isaac Sim to hang due to a thread wait issue in the rclcpp cyclonedds backend.

Crash

  1. There is a known issue where the host resolution must be at least 1920x1080 to launch Isaac Sim

  2. There is a known crash when the Isaac Sim window is being resized while it is still launching.

  3. There is a known crash when using the WebRTC mode in Isaac Sim container.