GANverse3D Extension

Overview

NVIDIA is using AI to power GANverse3D, an Omniverse extension that enables creators to take photos of cars and create virtual replicas with lights, physics models and PBR materials.

Reconstructing virtual assets from just one image is an incredibly hard problem, and we are continually exploring new ideas to increase the resulting asset’s fidelity. Our current AI model is explicitly trained only on sedans, so other types of vehicles will be transformed into sedans. As our research work progresses, expect to see better vehicle diversity as well as improved quality reconstructions.

Installation

GANverse3D can be easily installed to an existing Omniverse application through the Extension Manager. Currently, installation requires downloading the extension directly. Please follow these steps:

Method 1

  1. Bring up the Extension Manager by selecting Window->Extensions.

  2. Search for “aitoybox”. Select the aitoybox.ganverse3d extension and click the Install button. (this may take a few minutes, depending on your connection speed)

  3. Enable the extension by clicking on the toggle icon to bring up the GANverse3D main window.

Method 2 (Better for slower connections)

  1. Download the GANverse3D extension directly here.

  2. Once you have downloaded and saved the zip file, bring up the Extension Manager by choosing Window->Extensions.

  3. From there, click on the ‘Hamburger’ (☰) menu and choose “Import Extension”, then navigate to the GANverse3D zip file to import it.

  4. Once imported, you will be able to see the extension in the extensions list by searching for “aitoybox”. Select the extension and toggle it to the enabled state to bring up the GANverse3D main window.

Using GANverse3D

UI Overview

UI Car Settings Legend

Ref #

Option

Result

1

Add Image

Opens the file picker to select an image of a car

2

Test Drive

Opens the scene selection dialog. Once a scene is selected, the camera will move to a chase view, physics simulation will be enabled and the car will be respond to inputs

3

Reset

Resets the physics simulation and returns the cars to their original starting point

UI Car Settings Legend

Ref #

Option

Result

1

Input Image

Selects the 3D car model produced by this input image

2

Select Visualization

Selects the visualization mode of the car

3

Toggle Lights

Toggle the car’s lights on/off

4

Clear

Stop the simulation and remove the car from the scene

5

Change scene

Select and load a new scene

Driving a Car

After selecting a scene through the Test Drive menu, use the keyboard’s arrow keys to drive forward (⬆), backwards (⬇) and turn left (⬅) and right (➡). Hitting the down (⬇) key while driving will cause the car to brake.

Supporting Research

This extension is made possible by the research works the works Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering by Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, Sanja Fidler, and DatasetGAN: Efficient Labeled Data Factory With Minimal Human Effort by Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, Sanja Fidler

Release Notes

1.0.1

  • Add compatibility with Omniverse Create >= 2021.3.4

Known Issues

  • Image files stored on a Nucleus server can not be opened directly and must be downloaded to a local drive.