.. _GANverse3D: ============================================================================================================================================================================================================================= GANverse3D Extension ============================================================================================================================================================================================================================= .. raw:: html
Overview ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- NVIDIA is using AI to power GANverse3D, an Omniverse extension that enables creators to take photos of cars and create virtual replicas with lights, physics models and PBR materials. Reconstructing virtual assets from just one image is an incredibly hard problem, and we are continually exploring new ideas to increase the resulting asset's fidelity. Our current AI model is explicitly trained only on sedans, so other types of vehicles will be transformed into sedans. As our research work progresses, expect to see better vehicle diversity as well as improved quality reconstructions. Installation ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- GANverse3D can be easily installed to an existing Omniverse application through the `Extension Manager <../../extensions/extension-manager.html>`__. #. Bring up the Extension Manager by selecting **Window->Extensions**. #. Search for "aitoybox". Select the `aitoybox.ganverse3d` extension and click the `Install` button. (this may take a few minutes, depending on your connection speed) #. Enable the extension by clicking on the toggle icon to bring up the GANverse3D main window. Using GANverse3D ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- UI Overview ########### .. image:: /content/images/ext_ganverse3d_ui-overview.png :align: center :width: 600 :alt: UI Car Settings Legend ======= ======================= ========================================================================= Ref # Option Result ======= ======================= ========================================================================= 1 Add Image Opens the file picker to select an image of a car 2 Test Drive Opens the scene selection dialog. Once a scene is selected, the camera will move to a chase view, physics simulation will be enabled and the car will be respond to inputs 3 Reset Resets the physics simulation and returns the cars to their original starting point ======= ======================= ========================================================================= .. image:: /content/images/ext_ganverse3d_ui-car-settings.png :align: center :width: 600 :alt: UI Car Settings Legend ======= ======================= ========================================================================= Ref # Option Result ======= ======================= ========================================================================= 1 Input Image Selects the 3D car model produced by this input image 2 Select Visualization Selects the visualization mode of the car 3 Toggle Lights Toggle the car's lights on/off 4 Clear Stop the simulation and remove the car from the scene 5 Change scene Select and load a new scene ======= ======================= ========================================================================= Driving a Car ############# After selecting a scene through the `Test Drive` menu, use the keyboard's arrow keys to drive forward (⬆), backwards (⬇) and turn left (⬅) and right (➡). Hitting the down (⬇) key while driving will cause the car to brake. Supporting Research ------------------- This extension is made possible by the research works the works `Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering `__ by Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, Sanja Fidler, and `DatasetGAN: Efficient Labeled Data Factory With Minimal Human Effort `__ by Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, Sanja Fidler Release Notes ------------- 1.1.1 ##### - Add compatibility with Omniverse Create >= 2021.3.8 1.0.1 ##### - Add compatibility with Omniverse Create >= 2021.3.4 Known Issues ------------ - Image files stored on a Nucleus server can not be opened directly and must be downloaded to a local drive.