Simulation Synthetic Data#
The following diagram gives sample details of the integration of simulation and synthetic data with a factory digital twin. It displays a representational rather than an explicit workflow demonstrating how a factory digital twin can be used with simulation and synthetic data to perform tasks which can improve a physical factory. This reference architecture is meant to demonstrate only a few of many possible workflows and for each demonstrated workflow it is meant to give an idea of a few of the many tools and processes that could be used.
NIM Acceleration#
NVIDIA Inference Models (NIMs) are cloud-based, AI tools. Developers can deploy their own NIMs as NVIDIA NIM Microservices.
USD Search uses AI to identify USD assets from an asset library using only asset thumbnails.
USD Code is an LLM that has been trained on the USD specification and can assist in writing python extensions for Omniverse.
Augmentation can help augment synthetic data sets through AI features such as inpainting and outpainting.
Geometry Creation#
To create a factory digital twin, 3D geometry, content and textures are combined.
3D models can be created directly in CAD or other 3D DCC software.
Models can be acquired through asset libraries such as standard parts libraries.
Scenes can be created from manual or automated reality captures. These scenes can be captured one-time or through automated processes with individual or multiple hardware devices such as terrestrial laser scanners, mobile mapping systems and GNSS smart antennas.
Technical Asset Engineers can develop photorealistic textures for 3D assets that include artifacts such as wear and tear, dirt, and defects.
OpenUSD#
Data from heterogeneous sources can be unified with the OpenUSD format through connectors, converters, importers and DCC applications.
Factory Digital Twin#
Once geometry is in the OpenUSD format, a scene can be composed which is a digital twin of the factory in question.
Virtual Design Review#
One of the most straight-forward uses for such a digital twin is a virtual design review in which users can view the digital twin, comment on it, and revise it.
Computer Vision Model#
Another use of such a digital twin is to facilitate a computer vision workflow, such as defect detection, AMR or robotic manipulation.
Any elements of the scene such as background and lighting that the model should ignore can be randomized. This is called domain randomization. For example, extra lights can be added to the scene and then lighting parameters such as intensity and color temperature can be prepared for randomization.
With domain randomization ready, Omniverse Replicator can be used to randomize the data, render photo-realistic images of the scene, and annotate the images with bounding boxes or segmentations as needed by an AI model and thus create an annotated data set with sensor data such as images or lidar data.
The annotated data set can then be used with NVIDIA TAO or a third party AI training solution to create a computer vision model that can be used in a wide variety of workflows.
For deeper insights into multi-camera computer vision use cases, please refer to the Multi-Camera Tracking-Sim2Deploy Guide.
Workflow Simulation#
Yet another use of such a digital twin is workflow simulation.
With a factory digital twin, the environment can be mapped and simulated with such tools as Navigation meshes, NVIDIA Warp, NVIDIA CuOpt, Partner Simulations and custom extensions.
These tools can be used to run process simulations, detect potential issues and measure the time of manufacturing processes.
Factory Physical Twin#
Any of these virtual processes can be used to make process improvements in a physical factory.
Computer Vision Deployment#
The physical factory can be improved by deploying AI models trained in a virtual environment to the physical world. This is often called “crossing the sim to real gap”.
Process Improvement#
This physical factory can be improved by performing traditionally real-world activities such as lean and six sigma events in a virtual or augmented reality environment.
Conclusion#
Synthetic data and simulation can be used in a factory digital twin context to improve a physical factory. They can facilitate numerous workflows and activities that are either more difficult or not feasible in a physical factory.