DeepSearch

Intro

DeepSearch is a set of services and clients that allow Omniverse users to automatically compute AI-driven embeddings for their 3D models stored in Nucleus. Together with NGSearch service this allows Omniverse users to search for 3D content using natural language (NLP) queries.

Design

DeepSearch requires NGSearch to be available and discoverable through the Nucleus discovery service and an Omniverse Farm instance to be available for rendering. DeepSearch consists of several services, which are schematically illustrated on the diagram below.

../../../_images/deepsearch-diagram-design.png

Note, here only part of NGSearch functionality is illustrated. Please refer to NGSearch Design for a more detailed description of all the individual components.

Components

As depicted by the diagram above DeepSearch stack consists of several services:

  1. Embedding service - provides with the ability to run inference on either text or image data using CLIP deep learning model.

  2. Cache service - centralized persistent storage that is used by all the DeepSearch services to exchange information.

  3. Monitor service - listens to updates from the nucleus server and stores these updates in the database of the Cache service.

  4. Omni writer service - reads data for which processing has already been completed from the Cache service. It interacts with both Storage service from the NGSearch stack and Tagging service in order to update metadata for processed assets.

  5. Worker services:

    • Farm worker service - reads data that requires Omniverse Farm-based processing from the Cache service. Interacts with both Omniverse Farm and Embedding service to render several views for every asset in omniverse, extract embeddings from these views and store then in Elastic Search service through the NGSearch storage service API.

    • Non-farm worker service - reads data that does not require Omniverse Farm-based processing from the Cache service. It only interacts with Embedding service to compute embeddings from thumbnails of assets in nucleus and stores these embeddings in Elastic Search service through the NGSearch storage service API.

  6. Utility services:

    • Model updater service - this service gradually updates embeddings for nucleus assets stored in the Elastic Search engine that is accessed by the NGSearch stack (e.g. in the cases, when Deep Learning model from the Embedding service was changed). It interacts with Search and Storage services from NGSearch stack.