Design

DeepSearch consists of several services, which are schematically shown in the diagram below.

../../../_images/deepsearch-combined-diagram.png

Components

As depicted by the diagram above, the DeepSearch stack consists of various services:

NGSearch

NGSearch provides indexing and searching for content on the Nucleus server. It consists of the following micro-services:

  • Search service - provides APIs to query search backend engine (Elastic Search or OpenSearch) and is used by Omniverse Kit and Navigator (version 3.1.0 or greater).

  • Storage service - abstracts the search backend functionality and operates with APIs to write asset metadata that the search service can examine.

  • Metadata indexing service - listens to the updates coming from the Nucleus server and interacts with the Storage service to register these updates in the search backend engine.

  • Tags Crawling service - listens to the tag updates coming from the Nucleus Tagging Service and interacts with the Storage service to register these updates in the search backend engine.

Embedding service

Embedding service provides the ability to run inference on either text or image data using CLIP (Contrastive Language-Image Pre-training) deep learning model.

Projection service

Projection service provides the ability to covert CLIP embedding vectors to a lower dimensional space using some dimensionality reduction methods (at the moment UMAP and PCA approaches are supported).

Cache service

Cache service is centralized persistent storage that uses the majority of DeepSearch services to exchange information.

Monitor service

Monitor service listens to updates from the Nucleus server and stores these updates in the database of the Cache service.

Writer service

Writer service reads completed, processed data from the Cache service. It interacts with both the Storage service from the NGSearch stack, and the Nucleus Tagging Service to update metadata for processed assets.

Worker services:

  • Farm worker service - reads data that requires Omniverse Farm-based processing from the Cache service. Interacts with both Omniverse Farm and the Embedding service to render multiple views for every asset in Omniverse. It extracts embeddings from these views and stores them in the Elastic Search service through the NGSearch storage service API.

  • Non-farm worker service - reads data that does not require Omniverse Farm-based processing from the Cache service. It only interacts with the Embedding service to compute embeddings from thumbnails of assets in Nucleus and stores these embeddings in the Elastic Search service through the NGSearch storage service API.

Utility services:

  • Model updater service - this service gradually updates embeddings for Nucleus assets stored in the Elastic Search engine that is accessed by the NGSearch stack (e.g., when Deep Learning models from the Embedding service are changed). It interacts with Search and Storage services from NGSearch stack.