Overview#

This is an extension that uses AI-powered LangChain agents to answer questions, provide code samples, and guide users in the development of the Omniverse UI Framework. The key deliverable of the extension is to assist and guide users in writing and debugging omni.ui and omni.ui.scene code, as well as answer related questions. This extension does so via a conversational chat interface with a LangChain AI agent.

Implementation details#

Here’s an overview of the structure and key components of the omni.ai.langchain.agent.omni_ui extension:

graph LR A[omni.ai.langchain.agent.omni_ui] --> B[modifiers] A --> C[nodes] A --> D[tools] C --> E[systems]

The omni.ai.langchain.agent.omni_ui extension consists of several key components and submodules:

  1. modifiers submodule:

    • Contains modifier classes that can be applied to various aspects of the agent’s behavior.

    • Key components:

      • AsyncOmniUIClassAppender: Based on the questions, find related classes in codeAtlas and Appends dynamic input messages to agent.

      • NetworkLengthModifier: Calculate the network length.

      • OmniUICodeInterpreterModifier: Interprets AI generated code snippets.

      • OmniUICodeRagModifier: Based on the question, find the closest related code sample for AI.

      • RemoteCodeInterpreter: Interprets code snippets remotely to isolate possible crashes in code.

  2. nodes submodule:

    • Contains node classes that define the behavior of the extension.

    • Key components:

      • systems submodule: Contains system prompts for the AI agent.

        • agent_system: The general system prompt for the agent that contains the high level extension overview exposed to the agent (e.g., omni.ui, omni.ui.scene, omni.kit.widget.filter).

        • classes: This is used for AsyncOmniUIClassAppender as a selective system prompt inputs for agent to avoid token access issue.

        • omni_ui_scene_system: The system prompt for the agent to view examples of specific classes in the omni.ui.scene extension.

        • omni_ui_system: The system prompt for the agent to view examples of specific classes in the omni.ui extension.

      • OmniUIAgent: The UI agent to answer questions and provide code samples related to the Omniverse UI Framework.

      • OmniUICodeNetworkNode: The UI agent network which uses modifiers and tools to iteratively generate correct code.

  3. tools submodule:

    • Contains tools used by the agent for various tasks.

    • Key components:

      • create_ui_atlas: Creates the UI CodeAtlas for the extension. See Data section for more details

      • omni_ui_code_atlas_tool: A tool for managing the UI CodeAtlas.

      • omni_ui_rag_collection: Manages the RAG collection for omni.ui code. See Data section for more details

      • remote_code_interpreter_server: Spawn a remote kit instance for interpreting code snippets.

Data & Tool#

The agent has access to various tools and data to provide as accurate and relevant responses to user queries. Here’s an overview of the resources available to the agent:

The agent use the system prompt to chooses the most relevant extensions prompts. It then uses the CodeAtlas and RAG to generate responses:

  1. UI CodeAtlas: A collection of code snippets, examples, and documentation related to the omni.ui extension. The UI CodeAtlas is created and managed by the create_ui_atlas tool.

  2. RAG Collection: A collection of code snippets and relationships for the omni.ui extension. The RAG collection is managed by the omni_ui_rag_collection tool.

  3. Update the collection json with the collection tool.

  4. Update the RAG database with python -m lc_agent_faiss_rag_gen --input_json <collection_json> --output_db <faiss_database_directory> --text_field file_name --metadata_fields file_path method_name source_code --embedding_model ai-embed-qa-4 (--update_existing)

Adding an Extension#

To add a new extension to the existing agent’s knowledge base, follow these steps:

  1. Create a new system prompt in the systems submodule for the new extension.

  2. Add relevant classes and code examples to the new extension’s system prompt.

  3. Update the agent’s knowledge base with the new extension’s data sources

    1. Update a UI CodeAtlas for the extension using the create_ui_atlas tool.

    2. Update the RAG collection for the extension using the omni_ui_rag_collection tool.

  4. Include relevant dependencies and imports in the extension’s codebase.