Using Python Pip Packages

In some cases during application development, you may need to include a Python package that is not part of the default Python environment in the Kit. To do this, you can use the pip.toml file located at .\kit-app-template\tools\deps\pip.toml to specify which additional Python packages should be included during the build process. This informs the build system about the required dependencies.

Process Overview:

  1. Add pip packages to pip.toml

  2. Create a pip_bundle extension to manage the additional Python dependencies

  3. Add the pip_bundle extension into your application .kit file as a dependency

Prerequisites

The following prerequisites are required to complete this tutorial:

  • Clone the kit-app-template repository.

  • A development machine with an NVIDIA RTX GPU and latest drivers installed.

  • (recommended) Familiarity with Python.

Tutorial

In this tutorial, we will walk through how to add third-party Python libraries to your application via an extension using pip. The recommended approach is to add the additional dependencies at build time, and this guide will walk you through that process. With the approach taken in this tutorial, the additional dependencies will be accessible to all other extensions in your application.

1. Setup

We first need to create a basic application using the kit-app-template tooling. This step ensures that we have a foundation to which dependencies can be added.

Create a New Editor Application

From a fresh clone of the kit-app-template repository, create a new editor application using the template new tool. For this example, we will use Kit Base Editor.

Linux:

./repo.sh template new

Windows:

.\repo.bat template new

Follow the prompt instructions, choosing:

  • ? Select with arrow keys what you want to create: Application

  • ? Select with arrow keys your desired template: Kit Base Editor

  • ? Enter name of application .kit file [name-spaced, lowercase, alphanumeric]: tutorial.editor.pip

  • ? Enter application_display_name: Tutorial Pip Prebundle Application

  • ? Enter version: 0.1.0

2. Create an Extension for Pip Packages

Next, we will create an extension to manage all the Python dependencies. This extension will ensure that the required packages are included and available for use in our application.

Create a New Extension

Create a Basic Python Extension using the template new tool.

Linux:

./repo.sh template new

Windows:

.\repo.bat template new

Follow the prompt instructions, choosing:

  • ? Select with arrow keys what you want to create: Extension

  • ? Select with arrow keys your desired template: Basic Python Extension

  • ? Enter name of extension .py file [name-spaced, lowercase, alphanumeric]: tutorial.pip_prebundle_extension

  • ? Enter extension_display_name: Tutorial Pip Prebundle Extension

  • ? Enter version: 0.1.0

3. Add Python Dependencies via an Extension

Once the application and extension are set up, we will add the chosen Python dependencies via the extension. We will then add the extension to the main application as a dependency. This will ensure the new Python dependencies are made available for use within the application.

3a - Update the pip.toml File

In your project, navigate to the .\kit-app-template\tools\deps\pip.toml file. This is where you’ll list the Python packages you want to add to your project. You will also specify the folder where the packages will be installed.

Required pip.toml changes:

To add a new package, you can either uncomment the existing [[dependency]] section or add a new code snippet. For this example, we will use the openpyxl package, but feel free to replace it with any other required dependency.

[[dependency]]
python = "../../_build/target-deps/python"
packages = [
   "openpyxl==3.1.5",
]
target = "../../_build/target-deps/pip_prebundle"
  • python : This is the path to the Python interpreter that will be used for installation. It can be an absolute path or relative to the location of this configuration file.

  • packages : List the Pip packages you want to install, along with their specific versions to avoid compatibility issues. For example, with Kit 106.4, Python 3.10 is used. Some packages, like openpyxl==3.1.5, may not work with other Python versions.

  • target : This specifies the path where the packages will be installed. The installed packages will later be linked from your extension.

Verify the pip.toml file changes

After you have updated the pip.toml file, you can verify the requested packages are added by building the project.

Linux:

./repo.sh build

Windows:

.\repo.bat build

Now, locate to the .\_build\target-deps\pip_prebundle. If you see openpyxl and openpyxl-3.1.5.dist-info folders, you have successfully added Pip packages during the build.

3b - Point the Extension to Installed Dependencies

After making the required changes to the pip.toml file, we now need to let the build system know where the new dependencies are located for use in pip_prebundle_extension.

Within.\kit-app-template\source\extensions\tutorial.pip_prebundle_extension\premake5.lua, add a new prebuild link:

-- Link only those files and folders into the extension target directory
repo_build.prebuild_link {
    { "data", ext.target_dir.."/data" },
    { "docs", ext.target_dir.."/docs" },
    { "tutorial", ext.target_dir.."/tutorial" },
    { "%{root}/_build/target-deps/pip_prebundle", ext.target_dir.."/pip_prebundle" }, -- added line
}

3c - Update extension.toml

To make sure that the Python interpreter can find the installed packages, you need to add the pip_prebundle path in the [[python.module]] section of the extension.toml file.

Edit the extension.toml File

In your .\kit-app-template\source\extensions\tutorial.pip_prebundle_extension\config\extension.tomlfile, please replace [[python.module]] section with this:

[[python.module]]
path = "pip_prebundle"

note: remove the name=tutorial.pip_prebundle_extension

This configuration tells Python to include the pip_prebundle folder in the module search path, ensuring that your extension can import the installed packages.

3d - Adding the Extension to the Application

Once the pip_prebundle_extension is properly configured, we can add it to the application as a dependency and verify that the new packages are available.

Add tutorial.pip_prebundle_extension to the application .kit file

Within the .kit file for the application (\source\apps\tutorial.editor.pip.kit), add the new extension to the [dependencies] section:

[dependencies]
"tutorial.pip_prebundle_extension" = {}

4. Build, Launch, and Test

Finally, we can build, launch, and test it.

4a - Build

Use this command to build

Linux:

./repo.sh build

Windows:

.\repo.bat build

4b - Launch

After the build is complete, we can launch. For the sake of this simple test, we will use Script Editor in the developer bundle to verify our changes. The developer bundle can be activated by adding the -d flag to the launch command.

Linux:

./repo.sh launch -d

Windows:

.\repo.bat launch -d

The launch tool will prompt you to select an application .kit file to launch. Select tutorial.editor.pip.kit

Note that the first time startup of a rendering application may take 5-8 minutes as the shader cache is built.

4c - Test

Once the application launches, go to the Developer menu and click Script Editor. You should be able to use the dependencies you specified in the Script Editor.

Using Python Pip Packages Successfully Implemented

If successful, you will be able to import the openpyxl package.

5. Adding Multiple Dependencies

This is an optional section to demonstrate a use case where multiple additional Python dependencies are required. It is recommended that Python dependencies that will be used together are managed by the same pip_prebundle_extension.

5a - Update the pip.toml File

[[dependency]]
python = "../../_build/target-deps/python"
packages = [
   "openpyxl==3.1.5",
   "requests==2.32.3",
]
target = "../../_build/target-deps/pip_prebundle"

5b - Build

We cannot build as we did before because if the folder already exists, it will not be updated. Since the pip_bundle folder already exists, no changes would be made. To make sure the repository is cleaned before building, we need to add x after the build command.

Linux:

./repo.sh build -x

Windows:

.\repo.bat build -x

5c - Launch

After the build is complete, we can launch with the developer bundle enabled.

Linux:

./repo.sh launch -d

Windows:

.\repo.bat launch -d

The launch tool will prompt you to select an application .kit file to launch. Select tutorial.editor.pip.kit

5d - Test

Once the application launches, go to the Developer menu and click Script Editor. You should be able to use the dependencies you specified in the Script Editor.

Using Python Pip Packages Successfully Implemented Part 2

If successful, you will be able to import openpyxl and requests packages.

6. Cleanup

Before continuing to the next section, it is recommended to clean up the applications, extensions, and any other repository changes that were created during this exploration.

Either revert all changes made to the repository or delete the repository and clone it again before continuing.

Typical cleanup steps include:

  • Revert changes to top level premake5.lua file

  • Revert changes to repo.toml

  • Delete the source/ directory

  • Delete the _build/ directory, or run ./repo.sh build -c or .\repo.bat build -c


Additional Considerations

Dependency Management and Compatibility

When adding Python packages to your application through extensions, it’s important to consider the dependency management implications. When a Python package is installed via an extension and added as an application dependency (as done in this tutorial), all other extensions in the same application gain access to it. However, if an extension that relies on these shared packages is moved to another application without them, it may fail due to missing dependencies.

To ensure smooth functionality, consider these options:

  • Add the pip_bundle extension to the new application to make the required packages available to all extensions.

  • Set an extension dependency (in the extension.toml file) on the package-managing extension. This approach will ensure that all necessary Python packages are accessible to the extension regardless of where the extension is used.

Important Note: In both cases, additions to the pip.toml file are required as shown in the tutorial above.

Python Version Compatibility

When selecting specific versions of Python dependencies, it is important to ensure that the selected version is compatible with the version of Python being used by Kit SDK. In the case Kit 106.4, this is Python 3.10.


Additional Resources