Vendoring Python Dependencies

Concept

At a high level the idea is to vendor our dependencies at build time and consolidate tracking our dependencies into a single requirements.txt. Additionally regenerate that requirements.txt via Poetry and a pyproject.toml. This is all in the name of more easily tracking our dependencies and knowing what goes into our product.

How do I add or change dependencies?

In pyproject.toml you can add Python dependencies to the [tool.poetry.dependencies] block. Then run repo poetry to setup Poetry in a venv to generate poetry.lock and requirements.txt. Commit these three files, bump version + changelog and upload via ./upload.sh.

Implementation

This is written in the context of repo_man but it applies to anything that has access to repo_man at build time. We use repo_upload for packaging up our tools, and within repo.toml there is an exec block that vendors our dependencies via omni.repo.man.pip_install_requirements before packaging up repo_man.

On first run of repo_man in a downstream project within __init__.py we hash the omni/repo/man/_vendor directory and stash the hash at omni/repo/man/_vendor.md5. On future executions of repo_man if the hash does not match the directory, or the requirements.txt is newer than our hash file, we rebuild omni/repo/man/_vendor via pip.

In most use cases where your project pulls in repo_man and other tools via deps/repo-deps.packman.xml you will not notice this, as your _vendor directory will get unpacked and should never change. For local tool development or where you link a copy of repo_man via repo_source you might see a message like “Populating vendor directory:” which indicates your _vendor is out of sync and is being rebuilt.