Configuration#
repo_man comes with a powerful configuration system to improve flexibility of tools and promote code reusability.
TOML#
TOML is used as a configuration format. It has ini
like syntax, which makes it human readable and easy to copy and paste between configs. It was choosen as The configuration format for the Omniverse and repo_man followed it.
If you are unfamiliar with it I recommend using any online converter between JSON
and TOML
(or any other format). In the end they all build a nested dictionary and very similar.
Configuration files#
When any tool [tool_name]
runs at least 5 configurations merged on top of each other (in order):
repo_man’s
repo_tools.toml
for tools-wide settings[tool_name]
’srepo_tools.toml
for tool-specific settingsrepo.toml
for repo-specific settings~/.nvidia-omniverse/config/global_repo.toml
for global repo settings.user.repo.toml
for repo-specific per-user settings
Config settings get overridden the further down the sequence you go. So repo.toml
settings will override the repo_tools.toml
defaults, and user.repo.toml
will override anything set earlier in the stack.
Merging (or overriding) happens on leaf values (like integer, string, array), but not on dictionaries (or tables).
repo_tools.toml
serves as a source for all default settings for any tool. User can look into _repo/deps/[tool_name]/repo_tools.toml
for any settings and copy in repo.toml
. We highly recommended to put all settings default values in repo_tools.toml
with a comment for a better discoverability.
~/.nvidia-omniverse/config/global_repo.toml
is the second to last layer and should be used for setting local values unique to your environment. Examples include: forcing repo tools logging to be verbose, disabling repo_build ccache remote_storage, or setting repo_build max job count. It is not advisable to set core configuration values here such as repo_format clang_format version or repo_build target config. ~
resolves to C:\Users\{username}
on Windows at /home/{username}
on Linux.
user.repo.toml
is the last layer and can be used for setting local values unique to your environment and one particular project. For example you may only wish to build release
builds and can override the repo_build default config value for one project.
Example:#
repo_build
has repo_tools.toml
with:
[repo_build]
# All build configs supported
build_configs = ["debug", "release"]
Some repo can only build in release configuration. So it overrides it in repo.toml
:
[repo_build]
# Only "release" is supported because of Y:
build_configs = ["release"]
Tokens#
repo_man supports tokens to make configuration more flexible. They take a form of ${token}
or $token
.
They can be escapped using double $$
: $${token}
will be resolved to ${token}
.
Most of them are resolved when configuration is loaded. However some of them are unknown until tool actually runs. Those are left in config as is and it is each tool’s responsibility to resolve them.
Tokens supported by repo_man:#
${root}
- repo root path.
${config_root}
- current config folder path.
${platform}
- current host platform. For example windows-x86_64
or linux-x86_64
.
${lib_ext}
- current platform’s library extension. For example .dll
or .so
.
${lib_prefix}
- current platform’s library prefix. For example lib
or empty string.
${bindings_ext}
- current platform’s bindings extension. For example .pyd
or .so
.
${exe_ext}
- current platform’s executable extension. For example .exe
or empty string.
${shell_ext}
- current platform’s shell extension. For example .bat
or .sh
.
${in_ci}
- Set to True
if running on gitlab or TeamCity, False
otherwise.
${py_ver}
- current Python interpreter major and minor version. For example 3.10.4 => 310
.
${py_tag}
- Python tag per Python packaging guidelines. For example 3.10.4 => py310
.
${omni_data}
- OV data user system path to store persistent data
${omni_cache}
- OV system path to be used for caching
${omni_logs}
- OV system path to store logs
Tokens typically resolved by other tools:#
${config}
- current build config. For example debug
or release
.
${platform_target}
- target platform (can be different from host platform).
Tokens defined in a config:#
New tokens can be defined in any config (repo.toml
or repo_tools.toml
) in [repo.tokens]
section. For example:
[repo.tokens]
foo = 123
bar = "${foo}456"
[some_tool]
x = "${bar}"
Then some_tool.x
will be resolved to 123456
.
Tokens defined via CLI:#
You can also set a token with the --set-token
argument. It takes the form of a colon separated key:value pairing. This works nicely with a default token value defined in your repo.toml
with said token being overridden perhaps programmatically in CI.
Listing all tokens#
For any tool run repo -pt [tool_name]
(or --print-tokens
) to print all tokens known at configuration time.
Interpolations#
Some tokens can take a special form of ${operation:value}
. Where operation
is a custom interpolation to perform on a token:
Environment variables: env
#
${env:PATH}
- value of PATH
environment variable.
${env:DISABLE_CI_MESSAGE_BLOCK}
- If truthy, will silently bypass ci_message_block usage.
File content: file
#
${file:path}
- content of a file at path
. E.g.: "${file:${config_root}/VERSION}"
.
Other config value: conf
#
${conf:path}
- value of other config setting setting at path, e.g. x = "${conf:repo_build.build_configs}"
then x
would have the same value as repo_build.build_configs
setting.
API#
When writing a tool use omni.repo.man.resolve_tokens
to resolve tokens. Use omni.repo.man.set_token(name, value)
to set a new token at runtime. Example:
import omni.repo.man
# Set ${config} to be 'release'
omni.repo.man.set_token("config", "release")
# Now strings with config can be resolved (all other tokens work too):
value = omni.repo.man.resolve_tokens("${platform}-${config}")
print(value) # "windows-x86_64-release"
# dict, list, tuple also supported, but replaced in-place:
value = {"foo": "${platform}-${config}"}
omni.repo.man.resolve_tokens(value)
print(value) # {"foo": "windows-x86_64-release"}
# Token can be removed if None is passed:
omni.repo.man.set_token("config", None)
Filters#
Filters are used to apply settings conditionally. They are defined as a key in a dictionary.
To apply settings only on certain platforms use 'platform:[platform]'
. For example:
[tool_name]
foo.bar = 3
foo."platform:linux-aarch64".bar = 5
This will set foo.bar
to 3
on all platforms except linux-aarch64
where it will be set to 5
.
A whole section (dictionary) of settings can be overriden like that.
Equality, Inequality, and Existence filtering#
This feature has been expanded to support filtering against tokens and envvars. For example:
[repo.docker."token:abi==2.17".images] # Equality Check
registry = "some_registry"
image = "some_image_tag"
[repo.docker."token:abi!=2.17".images] # Inequality Check
registry = "some_other_registry"
"env:DISPLAY".registry = "some_other_registry_that_cares_about_DISPLAY" # Existence Check
image = "some_other_image_tag"
The filter must precede the key to be filtered e.g.:
"env:DISPLAY".some_key = "some_value"
The filter cannot follow the key to be filtered, e.g.:
some_key."env:DISPLAY" = "some_value"
If a filter of this format is encountered a ConfigurationError will be raised.
Filters and Booleans#
Unexpected behaviors can occur around booleans and how token and envvars are set. If a token with value true is set via a repo.toml [repo.tokens], repo_man sees this as the boolean True. But if a token is set via --set-token
with the value True, repo_man sees this as the string True. This is due to limitations of our run-time token setting parser not attempting to differentiate between basic types beyond string.
Because of this limitation, equality and inequality filtering will cast all filters to a string, and then lowercase the resulting string. So the following will evaluate and set repo.docker.images.registry:
[repo.tokens]
test = true
[repo.docker.images]
"token:test==tRuE".registry = "some_registry"
This is an understood limitation that will perhaps be improved in the future if more effort is allocated against filtering.
Array Merging#
Arrays can be merged using ++
key. For example,
repo_tools.toml
:
[tool_name]
foo = [1, 2]
repo.toml
:
[tool_name]
foo."++" = [3, 4]
Will result in tool_name.foo = [1, 2, 3, 4]
.
If array does not exist it will be created.
Importing Extra Configs#
Before applying final repo.toml
extra configs can be merged (and thus used as templates) using a import_configs
or import_optional_configs
key:
[repo]
# Path to additional toml configs merge before applying this one, which must exist
import_configs = [
"${root}/_repo/deps/repo_kit_tools/kit-template/repo.toml",
]
# Path to additional toml configs merge before applying this one, which can not exist
import_optional_configs = [
"/home/user/some_config.toml",
]
Paths and Folders#
Inside of repo_man
’s repo_tools.toml
there is a section [repo.folders]
which defines default (recommended) paths for various standard folders. This way each tool can use them and each repo can override them.
Here are some of them:
# Recommended/Default folder structure:
[repo.folders]
root = "${root}"
build = "${root}/_build"
compiler = "${root}/_compiler"
packages = "${root}/_build/packages"
host_deps = "${root}/_build/host-deps"
target_deps = "${root}/_build/target-deps"
repo_deps = "${root}/_repo/deps"
pip_packages = "${root}/_build/pip-packages"
...
Typically in tools it looks like this:
def run_tool(options: Dict, config: Dict):
repo_folders = config.get("repo", {}).get("folders", {})
# Path to repo root
root_path = repo_folders["root"]
# Path to build folder
build_path = repo_folders["build"]
Command Line Config Overrides#
To override (or add) any config setting from command line use a special syntax: --/[key]=[value]
. Where [key]
is path to a value in config separated by /
:
repo test --/repo_test/tracebacks_on_exit=1
Overrides repo_test.tracebacks_on_exit
setting to true.
If value is already present in config (which is usually the case) type of that value is used to convert to.
If value is not present (new) a reasonable conversion is attempted:
--/foo/bar=1
- Integer--/foo/bar=true
- Bool--/foo/bar=0.5
- Float
Quotes can be used to force string:
--/foo/bar="1"
- String
--/foo/bar='1'
- String
Tokens are supported, e.g.:
--/foo/bar=${root}
- Path to the root of repo
--/foo/bar=${env:HOME}
- HOME
environment variable
etc.
Overriding arrays is not supported.
Use --
to ignore or passthrough cmd arguments#
All command line arguments after --
are ignored. E.g. -- --/foo/bar=2 --help
will not override foo/bar
config setting and will not show help. Instead it strips ["--/foo/bar=2", "--help"]
from sys.argv
and passes them to the invoked tool as options.extra_args
.
This is useful to pass arguments to some other tool or process.
Typical example is repo_test
tool which runs some other test runner under the hood, like doctest
or pytest
. To pass extra arguments to doctest
put them after --
:
repo test --suite unittests -- -tc="*[wildcard]*"
Instead of each tool implemting own way to pass extra arguments (e.g. -e="--foo"
) they can just pass options.extra_args
. That also standardizes the way to pass extra arguments to any tool.
Debugging Configuration#
For any tool run repo -pr [tool_name]
to output resolved configuration.
Using Configuration in tools#
Within tools, Pydantic models are used to provide a validated configuration class.
Each tool should define their own configuration class which inherits from omni.repo.man.BaseRepoToolConfig
.
Nested models can inherit from the default Pydantic BaseModel
class.
In any tool the resolved configuration is passed into the tool run function as a Python dict
.
This can then be used to initialize the tools configuration class:
class Foo(pydantic.BaseModel):
"""A nested model for ToolConfigClass."""
bar: str = Field(..., description="A useful string")
class ToolConfigClass(omni.repo.man.BaseRepoToolConfig, section="tool_toml_section"):
"""A configuration class for a useful tool."""
foo: Foo = Field(..., description="A nested model of type Foo")
def run_tool(options: Dict, config: Dict):
tool_config = ToolConfigClass.from_config(config)
The configuration can then be used like so:
bar = tool_config.foo.bar