Repo Build#

omni.repo.build is a library and tool that is designed to generate projects and run a build toolchain, supporting cross compilation.

Description#

Two major steps are performed by this tool, as mentioned above.

  1. Generating projects

  2. Running a build toolchain

The Generate projects step utilizes the tool premake. It is possible to optionally provide premake description file. Specifying path to the premake tool is required.

Currently supported toolchains include:

  • MSVC for windows-x86_64 host platform, via MSBuild as included in Visual Studio installations.

  • gcc for linux-x86_64 and linux-aarch64 host platforms

By default, both the release and debug configurations are built, but it is possible to select only one if the relevant command line options are specified. Using the MSBuild toolchain also requires specifying additional parameters via the command line, such as toolchain path and path to a Visual Studio solution file.

There are a number of additional command line arguments that can be specified, such as build verbosity, number of jobs spawned by the toolchain, and repository root.

Prebuild file generation#

Examples in premake5.lua:

local repo_build = require("omni/repo/build")

-- One entry
repo_build.prebuild_copy("source/extensions/foo/*.py", "_build/$platform/$config/extensions/foo")

-- Multiple entries
repo_build.prebuild_link {
    { "source/extensions/foo", "_build/$platform/$config/extensions/foo" },
    { "source/extensions/bar", "_build/$platform/$config/extensions/bar" },
}

-- Only on linux debug:
repo_build.prebuild_copy("source/extensions/car", "_build/$platform/$config/", "linux.debug")

-- Change default prebuild.toml target file. Default is: "_build/generated/prebuild.toml"
repo_build.set_prebuild_file("foo/prebuild.toml")

Testing local changes for projects#

Let’s say that you are working in a repo_build-enabled repository, and are working on a secondary project that is built locally, and is a dependency of the current project.

You may want to temporarily link the dependency into the current project. repo-tool enabled repositories have a way to accomplish this.

It should even be possible to test a project against a changed local copy of repo_build itself.

Easy Method#

repo_source is a companion tool to repo_build and be added to any project relying on repo_man, by updating the appropriate packman dependency file in the repository.

If repo_source is present, the following would be able to link the dependency into the current project:

# Windows
cd C:\projects\current_project
repo.bat source link dependency_project C:\projects\dependency_project
# Linux
cd /path/projects/current_project
repo.sh source link dependency_project /path/projects/dependency_project

When done testing:

# Windows
repo.bat source unlink dependency_project
# Linux
repo.sh source unlink dependency_project

Manual Method#

It’s still possible to accomplish the same effect without repo_source.

You can manually set up a link to your local dependency. Find the matching packman dependency file that expresses your dependency, for example:

<project toolsVersion="5.0">
  <dependency name="dependency" linkPath="../_build/deps/dependency">
    <package name="dependency" version="0.1.0" />
  </dependency>
</project>

Create a matching “.user” file, such that the filepath and name is the same, but with extra “.user” on the end of the path, for example:

C:\projects\current_project\deps\target-deps.packman.xml.user

Then, put something like the following in (example using Windows path, use Linux path on Linux):

<project toolsVersion="5.6">
  <dependency name="dependency" linkPath="../_build/deps/dependency">
    <source path="C:\projects\dependency\" />
  </dependency>
</project>

Important to note is that the linkPath is the same, so that the source path is linked to where the dependency would normally be linked to.

Note that the .user file extension on target-deps.packman.xml.user. The .user file should not be checked in and is only used for local testing.

Once the .user files has been modified, running your build in C:\projects\current_project should rebuild the project using the local copy of that dependency

When done testing, simply delete the .user file and rebuild.

Versioning Changes#

User visible changes should be given a new version before publishing. The version number can be updated as follows:

  1. Update CHANGELOG.md with a description of the changes in the new version.

  2. Update VERSION / VERSION.md with the correct version number.

Publishing Changes (Currently NVIDIA only)#

Changes can be published as follows:

  1. Ensure the packaging system, packman, knows where to publish the package. Instructions can be found in packman’s documentation, “How to Configure S3 Cloudfront Full Access” section.

  2. Ensure you have updated the package’s version number (see Versioning Changes above).

  3. Create a merge request and have it approved.

  4. Create and upload your updated package with:

       repo upload

Consuming an Updated Package#

To update the version of a package consumed, make the package available via a publishing mechanism, and update the version tag in the related packman xml dependency file. Make sure to remove or modify any existing .user files that may have been used for testing or local validation.

Let’s say we wish to update current project (e.g. C:\projects\current_project) to use the latest version of repo_build (e.g. 0.62.3). After publishing version 0.62.3 of the package, consume the package in current_project with:

  1. Removing the repo_build dependency tag in C:\projects\projectname\deps\repo-deps.packman.xml.user that you added during testing/validation/development. Alternatively, you can simply delete this file if it isn’t being used.

  2. Update the repo_build package tag’s version number. In this instance, change it to 0.62.3:

      <dependency name="repo_build" linkPath="../_build/repo-deps/repo_build">
    <package name="repo_build" version="0.62.3" />
  </dependency>

  3. Build, run tests, send out a merge request, profit, etc.

ccache Info#

ccache can be automatically pulled for Windows, Linux, and MacOS platforms if --cache is specified to repo_build.

The CARB_CCACHE environment variable can also be set to enable the ccache functionality.

You’ll also need to enable the following values in your repo.toml (see repo_tools.toml for other values):

[repo_build.distributed_build_services]
enabled = false
"platform:linux-x86_64".cache_dir = "/tmp/ccache"  (or other location)
"platform:linux-aarch64".cache_dir = "/tmp/ccache"  (or other location)
"platform:linux-macos-univeral".cache_dir = "/tmp/ccache"  (or other location)
"platform:windows-x86_64".cache_dir = "C:\\ccache"  (or other location)
max_cache_size = "10GiB" (or other value)

Clang Support#

Starting with version 0.70.0, clang is now available on linux. You can enable clang through your repo.toml:

[repo_build.build.compiler]
compiler = "clang"

In addition, you can pass the clang option in via the command line, --compiler-toolset=clang.

Notes:

  • Windows (MSVC) has the restriction that PDBs cannot be generated by the compiler, so the following rules be observed to maximize your cacheable code:

    • Do not use “NoIncrementalLink”

    • Do not use the editandcontinue premake/MSVC option

    • Must use: debugformat "c7" for Debug builds

    • Must use: symbols "On", not "Full"

  • To check if the project is missing cache hits:

    • Go to your ccache binary path

      • Linux: type CCACHE_DIR=<path_to_cache_folder> ccache -s

      • Windows:

        • type set CCACHE_DIR=<path_to_cache_folder>

        • type ccache.exe -s

Build Metadata File#

repo_build will now emit a metadata file using the repo_man config option repo_man.folders.build_metadata which defaults to ${root}/_build/build_metadata.json. Currently this metadata file contains a json blob with the keys target_glibc, build_glibc, build_os, and build_arch:

  • target_glibc: the version of glibc the user desires. Typically set via the --abi CLI arg in the form of major.minor version e.g.: 2.35.

  • build_glibc: the version of glibc present on the build host. This can mismatch target_glibc if linbuild is not used or is not correctly setup to map glibc target versions with correct build containers.

  • build_os: Values of: windows, linux, macos

  • build_arch: Values of: x86_86, aarch64

Licensing#

This project is licensed with the MIT license. While we welcome your input and suggestions, we are not accepting contributions from the public at this time.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use. (See requirements.txt)

On this page