Packaging

All packaging and related operations are handled by packman. This repo contains the version of packman you should be using when working with the repo. It can be found under tools/packman. To execute packman simply run from Linux or Windows shell:

tools/packman/packman

from the root of the repo (or change directory to that path). This guide focuses on the specific operations you will be doing with packman in conjunction with this repo. If you want more information please consult the packman repo or use the packman help command, i.e. execute packman help.

NOTE: Never replace a package by overwriting it on the remote server unless you are the only person that has ever used that package. You can overwrite a package on the server by providing the --force argument but only use this when you know that nobody else has consumed the older package. If you cannot be certain of this please just bump the version number. If there is no actual change to the package other than fixes (maybe missed some files first time you packaged) then just add another version number at the end and set it to 1. For example: busted package had version 1.2.3 - your patched package is 1.2.3-1. It should also be noted that we use AWS CloudFront which is a proper CDN so even if you overwrite a package in good faith (because you have been the only consumer of it) it may take the CDN up to 24 hours to flush its cache and pick up your altered package.

Staging dependencies

Before you package your dependency, like a third party library, you should stage it within the packman packages cache. This cache is located in a central place on your system. The environment variable PM_PACKAGES_ROOT will inform you where it is. The best way to stage a dependency is to create a folder under PM_PACKAGES_ROOT that is named after the dependency and then a subfolder in there that is named after the version.

Depending on the nature and size of the dependency you may want to create a universal package, i.e one that works for all OSs and architectures or split it up into three packages, one for each OS and CPU architecture. The choice is yours. The only requirement is that if you create separate packages you specify exactly the platform by including one of

• windows-x86_64

• linux-x86_64

• linux-aarch64

in the package name or version.

Let’s take doctest as an example. This is a universal package so it does not have a platform qualifier in its name. If you were to stage this package for packaging you would have put the contents inside the folder:

$PM_PACKAGES_ROOT/doctest/2.3.5-0b0a374

This assumes the version we are using is 2.3.5-0b0a374. We will elaborate on version numbers a bit later (section 6).

After doing so you can now refer to the dependency in your dependency file. These files are stored under the deps folder in this repo. There is a target-deps if you want to see what they look like. Here is what it could look like:

<project toolsVersion="6.11" chroniclePath="../_build/chronicles">
  <dependency name="doctest" linkPath="../_build/target-deps/doctest">
    <package name="doctest" version="2.3.5-0b0a374" />
  </dependency>
  ...

When packman is run with this updated deps.packman.xml file it will find the package already in your local packman cache and will therefore not look for it on the remote server (where it doesn’t yet exist).

This example is rather trivial because DocTest is a library that works for all platforms. A more complex example would be the Vulkan SDK. This is what the dependency specification looks for Vulkan SDK:

  <dependency name="vulkansdk" linkPath="../_build/target-deps/vulkansdk">
    <package name="vulkansdk" version="1.1.73.0-win" platforms="windows-x86_64" />
    <package name="vulkansdk" version="1.1.73.0-linux-x86_64" platforms="linux-x86_64" />
    <package name="vulkansdk" version="1.1.73.0-linux-aarch64" platforms="linux-aarch64" />
  </dependency>

There are a few things to highlight here. Firstly, we only user lower case letters in package names since packman is case sensitive and so is the Linux file system. Secondly, the packages above were correctly versioned for both Linux architectures but not for Windows - should have been versioned as version="1.1.73.0-windows-x86_64". If that had been the case the dependency could have been simplified to the following:

  <dependency name="vulkansdk" linkPath="../_build/package-links/vulkansdk">
    <package name="vulkansdk" version="1.1.73.0-${platform}" />
  </dependency>

In this case we are taking advantage of packman’s variable substitution capability to insert the platform name into the version. Let’s say that in this example you are working on Linux and targeting linux-x86_64. You have already staged the corresponding Vulkan SDK in $PM_PACKAGES_ROOT/vulkansdk/1.1.73.0-linux-x86_64. When you run the build script to target this platform packman will find the version of the package which you have already staged. This will prevent packman from going out to the remote server and try to find the package there. Instead it will happily move on and perform the folder linking that is instructed by the linkPath attribute in the xml file above. In other words, packman will create a file system link from its internal cache, where you staged the dependency, to the target directory _build/package-links/vulkansdk. This allows you to test your dependency and make sure everything works as expected. Once that is done you can move to the next step.

Licensing

Please read the Package Licensing for packman and follow the steps in there - otherwise your package will fail license validation during the CI build process.

Packaging dependencies

It is assumed you have followed the steps in item 1. If the package is already available on a public HTTP(S) server, like GitHub, and can be consumed directly (zip/7z) there is no need to package it. You can simply configure a HTTP(s) remote server in the packman dependency file. This is for instance done with slang, like this:

  <remote name="github-slang" type="https" packageLocation=
   "github.com/shader-slang/slang/releases/download/v${version}/slang-${version}-win64.zip"/>
  <dependency name="slang" linkPath="../_build/package-links/slang">
    <package name="slang" version="0.10.29" platforms="windows-x86_64" remotes="github-slang" />
  </dependency>

And you are done! You can stop reading this guide.

If this is not the case you need to package your dependency. This is simply done with the following command:

./packman pack <path-to-version-folder>

In the example from item 1 this would be:

./packman pack $PM_PACKAGES_ROOT/doctest/2.3.5-0b0a374

Packman will tell you where the resulting package was placed (for more options on pack command simply run packman pack -h or get the detailed help by running packman help pack).

Pushing dependencies

You now want to push your package to remote storage. This allows others to get your package. Before you can do this you must configure credentials in your packman user settings that allow you to write to remote storage. The packman command will launch this information for you automatically if you attempt upload without proper credentials but here is the direct link. Once you have completed that configuration you can upload this way:

./packman push <path-to-package> --remote mycloudfront

This will upload the package to our CDN that is configured for Carbonite.

NOTE: If you feel really solid about this process you can perform packaging and pushing in one step, using the publish command. Execute packman publish -h to learn more.

Verification

Move the staged version of the dependency to a different location on your system or rename the version folder. When you run the project generation or build process the dependencies will be fetched and written to the packages cache. Note that they go under a ‘chk’ folder inside the PM_PACKAGES_ROOT. This is simply because packman has now generated a checksum for the package during unpacking so it can verified as intact later.

Capturing for traceability and maintenance

For legal reasons we need to capture the software in source control to simplify auditing. This is also beneficial when we are forced to make modifications to the external package, be it source code or build process. This way we can track the changes and reapply them when we upgrade to newer releases of the third party library. We collect these libraries under the omniverse/externals subgroup in GitLab. For historical reasons we also have some captured under Carbonite-externals. The process is straight-forward:

• Create a repo named after the third party library (preferably all in lowercase, matching the package name)

• Seed this repo via a mirror from GitLab.com or GitHub.com or simply with a copy of the source code

• Create a branch where all NVIDIA modifications are made, this branch should have nvidia or nv in the branch name, for example: nvidia and nv-main are both acceptable branch names.

• Often times we need to weed out runtime dependencies that are leaking out of the library, see Architecture for details.

• You must create a build.{bat|sh} script that builds the software, preferably not requiring any manually installed external dependencies (use packman to automatically provision them). You are exempt from this rule if the package is header-only.

• You must create a package.{bat|sh} script that packages the software, ready for pushing to packman remote server.

• Look at imgui repo for a good example of how to do this.

• If you are not using a mirror then upgrading to a newer version of the third party library is simply done by switching to the main branch, deleting all the files (except for the .git folder) and copy in the new snapshot. Commit those changes. Then merge that branch into your nvidia branch and resolve, picking up the changes that you had previously made on the older version.

You will notice from the above that we capture the packaging script as well in the repo and this makes it easier for others to produce a new version of the package after updating the repository.