Omniverse Wrapp Internal Change Log

1.1.0

1.1.0-b4

Changes in 1.1.0-b4

  • Added the same comment as in the changelog to the main documentation page, to make people aware of the redist installation requirement.

1.1.0-b3

Changes in 1.1.0-b3

  • Windows only: Removed the Microsoft Visual Studio redistributable files from the wheel and the standalone executable. Those files are to be found in any Microsoft Windows installation and do not need to be shipped. In case they are missing on a specific target system, download vcredist_x64.exe from https://aka.ms/vs/17/release/vc_redist.x64.exe and run the installer.

1.1.0-b2

Changes in 1.1.0-b2

  • Removed misleading version number for WRAPP internal S3 Storage.

  • Update to tagging client 3.1.26+3.1.26.gitlab.17325333.3274fee5

  • Fix pipeline for docs for internal Launcher release.

1.1.0-b1

Changes in 1.1.0-b1

  • OMPE-16858: handle exceptions raised during storage initialization more gracefully.

  • OMPE-16409: make sure WRAPP can copy files >5GB on S3.

  • OMPE-16046: make sure install, mirror and apply-patch do not ignore errors during patch execution.

  • OMPE-14390: make sure all commands supporting “–tags” work/fail as expected for storages not supporting tags.

  • OMPE-15320: make sure unsupported operations going from S3 to Nucleus or vice versa fail quickly with a resonable error message.

  • OMPE-17215: make sure apply-patch going from S3 to Nucleus or vice versa fail quickly with a resonable error message.

  • OMPE-16499: adopt create/freeze to new progress report/scheduler.

  • OMPE-17485: fix bucket A to bucket B copy operations.

  • OMPE-17417: make sure create prints unexpected error messages immediately.

  • Update to Client Library 2.49.2-hotfix.5255+gl.d6762ea1

  • Update to tagging client 3.1.12+3.1.25.gitlab.17086212.0190dff0

  • Update to discovery client 1.5.3+1.5.3-rc.5.gitlab.1154.e7388619

  • Updated to Python 3.10.14+nv2 from 3.10.14+nv1

API Changes in 1.1.0-b1

  • The API initialize function is now documented to possibly raise a RuntimeError when initialization fails.

    • Before it might have thrown any exception.

1.1.0-a3

  • Fix bug which did not allow WRAPP to start up if WRAPP_S3_ENDPOINT was not set.

  • Removed the libuuid.so.1 file from the Linux build, which is part the util-linux package. That dylib should be on the target system already.

1.1.0-a2

Changes in 1.1.0-a2

  • OMPE-15037: Cataloging is slow, fixed by not recreating the S3 connection. Added env variables WRAPP_S3_CONCURRENT_THREADS, which defaults to 24, and WRAPP_S3_CONNECT_TIMEOUT_S, which defaults to 5 seconds.

  • OMPE-15729: installing a package into a non-existing S3 bucket gives an exception instead of an error.

  • Make order of execution in Patch robust: add_tag has to be executed after the copy commands have been completed, same for delete folder.

  • Introduce progress report flag and additional options for number of tagging and file transfer jobs for install, mirror and apply-patch.

  • OMPE-15326 do not print “Downloading item …” when the file is read from local disk.

  • OMPE-16494 fix incompatibility problem between boto3 and minio causing MissingHeaderBodySeparatorDefect error messages.

  • OMPE-16496 increase number of connections in connection pool for boto3 to avoid “connection pool is full” error messages.

1.1.0-a1

Changes in 1.1.0-a1

  • Allow using the local file system as a package repository.

1.1.0-dev9

Changes in 1.1.0-dev9

  • Improve performance when iterating over S3 directories by using a recursive walk.

  • Fix copying files from S3 location to S3 location.

  • Catalogs uploaded to S3 are now using no longer etags as hashes, but Nucleus-style SHA256.

  • Catalogs for S3 no longer use a “/” at the end of folder paths.

    • This makes the catalogs compatible with the catalogs created through the client-library backend.

  • The S3 backend no longer uses aioboto3, instead it uses boto3 directly with a ThreadPoolExecutor.

    • This also fixes the issue that the wrapp executable was missing the aioboto3 and aiofiles module.

  • Fix the “ls” command for the root of a bucket.

  • Allow accessing S3 buckets using “https” links.

    • This change will for now route all “https” links to the S3 backend, and disable the ability to use client-library for those URLs.

    • For a later release, we are looking at adding a flag so the user can explicitly select the backend WRAPP uses.

  • Make sure connections to the tagging service for Nucleus are properly closed.

  • OMPE-15135: freeze would not create local file system folders when copying from S3

1.1.0-dev6

API Changes in 1.1.0-dev6:

  • The “get” API function used to download a file to the local disk is now accepting the optional parameters “context” and “scheduler”, like the other API functions.

  • The credentials used to sign in to S3 can be set using WRAPP_AWS_ACCESS_KEY_ID and WRAPP_AWS_SECRET_ACCESS_KEY, the endpoint can be set with WRAPP_S3_ENDPOINT.

    • If the variables are not set, boto3 tries to locate the credential in the usual way.

Changes in 1.1.0-dev6

  • Support upload to S3 buckets using “s3://…” URLs.

  • Update pyinstaller to 6.9.0

1.0.1

API changes in 1.0.1

  • Added AuthManager.get_last_status

Changes in 1.0.1

  • Update to Client Library 2.49.0-release.5347+tc.8d02373e

  • Update to tagging client 3.1.11+main.gitlab.14447049.09cf0298

  • Update to discovery client 1.5.2+1.5.2.gitlab.1075.e696a5d5

  • Updated to Python 3.10.14+nv1 from 3.10.13+nv3

  • Documentation update

  • Update to aiohttp 3.9.3, closing BDSA-2024-0214

  • Rename RELEASE_NOTES.md to INTERNAL_CHANGELOG.md

  • Detect expired auth token for verify tests.

  • Testing against Kit 106.0.0+release.150345.08f4f14b instead of Kit 105

  • Linux-binaries built on Ubuntu 20.04 LTS instead of CentOS 7, which is EOL

1.0.0

Changes in 1.0.0

  • Swapped CHANGELOG.md and RELEASE_NOTES.md

0.11.0

API changes in 0.11.0

  • OM-117593 We are renaming “remotes” to “repository”. This had a long discussion, but we think the name remote is too laden with git-connotation, while we think about it more as a folder/place for packages to be stored, an artifact store or repository. We have renamed all API parameters from remote to repo, so the API calls supplying this as a named parameter need to be changed.

  • We have kept the --remote CLI switch for now, but added a synonymous --repo switch, and the short version -r doesn’t change anyway. At some point we might remove the --remote, so please consider that deprecated. Same for the environment variables, we added WRAPP_REPO with the same meaning as WRAPP_REMOTE, please consider the latter as deprecated.

  • OM-117176 The ls command now supports all regular command line switches like --auth and --stats. The API command now returns a list of CatalogItem instances instead of omni.client.ListEntry

Changes in 0.11.0

  • OM-117922 Update to client-library 2.43.0

  • OM-114826 Use new client-library function “open_cached_file” instead of deprecated “get_local_file” for improved Hub cache garbage collection.

  • OM-117458 Use new client-library feature to use only single deletes, preventing the possibility of removing non-empty folders recursively on error.

  • OM-115687 Fix update install not ignoring top level wrapp file, which always changes. This prevented some downgrade installs

  • Disabled the ‘install’ command’s flag --resolve-conflicts for now, as the conflict resolution offered was not really clear and needs a bit more work.

0.10.0

API changes in 0.10.0

  • OM-116620 We added api methods for initialize and shutdown, as well as a simplifying wrapp.api.ContextManager

  • CommandContext is now called SchedulerContext

  • The context parameter in the Python API functions is now of type CommandParameters

  • There is a new scheduler keyword-only parameter in the API functions that can be used to hand in a pre-created scheduler, boosting performance for the case of multiple API calls in one program.

  • We greatly extended the documentation providing example programs on how to run a simple script, as well as how to embed wrapp and the async mechanics required into a larger program.

Changes in 0.10.0

  • OM-116602 Fixed some error messages which listed an exception instead of a proper description. Additional tests written to harden the CLI interface.

  • OM-117588 Fixed bug that would not allow installing from multiple remotes into the same top level package.

  • OM-116887 Give proper error message when running create with insufficient permissions to package repository

0.9.3

Re-release 0.9.2 to Pypi/PIP because of failed pipeline.

0.9.2

  • OM-117170 Fixed critical bug in uninstall that could remove a full folder tree while uninstalling a package in “non-dependency” mode, i.e. if no --package parameter was given to uninstall. This happened when the package was overlapping with existing data at least two levels deep, and caused by a failed check for empty folder.

  • OM-117372 Fix bug not collecting files within a mixed file/subpackage structure during create

  • OM-109662 Many persistent parameters like --auth and --remote now additionally can be filled by an environment variable like WRAPP_AUTH or WRAPP_REMOTE. The online help --help for each command gives details.

  • OM-117091 Adding a simple ls command that can be used to produce a recursive file list in tab separated format suitable for catalog operations of subsets of files with the --file-list switch of catalog.

  • OM-116391 Verification was hidden in the help as it is not yet included in the beta release.

  • OM-117088 Verify got a new parameter to make it create a file list tsv file for consumption by follow-up catalog command

  • OM-114827 Better exception logging. –debug will now make a full dump, while normal mode is to log to console and to file

0.9.1

  • OM-116938 Fix misleading message “Creating a fresh scheduler…” and remove progress report on commands where it was confusing

0.9.0

API changes

  • CommandContext moved from wrapp.app.CommandContext to wrapp.CommandContext.

  • The “context” parameter in the Python API functions is now a keyword-only parameter.

  • The “file-transfer-jobs” option now uses a default of 50 - which increases performance significantly for S3 downloads.

Changes

  • wrapp in beta channel now

  • Update to ClientLibrary 2.42.0.

  • OM-92317 Uninstall now works with nested packages.

  • OM-114832 Rename the stats output to make clear we are counting client library calls only, we do not know how many Nucleus server round-trips these actually represent if any.

  • OM-104866 add –auth parameter to all commands to allow authenticating against servers

  • OM-114343 Help is now available both as a command (wrapp help) as well as a single line option (-h). Additionally, the wrapp shell ignores it if you paste wrapp into it and will silently drop the superfluous wrapp statement.

  • OM-114924 Reworking all command line parameters to have sensible single letter short name where use is expected, and double dash abbreviations where typing is tedious. E.g. –hash-cache-file now is –hcf a well.

  • OM-115158 Deprecate source_url field in PackageInfo/wrapp file

  • OM-113347 “wrapp version” displays now the Hub version

  • OM-115335 Clean up task queue on Ctrl+C.

    • It is strongly recommended to create a Context class, and to call context.run_scheduler() before calling any Python API functions.

    • The “context” parameter in the Python API functions is now a keyword-only parameter.

  • OM-116904 Fix scheduler handling when the same API function is called several times without context parameter.

0.8.2

  • OM-115634 Fix missing _version.py file from wheel file uploaded to Artifactory, causing the pip installed version to fail (0.8.0 and 0.8.1 are missing this file)

0.8.1

  • Update to ClientLibrary 2.41.0 and test against Hub 0.33.0

  • Update to include Discovery Service client 1.5.0 and Tagging Service client 3.1.7

  • OM-102990 Remove the pull command. At the current point, it was unclear what it should do. Install does everything needed.

  • OM-105691, OM-115071

    • Replace “–memory-mode” command line parameter and parameter from import and export commands/functions with internal logic.

    • Make sure the catalog command only downloads file to the local client if the Hub cache is available.

  • OM-115110 reduce memory usage for long-running operations.

  • OM-115254 fix occasionally flickering tables in progress report

  • OM-115399 allow relative paths for the export “output” parameter. Improve logging for export command.

  • OM-112201 fix create-patch command does not handle “extra” files

  • OM-115437 improve output of import command with “–verbose”

0.8.0

  • OM-105691, OM-105693

    • fix S3 downloads.

    • optimize job scheduling.

    • Introduce progress report.

      • The wrapp commands catalog, import and export now display their on-going progress.

      • The verbosity of the progress report can be controlled with the new “–progress-report” option.

  • OM-107030 (partially) Use multi-threading for the hash calculation. Improves hash calculation performance in some cases.

  • OM-113180 Build a launcher package that can be installed via Launcher and allows to launch a command line with wrapp activated.

  • OM-108255 Changed the way we auto-ignore thumbnails created by the thumbnail service. We now only rely on the file name, and ignore all files ending on “auto.png”. The name of the user creating these files is no longer checked. This harmonized ignoring files from catalogs and from file systems, but required a breaking API change to the freeze_catalog() method.

  • OM-94537 Uninstall of files that overlap - have a file or folder both in their catalog installed into the same position - now can correctly identify which objects to keep on uninstall of one of the two packages if the dependency/top-level package is specified with the --package switch.

0.7.5

  • Bugfix - pytest-timeout was not required as a distribution dependency, just as a development dependency. Removed the requirement.

0.7.4

  • OM-112811 Do not use the filename of the tar file during “import” as the root catalog name. Disallow “export –catalog” for nested packages (catalogs containing .wrapp files).

  • OM-113201 Locking the Python version required to run wrapp to 3.10, as this is the version used by kit as of now

  • Fallback to implicit --local-hash during catalog operation when the server returns not supported.

  • Updated to client library 2.40.1. This turns default usage of Hub back on.

  • Changed to run the tests always against the Hub version referenced by client library. In this case, this is Hub 0.31.0-beta-0.

  • Our distribution artifactory changed from nv-shared-pypi-local to ct-omniverse-pypi-local, so the download URLs for pip installation have changed

0.7.3

  • Updated to client library 2.38.8-hotfix.5087+tc.3a93fb6e

  • Hub usage is turned off by default. You can turn it on by setting OMNICLIENT_USE_HUB=1 environment variable, and using an omni hub alpha build. Recommended Hub version is 0.20.4-alpha-3. Versions 0.30.x will not work.

0.7.2, 0.7.1, 0.7.0

Three identical builds which differ only in the NSPECT artifactory malware scan number.

  • Updated to client library 2.37.1-hotfix

  • OM-106994 we’re now publishing the latest wrapp documentation for the master branch to an internal page at https://omniverse.gitlab-master-pages.nvidia.com/wrapp/

  • OM-101027 A failed install will now write a patch file with the conflicts that caused the install failure if the --patch parameter is given. This file can be inspected, edited, and then applied manually with the wrapp apply-patch command to resolve any install conflicts.

  • OM-105814 Export can now optionally create tar files which deduplicate content. The normal mode still is a verbatim file tree suitable for extraction via tar -x in addition to wrapp import. In addition, wrapp export now takes an extra -dedup parameter that stores files with a content hashed based filename inside the tarfile. Import appropriately

  • OM-110381 Freeze now accepts non-checkpointed sources, replacing it with calculating the hashes locally if the storage does not provide hashes. This can be used for the new --catalog parameter for the export command, which allows to create a package tar directly from a catalogued source, which could be also an S3 bucket.

  • Python API change: All commands are now also available directly in the wrapp namespace, no need to say e.g. wrapp.commands.catalog but rather wrapp.catalog suffices.

  • OM-110492 All binaries published via Artifactory are undergoing malware scanning now

  • Fixed command context creation by adding a context parameter to the wrapp Python API functions

0.6.3

  • OM-100081 Instead of print now all output is logged via logging.info/.debug/.error. Specifying –verbose enables info level logging, specifying –debug enables full trace logging including ClientLibrary logs. Added more verbose logging to tagging service calls.

  • OM-101027 Install got a -dry-run parameter to check if an install would work. No output indicates no problems. To see what would happen add –verbose.

  • Bugfix - default loading of .wrappignore did not work when no –ignore-file was specified in some cases

  • Bugfix - on creation of nested packages, sometimes the files were included into the top level pacakge even though a dependency was registered.

  • Bugfix - drop the absolute install path from the dependency information, this fixes nested installs. Note the nesting feature is still alpha.

0.6.2

Build containing all changes now released in 0.6.3, but broken on Windows. Linux 0.6.2 works and should be identical to 0.6.3.

0.6.1

  • OM-104340 Implements a hash cache for local files and S3/mounts. Specify –hash-cache-file during catalog operation to create or use it

  • OM-104212 The catalog operation got a new parameter –no-checkpoints that allows to catalog a Nucleus project without requiring or even querying checkpoints. This catalogs the HEAD revision, which is suited for diff operations but not for create operations.

  • Automatic ignoring of thumbnails removes most need for an ignore file - .auto.png is now always ignored when created by user thumbnails_service

  • Apply-patch now also de-duplicates on upload when hash is known and available on target server

  • Nicer print formatting for diff’s –show command, adding –show –verbose to show all details

  • OM-104413 Fix for verify of MDL references. Verify does not report standard MDL modules as external references anymore

  • Bugfix: Verbose switch did not work as expected

0.6.0

  • OM-92317 Nesting of packages is supported, initial version

  • OM-92314 Verify command parses USD files to find external files outside of package

  • OM-104216 Extract API methods to use wrapp by Python scripts instead of command line

  • Added extract_catalog command to easily extract the catalog of a package to supply to diff or create_patch

  • Bugfix for ‘new’ command’s parameters

  • Bugfix to allow recording two dependencies to the same package in different relative locations

  • Added repo tool to allow repo docs build our documentation

0.5.2

  • OM-104211 Fix ignoring channels and objects properly during catalog and freeze operations

  • Updated to ClientLibrary release 2.35

  • Added the command count that takes a catalog as input and prints some statistics on data usage on disk and deduplication

  • Diff now has better summary output showing the number of unique files as well. Also, it can now diff content of two .wrapp files’ catalogs.

0.5.1

  • Fix install to allow for an -ignore-file parameter

0.5.0

  • OM-101032, OM-99367 Freeze will now try to create_with_hash before uploading anything. This can accelerate uploads during server to server copy

  • OM-94010 Catalog can be enabled to calculate the Nucleus hashes on local files or files it needs to download using the –local-hash option

  • OM-101030 Catalog now takes a csv file list as input as well using the –file-list option

  • OM-101028 Installing multiple packages into the same directory now is possible, but only one version per package of course

  • OM-98183 Shortcut for re-releasing the same package with a new version number, just specify –new-version on create

  • OM-101031 Remove TOML format in favor of using JSON everywhere. As the catalog format was JSON already and the catalog got integrated into the package file, adding a json list to a TOML file did not produce a readable file format.