Docker

General Notes

Almost all aspects of Omniverse Nucleus Docker stack are documented via it’s settings file (.env file) included in the stack tarball we provide. We try to keep our documentation on settings and options as close to the “code” as possible here.

That file should be self explanatory, with settings and comments talking about what they do.

This document should be considered an addendum - broad strokes on general approaches.

Monitoring

Monitoring your install is imperative to understand the general health of the system and if more resources are necessary. At a minimum, one should monitor:

  • Disk space

  • CPU and LA

  • Memory

Additionally, Nucleus stack itself exposes quite a few metrics about it’s load characteristics (such as amount of requests per user, per request type, etc). We recommend to take advantage of these metrics. We expose them to be consumable by Prometheus. As usual, the port for metrics can be found in the Stack’s .env file.

Data Management

Backups

Nucleus’s data directory can be backed up “hot”. To restore it, you have to shut down the stack, restore the directory, and start the stack back up.

Data is segregated into subdirs, and individual subdirs pertaining to specific services can be backed up and restored separately. Subdirs’ names should be self-explanatory.

Logs

Logs are text files and should be self-evident, with one major note: live logs (files that are being appended to by services) in general are not externally rotateable. However, our stack includes rotation and archival sidecars in it, and you can certainly blow away archives with no ill effects (aside from losing log data, of course).

Conversions and Upgrades

When changing your version of Nucleus, please bear in mind the following:

  • While we try our best, we currently do not guarantee data level compatibility between versions of Nucleus, especially between major revisions of the Core. For example, updating from 109.7 to 109.10 would be less risky than updating from 109.7 to 110.0

  • Nucleus data is only “forward compatible”, meaning that you can never go “back” and use an older version of Nucleus “over” data that was opened even once by a newer version of Nucleus. This is a strict clause. No matter if any changes to data were made, if a Nucleus Core process did start up “over” some data, that data is now not consumable by any Nucleus of a lower version

  • If deciding to bite the bullet and upgrade the version of Nucleus “over” old data, make sure to make a backup (or volume snapshot, or similar - do not confuse with Nucleus Snapshots!) of it, and confirm that everything works before making the change final

In our infrastructure, we practice “blue-green” type deployments when moving a major version - we would deploy a new version alongside the current production instance using data backup of the production instance, and confirm it’s health and data consistency extensively. We would then temporarily firewall the old version instance, do a final data sync, and move our DNS alias representing that instance to point to the setup with the new version running.

Authentication

Covered in Authentication and User Registration document.

Omniverse Navigator

Covered in Omniverse Navigator document.