Carbonite Tokens Interface#

Overview#

The Carbonite Tokens interface (carb::tokens::ITokens) provides a powerful system for storing and resolving string tokens throughout the Carbonite framework. Tokens are string pairs {name, value} that can be referenced in strings using the syntax "${token_name}", where the token name starts with the sequence "${" and ends with the first closing "}".

The tokens interface integrates seamlessly with the carb::settings::ISettings interface, storing tokens under the /app/tokens/ settings branch when available. This allows tokens to be configured through configuration files, command-line arguments, or programmatically at runtime.

The token system is fundamental to Carbonite applications, providing configuration flexibility and allowing for dynamic path resolution, environment-specific values, and hierarchical configuration management.

Token Syntax and Resolution#

Basic Token Syntax#

Tokens are referenced in strings using the ${token_name} syntax:

// Example token usage
std::string path = "${kit}/resources/icons/myicon.png";
std::string resolved = carb::tokens::resolveString(tokens, path);
// Results in something like: "C:/Program Files/Kit/resources/icons/myicon.png"

Special Characters and Escaping#

The $ symbol is considered special by the tokenizer and should be escaped by doubling it when you want to include a literal $ character:

// To include literal $ characters:
"some text with $ sign" should be specified as "some text with $ sign"

// Resolution examples:
"$"   becomes "$"     // Single unescaped $ becomes literal $
"$"  becomes "$"     // Doubled $ becomes single literal $
"$$" becomes "$"    // Three $ becomes two literal $

Single unescaped $ signs are acceptable and will be resolved to single $ signs without warnings, but doubling them is the recommended practice.

Complex Token Names#

Token names can contain $ and { characters, but this may result in warnings:

// This will work but generate a warning:
"${bar${}"  // Token name becomes "bar${"

To avoid tokenization entirely, use the escapeString() utility function from TokensUtils.h.

Recursive Token Resolution#

One of the most powerful features of the tokens interface is recursive resolution. Tokens can reference other tokens, and the system will resolve them recursively until all tokens are expanded or an error occurs.

Recursive Resolution Example#

carb::tokens::ITokens* tokens = carb::getCachedInterface<carb::tokens::ITokens>();

// Set up a chain of token dependencies
tokens->setValueS("a", "${b}");
tokens->setValueS("b", "${c}");
tokens->setValueS("c", "hello ${d}");
tokens->setValueS("d", "world");

// Resolve the chain
std::string result = carb::tokens::resolveString(tokens, "${a}");
// result will be "hello world"

Recursion Depth Limits#

The token resolution system protects against infinite recursion by limiting the recursion depth. If the maximum recursion depth is exceeded, the resolution will fail with a ResolveResult::eFailure. The maximum recursion depth is 100 recursive references.

Circular Dependencies#

Circular token dependencies will cause resolution to fail when the recursion depth limit is reached:

tokens->setValueS("circular1", "${circular2}");
tokens->setValueS("circular2", "${circular1}");
// Attempting to resolve "${circular1}" will fail due to infinite recursion

Environment Variable Integration#

Environment variables are automatically available as tokens using the special env: prefix:

// Access environment variables as tokens
std::string path = "${env:HOME}/myapp/config";
std::string resolved = carb::tokens::resolveString(tokens, path);

Environment Variable Features#

Environment variables are accessed with ${env:<var_name>} syntax
The env: prefix is reserved and cannot be used for regular tokens
Environment variables are read when needed and not cached
Undefined environment variables behave as undefined tokens (resolve to empty string)
Calls to setValue() or setInitialValue() with names starting with env: will be rejected

Code Examples#

Adding a New Token#

#include <carb/tokens/ITokens.h>
#include <carb/Framework.h>

// Get the tokens interface
carb::tokens::ITokens* tokens = carb::getCachedInterface<carb::tokens::ITokens>();

// Add a new token (preferred method using string views)
tokens->setValueS("my_app_path", "/path/to/my/application");

// Add a token only if it doesn't exist (initial value)
tokens->setInitialValueS("default_timeout", "30");

// Add a token with the legacy interface (deprecated but still available)
tokens->setValue("legacy_token", "some_value");

Resolving Tokens in a String#

#include <carb/tokens/TokensUtils.h>

// Method 1: Using the utility function (recommended)
std::string input = "Config file: ${my_app_path}/config/${config}.toml";
std::string resolved = carb::tokens::resolveString(tokens, input);

// Method 2: Using the interface directly with pre-allocated buffer
carb::tokens::ResolveResult result;
size_t bufferSize = tokens->calculateDestinationBufferSizeS(
    input, carb::tokens::StringEndingMode::eNullTerminator, 
    carb::tokens::kResolveFlagNone, &result);

if (result == carb::tokens::ResolveResult::eSuccess && bufferSize > 0)
{
    std::string output;
    output.resize(bufferSize - 1); // -1 for null terminator
    
    result = tokens->resolveString(input.c_str(), input.length(),
                                   &output[0], output.size(),
                                   carb::tokens::StringEndingMode::eNoNullTerminator,
                                   carb::tokens::kResolveFlagNone, nullptr);
}

// Method 3: Using the modern string view interface
omni::string resolvedString;
result = tokens->resolveStringS(input, resolvedString, carb::tokens::kResolveFlagNone);

Working with Resolution Flags#

// Default resolution: undefined tokens become empty strings
std::string result1 = carb::tokens::resolveString(tokens, "Path: ${undefined_token}/end");
// result1 = "Path: /end"

// Leave undefined tokens as-is
carb::tokens::ResolveFlags flags = carb::tokens::kResolveFlagLeaveTokenIfNotFound;
std::string result2 = carb::tokens::resolveString(tokens, "Path: ${undefined_token}/end", flags);
// result2 = "Path: ${undefined_token}/end"

Managing Token Lifecycle#

// Check if a token exists
if (tokens->existsS("my_token"))
{
    // Token exists, safe to use
}

// Delete a token
tokens->deleteValueS("temporary_token");

// Using the escapeString utility to prevent tokenization
std::string userInput = "User entered: $weird{stuff}";
std::string escaped = carb::tokens::escapeString(userInput);
// escaped = "User entered: $weird{stuff}"

Built-in Tokens#

Kit-kernel automatically registers numerous built-in tokens during startup. These tokens provide access to important paths, platform information, and runtime configuration.

Path Tokens#

Token	Description	Example Value
`kit`	Kit installation directory	`C:/Program Files/Kit`
`app`	Application/experience directory	`C:/MyApp`
`drive`	Root drive of the kit installation	`C:`
`resources`	Kit resources directory	`${kit}/resources`
`fonts`	Fonts directory	`${kit}/resources/fonts`
`glyphs`	Glyphs directory	`${kit}/resources/glyphs`
`icons`	Icons directory	`${kit}/resources/icons`
`sounds`	Sounds directory	`${kit}/resources/sounds`
`models`	Models directory	`${kit}/resources/models`
`materials`	Materials directory	`${kit}/resources/materials`

Omniverse Path Tokens#

Token	Description	Example Value
`omni_config`	User configuration directory	`~/Documents/Kit/config`
`omni_data`	User data directory	`~/Documents/Kit/data`
`omni_logs`	Logs directory	`~/Documents/Kit/logs`
`omni_cache`	Cache directory	`~/Documents/Kit/cache`
`omni_global_config`	Global configuration directory	`C:/ProgramData/NVIDIA/Kit/config`
`omni_global_data`	Global data directory	`C:/ProgramData/NVIDIA/Kit/data`
`omni_global_logs`	Global logs directory	`C:/ProgramData/NVIDIA/Kit/logs`
`omni_global_cache`	Global cache directory	`C:/ProgramData/NVIDIA/Kit/cache`
`omni_documents`	Documents directory	`~/Documents`
`shared_documents`	Shared documents directory	`~/Documents/Kit/shared`
`omni_program_data`	Program data directory	`C:/ProgramData`
`shared_program_data`	Shared program data directory	`C:/ProgramData/NVIDIA Corporation/kit`
`app_program_data`	App-specific program data directory	`C:/ProgramData/NVIDIA Corporation/kit/MyApp`

Platform and Build Tokens#

Token	Description	Example Value
`platform`	Current platform	`windows-x86_64`, `linux-x86_64`
`config`	Build configuration	`debug`, `release`
`lib_ext`	Library file extension	`.dll`, `.so`
`lib_prefix`	Library file prefix	`""` (Windows), `lib` (Linux)
`exe_ext`	Executable file extension	`.exe` (Windows), `""` (Linux)
`bindings_ext`	Python bindings extension	`.pyd` (Windows), `.so` (Linux)
`shell_ext`	Shell script extension	`.bat` (Windows), `.sh` (Linux)

Runtime Tokens#

Token	Description	Example Value
`temp`	Temporary directory	`/tmp/kit_temp_12345`
`python_version`	Python version	`3.10`
`python`	Python executable path	`${kit}/python/python.exe`
`kit_version`	Kit version string	`106.0.1`
`session_id`	Telemetry session ID	`123456789`
`exe-path`	Executable directory path	`C:/Program Files/Kit`
`exe-filename`	Executable filename	`kit.exe`

Discovering Registered Tokens#

You can discover all registered tokens in your application by using the --/app/printConfig=true command-line option:

kit.exe --/app/printConfig=true

This will output the complete settings database as JSON, including all tokens stored under the /app/tokens/ branch. The output will show:

{
  "app": {
    "tokens": {
      "kit": "C:/Program Files/Kit",
      "app": "C:/MyApp",
      "platform": "windows-x86_64",
      "config": "release",
      // ... all other registered tokens
    }
  }
}

Note that this listing will only include the tokens that are registered during app startup. If any tokens are added programmatically after startup, these will not be included in the output.

Programmatic Token Discovery#

You can also discover tokens programmatically through the settings interface:

carb::settings::ISettings* settings = carb::getCachedInterface<carb::settings::ISettings>();
const carb::dictionary::Item* tokensDict = settings->getSettingsDictionary("/app/tokens");

if (tokensDict)
{
    carb::dictionary::ScopedRead readLock(*carb::getCachedInterface<dictionary::IDictionary>(), tokensDict);

    // Iterate through all registered tokens
    for (auto item : tokensDict->getItems())
    {
        const char* tokenName = item.getKey();
        const char* tokenValue = item.getStringBuffer();
        printf("Token: %s = %s\n", tokenName, tokenValue);
    }
}

Interface Reference#

Core Functions#

The ITokens interface provides both legacy C-string functions and modern string view functions:

Modern Interface (Recommended)#

// Set a token value
bool setValueS(const carb::cpp::zstring_view& name, const carb::cpp::string_view& value);

// Resolve a string with tokens
ResolveResult resolveStringS(const carb::cpp::string_view& sourceBuf, 
                            omni::string& destBuf, 
                            ResolveFlags resolveFlags);

// Check if a token exists
bool existsS(const carb::cpp::zstring_view& tokenName);

// Delete a token
bool deleteValueS(const carb::cpp::zstring_view& name);

// Set initial value (only if token doesn't exist)
void setInitialValueS(carb::cpp::zstring_view name, carb::cpp::string_view value);

Legacy Interface (Deprecated)#

// Legacy functions (deprecated, use string view versions instead)
bool setValue(const char* name, const char* value);
bool removeToken(const char* name);
bool exists(const char* tokenName);
void setInitialValue(const char* name, const char* value);
ResolveResult resolveString(const char* sourceBuf, size_t sourceBufLen, ...);

Utility Functions#

The TokensUtils.h header provides helpful utility functions:

namespace carb::tokens
{

// Resolve a string with tokens (convenience function)
std::string resolveString(const ITokens* tokens,
                         carb::cpp::string_view str,
                         ResolveFlags resolveFlags = kResolveFlagNone,
                         ResolveResult* resolveResult = nullptr);

// Escape special characters to prevent tokenization
std::string escapeString(carb::cpp::string_view str);

}

Resolution Results#

Token resolution operations return a ResolveResult enum:

enum class ResolveResult
{
    eSuccess,    // Resolution completed successfully
    eTruncated,  // Resolution succeeded but output was truncated
    eFailure     // Resolution failed due to an error
};

Resolution Flags#

Control token resolution behavior with flags:

using ResolveFlags = uint32_t;
const ResolveFlags kResolveFlagNone = 0;                      // Default behavior
const ResolveFlags kResolveFlagLeaveTokenIfNotFound = 1;      // Leave unresolved tokens as-is

Best Practices#

Use String View Interface: Prefer the modern *S functions (e.g., setValueS, resolveStringS) over the legacy C-string functions.
Use Utility Functions: Use carb::tokens::resolveString() from TokensUtils.h for simple string resolution rather than the lower-level interface functions.
Escape User Input: Use carb::tokens::escapeString() to escape any user-provided strings that should not be tokenized.
Handle Recursion Carefully: Be aware of potential circular dependencies when setting up token chains.
Check Return Values: Always check the return values and ResolveResult to handle errors appropriately.
Use Initial Values: Use setInitialValueS() for default values that should not override existing tokens.
Environment Variables: Use the ${env:VAR_NAME} syntax to access environment variables rather than reading them directly.

Integration with Settings#

The tokens interface integrates seamlessly with the Carbonite settings system. Tokens are automatically stored under /app/tokens/ in the settings registry, allowing them to be:

Configured in TOML configuration files
Overridden via command-line arguments using --/app/tokens/token_name=value
Accessed programmatically through the settings interface
Included in configuration dumps when using --/app/printConfig=true

This integration provides a unified approach to configuration management across the entire Carbonite ecosystem.