Tutorial 20 - Tokens

Tokens are a method of providing fast access to strings that have fixed contents. All strings with the same contents can be translated into the same shared token. Token comparison is as fast as integer comparisons, rather than the more expensive string comparisons you would need for a general string.

One example of where they are useful is in having a fixed set of allowable values for an input string. For example you might choose a color channel by selecting from the names “red”, “green”, and “blue”, or you might know that a mesh bundle’s contents always use the attribute names “normals”, “points”, and “faces”.

Tokens can be accessed through the database methods tokenToString() and stringToToken(). Using the tokens keyword in a .ogn file merely provides a shortcut to always having certain tokens available. In the color case then if you have a token input containing the color your comparison code changes from this:

const auto& colorToken = db.inputs.colorToken();
if (colorToken == db.stringToToken("red"))
{
    // do red stuff
}
else if (colorToken == db.stringToToken("green"))
{
    // do green stuff
}
else if (colorToken == db.stringToToken("blue"))
{
    // do blue stuff
}

to this, which has much faster comparison times:

const auto& colorToken = db.inputs.colorToken();
if (colorToken == db.tokens.red)
{
    // do red stuff
}
else if (colorToken == db.tokens.green)
{
    // do green stuff
}
else if (colorToken == db.tokens.blue)
{
    // do blue stuff
}

In Python there isn’t a first-class object that is a token but the same token access is provided for consistency:

color_token = db.inputs.colorToken
if color_token == db.tokens.red:
    # do red stuff
elif color_token == db.tokens.green:
    # do green stuff
elif color_token == db.tokens.blue:
    # do blue stuff

OgnTutorialTokens.ogn

The ogn file shows the implementation of a node named “omni.graph.tutorials.Tokens”, which contains some hardcoded tokens to use in the compute method.

{
    "Tokens": {
        "version": 1,
        "categories": "tutorials",
        "scheduling": ["threadsafe"],
        "description": ["This is a tutorial node. It exercises the feature of providing hardcoded token values",
                        "in the database after a node type has been initialized. It sets output booleans to the",
                        "truth value of whether corresponding inputs appear in the hardcoded token list."
        ],
        "uiName": "Tutorial Node: Tokens",
        "inputs": {
            "valuesToCheck": {
                "type": "token[]",
                "description": "Array of tokens that are to be checked"
            }
        },
        "outputs": {
            "isColor": {
                "type": "bool[]",
                "description": "True values if the corresponding input value appears in the token list"
           }
        },
        "$comment": [
            "The tokens can be a list or a dictionary. If a list then the token string is also the name of the",
            "variable in the database through which they can be accessed. If a dictionary then the key is the",
            "name of the access variable and the value is the actual token string. Use a list if your token values",
            "are all legal variable names in your node's implementation language (C++ or Python)."
        ],
        "tokens": ["red", "green", "blue"],
        "tests": [
            {
                "inputs:valuesToCheck": ["red", "Red", "magenta", "green", "cyan", "blue", "yellow"],
                "outputs:isColor": [true, false, false, true, false, true, false]
            }
        ]
    }
}

OgnTutorialTokens.cpp

The cpp file contains the implementation of the compute method. It illustrates how to access the hardcoded tokens to avoid writing the boilerplate code yourself.

// SPDX-FileCopyrightText: Copyright (c) 2021-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: LicenseRef-NvidiaProprietary
//
// NVIDIA CORPORATION, its affiliates and licensors retain all intellectual
// property and proprietary rights in and to this material, related
// documentation and any modifications thereto. Any use, reproduction,
// disclosure or distribution of this material and related documentation
// without an express license agreement from NVIDIA CORPORATION or
// its affiliates is strictly prohibited.
#include <OgnTutorialTokensDatabase.h>


class OgnTutorialTokens
{
public:
    static bool compute(OgnTutorialTokensDatabase& db)
    {
        const auto& valuesToCheck = db.inputs.valuesToCheck();
        auto& isColor = db.outputs.isColor();

        size_t len = valuesToCheck.size();
        isColor.resize(len);
        if (len == 0)
        {
            return true;
        }

        // Walk the list of inputs, setting the corresponding output to true if and only if the input is in
        // the list of allowable tokens.
        for (size_t index = 0; index < len; index++)
        {
            const auto& inputValue = valuesToCheck[index];

            // When the database is available you can use it to access the token values directly. When it is not
            // you can access them statically (e.g. OgnTutorialTokensDatabase.token.red)
            if ((inputValue == db.tokens.red) || (inputValue == db.tokens.green) || (inputValue == db.tokens.blue))
            {
                isColor[index] = true;
            }
            else
            {
                isColor[index] = false;
            }
        }

        return true;
    }
};

REGISTER_OGN_NODE()

OgnTutorialTokensPy.py

The py file contains the implementation of the compute method in Python. The .ogn file is the same as the above, except for the addition of the implementation language key "language": "python". The compute follows the same algorithm as the cpp equivalent.

"""
Implementation of a node handling hardcoded tokens. Tokens are a fixed set of strings, usually used for things
like keywords and enum names. In C++ tokens are more efficient than strings for lookup as they are represented
as a single long integer. The Python access methods are set up the same way, though at present there is no
differentiation between strings and tokens in Python code.
"""


class OgnTutorialTokensPy:
    """Exercise access to hardcoded tokens"""

    @staticmethod
    def compute(db) -> bool:
        """
        Run through a list of input tokens and set booleans in a corresponding output array indicating if the
        token appears in the list of hardcoded color names.
        """

        values_to_check = db.inputs.valuesToCheck

        # When assigning the entire array the size does not have to be set in advance
        db.outputs.isColor = [value in [db.tokens.red, db.tokens.green, db.tokens.blue] for value in values_to_check]

        return True