Testing Extensions with C++

For information on testing extensions with Python, look here

Doctest

omni.kit.test has a doctest library as a runner for C++ tests.

Further, refer to Carbonite’s Testing.md to learn about using the Doctest testing system.

List of doctest command line arguments

You can also use –help on the command line for test.unit:

test.unit.exe --help

Although Omniverse adds additional options, there is also an online reference for Doctest command-line options.

Set up testable libraries

To write C++ tests, you first must have created a shared library with tests to be loaded:

project_ext_tests(ext, "omni.appwindow.tests")
    add_files("impl", "tests.cpp")

tests.cpp/AppWindowTests.cpp:

#include <carb/BindingsUtils.h>
#include <doctest/doctest.h>

CARB_BINDINGS("omni.appwindow.tests")

TEST_SUITE("some test suite") {
    TEST_CASE("test case success") {
        CHECK(5 == 5);
    }
}

Next specify in the test section to load this library:

[[test]]
cppTests.libraries = [
    "bin/${lib_prefix}omni.appwindow.tests${lib_ext}",
]

Run tests the same way with _build\windows-x86_64\release\tests-[extension_name].bat. omni.kit.test this:

  1. loads the library

  2. tests will be registered automatically in doctest

  3. runs doctest

In this setup C++ and python tests will run in the same process. A separate [[test]] entry can be created to run separate processes.

To run only subset of tests -f option can be used:

> _build\windows-x86_64\release\tests-omni.appwindow.bat -f foo

All arguments in the tested process after -- are passed to doctest. But to pass to the tested process, -- must be used. So to pass arguments to doctest, -- must be specified twice, like so:

> _build\windows-x86_64\release\tests-omni.appwindow.bat -- -- -tc=*foo* -s

When using the test.unit.exe workflow instead, check below.

Running a single test case

In order to run a single test case, use the -tc flag (short for –test-case) with wildcard filters:

_build\windows-x86_64\release\tests-omni.appwindow.bat -- -- -tc="*[rtx]*"

Commas can be used to add multiple filters:

_build\windows-x86_64\release\tests-omni.appwindow.bat -- -- -tc="*[rtx]*,*[graphics]*"

Unit Tests

Some tests are written using an older approach. Carbonite is used directly without kit and all the required plugins are manually loaded. To run those tests use:

> _build\windows-x86_64\release\test.unit.exe

Image comparison with a golden image

Some graphics tests allow you to compare visual tests with a golden image. This can be done by creating an instance of ImageComparison class. Each ImageComparison descriptor requires a unique GUID, and must be accompanied with the equivalent string version in C++ source as a comment for easy lookup.

Defining a test case:

ImageComparisonDesc desc =
{
    { 0x2ae3d60e, 0xbc3b, 0x48b6, { 0xa8, 0x67, 0xe0, 0xa0, 0x7c, 0xaa, 0x9e, 0xd0 } }, // 2AE3D60E-BC3B-48B6-A867-E0A07CAA9ED0
    "Depth-stencil-16bit",
    ComparisonMetric::eAbsoluteError,
    kBackBufferWidth,
    kBackBufferHeight
};

// Create Image comparison
imageComparison = new ImageComparison();

// register the test case (only once)
status = imageComparison->registerTestCase(&desc);
REQUIRE(status);

Regression testing of an executable with a golden image:

  • This is supported by any executable that uses carb.renderer (e.g. omnivserse-kit or rtx.example), in which it can capture and dump a frame.

  • NVF is not yet supported.

// 1- run an executable that supports CaptureFrame
std::string execPath;
std::string cmdLine;
ImageComparison::makeCaptureFrameCmdLine(500,             // Captures frame number 500
                                         &desc,           // ImageComparisonDesc desc
                                         "kit", // Executable's name
                                         execPath,        // Returns the executable path needed for executeCommand()
                                         cmdLine);        // Returns command line arguments needed for executeCommand()

// 2- Append any command line arguments you need to cmdLine with proper spaces
cmdLine += " --/rtx/debugView='normal'";

// 3- Run the application with a limited time-out
status = executeCommand(execPath, cmdLine, kExecuteTimeoutMs);
REQUIRE(status);

// 4- compare the golden image with the dumped output of the captured frame (located at $Grapehene$/outputs)
float result = 0.0f;
CHECK(m_captureFrame->compareResult(&desc, result) == ImageComparisonStatus::eSuccess);
CHECK(result <= Approx(kMaxMeanSqrError)); // With ComparisonMetric::eMeanErrorSquared

Regression testing of a rendering test using comparison of backbuffer with a golden image:

// 1- Create an instance of CaptureFrame and initialize it
captureFrame = new CaptureFrame(m_gEnv->graphics, m_gEnv->device);
captureFrame->initializeCaptureFrame(RenderOpTest::kBackBufferWidth, RenderOpTest::kBackBufferHeight);

// 2- Render something

// 3- copy BackBuffer to CaptureFrame
captureFrame->copyBackBufferToHostBuffer(commandList, backBufferTexture);

// 4- Submit commands and wait to finish

// 5- compare the golden image with the BackBuffer (or dump it into the disk $Grapehene$/outputs)
float result = 0.0f;
CHECK(imageComparison->compareResult(&desc, captureFrame->readBufferData(true), captureFrame->getBufferSize(), result) == ImageComparisonStatus::eSuccess);
CHECK(result == Approx(0.0f));

  • compareResult() also allows you to dump the BackBuffer into outputs folder on the disk. This can be done using the following option of test.unit executable:

test.unit.exe --carb-golden -tc="*[graphics]*,*[visual]*"

  • The following command allows executables to dump only the golden images of the test that fail our acceptable threshold:

test.unit.exe --carb-golden-failure -tc="*[graphics]*,*[visual]*"

How to update and upload a new golden image

  • Run the test in release mode.

// Example: regression testing of OmniverseKit executable
test.unit.exe -tc="*[omniverse-kit][rtx]*"

// Example: regression testing of visual rendering tests
test.unit.exe --carb-golden -tc="*[graphics]*,*[visual]*"

// Example: regression testing of visual rendering tests that fail our acceptable threshold
test.unit.exe --carb-golden-failure -tc="*[graphics]*,*[visual]*"

  • Verify and view the golden image that is added to outputs folder in Omniverse kit’s repo. It must be exactly what you expect to see. Name of the golden image is GUID of the test.

  • Copy the golden image from outputs folder to data\golden. This is a folder for git lfs data.

  • Open a merge request with git-lfs data changes.

Troubleshooting

  • All unit tests crash on textures or shaders:

    • You must have git lfs installed and initialize it.

    • Check files in data folder, and open them in a text editor. You should not see any URL or hash as content.

    • Install the latest driver (refer to readme.md)

  • executeCommand() fails:

    • A possible crash or assert in release mode.

    • A crash or hang during exit.

    • Time-out reached. Note that any assert dialog box in release mode may cause time-out.

  • compareResult() fails:

    • Rendering is broken, or a regression is introduced beyond the threshold.

  • outputs folder is empty for tests with a tag of [executable]:

    • A regression caused the app to fail.

How to use asset folder for storing data

  • Perform the following command to delete the existing _build\asset.override folder. That folder must be gone before proceeding further.

.\assets.bat clean

  • Stage assets. It copies data from assets to assets.override.

.\assets.bat stage

  • Modify any data under asset.override. Do NOT modify assets folder.

  • Upload and publish a new asset package:

.\assets.bat publish

  • Rebuild to download the new assets and run the the test to verify:

.\build.bat --rebuild

  • Open a merge request with new assets.packman.xml changes.

Skipping Vulkan or Direct3D 12 graphics tests

In order to skip running a specific backend for graphical tests, use --carb-no-vulkan or --carb-no-d3d12.

test.unit.exe --carb-no-vulkan"

Getting C++ Code Coverage Information

Code coverage is only supported on Linux currently.

To generate a C++ code coverage report, do the following steps:

  • Build with repo build --enable-gcov.

    • This must be a full rebuild.

    • This will disable optimization for best accuracy.

  • Run whatever tests you want coverage information for.

    • The code instrumentation will automatically put coverage data into the object directory.

  • Run repo cxx_coverage to generate a code coverage report with the data that was generated.

    • The report will be available in _build/coverage/index.html.

Understanding the Teamcity Coverage pipeline

Kit uses gcov on Linux to report coverage data. This has several components:

  • .gcno files: One of these is produced for every .o file during the build. These store information that’s needed during the ‘collection’ stage.

  • .gcda files: One of these is produced for every .o file when code that corresponds to that .o file is executed.

  • .info files: These are produced from lcov during the ‘collection’ stage.

The teamcity code coverage pipeline differs from the regular build pipeline in the following ways:

  • The ‘build’ stage of the pipeline packages its .gcno files and source files into the ‘coverage’ package.

    • Packaging these is essential on Teamcity. Sending these files loose takes 4 hours while the package takes 1 minute.

    • The source files are needed because our Teamcity test jobs don’t check out the repo, so this is how we get the C++ source code to those jobs.

  • The ‘test’ stages of the pipeline execute the tests normally, then they run repo cxx_coverage to generate a test report and then they package up the .gcda files.

    • The test reports here are just produced for debugging. The final test report is produced in the ‘publish’ stage.

    • These jobs produce a .info file, but it has hard-coded source paths, so we can’t use that later on.

  • The ‘publish’ stage collects all of the .gcda files and .gcno files from previous jobs, generates .info files from all of them, merges the .info files together and generates a report.

    • The data from this final .info file is reported to NVDF.

    • The publish script uses some manual lcov commands, but this functionality will be merged into repo cxx_covergae eventually.

Troubleshooting Code Coverage

Kit uses gcov to report coverage data. There are a number of details about gcov that could lead to lost coverage information.

  • gcov data needs to be explicitly flushed with __gcov_flush() if the program does not exit cleanly.

    • Gcov data is flushed from library destructors, so gcov data won’t be flushed if the program terminates in a way that doesn’t call library destructors.

    • The Kit ‘fast shutdown’ path does not exit cleanly.

    • The kit-gcov entrypoint in the kit-kernel package calls __gcov_flush() before shutdown.

  • The program entrypoint must be compiled with --coverage for __gcov_flush() to function properly.

    • This is most likely not a problem if the program exits cleanly and __gcov_flush() is not needed.

    • When the entrypoint doesn’t have gcov, the program ends up with multiple global states for gcov so calling __gcov_flush() will result in a random module being flushed.

      • If you think this is happening, load the program up in gdb, break in __gcov_flush and run info variables __gcov_master. Most instances of this variable should be the same and they should be inside the exectuable’s address space. If they all have different addresses, it’s likely you are hitting this bug.

  • gcov embeds absolute paths to its output directories.

    • For example, if you built in /buildAgent/work/56c64080b96508f3/kit, your code coverage data will output to /buildAgent/work/56c64080b96508f3/_build/intermediate/linux/....

    • This is a major problem on teamcity where the work directory may change between the build and test stages.

    • You need to use the GCOV_PREFIX and GCOV_PREFIX_STRIP environment variables to override this output path. Documentation available here.

  • lcov embeds absolute source code paths into its .info files.

    • There doesn’t seem to be an easy way to work around this. On CI, we transfer the .gcda files between jobs instead of lcov’s .info files to avoid this.

  • Using your system’s version of lcov results in this error: version 'A73R', prefer 'A94*'.

    • You need to run lcov out of linbuild to avoid version issues.

    • Try to use repo cxx_coverage instead of manually running lcov.

  • Teamcity logs have a spam of ‘cannot open source file’ errors from files in _build/target-deps.

    • This isn’t an actual error. This only happens for files that are external to the Kit repo and we remove these files from the report in a later step.

    • If a file that’s within the kit repo is missing, that will cause an error in the genhtml step. Check package.toml to see why that wasn’t getting packaged.