Testing

Testing is one of the cornerstones of Carbonite philosophy. Our goal is to keep all the plugins and the framework itself covered with tests and continuously run them on every MR. The isolated nature of the plugin interfaces makes unit testing easy and straightforward. This guide describes how to write new tests and work with existing ones.

Folder Structure and Naming

Everything test related is in the source/tests folder.

source/tests/test.unit contains all unit tests which compile in the test.unit executable.

All tests grouped by the interface/namespace they are testing. For instance, source/tests/test.unit/tasking contains all tests for carb.tasking.plugin while source/tests/test.unit/framework contains all framework tests.

Each unit test source file must start with Test prefix, e.g. tests/test.unit/tasking/TestTasking.cpp.

Generally you want to test against the interface. If the interface has multiple plugins implementing it you can just iterate over all of them in the test initialization code, without changing test itself or making small changes (like having separate shaders for different graphics backends).

If you want to write tests against a particular implementation and it is not convenient anymore to keep them in the same folder the naming guideline is to add impl name: /tests/test.unit/graphics.vulkan.

If you need to create special plugins for your testing they should be put into: source/tests/plugins/.

The Testing Framework

We use doctest as our testing framework of choice. While some parts of its functionality will be covered in this guide, it it is recommended you read the official tutorial. This will for instance help you understand the concept of SECTION() and how it can be used to structure tests. This CppCon 2017 talk is also useful.

Writing Tests

The typical unit test can look like this (from framework/TestFileSystem.cpp):

#if CARB_PLATFORM_WINDOWS
const char* kFileName = "carb.dll";
#else
const char* kFileName = "libcarb.so";
#endif

TEST_CASE("paths can be checked for existence",
        "[framework][filesystem]"
        "[component=carbonite][owner=adent][priority=mandatory]")
{
    FrameworkScoped f;
    FileSystem* fs = f->getFileSystem();
    REQUIRE(fs);

    SECTION("carb (relative) path exists")
    {
        CHECK(fs->exists(kFileName));
    }
    SECTION("app (absolute) path exists")
    {
        CHECK(fs->exists(f->getAppPath()));
    }
    SECTION("made up path doesn't exist")
    {
        CHECK(fs->exists("doesn't exist") == false);
    }
}

The general flow is to first get the framework, using the FrameworkScoped utility from common/TestHelpers. Then get the interface you need to test against, write your tests and clean up. Framework config can be used to control which plugins to load (at least to avoid loading overhead). It’s up to the test writer how to organize initialization code, the only important thing is to clean up everything after the test is done. In order to not have to write the same setup and teardown code over and over you can create a C++ object using the RAII pattern, like this:

class AutoTempDir
{
    char m_path[1024];

public:
    AutoTempDir()
    {
        FileSystem* fs = getFramework()->getFileSystem();
        bool res = fs->makeTempDirectory(m_path, sizeof(m_path));
        REQUIRE(res);
    }
    ~AutoTempDir()
    {
        FileSystem* fs = getFramework()->getFileSystem();
        bool res = fs->removeDirectory(m_path);
        CHECK(res);
    }

    const char* getPath()
    {
        return m_path;
    }
};

TEST_CASE("temp directory",
        "[framework][filesystem]"
        "[component=carbonite][owner=ncournia][priority=mandatory]")
{
    FrameworkScoped f;
    FileSystem* fs = f->getFileSystem();
    REQUIRE(fs);

    SECTION("create and remove")
    {
        AutoTempDir autoTempDir;

        SECTION("while creating empty file inside")
        {
            std::string path = autoTempDir.getPath() + std::string("/empty.txt");
            File* file = fs->openFileToWrite(path.c_str());
            REQUIRE(file);
            fs->closeFile(file);
        }

        SECTION("while creating empty utf8 file inside")
        {
            std::string path = autoTempDir.getPath() + std::string("/") + std::string(kUtf8FileName);
            File* file = fs->openFileToWrite(path.c_str());
            REQUIRE(file);
            fs->closeFile(file);
        }
    }
}

Every test must be tagged at least with the [my interface name] tag. Where “my interface name” is equal to the folder name. You can add additional tags to support even more elaborate filtering. Keep in mind that these tags are primarily there for people to focus their testing on a specific area of the code. We will explain later how to control this.

Running Tests

Once compiled all tests can be run with the test.unit executable. For instance on Windows release it is placed here:

_build\windows-x86_64\release\test.unit.exe

Command line examples:

Run all tests and output errors:

test.unit

Help can be found with the -h flag or by reading the official documentation:

test.unit -h

Run only tests tagged with [tasking]:

# Linux
test.unit -tc='*[tasking]*'

# Windows
test.unit.exe -tc=*[tasking]*

Run all tests, excluding tests tagged with [tasking]:

test.unit -tce=*[tasking]*

Run only tests tagged with [framework] or [tasking]:

test.unit -tc=*[framework]*,*[tasking]*

Run only tests which name starts with Acquire*:

test.unit -tc=Acquire*

Include successful tests in the output:

test.unit -s

Prints a list of all test cases:

test.unit --list-test-cases

Enable Carbonite error logging:

# -g or --carb-log
test.unit -g

Enable Carbonite logging and set log level to verbose:

test.unit -g --carb-log-level=-2

Use Debug Heap as provided by VC C runtime (Windows only).

test.unit --debug-heap