ABI Compatibility

Two binaries (modules, executables, etc.) must have agreement on how data is exchanged between them. While programmers typically think of this in terms of an API (Application Programming Interface) in source code, the hardware requires a binary interpretation of this. This is called the Application Binary Interface.

The ABI therefore is the binary contract between two entities (generally modules) of how functions are called and data is exchanged.

This document is written from the perspective of Carbonite C++ development. Other languages may have similar concepts but vary in implementation.

Importance

Each C/C++ source file (compilation unit) is processed by the compiler to form an object file. These object files are then combined together by a linker to form a binary (executable, static library, dynamic library, etc.). All of these items–object files, executables, static/dynamic libraries–can be generated at different times yet linked together, they must agree on how functions exchange data.

When using plugins as dynamic libraries as Carbonite does, it is monumentally important to ensure that changes can be made to both plugins and the Framework in such a way that ensures backwards compatibility. This allows the API to grow and evolve, but allows binaries built at different times to function correctly.

A similar topic to ABI is that of Interoperability, often shortened to interop. Interop is the ability for two different languages to be able to call functions and exchange data between themselves. Some languages are able to call functions directly with a C calling convention, whereas other languages may require a binding layer–code that is generated to translate function calls and data.

Terminology

“API”

Application Programmer Interface

This is a programmer’s contract between two entities that describes how functions are called and data is exchanged. This contract involves the names of functions that can be called, the parameters and return values that are passed, and other concepts such as atomicity and global state that are considered meta-concepts and not enforced specifically in the code.

“ABI”

Application Binary Interface

Defined above to be the binary contract between two entities (generally modules) of how functions are called and data is exchanged.

This is similar to an “API”, but is a contract at the machine level, describing items such as calling convention, parameter type, count and order, structure size and binary layout, enum values, etc. Whereas a programmer’s API might think of a function parameter as an int, the ABI considers this value as a 32-bit machine word and how it is passed (stack or register) with no inherent type safety, no concept of signed or unsigned, etc. While an API describes a struct, the ABI considers only the binary representation of the data. An API might describe a function by name, but to the ABI this is just an address.

“Breaking ABI”

In terminology below, “Breaking ABI” or “ABI-Breaking” means that a change is made that causes a compiled object with an understanding of the object’s binary layout before the change will no longer work correctly after the change.

Non-exhaustive examples of changes that break ABI (expounded further below):

Function calling convention
Function return type or parameters (including changing pass-by-value to pass-by-reference)
Ordering of members within a struct or class
Type of a member (i.e. changing size_t to unsigned)
Offset or alignment (such as by inserting a member)
Size of a member (i.e. changing char buffer[128] to char buffer[256])

Making these types of changes is acceptable for an InterfaceType by increasing the major version parameter in the CARB_PLUGIN_INTERFACE macro usage (or latest version given to the CARB_PLUGIN_INTERFACE_EX macro usage) by at least one.

Carbonite interfaces may support multiple versions in order to be backwards compatible.

“ABI Boundary”

An “ABI Boundary” in this document is a potentially fragile point in the code where an ABI-Breaking change may occur. Generally this represents the boundary of a module or executable, since that is the typical granularity of a binary that is built at a given point in time. Calling a function in a different binary module is considered calling across an ABI Boundary. If static libraries or object files are stored in source control and not rebuilt when linking a binary, functions in those object files or static libraries could be considered ABI Boundaries as well.

Any and all parameters, either passed to or returned from an object in a different binary is considered as crossing the ABI Boundary.

Inline functions do not cross a ABI Boundaries and the restrictions below do not apply.

“Semantically Compatible”

Certain changes are allowed to be made as they are semantically compatible with modules built prior to the change. Carbonite Interfaces use semantic versioning to determine compatibility. The version of an interface is specified in the CARB_PLUGIN_INTERFACE macro usage. There is also a CARB_PLUGIN_INTERFACE_EX macro that allows specifying latest and default versions.

Any ABI breaking change must be accompanied by a major version increase of at least one.

Any change that is semantically compatible with older modules built prior to the change must be accompanied by a minor version increased by at least one and retaining the same major version.

Carbonite interfaces may support multiple versions in order to be backwards compatible. In the case of a plugin supporting multiple versions the version in the CARB_PLUGIN_INTERFACE macro is the highest version supported, or the latest version passed to the CARB_PLUGIN_INTERFACE_EX macro.

“Interop”

As stated above, Interop (short for Interoperability) is the ability for two languages to exchange data and call functions between themselves. Many languages (such as Python and Rust) can call functions with a C calling convention, but cannot call functions with a C++ calling convention.

Generally for C++ types we require them to be trivially-copyable and conform to StandardLayoutType at a bare minimum to consider them Interop-Safe. Carbonite has the CARB_ASSERT_INTEROP_SAFE to ensure this.

Interop Safety allows two languages to agree on the data layout of a type, but the code that modifies that type still needs to ensure atomicity, memory ownership and expected data. This is up to the programmer to implement.

Calling Convention

This ABI has many different levels to it. For instance, system calls to an Operating System such as Linux or Windows generally require calls using the syscall CPU instruction, but the OS defines a system calling convention– a contract between the application and the OS that describes the registers or memory that correspond to function arguments and return values.

Applications are made from the building blocks of object files, static and dynamic libraries, and executables. Since these components can be built at different times, yet need to work together, a calling convention is also used to form a contract in how these different pieces call functions and exchange data.

Different compilers on the same platform should agree on the same convention, which would allow a binary with objects compiled by GCC to call functions in a library of objects compiled with Clang.

C supports different types of platform-specific calling conventions specified per function, such as __cdecl, __pascal, __stdcall, __fastcall, etc. The default for 32-bit x86 architecture was __cdecl. The calling convention indicates what registers (or where on the stack) arguments are placed in, where the return value is returned, and if the caller or callee is responsible for cleanup.

Some applicable ABI references that discuss calling conventions:

Itanium C++ ABI
System V AMD64 ABI (used by Linux and MacOS on x86_64)
Microsoft x64 Calling Convention (x86_64)
ARM AAPCS ABI (used by Linux and MacOS on aarch64)

While calling convention is typically not on the mind of a programmer writing an API, it does affect ABI and therefore should be considered.

Calling convention also comes into play with Interop. Many languages (such as Python and Rust) can call functions with a C calling convention, but cannot call functions with a C++ calling convention. The distinction here is important: A C calling convention specifies how built-in types (such as int and float) are passed, as well as pointer types (including const char*-style strings) and even struct-by-value types (provided that they are standard layout and trivially copyable). More complicated C++ types (such as omni::string and omni::function) can also be passed by value, but these types require additional specification on top of the C calling convention, such as who is responsible for destruction and how copying works. This is handled by the C++ calling convention, which makes passing these types by value interop-unsafe.

Carbonite provides the macro CARB_ASSERT_INTEROP_SAFE which ensures that a type is standard layout and trivially copyable.

Note

Though references are not part of the C calling convention (as they are a C++ feature), the ABI for references is essentially the same as pointers. They are therefore ABI-safe and interop-safe.

Warning

Changing calling convention, return value or parameters of a function will break its ABI. Changing almost anything about a function will break ABI. Ironically, changing the name of a function will typically not break ABI, but will affect compilation (“API”). This is because source code refers to names whereas the compiled binary code typically does not.

Built-in Types

C/C++ types (e.g. int, uint32_t, size_t, float, etc.) are always ABI-safe as their characteristics are guaranteed to never change. However, note that these may be different between platforms/architectures.

These types may be safely used as function parameters, return values, and members of structs and classes.

Pointers and References

Both pointers and references have a well-defined ABI described by the calling convention and are therefore safe to use as members, function parameters and return values provided that the types referenced or pointed to are also ABI safe.

Variadic Arguments

Declaring a function as having variadic arguments (e.g. log(const char* message, ...)) has an ABI described by the calling convention and is therefore safe, provided that the types passed through the variadic arguments are also ABI safe.

Endianness

Also typically not thought of by programmers, endianness should be considered part of the ABI. This is the ordering of bytes in memory of a binary word. A hardware architecture is one specific endianness: big or little. For big-endian hardware, a 32-bit hexadecimal word 0x01020304 would be represented in memory bytes in the same order: 01 02 03 04. For little-endian hardware, the bytes of 0x01020304 are stored backwards: 04 03 02 01. Carbonite focuses mostly on little-endian because all supported architectures are little-endian. However, Network Byte Order as specified by the TCP/IP standard is big-endian. If a function changed to keep the same datatype, but instead required a parameter to be in network byte order, this would break ABI.

Enum Values

Changing the values of existing enums is an ABI-breaking change for every function that uses them. Adding a new, previously-unassigned enum value is a semantically compatible change.

Struct/Class Layout

Changing the layout of a struct or class that is passed across an ABI boundary is likely to affect ABI. See Best Practices below for some semantically-compatible methods of changing structs and classes.

Consider the following struct:

struct Version {
    uint32_t major;
    uint32_t minor;
};

This struct has two members: major and minor. It’s size is 8 bytes. Those 8 bytes are layed out in memory as follows:

                         minor (bytes 4 - 7)
                      |---------|
00000000  00 00 00 00 00 00 00 00  .. .. .. .. .. .. .. ..  |........        |
          \_________/
              major (bytes 0 - 3)

          |---------------------|
            total size: 8 bytes

The binary layout of this object represents its ABI. Now consider what would happen if we change the class as follows:

struct Version {
    uint32_t major;
    char dummy; // Added field
    uint32_t minor;
};

The binary representation of this object changes:

00000000  00 00 00 00 00 00 00 00  00 00 00 00 .. .. .. ..  |............    |
          \__major__/  | (padding) \__minor__/
                       dummy
          |__________________________________|
             total size: 12 bytes

Notice how the object changed? The major member is still at the same size and location. However, the size of the struct changed to 12 bytes instead of 8, and the minor member no longer starts at byte 4 but now at byte 8.

Any module that was compiled with an understanding of Version in the top code block would not work properly with the bottom block.

Therefore, we cannot change Version in this manner without Breaking ABI.

Warning

Changing the type, size, order, or alignment of any member within a struct will break its ABI.

Inheritance

Changing inheritance or inheriting from multiple base classes may break ABI and is not recommended. Inheritance may be changed only if it does not affect the characteristics of any existing members.

Non-virtual Functions

Adding member functions that are non-virtual to an existing struct or class does not break ABI. Typically these functions will be declared as inline, otherwise they would be written into an object file (or static library) that must be linked in order to operate. Functions declared as inline that cannot actually be compiled in-line where called will be written into any object files that reference them and then typically be coalesced down to a single function at link time.

Changing parameters, return values, or the body of a non-virtual member function also does not break ABI since this function is essentially copied into whatever module calls it. However, a module will not take advantage of any changes to this function until it is rebuilt with the changes.

Virtual Functions

Changing a class that does not have a v-table to add a v-table by adding a virtual function will break ABI as it causes all members of the class to change characteristics. If the class is a base class for other classes, it will break the ABI of all descendant classes.

Since virtual functions in a class are in order of their declarations, adding a new virtual function to the end of a final class (that already has virtual functions) is allowed; this is a semantically compatible change. It is very important that the functions are added to the end of the declaration, and that the class/struct is final so that it may not be inherited from.

Members

Members of a struct or class must themselves be ABI-safe in order for a struct or class to be considered ABI-safe.

Caution

It is not ABI-safe to change a struct that is contained as a member within another struct.

Constructor and Initialization

Adding a new type initializer or changing the existing type initializer is a non-ABI-breaking (allowed) change and can be done at any time. However, keep in mind that older modules will not have this change, so previous values of the initializer must be anticipated.

Likewise, adding a new inline constructor to a class is a non-ABI-breaking (allowed) change.

Standard Layout

Data that is exchanged across the ABI Boundary must conform to the C++ named requirements of a standard-layout type. This is often checked with the is_standard_layout type-trait.

Copying and Moving

Data that is passed by value across the ABI Boundary must conform to the C++ is_trivially_copyable type-trait.

C++ Runtime Library Types

C++ Runtime Library Types, such as std::chrono::time_point, std::vector, std::string, std::optional, std::unique_ptr, std::function, std::shared_ptr, std::weak_ptr, et al have no guarantees about ABI safety; their layout could change with an update.

Carbonite considers passing C++ Runtime Library Types to be ABI unsafe.

However, it is acceptable to use these types within inline functions as they do not cross the ABI Boundary.

Warning

Do not pass any C++ Runtime Library Types (or pointers or references to them) across an ABI boundary.

Exceptions

Exceptions are generally C++ Runtime Library Types that inherit from std::exception and as such are not considered by Carbonite to be ABI safe.

It is recommended that all interface functions be declared as noexcept.

Warning

Do not allow any exceptions to cross an ABI Boundary when unwinding.

ABI-Breaking Changes Summarized

This is a non-exhaustive summation of the ABI-breaking changes mentioned above. Do not do the things listed here without an ABI-breaking InterfaceType version change (major version change).

Changing Interface function calling convention, parameters, or return type
Reordering members within an InterfaceType
Adding members to an InterfaceType not at the end (note: adding members to the end is a semantically compatible change)
Changing types, size, alignment or other characteristics of members

Best Practices

Use Built-In Types

Built-in data types (i.e. char, uint32_t, size_t, float, etc.) are always ABI-safe.

Pointers and references are ABI-safe only if the type pointed-to or referenced is also ABI-safe.

Use ABI-Safe Complex Types

Carbonite provides some helper classes that are ABI-safe:

omni::function - A replacement for std::function
omni::string - A replacement for std::string
carb::RString (and variations) - A fast string-interning mechanism

Note that not all Carbonite types are intended to be ABI-safe.

Warning

The omni::function and omni::string types, while ABI-safe, should only be passed by reference or pointer. This is because these types are not trivially copyable. This allows maintaining a C calling convention and interop safety.

Add New Interface Functions to the End

Adding interface functions to the end of an InterfaceType allows the change to be semantically-compatible (minor version change only).

Design Structs to be Changed

It can be advantageous to design structs for future change by including a version of some sort.

Many Windows API structs contain a member that must be initialized to the sizeof the struct. This is an effort to plan for future changes in a way that does not break ABI compatibility. New members can then be added to the end of the struct (avoid removing members, resizing them or adding them anywhere but the end as this drastically complicates processing).

This technique can be used to change a struct in a way that is safe and is employed by Carbonite structures, such as carb::tasking::TaskDesc and carb::AcquireInterfaceOptions.

As the struct grows, the size naturally changes, so the size of the struct makes a decent version.

At the first point of the code where the struct is used, promote it to the latest version:

static std::optional<AcquireInterfaceOptions> versionPromote(const AcquireInterfaceOptions& opt)
{
    CARB_LIKELY_IF(sizeof(opt) == opt.sizeofThis)
    {
        return opt;
    }

    // Version promotion to current goes here. Initial size of the struct was 48 bytes.

    CARB_LOG_ERROR("Unknown size of AcquireInterfaceOptions struct: %zu", opt.sizeofThis);
    return std::nullopt;
}

void* FrameworkImpl::acquireInterface(AcquireInterfaceOptions opt_)
{
    auto options = versionPromote(opt_);
    if (!options)
        return nullptr;

    auto& opt = options.value();
    // ...

When promoting, if the sizeof field is not equal to the current sizeof the struct, keep in mind that every member with an offset after the sizeof field is not initialized; reading it will result in undefined behavior.

It is also good practice to do the promote into a copy of the struct (remember from above that structs must be trivially copyable).

Warning

While it may seem like a good idea to pass a struct by value and promote it in-place, this is dangerous and can lead to stack corruption. This is because of the calling convention: the space for the struct is allocated by the caller, so only the space for the sizeof struct is allocated. Writing to members beyond that will corrupt the stack. Instead, pass the struct by const-reference and copy into a new local struct. Some older Carbonite APIs were written unaware of this and pass the struct by value.

Warning

When working with arrays of structs, keep in mind that the size of passed-in structs might be different than the current size, so standard pointer incrementing will not work. Instead, you will have to cast to uint8_t* (a byte pointer) and manually calculate the pointer offset.

Example of working with arrays of versioned structs:

void Scheduler::addTasks(TaskDesc* tasks, uint32_t numTasks, Counter* counter)
{
    CARB_CHECK(counter != detail::kListOfCounters, "addTasks() does not support list of counters");

    // Because the struct may be an older version (different size), we cannot increment the pointer itself. We need to
    // handle it at the byte level.
    uint8_t* p = reinterpret_cast<uint8_t*>(tasks);
    while (numTasks--)
    {
        TaskDesc* pTask = reinterpret_cast<TaskDesc*>(p);
        CARB_FATAL_UNLESS(pTask->size <= sizeof(TaskDesc), "Invalid TaskDesc size");
        p += pTask->size;
        addTask(*pTask, counter);
    }
}

It is good practice to have unit tests for all versions of a struct to ensure that they promote correctly.

Remember to ensure that your struct conforms to C++ type-trait is_standard_layout.

Consider Thread Safety

While not strictly about ABI, it is good Interface design to consider thread safety. For example, this could lead to problems as you cannot atomically get the count and then fill the buffer (also the fill function doesn’t take a count for safety!):

struct Widget;

struct MyInterface {
    CARB_PLUGIN_INTERFACE("MyInterface", 1, 0)

    size_t(CARB_ABI* getCount)();

    void(CARB_ABI* fillWidgets)(Widget** outWidgets);
};

Since no container is currently ABI-safe, one possibility might be to take a function/context:

using WalkWidgetsFn = void(Widget*, void*);

void (CARB_ABI* walkWidgets)(WalkWidgetsFn* fn, void* context);

// Call example:
std::vector<Widget*> widgets;
iWidget->walkWidgets([](Widget* w, void* context) {
    static_cast<std::vector<Widget*>*>(context)->push_back(w);
}, &widgets);