exotic

EXtatic.Org Test InfrastruCture — a minimal unit testing framework for C and C++.

A test is a function that returns non-zero to pass:

#include "magic.h"

EXO_TEST(test_function_magic_1, {
	int n = magic();
	return n == 42;
});

There is no framework to learn beyond that. Tests live in .tcc files, and the whole runtime is one small C file.

Building exotic

make && make check && sudo make install

or

cmake -B build && cmake --build build && ctest --test-dir build
cmake --install build

Both install libexotic, libexotic_main, the <exotic/exotic.h> header, the exotic generator, a pkg-config file, and (from CMake) a find_package package.

Using exotic from CMake

find_package(Exotic REQUIRED)

exotic_add_tests(mytests
	SOURCES   test_magic.tcc test_hash.tcc
	LIBRARIES magic
	DISCOVER)

DISCOVER registers every EXO_TEST as its own CTest entry, so ctest -R and ctest --output-on-failure work per test rather than per binary:

$ ctest
    Start 1: test_function_magic_1
1/2 Test #1: test_function_magic_1 ......   Passed
    Start 2: test_hash_empty
2/2 Test #2: test_hash_empty ............   Passed

Vendoring works the same way — no install step needed:

include(FetchContent)
FetchContent_Declare(exotic
	GIT_REPOSITORY https://github.com/janvidar/exotic.git
	GIT_TAG        v0.6.0)
FetchContent_MakeAvailable(exotic)

exotic_add_tests(mytests SOURCES test_magic.tcc LIBRARIES magic DISCOVER)

exotic_add_tests() accepts:

Argument	Meaning
SOURCES	The .tcc files (required)
LIBRARIES	Libraries the code under test needs
INCLUDE_DIRECTORIES, COMPILE_OPTIONS	Passed through to the target
LANGUAGE	C (default) or CXX
MODE	AUTO (default), GENERATE or STANDALONE
DISCOVER	One CTest entry per EXO_TEST
NO_CTEST	Build the binary but register nothing
PREFIX	Prepended to registered test names
WORKING_DIRECTORY, ARGS	Applied to the registered tests
TIMEOUT	CTest TIMEOUT, in seconds
RUN_SERIAL	Never run alongside another test
RESOURCE_LOCK	Serialise only against tests naming the same resource
LABELS	For ctest -L
ENVIRONMENT	VAR=value entries for the test process

The last five are set on every test the call registers — one entry, or one per EXO_TEST under DISCOVER.

Running tests in parallel

ctest -j runs registered tests concurrently, so two that bind the same port or write the same file will collide. RESOURCE_LOCK is usually the right tool — it serialises only the tests naming that resource, where RUN_SERIAL stops the test running beside anything at all:

exotic_add_tests(socket_tests
	SOURCES   test_http.tcc
	DISCOVER
	LABELS        network
	RESOURCE_LOCK loopback_port
	TIMEOUT       30)

This is worth setting up before adopting EXO_SKIP for "could not bind a port", not after: otherwise a collision between two parallel tests disguises itself as an environmental skip and CI stays green.

DISCOVER or one entry per binary?

They are not alternatives so much as two different jobs, and a suite generally wants both.

	DISCOVER	One entry per binary
A crash	fails that test only; CTest reports (SEGFAULT)	takes the binary down, unless --fork
ctest -R, --rerun-failed	per test	per binary
EXO_SETUP cost	paid once per test	paid once per file
A test needing another's leftovers	fails, since it runs alone	passes in source order; --sort=random finds it
A test broken by another's leftovers	passes, since it runs alone — invisible	--sort=random finds it

So DISCOVER gives isolation and triage, and pays for it by rebuilding each file's fixture for every test in it. A file whose EXO_SETUP is expensive is the case where registering the binary as a single entry earns its keep — and that choice is per call, so a suite can do both.

Note that --fork is redundant under DISCOVER: CTest has already given each test its own process, so forking inside it buys no isolation. Use --fork for the bulk runs.

A suite that wants both jobs covered registers twice:

exotic_add_tests(mytests SOURCES test_*.tcc DISCOVER)

# The same tests in one process, in a different order every run, to catch a
# test that only passes because another ran first.
exotic_add_tests(mytests_coupling
	SOURCES test_*.tcc
	PREFIX  coupling.
	ARGS    --sort=random)

The two ways to build a test binary

Self-registering (default)

Each EXO_TEST emits a constructor that registers the test before main() runs, and libexotic_main supplies the main(). Nothing is generated, so there is no Perl dependency and cross-compiling is unremarkable:

cc -o mytests -x c test_*.tcc -x none -lexotic_main -lexotic

Two things to know:

• .tcc is a C++ header extension as far as GCC and Clang are concerned, so the language has to be stated explicitly with -x c (or -x c++; MSVC uses /TC and /TP). exotic_add_tests() does this for you.
• Existing .tcc files do not include <exotic/exotic.h> themselves, because the generator used to include it for them. Add -include exotic/exotic.h (MSVC: /FIexotic/exotic.h) — again, exotic_add_tests() handles it.

Generated (the original workflow)

exotic -o autotest.c test_magic.tcc
cc -o autotest autotest.c -lexotic
./autotest

exotic --standalone inlines the entire runtime into the generated file, so the result links against no library at all.

Both generator modes #include every .tcc into a single translation unit and emit one main(), so they are whole-program: pass every .tcc to one invocation. Two separately generated files cannot be linked together — they would each define main() and the whole runtime. exotic_add_tests() does this correctly; a hand-rolled per-file invocation does not.

One translation unit also means static helpers with the same name in two files collide, and per-file EXO_SETUP is rejected outright. The self-registering path compiles each file separately and has neither problem, which is why it is the default and the better choice for a suite of any size.

If what you want is "no library to link" and separate compilation units, you do not need the generator at all — compile autotest.c and exotic_main.c alongside your tests:

cc -o mytests -x c test_*.tcc -x none autotest.c exotic_main.c

Writing tests

EXO_TEST(name, { ...; return 1; });   /* non-zero passes */
EXO_TEST_DISABLED(name, { ... });     /* still compiled, never run */

Top-level commas in the body are fine (int a, b;, std::map<int,int> m;).

Assertions report where they failed, which a bare return 0 cannot:

EXO_TEST(test_parser, {
	EXO_ASSERT(parse("x") != NULL);
	EXO_ASSERT_EQ_INT(count(), 3);
	EXO_ASSERT_STR_EQ(name(), "x");
	return 1;
});

* FAIL test 'test_parser'
    test_parser.tcc:3: assertion failed: count() == 3 (2 != 3)

Assertion	Notes
EXO_ASSERT(expr)
EXO_ASSERT_EQ_INT(a, b), EXO_ASSERT_NE_INT	compares through long
EXO_ASSERT_EQ_UINT(a, b), EXO_ASSERT_NE_UINT	unsigned long long; use these for 64-bit values
EXO_ASSERT_EQ_PTR(a, b), EXO_ASSERT_NE_PTR	pointer identity; two nulls are equal
EXO_ASSERT_NULL(a), EXO_ASSERT_NOT_NULL(a)
EXO_ASSERT_STR_EQ(a, b), EXO_ASSERT_STR_NE	STR_EQ fails if either side is null
EXO_ASSERT_MEM_EQ(a, b, n)	reports the first differing offset
EXO_FAIL(msg)	fail with a message

EXO_ASSERT_EQ_INT compares through long, which is 32 bits on Windows, so it truncates 64-bit values there. It is kept that way rather than quietly changing results; reach for EXO_ASSERT_EQ_UINT when the values may not fit.

Test names must be unique across all files in a binary, and must be valid C identifiers. In the default self-registering mode a duplicate is a link error naming exotic_test_<name>; the generator reports it by file and line.

Setup and teardown

EXO_SETUP and EXO_TEARDOWN run once around the tests of the file they are written in — for a fixture that is expensive, process-wide, or whose destruction has to be ordered against something else:

static Core* core;

EXO_SETUP({    /* before the first test in this file that will run */
	core = core_create(config);
	return core != NULL;
});

EXO_TEARDOWN({ /* after the last one */
	core_destroy(core);
	return 1;
});

• At most one of each per file. A second is a compile error.
• Setup runs immediately before the first test from its file that is actually going to run, so a file whose tests are all filtered out costs nothing, and --list-tests builds no fixture at all.
• A setup returning zero fails every test in its file instead of running them against a fixture that does not exist. The teardown still runs, so a partly built fixture is released. A setup that calls EXO_SKIP skips them instead.
• Each hook runs exactly once whatever --sort does. Under --sort=name files interleave, so two fixtures can be alive at once.
• Under --fork the hooks run inside each child, around the one test it was forked for — so the fixture is built per test.
• Hooks need one translation unit per file, so they require MODE AUTO. The generator rejects them rather than silently never building the fixture.

Skipping

A test that cannot run on this machine is neither a pass nor a failure. Saying so keeps an environmental limitation from being reported as a library defect:

EXO_TEST(multicast_joins_a_group, {
	if (!have_multicast_route()) EXO_SKIP("no multicast route on this host");
	...
});

* SKIP test 'multicast_joins_a_group'  (no multicast route on this host)

A skip is recorded out of band, the same way an assertion failure is, because every non-zero return already passes and zero already fails. A test that returns zero without calling EXO_SKIP therefore still fails exactly as before.

Skips appear as ok N - name # SKIP reason in TAP and as <skipped> in JUnit. They never on their own make the exit status non-zero, and exotic_add_tests() sets CTest's SKIP_RETURN_CODE, so ctest shows them as Skipped rather than burying them in output nobody reads.

Running tests

Usage: mytests [OPTIONS] [TEST]...

  --help        -h    Show this message
  --version     -v    Show version
  --summary     -s    Show only summary
  --fail        -f    Show only test failures
  --pass        -p    Show only test passes
  --list-tests  -l    List the test names, one per line, and exit
  --filter=GLOB       Run only tests matching GLOB (repeatable)
  --tap               Write TAP version 13 to stdout
  --junit=FILE        Write a JUnit XML report to FILE
  --sort=ORDER        Test order: 'source' (default), 'name',
                      'random' or 'random:SEED' to replay one
  --fork              Run each test in its own process

Finding dependencies between tests

--sort=random runs the tests in a different order every time and reports the seed it used, so a failure can be replayed exactly:

$ ./mytests --sort=random
exotic: --sort=random:1785865381
...
$ ./mytests --sort=random:1785865381     # the same order again

This finds tests coupled through process-wide state — a global, a cached handle, an open socket left behind. The seed is a TAP comment (# sort=random:N) under --tap, and --list-tests is never shuffled, since CTest discovery reads it and needs it stable.

Running each test in its own process — --fork, or DISCOVER — is not a substitute, and which way the coupling points decides what you see:

• A test that needs what another left behind fails when it runs alone. Per-process running catches this one, loudly.
• A test that is broken by what another left behind passes when it runs alone. Per-process running reports it green, and so does a source-ordered run whenever the source order happens to be the benign one. Nothing but running them together, in a different order, will show it.

CTest's own --schedule-random does not help either: it shuffles entries that are already isolated processes. Coupling is only observable when tests share a process, which is why this lives in the runtime rather than the build system.

ctest --repeat until-fail:20 pairs well with it for something intermittent.

Bare arguments select tests by name and accept * and ?:

./mytests 'test_parser_*'

Exit status	Meaning
0	everything passed, or passed or skipped
1	at least one test failed
2	usage error, or the binary contains no tests
77	everything that ran was skipped

77 is the autotools convention and what exotic_add_tests() sets CTest's SKIP_RETURN_CODE to. It is unreachable unless a test calls EXO_SKIP.

CI

GitLab and most other CI systems consume JUnit XML:

./mytests --junit=report.xml

Under CTest, ctest --output-junit report.xml produces the same thing, and with DISCOVER each EXO_TEST appears as its own case.

--fork runs each test in a child process, so a segfault fails only that test instead of ending the run. Without it, a crash still names the responsible test on stderr.

Compatibility

EXO_TEST requires a C99 or C++11 preprocessor (variadic macros). Test bodies themselves can be anything the compiler accepts.

Self-registration uses __attribute__((constructor)) on GCC/Clang, a static object constructor in C++, and .CRT$XCU on MSVC. On a compiler with none of those, use MODE GENERATE.

EXO_SETUP and EXO_TEARDOWN are registered the same way, so they need one of those mechanisms and cannot fall back to the generator. EXO_ASSERT_EQ_UINT and EXO_ASSERT_NE_UINT use long long, so C99 or C++11.

Upgrading from 0.5

Existing .tcc files need no changes, and nothing here is required — a suite that adopts none of it keeps working. struct exotic_handle is unchanged: the skip count and everything else added in 0.6 lives in private library state.

New: EXO_SETUP/EXO_TEARDOWN, EXO_SKIP, the assertions listed above, --sort=random, and the TIMEOUT/RUN_SERIAL/RESOURCE_LOCK/LABELS/ ENVIRONMENT arguments to exotic_add_tests(). Behaviour that did change:

• --fork no longer loses what a test printed. A forked child used to _exit() without flushing stdio, discarding its own output.
• Reported test durations no longer include a file's setup and teardown.
• The generator rejects EXO_SETUP/EXO_TEARDOWN instead of accepting them and never running the fixture.
• --sort=name now works in MODE GENERATE, where it was silently doing nothing: the sort ran before the generated main() added its tests.
• A binary exits 77 when everything that ran was skipped — reachable only if a test calls EXO_SKIP.

Upgrading from 0.4

Existing .tcc files need no changes. Behaviour that did change:

• A binary containing no tests now exits 2 instead of 0, and the generator refuses to emit one (--allow-empty overrides this).
• An unknown command-line option exits 2 instead of 0.
• The exit status is 1 when tests fail, rather than the number of failures — which reported success whenever the count was a multiple of 256.
• libexotic.so now carries a soname (libexotic.so.1).

License

MIT — see LICENSE.