How to Write a C++ Test Framework
When I started using TDD and Googletest as a developer, my first thought was: How is this implemented? At that time, I accepted that it worked, and within the last year, when I implemented CWT Cucumber, I came across the same question (again). But in order to understand this mechanism, this time I needed an answer. So lets create a prototype for a test framework. Note: This is really a prototype and demonstrates how to write such a framework. This is not meant to be production ready.
Lets consider a simple unittest with Googletest:
#include <gtest/gtest.h>
TEST(your_testsuite, your_test)
{
// ...
EXPECT_EQ(2+2, 4);
}Alternatively with Catch2:
#include <catch2/catch.hpp>
TEST_CASE("your test", "some description")
{
// ...
REQUIRE(2+2 == 4);
}And if you compile this test and link it against Googlest/Catch2, you can run the implemented tests. For me, the question was how these frameworks register and execute the tests, more specifically, what is behind the TEST or TEST_CASE macro. In the end, it turns out that creating such a mechanism is not difficult at all.
After some research on the internet, the "trick" is to use a struct and a global variable, where the constructor is called before main is executed. Let's break this down:
// our test does not return a value and has no arguments,
// which gives this simple alias for a void function:
using test_case = void (*)();
// we will store the tests then in a vector
std::vector<test_case> tests;
// we forward declare foo and here will the macro later begin
void foo();
// we open an unnamed namespace
namespace
{
// and this is our test struct for foo:
struct test_foo_t
{
// where we use its constructor to push foo() to our tests
test_foo_t()
{
tests.push_back(foo);
}
// and we create a global instance of test_foo_t
// which calls the constructor before main gets executed
} g_test_foo;
} // namespace
// here ends the macro and the actual foo implementation follows:
void foo()
{
std::cout << "foo called\n";
}
// ...
// and now we just need to iterate over all functions in our
// global vector and we call each element.
int main()
{
for (const auto &t : tests)
{
t();
}
return 0;
}
Run this code on Compiler Explorer.
TEST(.., ..) Macro and Test Execution
That wasn't too hard, so let's create the macro, because we don't want to write all this code by hand for every test:
// this short helper concatenates two terms from a macro
#define _CONCAT_(a, b) a##b
#define CONCAT(a, b) _CONCAT_(a, b)
// and this is the full macro representation, I decided to go with a function name
// and a more descriptive name as string. In the constructor we push our forward
// declared function to our test container. you'll find the test class below then
#define INTERNAL_TEST_MACRO(func, name) \
void CONCAT(func, _test_body)(); \
namespace \
{ \
struct CONCAT(func, _t) \
{ \
CONCAT(func, _t)() \
{ \
::cwt::tests().push_test( \
cwt::details::test_case(CONCAT(func, _test_body), name)); \
} \
} CONCAT(g_, func); \
} \
void CONCAT(func, _test_body)()
// and the actual macro users will use then
#define TEST(func, name) INTERNAL_TEST_MACRO(func, name)
// which means if we take foo() from above, we can now write:
TEST(foo, "some description")
{
std::cout << "foo called\n";
}
Now, lets create some classes, where we store our tests and their results, check out the comments in the code below:
// we put all in a details namespace
namespace cwt::details
{
// we represent a test in class test_case
// it only holds the call to the test and a description
class test_case
{
// a simple alias for the signature/function pointer
using func_t = void (*)();
public:
// a simple constructor
test_case(func_t cb, const std::string& name) : m_test(cb), m_name(name) {}
// a call function to execute a test
void call() const { m_test(); }
private:
// and the test and some description.
// the description is unused in this example ....
func_t m_test;
std::string m_name;
};
// and we need another class to hold everything together
class tests
{
public:
// this will represent the results from our tests
enum class result
{
passed,
failed
};
int run()
{
// when running our tests we need to iterate over them
// and by default the test is passed
for (const auto& test : test_cases())
{
test_results().push_back(result::passed);
test.call();
}
// this will print the results to the user
print_test_results();
// and the return value depending if all results are passed
return std::all_of(test_results().begin(), test_results().end(),
[](result r) { return r == result::passed; })
? EXIT_SUCCESS
: EXIT_FAILURE;
}
// push_tests adds a test
static void push_test(const test_case& tc) { test_cases().push_back(tc); }
// and this will set the test to failed (if so)
static void current_test_failed() { test_results().back() = result::failed; }
private:
// a simple short report which we call in run()
void print_test_results()
{
// lets see how many of our tests passed
const std::size_t passed_count =
std::count_if(test_results().begin(), test_results().end(),
[](result r) { return r == result::passed; });
// and just print the overall result
std::cout << "\n================================================\n";
std::cout << "Tests done, " << passed_count << '/' << test_results().size()
<< " passed.\n\n";
}
private:
// all the test cases in a static vector
static std::vector<test_case>& test_cases()
{
static std::vector<test_case> tests;
return tests;
}
// and all results in another static vector
static std::vector<result>& test_results()
{
static std::vector<result> results;
return results;
}
};
} // cwt::details
That's all we need for now, and for ease of use we'll create two free functions to finally run all the tests and give the user a chance to asssert a test.
// and now we create some free functions we'll use
namespace cwt
{
// this creates our tests
details::tests& tests()
{
static details::tests t;
return t;
}
// and this assert function we can use to assert our tests
void assert_true(bool condition)
{
// we just check if the given condition is false and set the current test failed
// test are passed by default because if there is no assertion during the test
// we want them to be passed
if (condition == false)
{
details::tests::current_test_failed();
}
}
} // namespace cwt
Finally we create two tests and execute them in main. We'll have one test passing and one test failing:
TEST(sunny_case, "This will pass")
{
cwt::assert_true(2+2 == 4);
}
TEST(failing_case, "This will fail")
{
cwt::assert_true(false);
}
int main()
{
return cwt::tests().run();
}
Of course, if we were really writing a test framework, we'd structure the project better and separate the code into different header files. And we'd include main in the framework so that users don't have to call cwt::tests().run() themselves. Like Googletest, you can give the user two targets, one that includes main and you don't need to implement it, or maybe you need to implement something in main before the test execution, then you can have a no-main target. But for simplicity and for the sake of this article, we'll keep everything in one file. Find this example here on Compiler Explorer and feel free to play around.
Typed Tests
Typed tests are quite handy when it comes to tests that you want to run for different types. Consider writing tests for empty() in std::vector, std::deque and std::list. In this case, we'd write three tests:
TEST(vector_empty, "vector is not empty")
{
std::vector<int> v{1,2,3};
cwt::assert_true(!v.empty());
}
TEST(list_empty, "list is not empty")
{
std::list<int> l{1,2,3};
cwt::assert_true(!l.empty());
}
TEST(deque_empty, "deque is not empty")
{
std::deque<int> d{1,2,3};
cwt::assert_true(!d.empty());
}
And there we have basically the same test three times. Here we can use typed tests, which makes our life a bit easier. In Catch2 we could write this like this:
// we use TEMPLATE_TEST_CASE this time
// where we append all types for which we want to run the test
TEMPLATE_TEST_CASE("container empty", "", std::vector<int>, std::list<int>, std::deque<int>)
{
// TestType uses the given types
// since every element type is int, we initialize the container with 1, 2 and 3
// and we check if the container is not empty
TestType container{1,2,3};
REQUIRE(!container.empty());
}
This means that we end up running this test three times, for std::vector<int>, std::list<int> and std::deque<int>. I'll prepare the implementation in a rather simple way and explain how you could do this. This is primarily a proof of concept and not supposed to be ready for production.
With that said, we are going to do the following:
- We'll use a typelist where we store all our types from the macro
- The testbody is going to be templated
- We add a call functions which calls the templated function with each type
Let's break this down with a simple function foo:
// this is our typelist
template<typename... Types>
struct typelist{};
// forward declared foo which is our test function
// this means the a possible macro TEMPLATED_TEST(.., ..., ...)
// begins here
template<typename TestType>
void foo();
// for an empty list we'll have this empty function
void foo_call_with_typelist(typelist<>) {}
// and for typelists with types we have this one
template<typename First, typename... Rest>
void foo_call_with_typelist(typelist<First, Rest...>)
{
// with the first type we call foo
foo<First>();
// with the rest of the types we call this again
// until we end up in an empty typelist
foo_call_with_typelist(typelist<Rest...>());
}
// this is our function which we'll store in class test_case later
// note: we need a void function without arguments to put it into our vector
void foo_caller()
{
// and inside this body we create the typelist.
// with the macro this will be typelist<__VA_ARGS__>{}
// but for this demonstration we leave int, float, double there
foo_call_with_typelist(typelist<int, float, double>{});
}
// and now we come to the function body where foo is implemented
// here is then the end for the macro later
template<typename TestType>
void foo()
{
std::cout << "executed with T=" << typeid(TestType).name() << std::endl;
}
Find this snippet here on Compiler Explorer.
This you could put now behind a macro to achieve the same mechanism here. But the problem is that I didn't start properly here. Because now you would need to add the number of types or the number of actual tests correctly to the test results. But lets recall, we add the test result initially before the call where we call all our tests in run:
int run()
{
for (const auto& test : test_cases())
{
// for templated tests this would be correct because
// most likely there would be more than one test
test_results().push_back(result::passed);
test.call();
}
// ...
}
I could create a dirty way to put the correct number of test results into our test result vector, but it won't change much, and I don't plan on writing an actual test framework. So I'll stop here. If you are interested in how to get the number of elements in a typelist, this would do the job:
// base template to define a typelist
template<typename... Types>
struct typelist {};
// base case: an empty Typelist has size 0
template<typename List>
struct size;
template<>
struct size<typelist<>> {
static const int value = 0;
};
// recursive case: a Typelist with at least one type
template<typename First, typename... Rest>
struct size<typelist<First, Rest...>> {
static const int value = 1 + size<typelist<Rest...>>::value;
};
int main() {
using some_typelist = typelist<int, double, char, float>;
std::cout << "Size: " << size<some_typelist>::value << std::endl;
return 0;
}
And that brings me to my conclusion. I think it's a cool project to do, and I think you'll learn a lot. As far as I'm concerned about the macros, it's probably not widely used in modern C++, but popular test frameworks like Googletest or Catch2 use it to achieve the demonstrated effect. I think if you plan to create such a framework, you'd need a good reason to do so. As far as I'm concerned, these frameworks are very flexible and you'll get the most out of them.
Maybe this paradigm of using macros to register functions will help you in other ways, like it did for me in CWT-Cucumber. And if not, and you have read this far, I hope you enjoyed it :)
Until the next one.
Cheers, Thomas
