CMake and Test Suites

The FindGTest module that ships with CMake provides the command GTEST_ADD_TESTS that registers all the test cases from a test executable with CMake so they are listed individually in the report generated by CTest. The following snippet shows its use:

enable_testing()
find_package(GTest REQUIRED)
include_directories(${GTEST_INCLUDE_DIRS})

set(SOURCES foo.cpp bar.cpp cow.cpp)
add_executable(foobar ${SOURCES})
target_link_libraries(foobar ${GTEST_BOTH_LIBRARIES})

GTEST_ADD_TESTS(foobar "" ${SOURCES})

As we see, the GTEST_ADD_TESTS command receives the list of source files. Starting from CMake version 3.1, we can also pass AUTO instead of a list of sources which tells GTEST_ADD_TESTS to retrieve the list of source files from the SOURCES target property. Either way, it then analyzes the source code with some regular expressions to find all the test cases.

This approach is problematic. Whenever we add or remove test cases, CMake needs to rerun. This approach also fails to find test cases where the header spans two lines …

TEST(SomeLongSectionName,
     AnEvenLongerTestName)
{
  ...
}

… and it wrongly finds test cases in comments and inactive #if sections:

/* Add tests like this:
TEST(Section, Test) {
  put the test code here.
}
*/

Why are the sources parsed at all? The list of test cases can be easily retrieved from the compiled test executable via the command line. Instead of parsing the source code, we could parse the output:

execute_process(COMMAND "/path/to/foobar" --gtest_list_tests
  OUTPUT_VARIABLE output)
string(REPLACE "\n" ";" lines "${output}")
set(section)
foreach(line IN LISTS lines)
  if(line MATCHES "^([A-Za-z_/0-9]+)\\.$")
    set(section "${CMAKE_MATCH_1}.")
  elseif(line MATCHES "^  ([A-Za-z_/0-9]+)")
    add_test("${section}${CMAKE_MATCH_1}"
      "/path/to/foobar" --gtest_filter="${section}${CMAKE_MATCH_1}")
  endif()
endforeach()

Putting the above snippet of CMake code into our CMakeLists.txt file requires the test executable to be available at configure time. Of course, that does not make any sense at all. No, we want this code to be in the CTestTestfile.cmake file!

What we need is a CMake command that writes the above block of code and we pass it just the options that are specific to the test framework in use.

Such a command might be used for GTest:

add_test_suite(foobar
  LIST_TEST_FLAGS   "--gtest_list_tests"
  SELECT_TEST_FLAGS "--gtest_filter=<SECTION><TEST>"
  SECTION_REGEX     "^([A-Za-z_/0-9]+)\\.$"
  TEST_REGEX        "^  ([A-Za-z_/0-9]+)"
  )

And for Catch:

add_test_suite(foobar
  LIST_TEST_FLAGS   "--list-test-names-only"
  SELECT_TEST_FLAGS "<TEST>"
  SECTION_REGEX     "^$"
  TEST_REGEX        "^(.+)$"
  )

And for GLib Test:

add_test_suite(foobar
  LIST_TEST_FLAGS   "-l"
  SELECT_TEST_FLAGS "-p <TEST>"
  SECTION_REGEX     "^$"
  TEST_REGEX        "^(.+)$"
  )

And for Qt Test:

add_test_suite(foobar
  LIST_TEST_FLAGS   "-functions"
  SELECT_TEST_FLAGS "<TEST>"
  SECTION_REGEX     "^$"
  TEST_REGEX        "^([^(]+)$"
  )

And a sad running joke: For Boost.Test it would be possible to, but only with the version on trunk.