Proposed testing framework for Standard Library facilities

This "testing framework" is based on the LLVM Integrated Tester,
which is used to implement the libc++ conformance test suite.
In turn, this conformance suite is also used by other non-LLVM
implementations to check their conformance.

Using this testing framework means that papers contributed to
the Beman project will already include tests that can be reused
by the major implementations, which is both a great time saver
for implementers but also a great way for implementers to get
experience with implementing the paper within their own
implementation and provide feedback to LEWG during design
reviews.

.ci/docker/rockylinux.Dockerfile

@@ -11,7 +11,8 @@ RUN dnf install -y \
         clang \
         g++ \
         ninja-build \
-        cmake
+        cmake \
+        python3.11-venv
 RUN dnf clean all
 
 # Copy code

.ci/docker/ubuntu.Dockerfile

@@ -8,7 +8,8 @@ RUN apt-get install -y \
         clang-tidy \
         g++ \
         ninja-build \
-        cmake
+        cmake \
+        python3.11-venv
 RUN apt-get clean
 
 WORKDIR /workarea

.github/workflows/cxx.yml

@@ -80,4 +80,4 @@ jobs:
           cache-to: type=gha,mode=max
       - name: Run tests
         run: |
-          docker run ${{ matrix.cfg.id }} ctest --test-dir build
+          docker run ${{ matrix.cfg.id }} ctest --test-dir build --output-on-failure

README.md

@@ -102,6 +102,61 @@ cmake -B /some/build/dir -S . -DCMAKE_CXX_CLANG_TIDY="clang-tidy;-checks=-*,cppc
 Otherwise follow the Basic Build workflow as described above.
 
 
+### Testing
+
+Tests are written using `lit`, the [LLVM Integrated Tester][]. Each test is an
+independent executable with a `main` function. Tests are written in a pretty
+basic way, using `assert`. If you want, you can use an arbitrary testing framework
+on top of that. However, we recommend against doing so because keeping the tests
+basic makes them very portable, and in particular it makes them compatible with
+the LLVM C++ Standard Library conformance test suite developed as part of libc++,
+which is also reused by other implementations like the MSVC STL and libstdc++.
+
+#### Writing tests
+
+Tests are written as standalone executables with a `main` function. Tested conditions
+are asserted using simple `assert(...)` statements. Various test flavors are supported:
+
+- `.pass.cpp`
+    These tests are built and run. The test succeeds if the program terminates normally
+    with a `0` exit code. The test fails if it doesn't compile, link or if it exits with
+    an error.
+- `.verify.cpp`
+    These tests run using `clang-verify`. This allows checking that specific diagnostics
+    are being emitted at compile-time.
+
+    Clang-verify supports various directives like `expected-error`, `expected-warning`, etc.
+    The full set of directives supported and how to use them is documented in [the Clang
+    documentation](https://clang.llvm.org/docs/InternalsManual.html#specifying-diagnostics).
+- `.compile.pass.cpp`
+    These tests don't run, they only compile. The test passes if the program compiles, and
+    fails otherwise. This can be used to test things like `aliases` or concept conformance,
+    which don't need to actually run anything.
+- `.sh.pass.cpp`
+    These tests run whatever shell commands are specified in the `RUN` commands specified
+    in the test. This provides a lot of flexibility for controlling how the test gets built
+    and run, and can be used to check things that are otherwise difficult to test (e.g.
+    compatibility between TUs built with different Standard modes).
+
+#### Running tests
+
+To run all the tests in the project, use:
+
+```shell
+$ ctest --test-dir /some/build/dir
+```
+
+You can run a single test in the project by using `lit` directly:
+
+```shell
+$ /some/build/dir/test/venv/bin/lit /some/build/dir/test/example-paper/test1.pass.cpp
+```
+
+That's not a typo, you must use the `/some/build/dir` directory as a prefix to the
+path of the test you want to run. This lets `lit` find the testing configuration
+that was generated by CMake.
+
+
 ## Usage
 
 ### From C++
@@ -141,6 +196,8 @@ Please do! Issues and pull requests are appreciated.
 Note that adding more C++ code will be out of scope for this project. Changes that further improve or simplify this project given that goal are appreciated. Enhancements to better support packaging ecosystems would also make sense.
 
 
+[LLVM Integrated Tester]: https://llvm.org/docs/CommandGuide/lit.html
+
 <!--
 Creative Commons Legal Code
 

src/example/example.hxx

@@ -1,2 +1,15 @@
 // Copyright © 2024 Bret Brown
 // SPDX-License-Identifier: MIT
+
+#ifndef EXAMPLE_HXX
+#define EXAMPLE_HXX
+
+#include <type_traits>
+
+template <class T>
+constexpr bool foo() {
+    static_assert(std::is_trivially_copyable_v<T>);
+    return true;
+}
+
+#endif

test/example/CMakeLists.txt

@@ -1,8 +1,30 @@
-add_executable(example.test)
-target_sources(example.test PRIVATE example.test.cxx)
-target_link_libraries(example.test PRIVATE example::example)
+# Setup the `lit` tool
+add_custom_command(
+    OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/venv/bin/lit"
+    COMMAND python3 -m venv "${CMAKE_CURRENT_BINARY_DIR}/venv"
+    COMMAND "${CMAKE_CURRENT_BINARY_DIR}/venv/bin/pip" install --upgrade pip
+    COMMAND "${CMAKE_CURRENT_BINARY_DIR}/venv/bin/pip" install --upgrade lit
+)
+
+# Setup the test suite configuration
+configure_file("${CMAKE_SOURCE_DIR}/test/support/lit.cfg.in"
+               "${CMAKE_CURRENT_BINARY_DIR}/lit.cfg")
+
+add_custom_target(test-depends
+    COMMAND true
+    DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/venv/bin/lit"
+            "${CMAKE_CURRENT_BINARY_DIR}/lit.cfg"
+            example
+    COMMENT "Setup the test dependencies"
+)
+
+add_test(
+    NAME setup-tests
+    COMMAND "${CMAKE_COMMAND}" --build "${CMAKE_BINARY_DIR}" --target test-depends
+)
 
 add_test(
   NAME example.test
-  COMMAND example.test
+  COMMAND "${CMAKE_CURRENT_BINARY_DIR}/venv/bin/lit" -sv "${CMAKE_CURRENT_BINARY_DIR}"
 )
+set_tests_properties(example.test PROPERTIES DEPENDS setup-tests)

test/example/test1.pass.cpp

@@ -0,0 +1,14 @@
+//
+// This test gets built and run. The test succeeds if the program
+// terminates normally with a 0 exit code. The test fails if it
+// doesn't compile, link or if it exits with an error.
+//
+
+#include <cassert>
+#include <example.hxx>
+
+int main(int, char**) {
+    assert(foo<int>());
+
+    return 0;
+}

test/example/test2.verify.cpp

@@ -0,0 +1,15 @@
+//
+// This test runs using clang-verify. This allows checking that specific
+// diagnostics are being emitted at compile-time.
+//
+// Clang-verify supports various directives like 'expected-error',
+// 'expected-warning', etc. The full set of directives supported and
+// how to use them is documented in https://clang.llvm.org/docs/InternalsManual.html#specifying-diagnostics.
+//
+
+#include <string>
+#include <example.hxx>
+
+void f() {
+    foo<std::string>(); // expected-error@*:* {{static assertion failed due to requirement 'std::is_trivially_copyable_v<std::string>'}}
+}

test/example/test3.compile.pass.cpp

@@ -0,0 +1,8 @@
+//
+// This test doesn't run, it only compiles.
+// The test passes if the program compiles, and fails otherwise.
+//
+
+#include <example.hxx>
+
+static_assert(foo<int>());

test/example/test4.sh.cpp

@@ -0,0 +1,24 @@
+//
+// This test runs whatever shell commands are specified in the RUN commands
+// below. This provides a lot of flexibility for controlling how the test
+// gets built and run.
+//
+
+// RUN: %{cxx} %{flags} %s -DTRANSLATION_UNIT_1 -c -o %t.tu1.o
+// RUN: %{cxx} %{flags} %s -DTRANSLATION_UNIT_2 -c -o %t.tu2.o
+// RUN: %{cxx} %{flags} %t.tu1.o %t.tu2.o -o %t.exe
+// RUN:  %t.exe
+
+#include <example.hxx>
+
+#ifdef TRANSLATION_UNIT_1
+void f() { }
+#endif
+
+#ifdef TRANSLATION_UNIT_2
+extern void f();
+
+int main() {
+    f();
+}
+#endif

test/support/lit.cfg.in

@@ -0,0 +1,21 @@
+import site, os
+site.addsitedir(os.path.join('@CMAKE_SOURCE_DIR@', 'test', 'support'))
+import testformat
+
+config.name = 'Beman project test suite'
+config.test_format = testformat.CxxStandardLibraryTest()
+config.test_exec_root = '@CMAKE_CURRENT_BINARY_DIR@'
+config.test_source_root = '@CMAKE_CURRENT_SOURCE_DIR@'
+flags = [
+    '-std=c++20',
+    '-isysroot @CMAKE_OSX_SYSROOT@' if '@CMAKE_OSX_SYSROOT@' else '',
+    '-isystem {}'.format(os.path.join('@CMAKE_SOURCE_DIR@', 'src', 'example')),
+    '@CMAKE_CXX_FLAGS@'
+]
+config.substitutions = [
+    ('%{cxx}', '@CMAKE_CXX_COMPILER@'),
+    ('%{flags}', ' '.join(filter(None, flags)))
+]
+config.available_features = []
+if '@CMAKE_CXX_COMPILER_ID@' in ('Clang', 'AppleClang'):
+    config.available_features.append('verify-support')

test/support/testformat.py

@@ -0,0 +1,91 @@
+import lit
+import os
+import re
+
+def _parseScript(test, preamble):
+    """
+    Extract the script from a test, with substitutions applied.
+
+    Returns a list of commands ready to be executed.
+
+    - test
+        The lit.Test to parse.
+
+    - preamble
+        A list of commands to perform before any command in the test.
+        These commands can contain unexpanded substitutions, but they
+        must not be of the form 'RUN:' -- they must be proper commands
+        once substituted.
+    """
+    # Get the default substitutions
+    tmpDir, tmpBase = lit.TestRunner.getTempPaths(test)
+    substitutions = lit.TestRunner.getDefaultSubstitutions(test, tmpDir, tmpBase)
+
+    # Parse the test file, including custom directives
+    scriptInTest = lit.TestRunner.parseIntegratedTestScript(test, require_script=not preamble)
+    if isinstance(scriptInTest, lit.Test.Result):
+        return scriptInTest
+
+    script = preamble + scriptInTest
+    return lit.TestRunner.applySubstitutions(script, substitutions, recursion_limit=10)
+
+class CxxStandardLibraryTest(lit.formats.FileBasedTest):
+    def getTestsForPath(self, testSuite, pathInSuite, litConfig, localConfig):
+        SUPPORTED_SUFFIXES = [
+            "[.]pass[.]cpp$",
+            "[.]compile[.]pass[.]cpp$",
+            "[.]sh[.][^.]+$",
+            "[.]verify[.]cpp$",
+        ]
+
+        sourcePath = testSuite.getSourcePath(pathInSuite)
+        filename = os.path.basename(sourcePath)
+
+        # Ignore dot files, excluded tests and tests with an unsupported suffix
+        hasSupportedSuffix = lambda f: any([re.search(ext, f) for ext in SUPPORTED_SUFFIXES])
+        if filename.startswith(".") or filename in localConfig.excludes or not hasSupportedSuffix(filename):
+            return
+
+        yield lit.Test.Test(testSuite, pathInSuite, localConfig)
+
+    def execute(self, test, litConfig):
+        supportsVerify = "verify-support" in test.config.available_features
+        filename = test.path_in_suite[-1]
+
+        if re.search("[.]sh[.][^.]+$", filename):
+            steps = []  # The steps are already in the script
+            return self._executeShTest(test, litConfig, steps)
+        elif filename.endswith(".compile.pass.cpp"):
+            steps = ["%dbg(COMPILED WITH) %{cxx} %s %{flags} -fsyntax-only"]
+            return self._executeShTest(test, litConfig, steps)
+        elif filename.endswith(".verify.cpp"):
+            if not supportsVerify:
+                return lit.Test.Result(
+                    lit.Test.UNSUPPORTED,
+                    "Test {} requires support for Clang-verify, which isn't supported by the compiler".format(test.getFullName()),
+                )
+            steps = ["%dbg(COMPILED WITH) %{cxx} %s %{flags} -fsyntax-only -Xclang -verify -Xclang -verify-ignore-unexpected=note -ferror-limit=0"]
+            return self._executeShTest(test, litConfig, steps)
+        elif filename.endswith(".pass.cpp"):
+            steps = [
+                "%dbg(COMPILED WITH) %{cxx} %s %{flags} -o %t.exe",
+                "%dbg(EXECUTED AS) %t.exe",
+            ]
+            return self._executeShTest(test, litConfig, steps)
+        else:
+            return lit.Test.Result(lit.Test.UNRESOLVED, "Unknown test suffix for '{}'".format(filename))
+
+    def _executeShTest(self, test, litConfig, steps):
+        if test.config.unsupported:
+            return lit.Test.Result(lit.Test.UNSUPPORTED, "Test is unsupported")
+
+        script = _parseScript(test, steps)
+        if isinstance(script, lit.Test.Result):
+            return script
+
+        if litConfig.noExecute:
+            return lit.Test.Result(lit.Test.XFAIL if test.isExpectedToFail() else lit.Test.PASS)
+        else:
+            _, tmpBase = lit.TestRunner.getTempPaths(test)
+            useExternalSh = False
+            return lit.TestRunner._runShTest(test, litConfig, useExternalSh, script, tmpBase)