Testing the Assertions

[tothambrus11]

When testing software, it is nice if we can test everything that we want to make sure work. Assertions, precondition checks are hopefully never executed in production because that would signify a bug in our code, but during development, they serve as an essential tool to discover bugs early on, and correctly refuse invalid inputs to our function when the developer using our API did something wrong.

As a library author, proper assertions are part of the delivered work, so it is arguably important to test their behaviour. If tests are too afraid to go into testing invalid territory, the assertions are never exercised, and might accidentally contain bugs in them or be completely removed without being noticed.

Note, making everything return a Result or Optional type is not a solution, sometimes it is handy to 'trust' the user in some way, e.g. when encoding all invariants of a type or properties of inputs would be overcomplicating our types. Therefore, the need for assert/precondition won't disappear.

Different languages have various solutions for supporting testing assertions:

Java

All assertion failures are just _Exception_s, so they can be caught using try/catch, just as regular exceptions. Note that AssertionError, IllegalStateException and their friends are unchecked runtime exceptions, meaning that they are not part of the method's signature nor should be documented. Generally, it is recommended to throw checked exceptions when we expect the caller to be able to recover and handle the exception, and throw unchecked exceptions when the problem came from an unexpected programming error that cannot be reasonably handled by the application, and the best thing to do is to crash the application.

The AssertJ library provides a quite impressive set of testing facilities for checking the type of exception, its error message, and more: https://www.baeldung.com/assertj-exception-assertion

assertThatExceptionOfType(IndexOutOfBoundsException.class)
  .isThrownBy(() -> { ... })
  .hasMessageMatching("Index: \\d+, Size: \\d+");

While this provides a lot of flexibility, it also has many downsides:

• People sometimes catch all exceptions at once, sometimes catching unintended runtime exceptions. (Can be hard to use right)
• This strategy is not ideal for a systems programming language, as expecting every function to be throwing and that all those errors are possibly recovered in an outer exception handler. (Note: as seen in Rust, they can do this with compiling their code with test unwinding enabled for testing purposes)

Swift

In Swift, there is currently no way to test precondition and assert statements, nor problems with unwrapping empty optionals or out of bounds access, other than by writing death tests/exit tests, which create a new thread or process which can be tested whether failed. These testing methods were not encouraged and were not straightforward to write. However, now there is a new proposal for exit tests for Swift Testing that could streamline this: https://github.com/swiftlang/swift-testing/blob/jgrynspan/exit-tests-proposal/Documentation/Proposals/NNNN-exit-tests.md

Death tests are 100-1000x slower than simple unit tests (based on my GTest death test experience in C++ they can take 1-2 seconds each), so they are not a viable solution for proper unit testing at scale.

People have the option to encode the possibility for failure in the return types, such as a Result/Optional type or by making their method throwing. By nature, Swift seems to require less assertions than other languages because we can encode our true intent much more accurately and easily than in other languages.

C++

In C++, there are multiple ways to do error handling of assertions:

• An implementation-defined assert(...) function that exits the program in debug mode (or also in production, if specified by a compiler flag) when the provided expression evaluates to false.
• Exceptions (many projects disable them for various reasons)
• People get creative and write their own assertion macros that print to console, then exit.

When exceptions are enabled in a C++ project, any function may throw exceptions, and we need to mark methods noexcept to guarantee to the callers that a function won't throw. This is arguably the wrong default, which is flipped by Swift, though concurrency with cancellations might be tricky to implement this way (see #1261 ).

In C++, many projects have exceptions disabled because of the following reasons:

• Executable size is increased, sometimes it's infeasible for embedded applications.
• In low-latency applications, it is hard to guarantee the maximal propagation time of an exception. (There are tools for this but they might not be available for all target platforms / toolchains.
• Exceptions are not well-defined to be used at ABI boundaries. Note: modules can expose a noexcept API while using exceptions internally.
• Immature optimization or bad management: the boss or headquarters told the team to disable exceptions, no trade-offs considered.

Testing asserts made by throwing exceptions is straightforward, just as in Java.
The usage of the assert function from C or custom logging assertion functions may either require death tests, or don't terminate the program early, causing possible UB afterwards. Death tests can be expensive, 1-2 seconds for startup using GTest.

Rust

Rust doesn't have tradition exceptions for error handling but they provide the Result type, which is used by Swift under the hood to work with exceptions. Rust doesn't provide any syntactic sugar for working with errors. Pattern matching and monadic operations can help a long way, and can be satisfying to write, but in my experience, too many maps, flatMaps, etc make the code hard to read, and sometimes they can be often confused with maps over collections when the types are implicitly written.

Rust provides the std::assert! and std::debug_assert! macros for asserting the program's correctness. The shorter version cannot be disabled even in production, while debug_assert only runs in debug builds. They both panic if their argument evaluates to false.

They provide an interesting approach to testing assertions. Rust programs are only supposed to panic if they are truly experiencing a bug or an unrecoverable state. But they can be still recovered using the catch_unwind(closure) -> Result function. This is great for testing, but also for interop with foreign code that doesn't properly handle unwinding panics. Panics can be turned into an immediate abort with a compiler flag, which makes it impossible to catch them in tests, but might come with binary size and performance improvements.
(I am not too familiar with the Rust ecosystem, so this needs to be researched further.)

Conclusions

• Being able to test assertions is important.
• Sacrificing small runtime cost during testing can be beneficial, especially if they make us avoid death tests.
• Unchecked exceptions are not meant to be caught (usually). Tests, or long-running services that want to keep running anyhow might want to catch these.
• Hylo may use a special unwinding mechanism like in Rust for testing unchecked assertions that are not part of the API.
• Normally, assertions could be turned into abort in production to avoid even worse things happening than immediate exiting. Debug assertions should be marked with a longer name (e.g. debug_assert()), which can be used in performance-intensive places where they would be removed in production builds.

I would love to hear your thoughts about this topic and hear about your related experience, opinions or implementation ideas.