Handling of assertions in python module

[schnellerhase]

Describe new/missing feature

When the python module is build in Debug mode, assertions triggered in the cpp part yield a breakdown of the process, and especially it is not treated as a python event, such as AssertionError.

This for example is problem when an assertion is hit running in a pytest call, the stack trace of the failing test is not meaningful and does not indicate that the program abortion was caused by an 'expected' exit. (Expected in the sense, that when we build the python module in Debug mode, we are aware that we might need further feedback/checks coming from assertions, most likely due to debugging).

This was firstly noted in #3331 and further addressed in wjakob/nanobind#666.

Fixing this is not straight forward. A couple of thoughts regarding this:

1. Assertion hit in Debug mode built python module should not yield the program termination in the cpp module, but be propagated back to the python level.
2. Debug mode needs to be a proper resemblance of the Release build. For example this prohibits the usage of exceptions to facilitate this, as this would change function specifiers depending on build type.
3. Hitting an assertion should yield immediate execution termination - we must not continue on after hitting an assertion and produce the handling at a later stage.
4. We want to maintain the behavior of the assertions as is, as long as we do not export to a python module. So if we just build the cpp/ project there should be no difference to the current state.
5. One possible solution might be the wrapping of every python callback with a signal handler, which recovers some viable state in case of an exit signal being triggered during execution, to facilitate to return to the python module in a reasonable program state.

Suggested user interface

No response

[schnellerhase]

numpy deals with this by separating the assertion levels into a strict c and python part. The c functionality uses standard assert calls and the python module lays another layer of 'input checks' on top of it, which produce python assertions/events, that the user can handle. If I understand correctly that means that ´numpy´ has no option to handle a c assert caused by a python call in the desired way either.