Example 3: Failure Isolation

Based on a test execution in adverse conditions using Thor, it is often not possible to identify a single injection site where an event injection will cause the test to fail (for example, see Example 2: Adverse Conditions,). However, such a single injection site often exists.

The failure isolation mechanism addresses this issue (for details, see "Systematic Execution of Android Test Suites in Adverse Conditions").

It is used by selecting the algorithm Minimize, and providing a range of injection sites (default is 0-INF, meaning all injection sites in the test). We determined that test from Example 2: Adverse Conditions failed due to an injection at site #0, #1, #2 or #3. Hence, we might as well specify the range 0-3.

The below image shows the result of applying failure isolation to testCalculatorPlus. As it follows, failure isolation for this execution lead to 5 executions of the test case. The result is interpreted as follows:

• minimize-pause-stop-destroy-create: injecting events at all injection sites using the Pause-Stop-Destroy-Create event (in order to determine which of the selected events cause the test failure); failure means that the algorithm successfully detected the problematic event type

• minimize-pause-stop-destroy-create-0-2-4: injecting Pause-Stop-Destroy-Create in range #2-#4 (including #2 and #4); failure means that the algorithm identified a problematic injection in the right range, and hence splits the right range into two and continues the search in the right half

• minimize-pause-stop-destroy-create-2-3-4: injecting Pause-Stop-Destroy-Create in range #3-#4; failure means that the algorithm identified a problematic injection in the right range, and hence splits the right range into two and continues the search in the right half

• minimize-pause-stop-destroy-create-3-4-4: injecting Pause-Stop-Destroy-Create in range #4-#4; success means that the algorithm did not identify a problematic injection in the right range, and hence continues the search in the left range

• minimize-pause-stop-destroy-create-3-3-3: injecting Pause-Stop-Destroy-Create in range #3-#3; failure means that the algorithm identified aproblematic injection in the range, and hence the search stops

Note: since the last test execution of the search failed, the failure isolation was successful in identifying a single event type, which injected in a single injection site cause the test to fail.

If the last test execution of the search succeeded, the failure isolation would have been unsuccessful. This could happen for e.g. flaky tests, where a flakiness could mislead the search.