Starter Project #2: Generating standalone e2e tests to prevent regression

[tylergu]

Description

We reported bugs to operator developers, and when developers fix the bugs, they often write regression tests along to prevent this bug to be introduced in the future.
We observed that these regression tests wrote by developers are very similar to how Acto runs the E2E tests, and it is possible for Acto to automatically generate them using the existing information we already have.
With this feature, it would save a lot time for developers to reproduce and fix the bug, it would also help ourselves to automatically reproduce alarms during inspection.

Actions

1. The first step would be to understand how Acto works and what results and information does Acto produce. So the first recommended action is to reproduce the cass-operator's experiment, go through the process of using Acto to get familiar with everything.
2. After the first step, you would be familiar with what information Acto produces for each alarm. I believe currently Acto already produces enough information to generate E2E tests for each alarm. So the second step would be to figure out the minimum steps to be included in the generated E2E test to reproduce the alarm. At this step, you can simply think about generating bash commands to reproduce the bug.
3. After having all the logical steps to reproduce the bug, the next step is to emit source code for regression test. You may need a bit of background in compiler (AST in particular) for this, but it can be quickly learned. Pick a widely-used test framework, and generate source code in that test framework to reproduce the bug from the 2nd step. I recommend ginkgo/gotest.
4. To push further, we can have some intermediate representation for the generated e2e test, so that we can generate code in different test frameworks. This requires us to have a clean interface to separate test logic and code generation.

[TwinIsland]

Will the result.csv file by Acto suffice for generating the e2e test?

[tianyin]

@TwinIsland Could you give more context about the question you are asking?

[TwinIsland]

I've observed that Acto generates a variety of outputs, such as result.csv, along with several intermediate files upon completing an operator test. In line with Taylor's insights:

I believe currently Acto already produces enough information to generate E2E tests for each alarm.

Based on my current understanding, the result.csv file appears to encompass all necessary information for the autonomous creation of comprehensive e2e tests. However, I seek confirmation to ensure my approach is accurate. Is the result.csv file indeed sufficient for this purpose on its own, or is there a requirement to utilize the intermediate files (Acto's output) as well?

[tianyin]

Got it. I believe you are thinking about using this project for the 523 course project?

is there a requirement to utilize the intermediate files

There is no any requirement in research :)) you should feel free to think about any ideas that can potentially solve the problem.

This task is a bit old (almost 10 months ago) and there is some limited regression tests @tylergu put (e.g., the tests we run for CI). I think we should sit down and discuss more about what are the more regression testing effort we will need.

[tylergu]

@TwinIsland The result.csv only contains the the metadata information of the alarms.
To reproduce the bug, you would need to required CRs to trigger the bug again, and more importantly, generating minimized code for each alarm.

[TwinIsland]

Got it. I believe you are thinking about using this project for the 523 course project?

is there a requirement to utilize the intermediate files

There is no any requirement in research :)) you should feel free to think about any ideas that can potentially solve the problem.

This task is a bit old (almost 10 months ago) and there is some limited regression tests @tylergu put (e.g., the tests we run for CI). I think we should sit down and discuss more about what are the more regression testing effort we will need.

Thanks for the clarification! I've already outlined a plan for my 523 course project. However, I remain genuinely interested in this task and am considering it as a hobby project, especially since my schedule is relatively chill this semester. Would pursuing this task in that context be a feasible approach?

[TwinIsland]

@TwinIsland The result.csv only contains the the metadata information of the alarms. To reproduce the bug, you would need to required CRs to trigger the bug again, and more importantly, generating minimized code for each alarm.

Got it, thanks!