Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GHA to run the MCS conformance tests #1630

Open
tpantelis opened this issue Aug 26, 2024 · 3 comments · May be fixed by #1642
Open

Add GHA to run the MCS conformance tests #1630

tpantelis opened this issue Aug 26, 2024 · 3 comments · May be fixed by #1642
Assignees
Labels
enhancement New feature or request

Comments

@tpantelis
Copy link
Contributor

This would identify and keep us aware of which aspects of the MCS spec where we may be non-compliant. Initially, we could just have the job run on-demand and eventually run on every PR once we're fully compliant.

@tpantelis tpantelis added the enhancement New feature or request label Aug 26, 2024
@tpantelis tpantelis self-assigned this Aug 26, 2024
@dfarrell07 dfarrell07 self-assigned this Aug 27, 2024
@dfarrell07
Copy link
Member

There's likely a way to include these tests in our E2E suite.

@tpantelis
Copy link
Contributor Author

tpantelis commented Aug 27, 2024

There's likely a way to include these tests in our E2E suite.

I think these should be separate from the E2E tests. I propose we create a "conformance" dir under "test" (sibling to "e2e") with a conformance_test.go file that imports sigs.k8s.io/mcs-api/conformance and invokes TestConformance. The GHA would run go test in "test/conformance". There's probably more details to work out but that's the general idea.

@tpantelis
Copy link
Contributor Author

I tried this out and got it to run the conformance tests. The problem though is that LH imports v0.1.0 of sigs.k8s.io/mcs-api but it doesn't have the conformance tests and the current master branch has some incompatible changes to the MCS APIs so we can't import it in the top-level go.mod. So I added a separate go.mod in the test/conformance module to get around that but that causes issues with the Go linting job. I don't think think this approach is viable.

There is a new sigs.k8s.io/mcs-api release planned soon so we can wait for that although we'll have to do that in the next release due to MCS API changes. Either way, I think it would be better to clone the sigs.k8s.io/mcs-api repo with the desired commit in the GHA and run go test directly in the directory since the conformance tests are really tied to the MCS spec and not specifically to the API version so they'll likely evolve separately.

@tpantelis tpantelis linked a pull request Sep 19, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Next Version Candidate
Development

Successfully merging a pull request may close this issue.

2 participants