e2e tests for Agones integration

[markmandel]

Is your feature request related to a problem? Please describe.

Once we have the Agones integration from #506, we don't currently have a way to automatically test the integration end to end (which is not a good thing).

Describe the solution you'd like

So we need a way to run the e2e tests both locally and also in Cloud Build CI. We can probably break this down into a few steps/built in tools into our Makefile.

The only tricky part with local testing is that connectivity to something like Minikube on Agones can be tricky, because of the layers of virtualisation (see guide)

I believe we can mimic what we did in Agones to do a similar thing in Rust as we did with Go (sacrificial draft):

• We'll need a make target to install the latest version of Agones on the current authorised cluster (it's a helm install - pretty easy). (be good to have a terraform script that sets up the cluster too).
• All the e2e tests in Agones are standard Go tests, that use the local .kube credentials to run. It looks like Rust's kube client will happily do the same. This has two benefits - you can run individual tests in your IDE, and also run the whole suite with standard Rust tooling.
• For CI Agones has a custom build step that does the installation and running of tests - we could do similar things. The fun part here is that we need to use a shared lock to make sure no two tests are running at the same time (unless there is a way to make it so that it doesn't matter 🤔). In Agones we use consul to do the distributed lock, although I have debated implementing something using gcslock or similar.

Describe alternatives you've considered

None. We'll need automated testing for this.

Additional context

I would probably suggest starting with getting tests running locally in a Rust context, and then build out the CI platform from there.

[markmandel]

Starting work on this - starting with creating a simple terraform script for creating a agones powered GKE cluster, and creating a module that will create some kind of simple test we can expand on.

Will probably add some of the tooling (terraform, helm, gcloud etc) into the build image as well for convenience if they are not installed locally.

@XAMPPRocky it would handy if you could split out the Agones k8s client work in https://github.com/googleforgames/quilkin/blob/edf6d0d4fba93d5771b993af48f670563724cb6a/src/xds/provider/agones.rs into its own PR? Then I should be able to start building out more e2e tests.

No rush though, I'll get a PR with a Pod created first, and swing back around.

[markmandel]

Got this in a happy place locally:

• Setup the build image to have gcloud, kubectl and terraform in it, so that if you don't have it locally and want a quick fix, you can use make shell create and work with your K8s cluster
• Small terraform scripts to create a GKE cluster with Agones installed and appropriate Agones firewall rules (we may want these since creating Services can be slow.
• Small framework for running tests in it's own module, that creates a temporary namespace on startup for all resources to go in. Also removes all other testing namespaces each time it starts up.
• A singular test that creates Quilkin in a pod and makes sure it moves to a running state.
• Some make targets for integration with CI when ready.

Splitting out into PR's and submitting shortly 👍🏻 Just wanted to give you a headsup, since I know you've been waiting on this for xDS testing.

[markmandel]

Writing this down, so I remember (and commentary welcome) what to work on once #545 is merged:

• Write tooling to manage build, push and test (probably make targets, since that is the easiest) flow
• Do the same thing, but for minukube (likely do this in conjunction with above)
• Build out CI tooling so tests can be run for each PR (Also realised we don't need a global lock, and test can be run in parallel as long as we handle test Namespace cleanup appropriately)
• Document the development flow in build/README.md

Once xDS PRs have been merged:

• Write some tests that using GameServer, and expand framework (sidecar integration seems like the best starting point)
• Work out load balancer testing strategy that will both (a) work on GKE and Minukube and (b) be fast too test (spinning up external LBs in GKE can take minutes)
• Write an xDS test

I think at that point, we can close this ticket!

[XAMPPRocky]

I can write the xDS test once merged, I want to get more familiar with agones.

[markmandel]

Just adding a note I'm starting on "Write tooling to manage build, push and test (probably make targets, since that is the easiest) flow", and will move into "Do the same thing, but for minukube (likely do this in conjunction with above)" as well.

[markmandel]

😅 took some documentation hunting, but locally I've got the Agones integration tests working with the Cloud Build setup.

Going to optimise it to remove some duplication of work that is happening, but hoping to have this in before you all return back from vacation.

[markmandel]

Working on an integration test to spin up a single Agones GameServer and connect to it. Most of the way there, PR incoming shortly.

[markmandel]

Assuming #582 is merged 😄 I'd love to get started on setting up the some preliminary xDS integration tests. Also will give me an opportunity to get hands on with the feature, which will help doing some examples, probably a quickstart and other onboarding improvements I'd like to do.

[markmandel]

Work in progress on xDS tests: https://github.com/markmandel/quilkin/blob/agones/xds-test/agones/src/xds.rs

[XAMPPRocky]

Closing as complete, we can open up new issues for specific tests.