Parallel e2e jobs may hang indefinitely

[anmazzotti]

Sometimes the long running jobs on self-hosted runners will hang indefinitely.
I found the root cause to be the following restriction: https://cloud.google.com/compute/docs/machine-images/create-instance-from-machine-image#restrictions

The triggered job will fail creating the new VM with:

Operation rate exceeded for resource 'projects/my-project-name/global/machineImages/my-runner-image'. Too frequent operations from the source resource.

Currently this is not a problem as there are exactly 6 jobs running in parallel, only once a day. Adding a following job however will make it fail, as the limit is 6 per hour. A possible solution is documented in the link above, by taking the snapshot of a VM and use that as image.

[kkaempf]

Does this also happen with self-hosted runners ?

[anmazzotti]

Only with self-hosted runners.

[kkaempf]

Then please reach out to EIO on this.