Fix preemptibles and maxRetries on GCP Batch [AN-274] #7684
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mcovarr
force-pushed
the
an_274_fix_preemptibles
branch
2 times, most recently
from
6bf93e5 to
1a88d90
Compare
mcovarr
changed the title
mcovarr
force-pushed
the
an_274_fix_preemptibles
branch
from
21ab47b to
f87aff2
Compare
aednichols
approved these changes
Feb 5, 2025
| zone=$(basename "$fully_qualified_zone") | ||
|
|
||
| if [ "$preemptible" = "TRUE" ]; then | ||
| gcloud beta compute instances simulate-maintenance-event $(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/name" -H "Metadata-Flavor: Google") --zone=$zone -q |
|
|
||
| override val prettyPrintedError: String = "The job was aborted" | ||
| // TODO: Use this when detecting a preemption or remove it |
There was a problem hiding this comment.
Looks like we did use it so comment can be deleted
Comment on lines
+39
to
+40
| def toProvisioningModel(preemptible: Boolean): ProvisioningModel = | ||
| if (preemptible) ProvisioningModel.SPOT else ProvisioningModel.STANDARD |
There was a problem hiding this comment.
I appreciate this type fixup
| @@ -51,7 +51,6 @@ final case class GcpBatchRuntimeAttributes(cpu: Int Refined Positive, | |||
| object GcpBatchRuntimeAttributes { | |||
|
|
|||
| val ZonesKey = "zones" | |||
|
|
|||
| private val ZonesDefaultValue = WomString("us-central1-b") | |||
There was a problem hiding this comment.
For a different PR, but all zones in the region is a better default
| def StandardException(message: String, jobTag: String): Exception = | ||
| new Exception(s"Task $jobTag failed: $message") | ||
| private val VM_PREEMPTION_PATTERN = Pattern.compile( | ||
| "failed due to the following task event: \"Task state is updated from RUNNING to FAILED on zones/\\S+ due to Spot VM preemption with exit code 50001.\"" |
There was a problem hiding this comment.
Optional nice-to-have: document this message in the Batch section of RTD and maybe show how to locate it the operation details
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes the behavior of preemptibles and maxRetries on the GCP Batch backend. The tl;dr is that preemptibles and maxRetries should work essentially the same as they do on PAPI v2. For review purposes the "interesting" parts start at the "Fix preemptible / maxRetries on GCP Batch" commit; the preceding commits are just reverting earlier preemptible work and then merging develop in small steps.
Several preemptible and maxRetries Centaur tests now have GCP Batch versions thanks to the magic of
gcloud beta compute instances simulate-maintenance-event.The main problem addressed here is that while GCP Batch offers the ability to manage task retries, it turns out this is not a good fit for Cromwell:
With these changes Cromwell now manages preemptible and maxRetries retries itself, just as it does on PAPI v2, with the same
attempt-X"subdirectories" created for each preemptible or maxRetry attempt.Release Notes Confirmation
CHANGELOG.mdCHANGELOG.mdin this PRCHANGELOG.mdbecause it doesn't impact community usersTerra Release Notes