Skip Mode #6039
Open
Skip Mode #6039
+2,873
−859
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wxtim
force-pushed
the
feature.skip_mode
branch
2 times, most recently
from
b43d296
to
d26c315
Compare
wxtim
force-pushed
the
feature.skip_mode
branch
from
d378e3a
to
e9c2e25
Compare
wxtim
requested review from
oliver-sanders,
markgrahamdawson and
MetRonnie
and removed request for
markgrahamdawson
MetRonnie
reviewed
Apr 23, 2024
wxtim
force-pushed
the
feature.skip_mode
branch
4 times, most recently
from
3d002ee
to
dd0a9fc
Compare
MetRonnie
reviewed
May 8, 2024
wxtim
force-pushed
the
feature.skip_mode
branch
3 times, most recently
from
0d42eab
to
c37ef72
Compare
wxtim
force-pushed
the
feature.skip_mode
branch
2 times, most recently
from
3839a8a
to
d7ee653
Compare
MetRonnie
reviewed
May 20, 2024
* Add `[runtime][<namespace>]run mode` and `[runtime][<namespace>][skip]`. * Spin run mode functionality into separate modules. * Run sim mode check with every main loop - we don't know if any tasks are in sim mode from the scheduler, but it doesn't cost much to check if none are. * Implemented separate job "submission" pathway switching. * Implemented skip mode, including output control logic. * Add a linter and a validation check for tasks in nonlive modes, and for combinations of outputs * Enabled setting outputs as if task ran in skip mode using `cylc set --out skip`. * Testing for the above. Schema: use `Enum` for task run mode instead of `String` (#61) * Schema: use `Enum` for task run mode instead of `String` * Tidy fixup merge fix broken functional test Improve cylc set --out skip * Improve documentation of feature in cylc set --help * Allow cylc set --out=skip,optional_output * Test the above Remove test: We don't want users opting out of validating [runtime][ns][simulation/skip] because we can now changes these in a running workflow. stop users opting out of validating workflows without validating ski/simulation taskdef sections added tests for db entries in nonlive mode ensure db entries for all four modes are correct. move the change file toi the correct name get localhost of platforms 'simulation' 'skip' or 'dummy' not defined. (They probably shouldn't be, but that's a site specific choice...) fix tests with extra messages surfaces by using log_filter make cylc clean and remote tidy not try to clean or tidy platforms stop dummy mode appearing to submit twice Prevent cleanup from attempting to remote clean platforms skip and simulation Update cylc/flow/run_modes/skip.py Co-authored-by: Oliver Sanders <[email protected]> fix small issues from OS review response to review: Make satisfaction method correct according the proposal Response to review * Allow only run modes skip and live to be selectable in the config. * Disallow workflow run modes sim and dummy from being overridden. * Attach Run mode to task_proxy rather than the task def. Response to review * Allow only run modes skip and live to be selectable in the config. * Disallow workflow run modes sim and dummy from being overridden. * Attach Run mode to task_proxy rather than the task def. don't run sim time check unless workflow in sim mode test that skip mode is only applied to a single task. remove now illegal items from test Response to review: - Remove Workflow Mode for tasks and make them default to live. - Ensure that we are checking (assert) log_filters. - Remove need for polling in functional test. :) usin enums Apply suggestions from code review Co-authored-by: Oliver Sanders <[email protected]>
Co-authored-by: Oliver Sanders <[email protected]>
values for ___retry_delays, as the lack of these causes failure on task retry or re-triggering.
Co-authored-by: Ronnie Dutta <[email protected]>
* Rename generalized run mode validation function to only checking skip mode validation.
Co-authored-by: Ronnie Dutta <[email protected]>
Co-authored-by: Ronnie Dutta <[email protected]>
wxtim
force-pushed
the
feature.skip_mode
branch
from
a78b95d
to
0c5e639
Compare
MetRonnie
reviewed
Oct 31, 2024
cylc/flow/task_outputs.py
Outdated
Comment on lines
637
to
638
| exclude: Exclude one possible required message, allowing | ||
| specification of all required outputs if succeeded or failed. |
There was a problem hiding this comment.
Could this be explained better? I am not sure what this means (though judinging from the tests it is not as simple as just excluding the specified output from the return value, which was my first impression)
wxtim
force-pushed
the
feature.skip_mode
branch
from
aeca3e4
to
2aa3f1c
Compare
wxtim
force-pushed
the
feature.skip_mode
branch
from
2aa3f1c
to
51f6585
Compare
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #5641 (also fixes #5975, and fixes #5820 )
Skip Mode Proposal Doc
This branch includes (marked against skip mode proposal):
[runtime][<namespace>]run mode.a. Broadcast can change
run modefor future task job submissions.b. Cylc Validate and lint will warn about the setting not being live.
cylc set --out skipsets outputs from skip mode.run mode = skiprespectsis_heldflag.Extras
7. Run Mode is available as an task attribute in the UI
8. When tasks are run in skip mode, the prerequisites which correspond to the outputs they generate should be marked as satisfied by skip mode rather than satisfied naturally for provenance reasons. For the purpose of cylc remove logic, satisfied by skip mode should be treated the same as satisfied naturally.
There are two extensions, which I haven't dealt with yet, because I want to ensure that the basic functionality works, and move to the substantial documentation PR which need follow this.
Check List
CONTRIBUTING.mdand added my name as a Code Contributor.setup.cfg(andconda-environment.ymlif present).CHANGES.mdentry included if this is a change that can affect users