Implement "cylc set" command

[hjoliver]

Implement accepted proposal A New Command for Manual Task Forcing

• Close set: implement the new command #5642
• Close Get rid of the hidden pool for partially satisfied tasks #5638
• Close Spawning parentless clock-xtriggered task #5572
• Close optional outputs: tasks which complete outputs on multiple tries marked as incomplete #5882

Supersede #5412 (fix expire triggers)

• Close :expire trigger broken #5364
• Close graph parser: expire-all ? #5354
• Close expiry: broken interaction with runahead limit #5447
• Close task "expired" event #4527

TODO

• Rename set-outputs to set
• Update the CLI and GraphQL schema
• Custom outputs: prohibit "all", "required", "optional" keywords, "," character, and "_cylc" prefix.
• Reproduce old set-outputs behaviour with new command options and query
• Implement the new logic for setting prerequisites [FIRST CUT done]
• Implement the new logic for setting outputs
• Test new schema works as expected in the UI
• Tests for all new functionality
• Update documentation -> Punted to document: cylc set cylc-doc#688
• This is a breaking change, add an entry into the cylc-doc changes page -> Punted to document: cylc set cylc-doc#688

Side effects:

• rename cylc set-verbosity to cylc verbosity to avoid command ambiguity apocalypse
• command UUIDs
• cleaner log message prefix for tasks

Follow-up:

• cylc set --pre=cycle

PR Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users

[hjoliver]

Rebased. Tests fixed. Will finish this off ASAP.

[hjoliver]

Unrelated (?) mypy error from the Py 3.10 and 3.11 fast tests:

/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/matplotlib/pyplot.py:2700:
 error: Positional-only parameters are only supported in Python 3.8 and greater  [syntax]
Found 1 error in 1 file (errors prevented further checking)

[update: fixed by #5735]

[hjoliver]

@oliver-sanders - I just took a quick look at your "Extension" part of the proposal:

When setting prereqs/outputs it would be useful to provide visual confirmation of the result.
We could display the cylc show output (which can be generated Scheduler side without the requirement for a second API call).

Correct me if I'm wrong, but to do this we need to take the set call out of the asynchronous command queue in the scheduler, and instead call it directly via a wrapper method in the resolver.

The trouble is, the set call updates task DB tables, and I get this:

self.conn.executemany(stmt, stmt_args_list)
    sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. 

[hjoliver]

@oliver-sanders - this now incorporates (and supersedes) #5412 because I needed to handle (and test consequences of, such as triggering) set to expired. So there will be a bit of overlap and conflict with #5651, but hopefully not too much.

[hjoliver]

@oliver-sanders - I think there's a small error in the proposal example 5

$ cylc set //foo --out=a,succeeded --wait

The database will record:

foo: state=waiting, outputs={a}

The DB should record state=succeeded and outputs={a, submitted, started, succeeded} because:

• completed outputs affect task status
• we set succeeded, and it has implied outputs

Agree?

[oliver-sanders]

I've gone back over all the unresolved comments, pretty sure everything has been addressed now

Yep all happy, can leave the lint stuff, just making a last pass...

Sorry, found another issue :(, hopefully a quick one:

If I pause a workflow (any graph where success is required):

- ~me
   - sequential/run1 - paused
      - ̿○ 1
           ̿○ a
      - ○ 2
           ○ a

Then set the failed output on any queued task:

cylc set //1/a --out=failed

- ~me
   - sequential/run1 - paused 1■
      - ̿⊗ 1
           ̿⊗ a
      - ○ 2
           ○ a

Then resume the workflow:

cylc play

The task submits a job and goes from failed to running:

- ~me
   - sequential/run1 - running 1■
      - ⊙ 1
         + ⊙ ■ a
      - ○ 2
           ○ a
      - ○ 3
           ○ a

Whereas, if I set 2/a:failed instead:

- ~me
   - sequential/run1 - paused 1■
      - ̿○ 1
           ̿○ a
      - ⊗ 2
           ⊗ a

Then the workflow stalls as expected:

- ~me
   - sequential/run1 - running 1■ 1■
      - ● 1
         + ● ■ a
      - ⊗ 2
           ⊗ a
      - ○ 3
           ○ a

I think setting a finished output on a queued task should cause it to be removed from the queue?

Curiously, the same doesn't appear to happen if I set 1/a:succeeded so presumably there's already some logic protecting against a completed task being (re)run?

[oliver-sanders]

I also spotted that the --pre option doesn't appear to work with the --wait option, which makes sense.

Another one for validation?

[hjoliver]

I also spotted that the --pre option doesn't appear to work with the --wait option, which makes sense.

Another one for validation?

👍 Added a comment to #6013

[hjoliver]

I think setting a finished output on a queued task should cause it to be removed from the queue?

Yes it should.

Curiously, the same doesn't appear to happen if I set 1/a:succeeded so presumably there's already some logic protecting against a completed task being (re)run?

That's because completed tasks leave the task pool (and the queue).

---

I've pushed a small fix. Still needs a test, and to check if I overdid the new DB updates, but I'm out of time for today.

It also cleans up the failed(queued) and failed(runahead) bits, which I'd been meaning to come back to anyway.

[wxtim]

needs a test

I'll have a go at that: hjoliver#48

and to check if I overdid the new DB updates

I'm not quite sure which updates these are... else I'd test for them too.

[oliver-sanders]

The workflow shuts down after 1069, rather than jumping the 1070 cycle and going to 1071. Is this expected behaviour?

Two things going on here:

1. The handling of task expiry.

   On this branch expired tasks are removed from the pool instantly, expiry can not count towards task completion.

   This will change with optional outputs: implement new proposal #5640 (also 8.3.0). After this change your example should stall rather than shutting down as completed.

2. The handling of parentless tasks.

   This part is working as expected, parentless tasks work by a somewhat similar logic to SoS in that task proxies spawn their successors (or more technically the task pool spawns parentless successors according to what is in the pool).

   E.G. If you remove all instances of a parentless task in the pool then you are also removing future instances from the workflow. SoD spawns on upstream outputs, parentless tasks have no upstream outputs so they don't fit the model great, it's not perfect, but it's good enough and we haven't come up with a better approach.

[wxtim]

Do we want to offer this suggestion?

Suggested change

	return (await _list_prereqs_and_outputs(tokens_list))[1]
	return (await _list_prereqs_and_outputs(tokens_list))[1] + ['required']

[oliver-sanders]

required applies only to outputs, this interface returns outputs AND prerequisites.

[wxtim]

I'm pretty sure that [1] is just the outputs:

prereqs, outputs = (await _list_prereqs_and_outputs(tokens_list))

[hjoliver]

I'll leave this to you two to agree as a follow-up.

[oliver-sanders]

Had another play today, I couldn't find any ways to break it 😓 🎉 !

Already building the next part on top of this branch, expect a PR in a week or so, distractions permitting.

Might want a rebase-squash, much as a love commit messages like "Fix screwed conflict resolution".

[MetRonnie]

I didn't do a full review but all my points have been addressed

[hjoliver]

Might want a rebase-squash, much as a love commit messages like "Fix screwed conflict resolution".

Squashed. Although I do genuinely love commit messages like that.

[hjoliver]

If the tests pass, this bad boy is going in.

[hjoliver]

Phew 😌