Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scheduler: task failure not logged when retries are configured #6151

Open
oliver-sanders opened this issue Jun 18, 2024 · 0 comments · May be fixed by #6169
Open

scheduler: task failure not logged when retries are configured #6151

oliver-sanders opened this issue Jun 18, 2024 · 0 comments · May be fixed by #6169
Assignees
Labels
bug Something is wrong :( small
Milestone

Comments

@oliver-sanders
Copy link
Member

A small logging bug where task failures can be omitted from the logs when execution retries are configured.

We are currently logging all task state changes, however, when retries are configured, the a failed task does not change state until all retries have been exhausted.

Reproducible Example

[scheduling]
    [[graph]]
        R1 = foo

[runtime]
    [[foo]]
        execution retry delays = PT1S
        script = """
            if [[ $CYLC_TASK_SUBMIT_NUMBER -gt 1 ]]; then
                true
            else
                false
            fi
        """
$ cylc play -N <workflow>

 ▪ ■  Cylc Workflow Engine 8.3.0.dev
 ██   Copyright (C) 2008-2024 NIWA
▝▘    & British Crown (Met Office) & Contributors

...
INFO - [1/foo:waiting(runahead)] => waiting
INFO - [1/foo:waiting] => waiting(queued)
INFO - [1/foo:waiting(queued)] => waiting
INFO - [1/foo:waiting] => preparing
INFO - [1/foo/01:preparing] submitted to localhost:background[91088]
INFO - [1/foo/01:preparing] => submitted
INFO - [1/foo/01:submitted] => running
INFO - [1/foo/01:running] => waiting
WARNING - [1/foo:waiting] retrying in PT1S (after 2024-06-18T09:56:09+01:00)
INFO - xtrigger satisfied: _cylc_retry_1/foo = wall_clock(trigger_time=1718700969.3399053)
INFO - [1/foo:waiting] => waiting(queued)
INFO - [1/foo:waiting(queued)] => waiting
INFO - [1/foo:waiting] => preparing
INFO - [1/foo/02:preparing] submitted to localhost:background[91380]
INFO - [1/foo/02:preparing] => submitted
INFO - [1/foo/02:submitted] => running
INFO - [1/foo/02:running] => succeeded
INFO - Workflow shutting down - AUTOMATIC
INFO - DONE
@oliver-sanders oliver-sanders added bug Something is wrong :( small labels Jun 18, 2024
@oliver-sanders oliver-sanders added this to the 8.3.1 milestone Jun 18, 2024
@wxtim wxtim self-assigned this Jun 25, 2024
@wxtim wxtim linked a pull request Jun 25, 2024 that will close this issue
8 tasks
@wxtim wxtim linked a pull request Jun 25, 2024 that will close this issue
8 tasks
@oliver-sanders oliver-sanders modified the milestones: 8.3.1, 8.3.x Jun 27, 2024
@oliver-sanders oliver-sanders modified the milestones: 8.3.x, 8.3.5 Oct 4, 2024
@oliver-sanders oliver-sanders modified the milestones: 8.3.5, 8.3.6 Oct 14, 2024
@oliver-sanders oliver-sanders modified the milestones: 8.3.6, 8.3.7 Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is wrong :( small
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants