Add PR-Based Config Testing Workflow

[CodeGat]

Background

In a similar vein to the existing access-om2-configs/access-esm1.5-configs CI infrastructure, create an access-om3-specific PR-based workflow. Work with @anton-seaice to develop requirements - they could be different to the above config repositories.

In any case, use the generic access-nri/model-config-tests reusable workflows that already exist (may have to be modified to support new requirements), to allow for easy porting of features/fixes to other config repositories. This doesn't mean that the CI infrastructure that is specific to this repository (i.e., the stuff in $this_repo/.github/workflows that isn't calls to generic workflows) has to remain the same.

TODO

• PR that adds the base functionality as in access-om2-configs, access-esm1.5-configs, etc.
• Propagate this change into all existing dev-*/release-* branches
• PR that adds the !test repro [commit] command to allow easy repro checks on non-release-* config branches
• PR that augments the existing functionality to allow the committing of docs as is a requirement for access-om3-configs. This will be handled in the driver.

[anton-seaice]

We are planning to rename our branches to follow the same naming format as other repositories, e.g. with release-, dev- and feature- prefixes to standardize with the other workflows.

We would like on creation/update of a PR, for it to trigger the workflow run and then have a mechanism for the developer to confirm they want the checksum to be updated if they change (e.g. a trigger word for the bot, which then commits the new checksum to the branch).

We would also like to capture the four files from the output with the file names MOM_parameter_doc.* (as artifacts), and commit them to the branch under test. (ping @minghangli-uni - is this correct?)

Its probably also worth rerunning the workflow after a merge commit to confrim the merge hasn't changed anything.

The main thing initially is to get the workflow running, and over time, we can improve and add tests to the model-config-tests repo.

[CodeGat]

I linked the wrong issue, whoops

[CodeGat]

Thanks @anton-seaice, getting to this now.
Note that the feature- branches don't have to literally have feature- on them - they can be any branch.
All those features you want can be supported - although the merge commit part really shouldn't change the answers since it's a requirement that the branch be rebased onto the HEAD of release-, but it can be worked in if you want it.

[anton-seaice]

@minghangli-uni Just confirming, we want to copy archive/output000/MOM_parameter_doc.* to the docs directory in the config? (Is that the right files?)

@aekiss - Do we want the test to fail if the files have changed, or commit them? (Or fail and the files can be committed with a keyword action in a comment(e.g. !update_docs)

[anton-seaice]

All those features you want can be supported - although the merge commit part really shouldn't change the answers since it's a requirement that the branch be rebased onto the HEAD of release-, but it can be worked in if you want it.

Ok - so we need to confirm that is the setting in the repository settings ?

edit: I guess I mean, if you click Rebase and Merge, that might change the answer right ? Because the checksum would have been from before the Rebase ?

[CodeGat]

There is a branch protection setting that enforces that merging is only available if your are on the HEAD of the PR target branch

[anton-seaice]

@chrisb13 - Can you turn on the setting "Require branches to be up to date before merging"

(https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule)

[aekiss]

Linking to issue discussing MOM_parameter_doc.*: COSIMA/access-om3#117

@minghangli-uni Just confirming, we want to copy archive/output000/MOM_parameter_doc.* to the docs directory in the config? (Is that the right files?)

Sounds good to me. We could put them in the root directory, but using a docs dir in each config makes it clear they aren't input files. Ideally payu would also copy the most recent archive/output*/MOM_parameter_doc.* to the docs directory after each run so that the git runlog would record any changes.

@aekiss - Do we want the test to fail if the files have changed, or commit them? (Or fail and the files can be committed with a keyword action in a comment(e.g. !update_docs)

I think fail and the files can be committed with a keyword action in a comment (e.g. !update_docs)

[CodeGat]

Thanks for the clarification @aekiss

In terms of the case where the files in docs/ change, the workflow gives a big 🚫 and you're given a comment that says "docs/ was modified in this run, commit these using !update_docs" - under which circumstance would you not do !update_docs? The current state would mean that the PR isn't mergable in this state because of the 🚫 (which is resolved by !update_docs).

[anton-seaice]

Thanks - that sounds correct @CodeGat

You would only not update docs if it the PR had some issue with it (e.g. changing defaults from a new binary that weren't expected) that is addressed in some other way (e.g. by changing the config).

[chrisb13]

@chrisb13 - Can you turn on the setting "Require branches to be up to date before merging"

Now done, I think. Let me know if it hasn't worked please. Or if we add further branches that it needs to be applied to.

[CodeGat]

The changes in #136 (specifically the stuff in .github/workflows will now have to be propagated to all the release-*/dev-* branches in the repository.