Add a "Moving to a new version of ESMValTool" (or equivalent) episode

[ehogan]

Hi @ESMValGroup/userengagementteam 👋 😁

As promised (but a little later than expected!), here is the issue I mentioned at the last User Engagement meeting about introducing an episode to help recipe developers handle backwards-incompatible changes. The draft policy will be circulated shortly, but the key phrases from the draft policy that (I hope!) will demonstrate that such an episode is required (and that I hope will make sense out of context!) are:

Recipe users of ESMValTool will be able to successfully run integrated recipes using a release, since all backwards-incompatible changes introduced between releases will have been fixed before the release is created.

However, recipe developers working on user recipes must be provided with information to enable them to adapt their code to resolve issues related to backwards-incompatible changes when backwards-incompatible changes are introduced to the main branch / when a release of ESMValTool is created.

Short description of the material
As part of work around developing a policy for backward-incompatible changes in ESMValTool, I would like to propose introducing a general “Moving to a new version of ESMValTool” (or equivalent) episode to the ESMValTool tutorial that provides general guidance to detail the steps a recipe developer should follow when moving from one release to another. For example, a recipe developer should:

• run their user recipe using the old release and store the outputs
• adapt their code following the clear instructions provided in the “Backwards-incompatible changes” section of the release notes for the new release
• run their updated user recipe using the new release and store the outputs
• compare the outputs (using Bouwe’s comparison tool):
  • if the adapted code is expected to introduce a science change, the outputs should differ, so the outputs must be validated
  • if the developer needs to exactly reproduce outputs, they should continue to use the old release

Thoughts? :)

Branch and pull request
Once you've started working, add the branch (and pull request) link.

[ehogan]

See ESMValGroup/Community#7 (comment) for the draft policy :)

[remi-kazeroni]

Hi @ehogan, thanks for opening the discussion here! I think what you're proposing here looks good. I was wondering if you are also planning on including some technical tips on how to update an ESMValTool installation (environment updates, handling of uncommitted files, ...). This is not always straightforward for various reasons (see a recent experience in ESMValGroup/ESMValTool#2805 ). Also, the proposed workflow may require to have 2 different installations of the Tool at the same time. This might not be easy for all and it could be helpful to write a few tips on how to do that. Many of our users need to work with their own installations and it could be confusing to handle different versions of the Tool.

[ehogan]

Thank you for engaging @remi-kazeroni! :) You make some great points, and I think they would fit nicely into this episode :)