Technical Lead Team rolling issues, shoutouts, and meeting deciders - Part 2, to be discussed at TLT Meeting 13 Dec 2022

[valeriupredoi]

We have had our first batch of Tech-related discussion topics listed in #56 for which we had a very fruitful 🥝 call on 8 November 2022 (https://github.com/ESMValGroup/Community/blob/main/Tech%20Lead/Minutes/20221108.md). We are, however, porting here a couple issues from #56 that we didn't touch upon, and opening the floor yet again for the @ESMValGroup/technical-lead-development-team to bring forward more issues they want to discuss in an upcoming TLT call (TBA, but most prob towards the end of November):

Discussion items from #56 that we still have to talk about

@bouweandela Release procedure

Should a feature freeze for the release mean that we allow only bugfixes after that, or will any pull request that is merged after the feature freeze but before the release be included in the release?

It looks like there are different opinions on this (see e.g. ESMValGroup/ESMValTool#2734 (comment)), so it would be good to talk about this at the upcoming tech lead team meeting.

@valeriupredoi Release stuff too

release procedure for ESMValTool is involving and not well documented - I'd like to discuss ESMValGroup/ESMValTool#2895 and also the spread of release workload - I think we can safely say that 2 people (release managers) would be needed in the eventuality that RTW doesn't come online before the next release (even so, it might be experimental, and a full run of all the recipes might still be needed one last release). I would also want to touch upon ESMValGroup/ESMValTool#2895 (also mentioned in my previous comment) and specifically data dirs on The Big Levante

@bouweandela adds to this: This would also be a good and related discussion topic: ESMValGroup/ESMValTool#2883

Other topics: please comment below

[bouweandela]

The approach for setting up running all recipes would also be a good topic. Should we use an existing framework (e.g. cylc, ESMValGroup/ESMValTool#2722) or write our own scripts (e.g. ESMValGroup/ESMValTool#2883)?

[bouweandela]

Follow up on #56 (comment)

[remi-kazeroni]

@valeriupredoi will look into a Gmail-based solution that can be used by some Core developers and/or the TLT. See previous discussion in #56 (comment)

[remi-kazeroni]

Administration of the VM: Bouwe and I met today to discuss the topic as decided in the last TLT Meeting (see this comment).

Done today:

• changed the configuration of ESMValBot so that missing data from the official CMIP data pool at DKRZ would be downloaded locally. This would solve the issue with reading data from the download_dir of bd0854.
• changed the permissions so that Release Managers do not require sudo rights to upload recipe results or use their own conda environments (e.g. to run the comparison tool)

To be check:

• enabling the bot to read Tier3 datasets

To be discussed at the next TLT Meeting:

• Could we delete intermediate recipe results not produced with the last release candidate?
• Could we only keep the results of the final run (success or failure) for each recipe? Do we need to write a script to help the Release Manager cleaning that up?
• Could we avoid uploading results from large work directories to the VM? Either by adding an option to the few problematic recipe not to generate large/many nc-files or with a script that we clean that up?
• We would need to improve the documentation on how to use the VM for a release. Readthedocs may not be the right place for that. Probably better to have a Readme in the VM.

[remi-kazeroni]

Organization of the meetings: I offered to support @valeriupredoi in organizing and running these meetings if that becomes too much work for one person. How shall we do that?

[valeriupredoi]

Cheers very much @remi-kazeroni 🍺 I propose we decide who does a number of the following:

• sets up a doodle and sends it around
• talks during the call
• takes minutes during the call
• collates and PRs minutes after the call

4 tasks - 2 each? 😁

[remi-kazeroni]

Schedule: once a month unless pressing issues to discuss

@valeriupredoi will set the doodle, call and discussion page
@remi-kazeroni will prepare the agenda and take the minutes

[valeriupredoi]

so do people think it's a good idea to organize a Techy meeting one last time before we all vanish for so-called religious holidays purposes and in fact it's all about 🍺 🍷 🦃 🐷 consumption? @ESMValGroup/technical-lead-development-team react with 👍 to this comment if you think so please

[bouweandela]

Hi @valeriupredoi or @remi-kazeroni,

Would it be possible to have an agenda today? I think that would help streamline the discussion. It could be compiled from the issues mentioned here.

[valeriupredoi]

@bouweandela if we just go through these issues here and comment on them as we discuss them, we shouldn't need yet another file/agenda/do would we? Then I can just collate the discussed items in an md file and that'd be the minutes

[bouweandela]

That's what we did last time and it didn't work very well

[remi-kazeroni]

I agree it would be more efficient to run this meeting if we had a list of discussion items ready beforehand. This would include topics proposed in this discussion and some follow-ups from last time. Shall I try to make a list of discussion items at the bottom of this page? We could quickly revise the order of these items at the beginning of the meeting if people think some topics should be more urgently discussed.

[valeriupredoi]

that'd be wonderful, very many thanks @remi-kazeroni 🍺

[bouweandela]

I would also like to talk about our strategy regarding supporting features that make it easier to use ESMValCore without doing a from source installation and modifying the source code if you need additional features. Examples could be:

• Configuring the default paths to ESMValTool recipes, diagnostics, and the references (config-references.yml and references folder), i.e. make it easy to use ESMValCore without ESMValTool
• Using fixes from an external location as proposed in Custom user-defined location for custom project CMOR fixes ESMValCore#1852 Local data fixes and custom preprocessors ESMValTool#2570
• Using arbitrary python functions as preprocessor functions Local data fixes and custom preprocessors ESMValTool#2570

[remi-kazeroni]

Do we want to revise the current policy? Many users may not contribute their features (like fixes) since the level to contribute can be high (via PR) and the timescale (until next Core release) too long. For example, fixes could be decoupled from the Core. Fixes could live in a separate package with config files. We could also have different packages for different projects. TLT supports the idea. Agreement that we may lose contributors in the current situation. To be discussed with the PIs and the Science Lead Team.

Downside: more package maintenance and/or too many packages to be maintained

[bouweandela]

@zklaus Did you bring this topic up in a cross-team meeting already?

[remi-kazeroni]

Agenda - Dec 13, 2022

1. Organization of the meetings
2. Strategy for supporting ESMValCore features that do not require a source installation
3. Maintaining Zenodo records - also discussed in the previous meeting here
4. Release procedure: What can go in after the feature freeze?
5. Release procedure: documentation of the release process
6. Release procedure: Strategy to run all recipes
7. Release procedure: Administration of the VM and its usage during the release

This list contains links to all suggested topics for this meeting. We can reorder the list at the beginning of the meeting. I'm going to move text from the first post to separate ones to ease note taking and linking to the agenda (edits: done).

[remi-kazeroni]

• Items discussed: 1 to 4 and partially 5
• Items postponed to next meeting: 6, 7

[bouweandela]

Thanks for organizing @valeriupredoi and @remi-kazeroni!

[remi-kazeroni]

Release procedure

[originally posted by @bouweandela]

Should a feature freeze for the release mean that we allow only bugfixes after that, or will any pull request that is merged after the feature freeze but before the release be included in the release?

It looks like there are different opinions on this (see e.g. ESMValGroup/ESMValTool#2734 (comment)), so it would be good to talk about this at the upcoming tech lead team meeting.

[remi-kazeroni]

Context.
@bouweandela: originally, feature freeze means: only bug fixes after the deadline. Bug fixes are first merged into main and then cherry-picked into the release branch. Later, the release procedure was changed and Tool and Core were released one after the other. In this case, release candidates are used for the Core and all recipes are tested over and over with each release candidates. The notion of feature freeze appeared to be less important. The documentation has not been fully adjusted to the latter procedure.

Discussion
@zklaus: supports the idea of release Core and the tool shortly after.
@valeriupredoi: important to keep feature freeze.
@ehogan: unlikely that the recipe test workflow would be available for the next release (v2.8)
@remi-kazeroni: prefers working with release candidates and only doing thorough recipe testing for the Tool (and last Core release candidate).
@valeriupredoi: try to keep the number of release candidates for the Core to a minimum and not necessary to have rcs for the Tool.
@bouweandela: can I still merge new features to main during the release procedure? @zklaus: suggests veto rights to the RM. Also others may ignore the ongoing release if new features can still be merged.
@valeriupredoi: It's the responsibility of the RM to ensure good communication and the community is actively supporting the release, the testing
@bouweandela: we should aim at keeping release cycles as short as possible so that contributions are not put on hold for too long. Also, make sure the release documentation is up-to-date
@zklaus: Core feature freeze would be the point of first release candidate.
@bouweandela: Make sure all the commits related to a tag are included into the main branch. Doubts about the handling of release candidate tags (e.g. v2.8.0_rc1), to be checked

Documentation @valeriupredoi will update the release documentation early January, previous release managers can then take a look

[remi-kazeroni]

Release procedure: documentation

[Originally posted by @valeriupredoi]

release procedure for ESMValTool is involving and not well documented - I'd like to discuss ESMValGroup/ESMValTool#2895 and also the spread of release workload - I think we can safely say that 2 people (release managers) would be needed in the eventuality that RTW doesn't come online before the next release (even so, it might be experimental, and a full run of all the recipes might still be needed one last release). I would also want to touch upon ESMValGroup/ESMValTool#2895 (also mentioned in my previous comment) and specifically data dirs on The Big Levante

@bouweandela adds to this: This would also be a good and related discussion topic: ESMValGroup/ESMValTool#2883

[valeriupredoi]

@Cuppingan is now blocked because their comments are of unsolicited commercial nature, most probably a bot - if this is a genuine user interested in ESMValTool then please state here