Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to data indices in anemoi models #17

Merged
merged 23 commits into from
Sep 9, 2024

Conversation

sahahner
Copy link
Member

@sahahner sahahner commented Aug 12, 2024

This PR brings the changes in anemoi-models to introduce the remapper into anemoi-training.

  • In validation mode, the preprocessor needs to be applied out-of-place instead of in-place since the remapper changes the number of variables. Therefore, in the callbacks the preprocessor needs to be applied.
    • The impact on memory consumption and runtime as been thoroughly tested using the profiler. The results on mlflow show that the increase is not significant.
  • Incorporate the changes to the data-indices into anemoi-training. After the preprocessor and before the preprocessor we need to use the internal set of indices now.
  • Changes to config files
  • Cycical colour maps are used to plot variables in degrees.

Some tests will fail because this PR depends on the following PR on anemoi-models: ecmwf/anemoi-models#21.
When this branch is checked out, all tests pass.

@FussyDuck
Copy link

FussyDuck commented Aug 12, 2024

CLA assistant check
All committers have signed the CLA.

@sahahner sahahner marked this pull request as ready for review August 13, 2024 06:55
Copy link
Member

@JesperDramsch JesperDramsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JesperDramsch JesperDramsch merged commit 1a377de into develop Sep 9, 2024
53 of 105 checks passed
@JesperDramsch JesperDramsch deleted the changes-to-data-indices-in-anemoi-models branch September 9, 2024 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants