Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add sea-ice graph partitioning to files_for_e3sm test case #489

Merged
merged 8 commits into from
Feb 18, 2023
Merged

Add sea-ice graph partitioning to files_for_e3sm test case #489

merged 8 commits into from
Feb 18, 2023

Conversation

xylar
Copy link
Collaborator

@xylar xylar commented Dec 27, 2022 •
edited
Loading

This merge also generalizes files_for_e3sm so it can be run with an existing mesh and graph file, not just one generated by compass. Thus, it replaces make_diagnostics_files, which performed a subset of this functionality for an existing mesh.

Checklist

  • User's Guide has been updated
  • Developer's Guide has been updated
  • API documentation in the Developer's Guide (api.rst) has any new or modified class, method and/or functions listed
  • Documentation has been built locally and changes look as expected
  • Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

Closes #474

@xylar xylar requested a review from akturner December 27, 2022 08:55
@xylar xylar self-assigned this Dec 27, 2022
@xylar
Copy link
Collaborator Author

xylar commented Dec 27, 2022

This will require a new version of mpas_tools with MPAS-Dev/MPAS-Tools#482, MPAS-Dev/MPAS-Tools#484 and MPAS-Dev/MPAS-Tools#485

@xylar xylar added enhancement New feature or request sea ice labels Jan 16, 2023
@xylar xylar force-pushed the use_better_seaice_partitioning branch from 51b5eb2 to b57eb02 Compare January 17, 2023 12:50
@xylar xylar marked this pull request as ready for review January 17, 2023 13:09
@xylar xylar force-pushed the use_better_seaice_partitioning branch from b57eb02 to 1a6c0cd Compare January 17, 2023 13:10
@xylar
Copy link
Collaborator Author

xylar commented Jan 17, 2023

Testing

I was able to run both the qu240_for_e3sm suite and the files_for_e3sm test case with this branch. For the latter, I used the SORRM initial condition and graph file.

See:

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20230117/seaice_part_qu240_for_e3sm
/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20230117/seaice_part_sorrm_files_for_e3sm

The next stage of testing will be writing out debugging/plotting information for the sea-ice partitions to make sure they're behaving as expected.

@xylar xylar force-pushed the use_better_seaice_partitioning branch from 80b2538 to 95887d4 Compare January 17, 2023 21:17
@xylar xylar force-pushed the use_better_seaice_partitioning branch from 95887d4 to e2b4e7a Compare January 28, 2023 19:41
@xylar
Copy link
Collaborator Author

xylar commented Jan 28, 2023

Plotting

I plotted the QU240 (not shown) and SORRM partitions for several processor counts. Below are images for 96 processors in the northern and southern hemispheres:
north
south

Given the higher resolution around Antarctica, it makes sense that most processors are allocated there. It is also clear that each processor are being allocated a polar and an equatorial region, as expected.

@xylar
Copy link
Collaborator Author

xylar commented Jan 28, 2023

@akturner, could you review these results either just using the above images or perhaps by taking a look at the following output directories?

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20230117/seaice_part_qu240_for_e3sm/ocean/global_ocean/QU240/PHC/files_for_e3sm/seaice_graph_partition
/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20230117/seaice_part_sorrm_files_for_e3sm/
ocean/global_ocean/files_for_e3sm/seaice_graph_partition

xylar
xylar commented Jan 28, 2023
View reviewed changes
compass/ocean/tests/global_ocean/files_for_e3sm/seaice_graph_partition.py Outdated
Comment on lines 75 to 81
args = ['create_seaice_partitions',
'-m', 'restart.nc',
'-o', '.',
'-p', f'mpas-seaice.graph.info.{creation_date}',
'-g', 'gpmetis',
'--plotting',
'-n']
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akturner, just a note that --plotting is always used here so we'll have the plot files to look at whenever we want them.

xylar added 7 commits February 17, 2023 17:42
@xylar
Add a step for creating sea-ice graph partitions
76fe2a8 
Pull out common functionality into graph_partition module
@xylar
Support running files_for_e3sm with existing mesh
a9f67f8 
This merge is a major reorganization of files_for_e3sm to have
a superclass for each step to handle common tasks such as linking
to a restart file, detecting whether ice-shelf cavities are in the
mesh, and getting attributes of the mesh such as the short name
and the creation date.

The DiagnosticsFiles step has been split into 2 steps since making
mapping files uses MPI parallelism whereas making region masks uses
thread parallelism and these need to be handled separately.

The MakeDiagnosticsFiles test case has been removed in favor of
this more complete solution.
@xylar
Add vertex mapping files
139196d
@xylar
Update docs
3f3f713
@xylar
Add docs for files_for_e3sm on existing meshes
9113926
@xylar
Add files_for_e3sm with exiting mesh to user's guide
dfdff5d
@xylar
Fix handling of both new and old creation date
4c10b26 
Some older MPAS-Ocean initial conditions have the full date and
time of creation, while newer meshes just have a YYYYMMDD date
string.  This merge should handle both.
@xylar xylar force-pushed the use_better_seaice_partitioning branch from e2b4e7a to 4c10b26 Compare February 17, 2023 23:42
akturner
akturner approved these changes Feb 18, 2023
View reviewed changes
Copy link

@akturner akturner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved based on examination of output partition diagnostic files

@xylar
Copy link
Collaborator Author

xylar commented Feb 18, 2023

Thanks @akturner! Very much appreciated.

@xylar xylar merged commit 688cc6d into MPAS-Dev:main Feb 18, 2023
@xylar xylar deleted the use_better_seaice_partitioning branch February 18, 2023 17:17
@xylar xylar mentioned this pull request Mar 12, 2023
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@akturner akturner akturner approved these changes

Assignees

@xylar xylar

Labels
enhancement New feature or request sea ice
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Use sea-ice load balanced partitioning in files_for_e3sm
2 participants
@xylar @akturner