Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FATES cross-grid seed dispersal mechanism #2077

Merged
merged 45 commits into from
Oct 25, 2023

Conversation

glemieux
Copy link
Collaborator

@glemieux glemieux commented Jul 27, 2023

Description of changes

This PR enables FATES to disperse seeds across neighboring grid cells using MPI. The API update includes calls to new fates dispersal procedures and as such should be coordinated with NGEET/fates#1005.

The design documentation can be found in the FATES Developer's Guide.

Specific notes

Contributors other than yourself, if any: @YanlanLiu

CTSM Issues Fixed (include github issue #):

Are answers expected to change (and if so in what way)? Answer should not change for non seeddispersal mode

Any User Interface Changes (namelist or namelist defaults changes)?
New fates_seeddisp_cadence item and requires changes to the FATES parameter file

Testing performed, if any: FATES testing, will do aux_clm

@glemieux glemieux marked this pull request as ready for review August 21, 2023 23:12
@glemieux
Copy link
Collaborator Author

@ekluzek just a heads up that from the FATES software meeting today, I'm going to move forward with this as the next FATES api update.

Rename hist fields to track them down more easily

Renaming history fields to make easier to find in lists, e.g. when
using ncview. For example, litter fields like MET_LIT and STR_LIT
will be LIT_MET and LIT_STR.
If fates is not using dispersal mode, which is set via the parameter
file, do not run the nearest neighbor algorithm to avoid performace impact.
Change order of history fields to improve performance on derecho

Instead of just ordering history fields alphabetically, order them first
by the name of their level dimension (with fields without a level
dimension appearing first), then alphabetically within a given level
dimension. This changed ordering gives a significant performance
improvement especially noticeable on lustre file systems such as on
derecho.
This corresponds to a fates-side refactor that moves the seed_out and seed_in variables from the bc_in/out types to site.
Corrections to the column type names are also made
@glemieux
Copy link
Collaborator Author

glemieux commented Aug 31, 2023

Cross posting that running fates regression tests on cheyenne, I'm seeing all ERP tests failing when seed dispersal is on regardless of grid size, debug mode or threading: NGEET/fates#1005 (comment)

As discussed in the ctsm software meeting this morning, this seems likely to be due the MPI_AllReduce not being able to maintain b4b when layouts change.

@glemieux
Copy link
Collaborator Author

glemieux commented Aug 31, 2023

Also per discussion during the software meeting, I'm going to add a namelist option to allow user's to turn off seed dispersal even if the seed dispersal parameters are not unset in the parameter file.

  • Add seed dispersal namelist option

@glemieux
Copy link
Collaborator Author

glemieux commented Sep 1, 2023

Cross posting that running fates regression tests on cheyenne, I'm seeing all ERP tests failing when seed dispersal is on regardless of grid size, debug mode or threading: NGEET/fates#1005 (comment)

As discussed in the ctsm software meeting this morning, this seems likely to be due the MPI_AllReduce not being able to maintain b4b when layouts change.

@billsacks was spot on with this being due to the non-associativity of floating point sums with MPI. I refactored the code to utilize a non-reducing MPI collective and it appears that the ERP tests are passing now.

src/main/clm_varctl.F90 Outdated Show resolved Hide resolved
src/main/controlMod.F90 Show resolved Hide resolved
src/utils/clmfates_interfaceMod.F90 Outdated Show resolved Hide resolved
src/utils/clmfates_interfaceMod.F90 Outdated Show resolved Hide resolved
@glemieux
Copy link
Collaborator Author

glemieux commented Oct 13, 2023

Per discussion today around #2151, I'm going to update the testmod to adopt a shell script from @adrifoster to call ncgen to avoid having to manually build and store the necessary parameter file for this test everytime the API updates.

  • Update testmod with shell script calling ncgen

This update implments a shell script to call ncgen
and the fates modify parameter file tooling to generate
and update the default fates parameters on the fly with
values appropriate for seed dispersal mode.
Zeroing of wood product fluxes on fates columns

products are otherwise zero'd in a p2c() routine that is incompatible with fates. When
wood product fluxes become available via fates, these routines will be updated. These fluxes
were previously left as uninitialized, which was causing math issues on some compilers.
@glemieux
Copy link
Collaborator Author

@ekluzek and @adrifoster I think I've resolved all the requested changes. Would you take a look when you get a chance to see if they look good?

SNICAR snow albedo scheme updates

A few substantial changes in the SNICAR module for the following updated snow albedo calculation features:
 - Updated ice optical properties from Flanner et al. (2021), with multiple types for ice refractive indices.
 - Updated aerosol optical properties from Flanner et al. (2021) with multiple dust types & new BC and OC optics.
 - Updated downward solar spectra from Flanner et al. (2021) for multiple condition types.
 - More accurate radiative transfer solver (adding-doubling) from Dang et al. (2019).
 - Nonspherical snow grain scheme from He et al. (2017).
 - BC-snow internal mixing scheme from He et al. (2017).
 - Dust-snow internal mixing scheme from He et al. (2019).
 - Hyperspectral (480-band, 10-nm spectral res) capability with all the above features.
 - New namelist controls for aerosol in snow and additional snow albedo diagnostic output variables.

 Specific notes
 - Code contributors: Cenlin He (NCAR/RAL) with advice from Dave Lawrence (NCAR/CGD) and Mark Flanner (UMich).
 - The manuscript to report this update is
 Cenlin He, Mark Flanner, David M Lawrence, Yu Gu: New features and enhancements in Community Land Model (CLM5) snow albedo modeling: description, sensitivity, and evaluation. Authorea. June 08, 2023. DOI:10.22541/essoar.168626390.01530324/v1
 - These updates will change the snow and surface albedo results along with other surface fluxes changes.
 - There are a few new namelist options related to SNICAR scheme added to the namelist control.

New namelist variables:
 snicar_numrad_snw = 5 or 480 wavelength bands, default 5
 snicar_solarspec, default "mid_latitude_winter" among six available options
 snicar_dust_optics, default "sahara" among three avail. options
 snicar_snw_shape, default "hexagonal_plate" among fourn avail. options
 snicar_use_aerosol, default .true.
 snicar_snobc_intmix and snicar_snodst_intmix, default .false. means do not activate bc-snow and dust-snow internal mixing

 do_sno_oc, default .false., already appeared in previous code but in caps
 use_snicar_frc, default .false., existed before
 fsnowoptics now points to an updated 5-band file and gives the option for a 480-band file
@glemieux
Copy link
Collaborator Author

glemieux commented Oct 20, 2023

Regression testing with aux_clm is coming back with all expected tests as b4b on cheyenne. The only differences are NLCOMP diffs due to the new fates_seeddisp_cadence option.

Cheyenne location: /glade/u/home/glemieux/scratch/ctsm-tests/tests_pr2077-aux_clm

Testing aux_clm on izumi is currently underway.

@glemieux
Copy link
Collaborator Author

Regression testing on izumi completed over the weekend. All expected tests are b4b/

Location: /home/glemieux/scratch/ctsm-tests/tests_pr2077-aux_clm

@ekluzek ekluzek merged commit 8f6b6f1 into ESCOMP:master Oct 25, 2023
@ekluzek ekluzek deleted the fates-seeddispersal branch October 25, 2023 02:17
@samsrabin samsrabin added the science Enhancement to or bug impacting science label Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement new capability or improved behavior of existing capability FATES API update Changes to the FATES version that also REQUIRE an API change in CTSM science Enhancement to or bug impacting science
Projects
Status: Done (non release/external)
Development

Successfully merging this pull request may close these issues.

4 participants