noresm2.5_alpha09: Update CTSM, BLOM config in compsets. #632

mvdebolskiy · 2025-01-29T18:29:03Z

Summary: Update CTSM to include megan refactor, ccs_config to checkout updated modules

Contributors: mvdebolskiy,
Reviewers: mvertens
Purpose of changes:
Github PR URL:
Changes made to build system: ESMF 8.6.1 and intel toolchain to 2023b
Changes made to the namelist: None
Changes to the defaults for the boundary datasets: None
Substantial timing or memory changes: None

Testing: See test results:

Issues addressed by this PR:
fixes #582

mvdebolskiy · 2025-01-29T18:31:00Z

@gold2718 should the NorESMhub/cime#93 be here too?

jmaerz · 2025-01-29T18:35:24Z

Hi @mvdebolskiy , since I stumbled over this issue recently: have you also committed the new tag version (as in git add, git commit the changed submodules)? - to stay in sync between what git submodules would check out and when using git fleximod.

gold2718 · 2025-01-29T19:45:04Z

@gold2718 should the NorESMhub/cime#93 be here too?

I think it is a good idea in that it at least fixes some issues that kept it running on Python 3.12.

mvdebolskiy · 2025-01-30T08:16:17Z

Failing tests:

  PASS ERI_Ly15.f09_g17_gris4.T1850Gg.betzy_intel.cism-isostasy_period4
  PASS ERI.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio
  FAIL ERP_D_Ln9_P256.f19_f19_mtn14.F1850.betzy_intel.cam-outfrq9s
  FAIL ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio
  FAIL ERS_Ld5.ne30pg3_tn14.N1850fates-sp.betzy_intel.allactive-defaultio
  FAIL ERS_Ld5.ne30pg3_tn14.N1850mam4.betzy_intel.allactive-defaultio
  PASS ERS_Ly7.f09_g17_gris4.T1850Gg.betzy_intel
  PASS MULTINOAIS_Ly2.f19_f19_ais8gris4_mtn14.I1850Clm50SpRsGag.betzy_intel.cism-change_params
  FAIL SMS_D_Ld1.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio
  FAIL SMS_D_Ld1.ne30pg3_tn14.N1850clmbgc.betzy_intel.allactive-defaultio
  FAIL SMS_D_Ld1.ne30pg3_tn14.N1850fates-sp.betzy_intel.allactive-defaultio
  FAIL SMS_D_Ln9_P256.ne30pg3_ne30pg3_mtn14.F2000climo.betzy_intel.cam-outfrq9s
  FAIL SMS_D_Ly1.f09_g17_ais8.T1850Ga.betzy_intel
  PASS SMS_Lm13.f19_f19_mtn14.I1850Clm50SpG.betzy_intel

According to the logs, the error comes from the CICE mesh, most likely related to #626

cesm.exe           00000000062DA233  ice_comp_nuopc_mp         726  ice_comp_nuopc.F90

In the _D_ CICE and also BLOM stop at:

1335: cesm.exe           000000000C9906D7  mo_profile_gd_mp_          96  mo_profile_gd.F90
1335: cesm.exe           000000000C90C69B  mo_ini_fields_mp_         144  mo_ini_fields.F90
1335: cesm.exe           000000000C8E814F  mo_hamocc_init_mp         194  mo_hamocc_init.F90
1335: cesm.exe           000000000C7F3C30  mod_tracers_updat         146  mod_tracers_update.F90
1335: cesm.exe           000000000BF34BC8  mod_blom_init_mp_         216  mod_blom_init.F90
1335: cesm.exe           000000000BECCF13  ocn_comp_nuopc_mp         459  ocn_comp_nuopc.F90

TomasTorsvik · 2025-01-31T13:09:18Z

I consistently get FAIL on ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaulti and ERS_Ld5.ne30pg3_tn14.N1850mam4.betzy_intel.allactive-defaultio.
For the other tests I have had at least one PASS result (I ran the tests twice).

mvdebolskiy · 2025-01-31T15:00:22Z

@TomasTorsvik it's ESMF issue. that goes all the way to openmpi. I am trying to bisect where the problem is right now.
We can postpone the ccs_config update, if this tag is needed asap. I'll then just do the compset aliases and add extra tests for blom.

TomasTorsvik · 2025-01-31T15:06:22Z

@mvdebolskiy - Thanks for checking! In this case I suggest to proceed with the alpha09 tag without the ccs_config update, and just do the compset aliases and extra tests for blom. Will the CTSM tag update be in as well?
Never mind, I see that it is already included.

mvdebolskiy · 2025-02-01T17:14:21Z

20250201_151426_2v4kog: 15 tests
ERI_Ly15.f09_g17_gris4.T1850Gg.betzy_intel.cism-isostasy_period4 (Overall: PASS)
ERI.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio (Overall: PASS)
ERP_D_Ln9_P256.f19_f19_mtn14.F1850.betzy_intel.cam-outfrq9s (Overall: PASS)
ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio (Overall: FAIL) details:
PASS ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio CREATE_NEWCASE
PASS ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio XML
PASS ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio SETUP
PASS ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio SHAREDLIB_BUILD time=14
PASS ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio MODEL_BUILD time=377
PASS ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio SUBMIT
PASS ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio RUN time=414
FAIL ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio COMPARE_base_rest
PASS ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio MEMLEAK
PASS ERR_Ld3.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio SHORT_TERM_ARCHIVER
ERS_D_Ld5.ne30pg3_tn14.N1850nohyb.betzy_intel.allactive-defaultio (Overall: FAIL) details:
PASS ERS_D_Ld5.ne30pg3_tn14.N1850nohyb.betzy_intel.allactive-defaultio CREATE_NEWCASE
PASS ERS_D_Ld5.ne30pg3_tn14.N1850nohyb.betzy_intel.allactive-defaultio XML
PASS ERS_D_Ld5.ne30pg3_tn14.N1850nohyb.betzy_intel.allactive-defaultio SETUP
PASS ERS_D_Ld5.ne30pg3_tn14.N1850nohyb.betzy_intel.allactive-defaultio SHAREDLIB_BUILD time=36
PASS ERS_D_Ld5.ne30pg3_tn14.N1850nohyb.betzy_intel.allactive-defaultio MODEL_BUILD time=141
PASS ERS_D_Ld5.ne30pg3_tn14.N1850nohyb.betzy_intel.allactive-defaultio SUBMIT
FAIL ERS_D_Ld5.ne30pg3_tn14.N1850nohyb.betzy_intel.allactive-defaultio RUN time=3635
ERS_Ld5.ne30pg3_tn14.N1850fates-sp.betzy_intel.allactive-defaultio (Overall: PASS)
ERS_Ld5.ne30pg3_tn14.N1850mam4.betzy_intel.allactive-defaultio (Overall: PASS)
ERS_Ly7.f09_g17_gris4.T1850Gg.betzy_intel (Overall: PASS)
MULTINOAIS_Ly2.f19_f19_ais8gris4_mtn14.I1850Clm50SpRsGag.betzy_intel.cism-change_params (Overall: PASS)
SMS_D_Ld1.ne30pg3_tn14.N1850.betzy_intel.allactive-defaultio (Overall: PASS)
SMS_D_Ld1.ne30pg3_tn14.N1850clmbgc.betzy_intel.allactive-defaultio (Overall: PASS)
SMS_D_Ld1.ne30pg3_tn14.N1850fates-sp.betzy_intel.allactive-defaultio (Overall: PASS)
SMS_D_Ln9_P256.ne30pg3_ne30pg3_mtn14.F2000climo.betzy_intel.cam-outfrq9s (Overall: PASS)
SMS_D_Ly1.f09_g17_ais8.T1850Ga.betzy_intel (Overall: PASS)
SMS_Lm13.f19_f19_mtn14.I1850Clm50SpG.betzy_intel (Overall: PASS)

` ERS_D_Ld5.ne30pg3_tn14.N1850nohyb.betzy_intel.allactive-defaultio` failed due to timelimit. Not sure why the throuput got worse, might be just betzy acting up.

@matsbn can you check the compsets? Do nay of the `user_mods` have to be changed for blom to have things you added in 1.7.0 as defaults?

I will redo the tests and add new baselines, once the changes for blom are confirmed.

mvdebolskiy · 2025-02-03T09:53:30Z

The time limit happens because of the test is run in DEBUG and the throughput is around 0.52 y/d. Will reduse to Ld4, to see if it gets through.

mvdebolskiy · 2025-02-04T15:33:16Z

@matsbn can you check the compsets and if there are changes to user_mods needed for hybrid coordinate?
Will generate baselines after PR is approved. 2 tests failing due to SFDMS issue #625.

gold2718

This looks good (assuming it passes tests), thanks for getting this done!

cime_config/config_compsets.xml

matsbn

The modified BLOM entries in the compsets looks fine. Thanks!

update .gitmodules

2cf9e04

make hybrid coord default in blom. Add a test for non-hybryd coords

638c733

revert ccs_config, add clm submodule

e5577c5

change test length for nonhyb to avoid time limit

08b927a

mvdebolskiy marked this pull request as ready for review February 4, 2025 15:31

mvdebolskiy requested review from matsbn and gold2718 February 4, 2025 15:31

gold2718 approved these changes Feb 5, 2025

View reviewed changes

TomasTorsvik reviewed Feb 5, 2025

View reviewed changes

cime_config/config_compsets.xml Outdated Show resolved Hide resolved

rename blom nohyb compset to blom compsets scheme

3b560d9

matsbn approved these changes Feb 5, 2025

View reviewed changes

mvdebolskiy changed the title ~~noresm2.5_alpha09: Update CTSM, ccs_config~~ noresm2.5_alpha09: Update CTSM, BLOM config in compsets. Feb 5, 2025

add changelog

c922239

mvdebolskiy merged commit 196d9b6 into NorESMhub:noresm_develop Feb 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

noresm2.5_alpha09: Update CTSM, BLOM config in compsets. #632

noresm2.5_alpha09: Update CTSM, BLOM config in compsets. #632

mvdebolskiy commented Jan 29, 2025 •

edited by gold2718

Loading

mvdebolskiy commented Jan 29, 2025

jmaerz commented Jan 29, 2025

gold2718 commented Jan 29, 2025

mvdebolskiy commented Jan 30, 2025

TomasTorsvik commented Jan 31, 2025 •

edited

Loading

mvdebolskiy commented Jan 31, 2025

TomasTorsvik commented Jan 31, 2025 •

edited

Loading

mvdebolskiy commented Feb 1, 2025

mvdebolskiy commented Feb 3, 2025

mvdebolskiy commented Feb 4, 2025 •

edited

Loading

gold2718 left a comment

matsbn left a comment

noresm2.5_alpha09: Update CTSM, BLOM config in compsets. #632

noresm2.5_alpha09: Update CTSM, BLOM config in compsets. #632

Conversation

mvdebolskiy commented Jan 29, 2025 • edited by gold2718 Loading

mvdebolskiy commented Jan 29, 2025

jmaerz commented Jan 29, 2025

gold2718 commented Jan 29, 2025

mvdebolskiy commented Jan 30, 2025

TomasTorsvik commented Jan 31, 2025 • edited Loading

mvdebolskiy commented Jan 31, 2025

TomasTorsvik commented Jan 31, 2025 • edited Loading

mvdebolskiy commented Feb 1, 2025

mvdebolskiy commented Feb 3, 2025

mvdebolskiy commented Feb 4, 2025 • edited Loading

gold2718 left a comment

Choose a reason for hiding this comment

matsbn left a comment

Choose a reason for hiding this comment

mvdebolskiy commented Jan 29, 2025 •

edited by gold2718

Loading

TomasTorsvik commented Jan 31, 2025 •

edited

Loading

TomasTorsvik commented Jan 31, 2025 •

edited

Loading

mvdebolskiy commented Feb 4, 2025 •

edited

Loading