A RDAS branch re-organized after the current GDASapp using submodules and mpas related component added #9

TingLei-NOAA · 2024-03-12T12:48:55Z

Following the current GDASapp which had switched to Git submodules to manage components, re-organize RDAS repo including updating its' module setup according to the current GDASapp.
The mpas-jedi related components are also added while they are not included in the current GDASapp.
The building of this RDAS bundle had been successfully done on Hera .
More tests are to be done including a complete/clean "clone" and "build" process and on orion.
But this branch could be used as an starting point for current work on RDAS.

…ain branches

CoryMartin-NOAA

It looks like a bunch of things were just copied from GDASApp to here without 1) thinking if they were relevant to the regional system or 2) renaming variables, scripts, etc.

I think this needs to be cleaned up before it gets merged or else these things will persist.

CoryMartin-NOAA · 2024-03-12T12:51:54Z

CMakeLists.txt

-  endif(CLONE_JCSDADATA)
-
-endif(BUILD_RDASBUNDLE)
+option(WORKFLOW_TESTS "Include global-workflow dependent tests" OFF)


I kept them there , which would be a good place holder for future regional workflow if it is decided to be included in rdas.

CoryMartin-NOAA · 2024-03-12T12:52:35Z

build.sh

-DYCORE="FV3"
-
-while getopts "p:t:c:m:hvdfa" opt; do
+#cltorg CLONE_JCSDADATA="NO"


These commented out/changed make the build script options worthless

I take them as a simplified version at the beginning. I am still not sure if components like gdas-utils should be included in rdas.

CoryMartin-NOAA · 2024-03-12T12:53:10Z

build.sh

-    echo "Building RDASApp on $BUILD_TARGET"
+  hera | orion | hercules)
+    echo "Building GDASApp on $BUILD_TARGET"
+    export PS1="[\u@\h \W]\$ " #clt to avoid unbound PS1 error


We just need to move away from Conda. Spack-stack has the python modules we need now. PS1 should not be exported in a non-login shell

This line was added for building on herea, because, otherwise, the building process would abort complain unbound PSI.
Seems conda was invoked somewhere. I will see if a better fix could be found

CoryMartin-NOAA · 2024-03-12T12:53:18Z

build.sh

-  hera | orion)
-    echo "Building RDASApp on $BUILD_TARGET"
+  hera | orion | hercules)
+    echo "Building GDASApp on $BUILD_TARGET"


CoryMartin-NOAA · 2024-03-12T12:53:26Z

build.sh

    source $dir_root/ush/module-setup.sh
    module use $dir_root/modulefiles
-    module load RDAS/$BUILD_TARGET
-    CMAKE_OPTS+=" -DMPIEXEC_EXECUTABLE=$MPIEXEC_EXEC -DMPIEXEC_NUMPROC_FLAG=$MPIEXEC_NPROC -DBUILD_GSIBEC=ON"
+    module load GDAS/$BUILD_TARGET.$COMPILER


CoryMartin-NOAA · 2024-03-12T12:55:50Z

bundle/CMakeLists.txt

+  endif()
+
+# Gibbs seawater
+#clt  ecbuild_bundle( PROJECT gsw SOURCE "../sorc/gsw" )


don't need this, no soca

Currently, soca building failed in my tries and building of them were turned off or commented out.
But, I am wondering if they could be included as an optional components when some developments in it are interesting to regional jedi too. But this might be done without keeping them in rdas. I would follow the decision made during this reviewing process.

CoryMartin-NOAA · 2024-03-12T12:56:02Z

bundle/CMakeLists.txt

+
+# FMS and FV3 dynamical core
+  ecbuild_bundle( PROJECT fms SOURCE "../sorc/fms" )
+  ecbuild_bundle( PROJECT fv3 SOURCE "../sorc/fv3" )


no need for fv3 anymore ,right?

For being now, I still hope to keep fv3/fv3-jedi still in rdas. At least, it includes rich examples for our jedi work.

CoryMartin-NOAA · 2024-03-12T12:56:13Z

bundle/CMakeLists.txt

+  ecbuild_bundle( PROJECT mpas-jedi-data SOURCE "../sorc/mpas-jedi-data" )
+  ecbuild_bundle( PROJECT mpas-jedi SOURCE "../sorc/mpas-jedi" )
+
+# SOCA associated repositories


no soca in regional

CoryMartin-NOAA · 2024-03-12T12:57:15Z

ush/check_yaml_keys.py

@@ -0,0 +1,69 @@
+#!/usr/bin/env python3


we shouldn't use this anymore, global is migrating to wxflow YAML tools. Ask @aerorahul for details

@CoryMartin-NOAA Thanks a lot for your comments/suggestions in such a prompt way.
I would dig more on this and talk to @aerorahul on this .

CoryMartin-NOAA · 2024-03-12T12:57:47Z

ush/convert_yaml_ewok2gdas.py

@@ -0,0 +1,83 @@
+#!/usr/bin/env python3


@danholdaway hopes to make this redundant with JCB, I suggest this script is not even needed anyways as of now

TingLei-NOAA · 2024-03-12T13:04:10Z

It looks like a bunch of things were just copied from GDASApp to here without 1) thinking if they were relevant to the regional system or 2) renaming variables, scripts, etc.

I think this needs to be cleaned up before it gets merged or else these things will persist.

Yes, in this rdas re-organizing, I preferred keeping/copying components in gdasapp into rdas if they didn't cause much troubles for radas building. I (or other RDAS developers) would clean them through this PR process and future use/developments of radas.

CoryMartin-NOAA · 2024-03-12T13:26:14Z

@TingLei-NOAA I think we (@ShunLiu-NOAA @danholdaway ) should chat soon. GDASApp is containing the GDAS-specific implementation of JEDI, and will be where releases are tagged, and the source code is controlled with hashes/tags. I am nervous about blindly copying things for RDAS, as these two systems will be on a different upgrade cycle, and thus need different modulefiles, different hashes, and because of MPAS vs FV3 and other reasons, different utilities (like blending would not need to be in the global)

ShunLiu-NOAA · 2024-03-12T13:30:54Z

Ting, Thank you for working on this.
@CoryMartin-NOAA @danholdaway, I have the same concerns about how to keep RDASApp and GDASApp on the same page.

TingLei-NOAA · 2024-03-12T13:30:55Z

@TingLei-NOAA I think we (@ShunLiu-NOAA @danholdaway ) should chat soon. GDASApp is containing the GDAS-specific implementation of JEDI, and will be where releases are tagged, and the source code is controlled with hashes/tags. I am nervous about blindly copying things for RDAS, as these two systems will be on a different upgrade cycle, and thus need different modulefiles, different hashes, and because of MPAS vs FV3 and other reasons, different utilities (like blending would not need to be in the global)

Sure. a clear plan for how to use rdas is needed. The current PR is supposed to make rdas could at least work (be able to be built on hera/orion) by, at least, EMC developers for being now.
To use the current heads of submodules rather than using hashed commits in gdasapp (or rdas), a submodule updating script is added : checkout_submodules_branches.sh.

delippi · 2024-03-12T14:05:45Z

modulefiles/RDAS/hera.intel.lua

@TingLei-NOAA , This is bit-for-bit the same file as the GDAS/hera.intel.lua. Do we need the GDAS directory at all? Also, being bit-for-bit there are mentions to GDAS instead of RDAS. The same goes for the hercules and orion *lua files under the RDAS/ directory.

@delippi Yes. now it is just a mirror of GDAS modules. To have a RDAS dir, it is more readable for like the build.sh. We could make it a link pointing to the GDAS. However, at the same time, I am wondering if we need the flexibilities to use different setting up modules from GDAS and, hence, use the current practice.

delippi · 2024-03-12T14:07:59Z

build.sh

@@ -24,7 +24,6 @@ usage() {
  echo "  -f  force a clean build             DEFAULT: NO"
  echo "  -d  include JCSDA ctest data        DEFAULT: NO"
  echo "  -a  build everything in bundle      DEFAULT: NO"
-  echo "  -m  select dycore                   DEFAULT: FV3"


@TingLei-NOAA , why remove this option and not just change to default to MPAS?

I need to re-sort out on this part of this part. Thanks.

aerorahul · 2024-03-12T15:21:44Z

@TingLei-NOAA I think we (@ShunLiu-NOAA @danholdaway ) should chat soon. GDASApp is containing the GDAS-specific implementation of JEDI, and will be where releases are tagged, and the source code is controlled with hashes/tags. I am nervous about blindly copying things for RDAS, as these two systems will be on a different upgrade cycle, and thus need different modulefiles, different hashes, and because of MPAS vs FV3 and other reasons, different utilities (like blending would not need to be in the global)

FWIW, I am with @CoryMartin-NOAA here. The RDASApp should use GDASApp as a guide and add features that are needed and build on them, rather than import everything and then spend time removing them later.

ShunLiu-NOAA · 2024-03-21T18:23:16Z

@TingLei-NOAA Can we close this PR and open an new PR to merge other changes that you want to do to the current "develop"

TingLei-NOAA · 2024-03-21T18:29:48Z

This PR is closed for #11 has been merged into develop branch.
Thanks to all who had contributed to the discussions on this PR.

TingLei-daprediction added 8 commits March 12, 2024 01:15

remove fv3-jedi with errors setup with errors

9373488

update oops submodule to the latest commit

b322eee

Change the develop branches in some public repo to the right master/m…

9878341

…ain branches

re-organize rdas folllowing gdas using submodules

80d25f9

Update submodules to latest commits

ba0deed

add mpas-jedi to rdasapp

dc2b764

add mpas-jedi to RDAS

40d4f61

--amend

d5d0b62

TingLei-NOAA requested review from CoryMartin-NOAA, ShunLiu-NOAA and HuiLiu-NOAA March 12, 2024 12:50

TingLei-NOAA self-assigned this Mar 12, 2024

TingLei-NOAA requested a review from xyzemc March 12, 2024 12:52

CoryMartin-NOAA requested changes Mar 12, 2024

View reviewed changes

TingLei-NOAA requested review from CoryMartin-NOAA and JacobCarley-NOAA March 12, 2024 12:58

fixs according this PR reviewers's comments

886d6f8

delippi self-requested a review March 12, 2024 13:14

fixs according this PR reviewers's comments

ba2a517

fixs according this PR reviewers's comments

b8cc27e

delippi reviewed Mar 12, 2024

View reviewed changes

ShunLiu-NOAA previously approved these changes Mar 13, 2024

View reviewed changes

Remove submodule mom6

f28e48a

TingLei-NOAA dismissed ShunLiu-NOAA’s stale review via f28e48a March 13, 2024 13:48

CoryMartin-NOAA mentioned this pull request Mar 18, 2024

Update RDASApp to use git submodules + Rocky 8 for Hera #11

Merged

TingLei-NOAA closed this Mar 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A RDAS branch re-organized after the current GDASapp using submodules and mpas related component added #9

A RDAS branch re-organized after the current GDASapp using submodules and mpas related component added #9

TingLei-NOAA commented Mar 12, 2024

CoryMartin-NOAA left a comment

CoryMartin-NOAA Mar 12, 2024

TingLei-NOAA Mar 12, 2024

CoryMartin-NOAA Mar 12, 2024

TingLei-NOAA Mar 12, 2024

CoryMartin-NOAA Mar 12, 2024

TingLei-NOAA Mar 12, 2024

CoryMartin-NOAA Mar 12, 2024

CoryMartin-NOAA Mar 12, 2024

CoryMartin-NOAA Mar 12, 2024

TingLei-NOAA Mar 12, 2024

CoryMartin-NOAA Mar 12, 2024

TingLei-NOAA Mar 12, 2024

CoryMartin-NOAA Mar 12, 2024

CoryMartin-NOAA Mar 12, 2024

TingLei-NOAA Mar 12, 2024

CoryMartin-NOAA Mar 12, 2024

TingLei-NOAA commented Mar 12, 2024

CoryMartin-NOAA commented Mar 12, 2024

ShunLiu-NOAA commented Mar 12, 2024

TingLei-NOAA commented Mar 12, 2024 •

edited

Loading

delippi Mar 12, 2024

TingLei-NOAA Mar 12, 2024

delippi Mar 12, 2024

TingLei-NOAA Mar 12, 2024

aerorahul commented Mar 12, 2024

ShunLiu-NOAA commented Mar 21, 2024

TingLei-NOAA commented Mar 21, 2024

A RDAS branch re-organized after the current GDASapp using submodules and mpas related component added #9

A RDAS branch re-organized after the current GDASapp using submodules and mpas related component added #9

Conversation

TingLei-NOAA commented Mar 12, 2024

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TingLei-NOAA commented Mar 12, 2024

CoryMartin-NOAA commented Mar 12, 2024

ShunLiu-NOAA commented Mar 12, 2024

TingLei-NOAA commented Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul commented Mar 12, 2024

ShunLiu-NOAA commented Mar 21, 2024

TingLei-NOAA commented Mar 21, 2024

TingLei-NOAA commented Mar 12, 2024 •

edited

Loading