-
Notifications
You must be signed in to change notification settings - Fork 10
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #205 from NOAA-GFDL/204-improve-fre-cmor-run-funct…
…ionality cmor tool improvements
- Loading branch information
Showing
9 changed files
with
810 additions
and
405 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,83 +1,81 @@ | ||
| old usage notes at the top of `cmor_mixer.py` | ||
|
|
||
| # Before start this script in common way run these 2 command in terminal where you are going to execute this script: | ||
| # module load python/3.9 | ||
| # conda activate cmor | ||
|
|
||
| # another possible runs without any preparation in terminal: | ||
| # /home/san/anaconda/envs/cmor_dev/bin/python | ||
| # /app/spack/v0.15/linux-rhel7-x86_64/gcc-4.8.5/python/3.7.7-d6cyi6ophaei6arnmzya2kn6yumye2yl/bin/python | ||
|
|
||
|
|
||
| # How to run it (simple examples): | ||
| # ~/fms_yaml_tools/CMOR_3/cmor_mixer.py | ||
| # -d /archive/oar.gfdl.cmip6/CM4/warsaw_201710_om4_v1.0.1/CM4_1pctCO2_C/gfdl.ncrc4-intel16-prod-openmp/pp/atmos/ts/monthly/5yr | ||
| # -l /home/san/CMOR_3/GFDL-CM4_1pctCO2_C_CMOR-Amon.lst | ||
| # -r /home/san/CMOR/cmor/cmip6-cmor-tables/Tables/CMIP6_Amon.json | ||
| # -p /home/san/CMOR/cmor/Test/CMOR_input_CM4_1pctCO2_C.json | ||
|
|
||
| # ~/fms_yaml_tools/cmor_mixer.py | ||
| # -d /archive/Fabien.Paulot/ESM4/H2/ESM4_amip_D1_soilC_adj/gfdl.ncrc3-intel16-prod-openmp/pp/land/ts/monthly/5yr | ||
| # -l /home/san/CMOR_3/GFDL-ESM4_amip_CMOR-landCML.lst | ||
| # -r /home/san/CMOR/cmor/cmip6-cmor-tables/Tables/CMIP6_Lmon.json | ||
| # -p /home/san/CMOR/cmor/Test/CMOR_input_ESM4_amip.json | ||
|
|
||
| # ~/fms_yaml_tools/cmor_mixer.py | ||
| # -d /archive/oar.gfdl.cmip6/CM4/warsaw_201710_om4_v1.0.1/CM4_historical/gfdl.ncrc4-intel16-prod-openmp/pp/atmos/ts/monthly/5yr | ||
| # -l /home/san/CMOR_3/GFDL-CM4_historical_CMOR-Amon.lst | ||
| # -r /home/san/CMOR/cmor/cmip6-cmor-tables/Tables/Atmos_Monthly.json | ||
| # -p /home/san/CMOR/cmor/Test/CMOR_input_CM4_historical.json | ||
|
|
||
| # ~/fms_yaml_tools/cmor_mixer.py | ||
| # -d /archive/oar.gfdl.cmip6/ESM4/DECK/ESM4_historical_D1/gfdl.ncrc4-intel16-prod-openmp/pp/atmos_cmip/ts/daily/5yr | ||
| # -l /home/san/CMOR_3/GFDL-ESM4_CMOR-day_historical.lst | ||
| # -r /home/san/CMOR/cmor/cmip6-cmor-tables/Tables/CMIP6_day.json | ||
| # -p /home/san/CMOR/cmor/Test/CMOR_input_ESM4_historical.json | ||
| # -o /net2/san | ||
|
|
||
| # ~/fms_yaml_tools/cmor_mixer.py | ||
| # -d /archive/oar.gfdl.cmip6/ESM4/DECK/ESM4_historical_D1/gfdl.ncrc4-intel16-prod-openmp/pp/atmos/ts/6hr/5yr | ||
| # -l /home/san/CMOR_3/GFDL-ESM4_CMOR-6hr.lst | ||
| # -r /home/san/CMOR/cmor/cmip6-cmor-tables/Tables/CMIP6_6hrPlev.json | ||
| # -p /home/san/CMOR/cmor/Test/CMOR_input_ESM4_historical.json | ||
|
|
||
| # ~/fms_yaml_tools/cmor_mixer.py | ||
| # -d /archive/oar.gfdl.cmip6/ESM4/DECK/ESM4_historical_D1/gfdl.ncrc4-intel16-prod-openmp/pp/atmos_cmip/ts/3hr/5yr | ||
| # -l /home/san/CMOR_3/GFDL_ESM4_historical_CMOR-3hr.lst | ||
| # -r /home/san/CMOR/cmor/cmip6-cmor-tables/Tables/CMIP6_3hr.json | ||
| # -p /home/san/CMOR/cmor/Test/CMOR_input_ESM4_historical.json | ||
| # -o /net2/san | ||
|
|
||
| # Additional tables containing in /home/san/CMIP6_work/cmor/cmip6-cmor-tables/Tables: | ||
| # CMIP6_CV.json | ||
| # CMIP6_formula_terms.json | ||
| # CMIP6_grids.json | ||
| # CMIP6_coordinate.json | ||
|
|
||
| # Detailed description of program is placed at | ||
| # https://docs.google.com/document/d/1HPetcUyrVXDwCBIyWheZ_2JzOz7ZHi1y3vmIlcErYeA/edit?pli=1 | ||
|
|
||
| # Keep in mind rule for input ../cmor/cmip6-cmor-tables/Tables/*.json: | ||
| # output variables can not contain "_" in out_name, though name (and standard_name) itself can have it; example: | ||
| # "alb_sfc": { | ||
| # "frequency": "mon", | ||
| # "modeling_realm": "atmos", | ||
| # "standard_name": "alb_sfc", | ||
| # "units": "percent", | ||
| # "cell_methods": "area: time: mean", | ||
| # "long_name": "surface albedo", | ||
| # "comment": "", | ||
| # "dimensions": "longitude latitude time", | ||
| # "out_name": "albsfc", | ||
| # "type": "real", | ||
| # "positive": "", | ||
| # "valid_min": "", | ||
| # "valid_max": "", | ||
| # "ok_min_mean_abs": "", | ||
| # "ok_max_mean_abs": "" | ||
| # } | ||
|
|
||
| # Problems with standard CMOR library: | ||
| # - monthly variable "enth_conv_col" produces error - CMOR expects 4 dimensions but it has only 3; | ||
| # - variable /archive/oar.gfdl.cmip6/CM4/warsaw_201710_om4_v1.0.1/CM4_historical/gfdl.ncrc4-intel16-prod-openmp/pp/atmos_cmip/ts/3hr/5yr/atmos_cmip.1965010100-1969123123.clt.nc | ||
| # is not readable. | ||
| UNDER CONSTRUCTION: old usage notes at the top of `cmor_mixer.py`, re-rigged for markdown and CMIP7. | ||
|
|
||
| at PP/AN, module load the latest `fre-cli` that's been pushed to the main branch: | ||
| ``` | ||
| > module load fre/canopy | ||
| > which fre | ||
| /home/fms/local/opt/fre-commands/canopy/bin/fre | ||
| ``` | ||
|
|
||
| alternatively, with access to conda: | ||
| ``` | ||
| > conda activate /nbhome/fms/conda/envs/fre-cli | ||
| > which fre | ||
| /nbhome/fms/conda/envs/fre-cli/bin/fre | ||
| ``` | ||
|
|
||
| this subtool's help, and command-specific `run` help: | ||
| ``` | ||
| > fre cmor --help | ||
| Usage: fre cmor [OPTIONS] COMMAND [ARGS]... | ||
| - access fre cmor subcommands | ||
| Options: | ||
| --help Show this message and exit. | ||
| Commands: | ||
| run Rewrite climate model output | ||
| # subtool command-specific help, e.g. for run | ||
| > fre cmor run --help | ||
| Usage: fre cmor run [OPTIONS] | ||
| Rewrite climate model output | ||
| Options: | ||
| -d, --indir TEXT Input directory [required] | ||
| -l, --varlist TEXT Variable list [required] | ||
| -r, --table_config TEXT Table configuration [required] | ||
| -p, --exp_config TEXT Experiment configuration [required] | ||
| -o, --outdir TEXT Output directory [required] | ||
| --help Show this message and exit. | ||
| ``` | ||
|
|
||
|
|
||
| the tool requires configuration in the form of variable tables and conventions to work appropriately | ||
| clone the following repository and list the following directory contents to get a sense of what | ||
| the code needs from you to work. a few examples shown in the output below. the CV file is of particular interest/necessity | ||
| ``` | ||
| > git clone https://github.com/PCMDI/cmip6-cmor-tables.git fre/tests/test_files/cmip6-cmor-tables | ||
| > ls fre/tests/test_files/cmip6-cmor-tables/Tables | ||
| ... | ||
| CMIP6_CV.json | ||
| CMIP6_formula_terms.json | ||
| CMIP6_grids.json | ||
| CMIP6_coordinate.json | ||
| CMIP6_input_example.json | ||
| ... | ||
| CMIP6_3hr.json | ||
| ... | ||
| CMIP6_Efx.json | ||
| ... | ||
| CMIP6_IyrGre.json | ||
| ... | ||
| ``` | ||
|
|
||
|
|
||
| Simple example call(s) using fre-cli in the root directory of this repository note the line-continuation character at the end for readability, | ||
| you may wish to avoid it when copy/pasting. | ||
| ``` | ||
| > fre cmor run \ | ||
| -d fre/tests/test_files/ocean_sos_var_file \ | ||
| -l fre/tests/test_files/varlist \ | ||
| -r fre/tests/test_files/cmip6-cmor-tables/Tables/CMIP6_Omon.json \ | ||
| -p fre/tests/test_files/CMOR_input_example.json \ | ||
| -o fre/tests/test_files/outdir | ||
| ``` | ||
|
|
||
|
|
||
|
|
Oops, something went wrong.