Merge pull request #196 from NOAA-GFDL/main

Merging main into the branch

fre/make/README.md

@@ -2,26 +2,89 @@
 Through the fre-cli, `fre make` can be used to create and run a checkout script, makefile, and compile a model.
 
 * Fremake Canopy Supports:
-   - multiple targets, would have to use one `-t` flag for each one
+   - multiple targets; use `-t` flag to define each target
    - bare-metal build
    - container creation
    - parallel checkouts for bare-metal build**
 
-* **Note: Users will not be able to create containers without access to podman**
+** **Note: Users will not be able to create containers without access to podman**
 
-The fremake canopy fre-cli subcommands are described below ([Subcommands](#subcommands)), as well as a Guide on the order in which to use them ([Guide](#guide)).
+The fremake canopy fre-cli subcommands are described below ([Subtools](#subtools)), as well as a Guide on the order in which to use them ([Guide](#guide)).
 
 Additionally, as mentioned, multiple targets can be used more multiple target-platform combinations. Below is an example of this usage for both the bare-metal build and container build, using the AM5 model
 
-- [Bare-metal Example](#bare-metal-build-multi-target-example)  
-- [Container Example](#container-build-multi-target-example)
+- [Bare-metal Example](#bare-metal-build) 
+- [Bare-metal Multi-target Example](#bare-metal-build-multi-target)  
+- [Container Example](#container-build)
 
 ## **Usage (Users)**
 * Refer to fre-cli [README.md](https://github.com/NOAA-GFDL/fre-cli/blob/main/README.md) for foundational fre-cli usage guide and tips.
-* Fremake package repository located at: https://gitlab.gfdl.noaa.gov/portable_climate/fremake_canopy/-/tree/main
 
+## **Quickstart** 
+### **Bare-metal Build:**
+```bash
+# Create checkout script
+fre make create-checkout -y am5.yaml -p ncrc5.intel23 -t prod
+
+# Create and run checkout script
+fre make create-checkout -y am5.yaml -p ncrc5.intel23 -t prod --execute
+
+# Create Makefile
+fre make create-makefile -y am5.yaml -p ncrc5.intel23 -t prod
+
+# Create the compile script
+fre make create-compile -y am5.yaml -p ncrc5.intel23 -t prod
+
+# Create and run the compile script
+fre make create-compile -y am5.yaml -p ncrc5.intel23 -t prod --execute
+```
+### **Bare-metal Build Multi-target:**
+```bash
+# Create checkout script
+fre make create-checkout -y am5.yaml -p ncrc5.intel23 -t prod -t debug
+
+# Create and run checkout script
+fre make create-checkout -y am5.yaml -p ncrc5.intel23 -t prod -t debug --execute
+
+# Create Makefile
+fre make create-makefile -y am5.yaml -p ncrc5.intel23 -t prod -t debug
+
+# Create the compile script
+fre make create-compile -y am5.yaml -p ncrc5.intel23 -t prod -t debug
+
+# Create and run the compile script
+fre make create-compile -y am5.yaml -p ncrc5.intel23 -t prod -t debug --execute
+```
+
+### **Container Build:**
+In order for the container to build successfully, a `-npc`, or `--no-parallel-checkout` is needed.
+```bash
+# Create checkout script
+fre make create-checkout -y am5.yaml -p hpcme.2023 -t prod -npc
+
+# Create and run checkout script
+fre make create-checkout -y am5.yaml -p hpcme.2023 -t prod -npc --execute
+
+# Create Makefile
+fre make create-makefile -y am5.yaml -p hpcme.2023 -t prod
+
+# Create Dockerfile
+fre make create-dockerfile -y am5.yaml -p hpcme.2023 -t prod
+
+# Create and run the Dockerfile
+fre make create-dockerfile -y am5.yaml -p hpcme.2023 -t prod --execute
+```
 
-## Subcommands
+### **Run all of fremake:**
+```bash
+# Bare-metal 
+fre make run-fremake -y am5.yaml -p ncrc5.intel23 -t prod
+
+# Container
+fre make run-fremake -y am5.yaml -p hpcme.2023 -t prod -npc 
+```
+
+## Subtools
 - `fre make create-checkout [options]`
    - Purpose: Creates the checkout script and can check out source code (with execute option)
    - Options:
@@ -69,87 +132,49 @@ Additionally, as mentioned, multiple targets can be used more multiple target-pl
         - `-n, --parallel [number of concurrent modile compiles]`
 
 ## Guide
-### **Bare-metal Build:**
-```bash
-# Create checkout script
-fre make create-checkout -y [experiment yaml file] -p [platform] -t [target]
-
-# Create and run checkout script
-fre make create-checkout -y [experiment yaml file] -p [platform] -t [target] -e
+In order to use the `fre make` tools, remember to create a combined yaml first. This can be done with the `fre yamltools combine-yamls` tool. This combines the model, compile, platform, experiment, and any analysis yamls into ONE yaml file for parsing and validation. 
 
-# Create Makefile
-fre make create-makefile -y [experiment yaml file] -p [platform] -t [target]
-
-# Creat the compile script
-fre make create-compile -y [experiment yaml file] -p [platform] -t [target]
-
-# Create and run the compile script
-fre make create-compile -y [experiment yaml file] -p [platform] -t [target] -e
+To combine: 
+`fre yamltools combine-yamls -y [model yaml file] -e [experiment name] -p [platform] -t [target]`
 
-# Run all of fremake 
-fre make run-fremake -y [experiment yaml] -p [platform] -t [target] [other options...]
-```
-
-### **Bare-metal Build (Multi-target example):**
+### **Bare-metal Build:**
 ```bash
 # Create checkout script
-fre make create-checkout -y am5.yaml -p ncrc5.intel -t prod-openmp -t debug
+fre make create-checkout -y [model yaml file] -p [platform] -t [target]
 
 # Create and run checkout script
-fre make create-checkout -y am5.yaml -p ncrc5.intel -t prod-openmp -t debug -e
+fre make create-checkout -y [model yaml file] -p [platform] -t [target] --execute
 
 # Create Makefile
-fre make create-makefile -y am5.yaml -p ncrc5.intel -t prod-openmp -t debug
+fre make create-makefile -y [model yaml file] -p [platform] -t [target]
 
 # Creat the compile script
-fre make create-compile -y am5.yaml -p ncrc5.intel -t prod-openmp -t debug
+fre make create-compile -y [model yaml file] -p [platform] -t [target]
 
 # Create and run the compile script
-fre make create-compile -y am5.yaml -p ncrc5.intel -t prod-openmp -t debug -e
+fre make create-compile -y [model yaml file] -p [platform] -t [target] --execute
 
 # Run all of fremake 
-fre make run-fremake -y am5.yaml -p ncrc5.intel -t prod-openmp -t debug [other options...]
+fre make run-fremake -y [model yaml file] -p [platform] -t [target] [other options...]
 ```
 
 ### **Container Build:**
-For the container build, parallel checkouts are not supported, so the `-npc` options must be used for the checkout script. In addition the platform must be a container platform. ***To reiterate, users will not be able to create containers unless they have podman access on gaea.***
+For the container build, parallel checkouts are not supported, so the `-npc` options must be used for the checkout script. In addition the platform must be a container platform. 
+
+***To reiterate, users will not be able to create containers unless they have podman access on gaea.***
 ```bash
 # Create checkout script
-fre make create-checkout -y [experiment yaml file] -p [CONTAINER PLATFORM] -t [target] -npc
+fre make create-checkout -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] -npc
 
 # Create and run checkout script
-fre make create-checkout -y [experiment yaml file] -p [CONTAINER PLATFORM] -t [target] -e -npc
+fre make create-checkout -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] --execute
 
 # Create Makefile
-fre make create-makefile -y [experiment yaml file] -p [CONTAINER PLATFORM] -t [target] 
-
-# Create the compile script
-fre make create-compile -y [experiment yaml file] -p [CONTAINER PLATFORM]-t [target] 
+fre make create-makefile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target]
 
 #Create a Dockerfile
-fre make create-dockerfile -y [experiment yaml file] -p [CONTAINER PLATFORM] -t [target] 
+fre make create-dockerfile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target]
 
 # Create and run the Dockerfile
-fre make create-dockerfile -y [experiment yaml file] -p [CONTAINER PLATFORM] -t [target]
+fre make create-dockerfile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] --execute
 ```
-### **Container Build (Multi-target example):**
-```bash
-# Create checkout script
-fre make create-checkout -y am5.yaml -p hpcme.2023 -t prod-openmp -t debug -npc
-
-# Create and run checkout script
-fre make create-checkout -y am5.yaml -p hpcme.2023 -t prod-openmp -t debug -npc -e
-
-# Create Makefile
-fre make create-makefile -y am5.yaml -p hpcme.2023 -t prod-openmp -t debug
-
-# Creat the compile script
-fre make create-compile -y am5.yaml -p hpcme.2023 -t prod-openmp -t debug
-
-# Create and run the compile script
-fre make create-compile -y am5.yaml -p hpcme.2023 -t prod-openmp -t debug -e
-
-# Run all of fremake 
-fre make run-fremake -y am5.yaml -p hpcme.2023 -t prod-openmp -t debug [other options...] -npc 
-```
-

fre/make/createCheckout.py

@@ -1,17 +1,19 @@
 #!/usr/bin/python3
 
 import os
+import subprocess
 import logging
 import sys
 import click
+from pathlib import Path
 from .gfdlfremake import varsfre, platformfre, yamlfre, checkout, targetfre
+import fre.yamltools.combine_yamls as cy
 
 @click.command()
 def checkout_create(yamlfile,platform,target,no_parallel_checkout,jobs,execute,verbose):
     # Define variables
     yml = yamlfile
-    ps = platform
-    ts = target
+    name = yamlfile.split(".")[0]
     run = execute
     jobs = str(jobs)
     pcheck = no_parallel_checkout
@@ -34,11 +36,16 @@ def checkout_create(yamlfile,platform,target,no_parallel_checkout,jobs,execute,v
     plist = platform
     tlist = target
 
+    # Combine model, compile, and platform yamls
+    # Default behavior - combine yamls / rewrite combined yaml
+    comb = cy.init_compile_yaml(yml,platform,target)
+    full_combined = cy.get_combined_compileyaml(comb)
+
     ## Get the variables in the model yaml
-    freVars = varsfre.frevars(yml)
+    freVars = varsfre.frevars(full_combined) 
 
-    ## Open the yaml file and parse as fremakeYaml
-    modelYaml = yamlfre.freyaml(yml,freVars)
+    ## Open the yaml file, validate the yaml, and parse as fremakeYaml
+    modelYaml = yamlfre.freyaml(full_combined,freVars)
     fremakeYaml = modelYaml.getCompileYaml()
 
     ## Error checking the targets
@@ -53,7 +60,7 @@ def checkout_create(yamlfile,platform,target,no_parallel_checkout,jobs,execute,v
          if modelYaml.platforms.hasPlatform(platformName):
               pass
          else:
-              raise SystemExit (platformName + " does not exist in " + modelYaml.platformsfile)
+              raise ValueError (platformName + " does not exist in platforms.yaml") 
          (compiler,modules,modulesInit,fc,cc,modelRoot,iscontainer,mkTemplate,containerBuild,ContainerRun,RUNenv)=modelYaml.platforms.getPlatformFromName(platformName)
 
     ## Create the source directory for the platform
@@ -67,22 +74,34 @@ def checkout_create(yamlfile,platform,target,no_parallel_checkout,jobs,execute,v
                    freCheckout = checkout.checkout("checkout.sh",srcDir)
                    freCheckout.writeCheckout(modelYaml.compile.getCompileYaml(),jobs,pc)
                    freCheckout.finish(pc)
-                   click.echo("\nCheckout script created at " + srcDir + "/checkout.sh" + "\n")
+                   click.echo("\nCheckout script created in "+ srcDir + "/checkout.sh \n")
 
-              # Run the checkout script
-              if run:
-                   freCheckout.run()
+                   # Run the checkout script
+                   if run == True:
+                        freCheckout.run()
+                   else:
+                        sys.exit()
               else:
-                   sys.exit()
+                   print("\nCheckout script PREVIOUSLY created in "+ srcDir + "/checkout.sh \n")
+                   if run == True:
+                        os.chmod(srcDir+"/checkout.sh", 0o744)
+                        try:
+                             subprocess.run(args=[srcDir+"/checkout.sh"], check=True)
+                        except:
+                             print("\nThere was an error with the checkout script "+srcDir+"/checkout.sh.",
+                                   "\nTry removing test folder: " + modelRoot +"\n")
+                             raise
+                   else:
+                        sys.exit()
+
          else:
-              ## Run the checkout script
               image="ecpe4s/noaa-intel-prototype:2023.09.25"
               bldDir = modelRoot + "/" + fremakeYaml["experiment"] + "/exec"
               tmpDir = "tmp/"+platformName
               freCheckout = checkout.checkoutForContainer("checkout.sh", srcDir, tmpDir)
               freCheckout.writeCheckout(modelYaml.compile.getCompileYaml(),jobs,pc)
               freCheckout.finish(pc)
-              click.echo("\nCheckout script created at " + srcDir + "/checkout.sh" + "\n")
+              click.echo("\nCheckout script created at " + tmpDir + "/checkout.sh" + "\n")
 
 
 if __name__ == "__main__":

fre/make/createCompile.py

@@ -3,17 +3,17 @@
 import os
 import sys
 import logging
+from pathlib import Path
 from multiprocessing.dummy import Pool
-
 import click
 from .gfdlfremake import varsfre, platformfre, yamlfre, targetfre, buildBaremetal
+import fre.yamltools.combine_yamls as cy
 
 @click.command()
 def compile_create(yamlfile,platform,target,jobs,parallel,execute,verbose):
     # Define variables
     yml = yamlfile
-    ps = platform
-    ts = target
+    name = yamlfile.split(".")[0]
     nparallel = parallel
     jobs = str(jobs)
     run = execute
@@ -31,11 +31,18 @@ def compile_create(yamlfile,platform,target,jobs,parallel,execute,verbose):
     plist = platform
     tlist = target
 
+    # Combined compile yaml file
+    combined = Path(f"combined-{name}.yaml")
+
+    ## If combined yaml exists, note message of its existence
+    ## If combined yaml does not exist, combine model, compile, and platform yamls
+    full_combined = cy.combined_compile_existcheck(combined,yml,platform,target)
+
     ## Get the variables in the model yaml
-    freVars = varsfre.frevars(yml)
+    freVars = varsfre.frevars(full_combined)
 
     ## Open the yaml file and parse as fremakeYaml
-    modelYaml = yamlfre.freyaml(yml,freVars)
+    modelYaml = yamlfre.freyaml(full_combined,freVars)
     fremakeYaml = modelYaml.getCompileYaml()
 
     ## Error checking the targets
@@ -50,7 +57,8 @@ def compile_create(yamlfile,platform,target,jobs,parallel,execute,verbose):
          if modelYaml.platforms.hasPlatform(platformName):
               pass
          else:
-              raise SystemExit (platformName + " does not exist in " + modelYaml.platformsfile)
+              raise ValueError (platformName + " does not exist in " + modelYaml.combined.get("compile").get("platformYaml"))
+
          (compiler,modules,modulesInit,fc,cc,modelRoot,iscontainer,mkTemplate,containerBuild,ContainerRun,RUNenv)=modelYaml.platforms.getPlatformFromName(platformName)
     ## Make the bldDir based on the modelRoot, the platform, and the target
          srcDir = modelRoot + "/" + fremakeYaml["experiment"] + "/src"