snakemake #14

jsadler2 · 2022-05-19T18:13:23Z

Adding snakefile to gridmet repo.

For some reason, this isn't working quite right yet.

The *gdf.pickle file isn't being written out. So I sill need to do some debugging.

msleckman

Made several attempts at running this in hpc and ran into errors as I describe below.

@jzwart could you give this a try as well?

msleckman · 2022-09-21T01:42:58Z

README.md

+
+You can run the workflow via `sbatch` 
+```
+sbatch -A iidd slurm/launch_snakemake.slum config_nhd.yml


Suggested change

sbatch -A iidd slurm/launch_snakemake.slum config_nhd.yml

sbatch -A iidd slurm/launch_snakemake.slurm config_nhd.yml

msleckman · 2022-09-21T01:43:30Z

README.md

+```
+salloc -N 1 -n 8 -t 10:00:00 -p cpu -A iidd
+module load singularity
+singularity exec-agg_v0.3.sif /opt/conda/bin/snakemake -j --configfile config_nhd.yml 


I think this is a typo right?

Suggested change

singularity exec-agg_v0.3.sif /opt/conda/bin/snakemake -j --configfile config_nhd.yml

singularity exec gridmet-agg_v0.3.sif /opt/conda/bin/snakemake -j --configfile config_nhd.yml

msleckman · 2022-09-21T01:48:44Z

README.md

+salloc -N 1 -n 8 -t 10:00:00 -p cpu -A iidd
+module load singularity
+singularity exec-agg_v0.3.sif /opt/conda/bin/snakemake -j --configfile config_nhd.yml 


When I run:

salloc -N 1 -n 8 -t 10:00:00 -p cpu -A iidd module load singularity singularity exec gridmet-agg_v0.3.sif /opt/conda/bin/snakemake -j --configfile config_nhd.yml

from the pump/gridmet/drb_gridmet_tools/ (following option A).
I get a Permission error
Permission denied: '/caldera/projects/usgs/water/impd/pump/gridmet/drb_gridmet_tools/.snakemake/log/2022-09-20T203254.660009.snakemake.log'

There isn't write access to all the files / directories (see below) so everyone but Jeff will get this error. We'll have to ask HPC folks to give group permissions in this folder because I'm pretty sure Jeff doesn't have access to USGS HPC anymore

[jzwart@tg-login1 drb_gridmet_tools] ls -la total 995356 drwxr-sr-x 11 jsadler pump 4096 Jun 17 12:24 . drwxr-sr-x 3 jsadler pump 4096 Jan 28 2022 .. drwxr-sr-x 3 jsadler pump 4096 Jun 17 12:22 archive -rw-r--r-- 1 jsadler pump 509 Jun 15 13:43 config_nhd.yml -rw-r--r-- 1 jsadler pump 536 Jun 15 16:04 config_nhm.yml drwxr-sr-x 4 jsadler pump 4096 Jun 14 11:09 drb-gridmet -rw-r--r-- 1 jsadler pump 536 Apr 8 11:42 environment.yml drwxr-sr-x 8 jsadler pump 4096 Jun 17 12:24 .git -rw-r--r-- 1 jsadler pump 123 Jan 28 2022 .gitignore -rw-r--r-- 1 jsadler pump 3079 Jun 15 12:34 gridmet_aggregation_PRMS.py -rwxrwxr-x 1 jsadler pump 1019154432 Jun 15 13:40 gridmet-agg_v0.3.sif -rw-r--r-- 1 jsadler pump 7329 Jun 14 15:51 gridmet_split_script.py drwxr-sr-x 2 jsadler pump 4096 Apr 21 15:57 .ipynb_checkpoints drwxr-sr-x 2 jsadler pump 4096 Jun 15 12:34 __pycache__ -rw-r--r-- 1 jsadler pump 4081 Jun 15 16:06 README.md drwxr-sr-x 5 jsadler pump 4096 Jun 15 13:46 scratch drwxr-sr-x 2 jsadler pump 4096 Jun 17 12:23 slurm drwxr-sr-x 2 jsadler pump 4096 Jun 17 12:23 slurm_out -rw-r--r-- 1 jsadler pump 6400 Jun 15 16:02 Snakefile drwxr-sr-x 11 jsadler pump 4096 Jun 9 16:57 .snakemake

I'll ask in #tallgrass-help channel

that makes sense 👍

did we get any response on this?

Yes (see here), they changed who owns the files and permissions should be updated.

[jzwart@tg-login2 drb_gridmet_tools] ls -la total 995356 drwxrwsr-x 11 jzwart pump 4096 Jun 17 12:24 . drwxrwsr-x 3 jzwart pump 4096 Jan 28 2022 .. drwxrwsr-x 3 jzwart pump 4096 Jun 17 12:22 archive -rw-rwxr-- 1 jzwart pump 509 Jun 15 13:43 config_nhd.yml -rw-rwxr-- 1 jzwart pump 536 Jun 15 16:04 config_nhm.yml drwxrwsr-x 4 jzwart pump 4096 Jun 14 11:09 drb-gridmet -rw-rwxr-- 1 jzwart pump 536 Apr 8 11:42 environment.yml drwxrwsr-x 8 jzwart pump 4096 Jun 17 12:24 .git -rw-rwxr-- 1 jzwart pump 123 Jan 28 2022 .gitignore -rw-rwxr-- 1 jzwart pump 3079 Jun 15 12:34 gridmet_aggregation_PRMS.py -rwxrwxr-x 1 jzwart pump 1019154432 Jun 15 13:40 gridmet-agg_v0.3.sif -rw-rwxr-- 1 jzwart pump 7329 Jun 14 15:51 gridmet_split_script.py drwxrwsr-x 2 jzwart pump 4096 Apr 21 15:57 .ipynb_checkpoints drwxrwsr-x 2 jzwart pump 4096 Jun 15 12:34 __pycache__ -rw-rwxr-- 1 jzwart pump 4081 Jun 15 16:06 README.md drwxrwsr-x 5 jzwart pump 4096 Jun 15 13:46 scratch drwxrwsr-x 2 jzwart pump 4096 Jun 17 12:23 slurm drwxrwsr-x 2 jzwart pump 4096 Jun 17 12:23 slurm_out -rw-rwxr-- 1 jzwart pump 6400 Jun 15 16:02 Snakefile drwxrwsr-x 11 jzwart pump 4096 Jun 9 16:57 .snakemake

msleckman · 2022-09-21T01:56:53Z

README.md

+git clone [email protected]:USGS-R/drb_gridmet_tools.git
+cd drb_gridmet_tools
+```
+2. Pull down Docker image into a Singularity file
+```
+singularity pull docker://jsadler2/gridmet-agg:v0.3
+```
+3. Do Steps 2-4 from Option A.
+


Ran this in my own user dir in tallgrass (as well as on a different project dir) in impd/pump/ and in both cases got the following error:

ERROR 1: PROJ: proj_create_from_database: Open of /opt/conda/share/proj failed Building DAG of jobs...

and botocore.exceptions.NoCredentialsError: Unable to locate credentials

I think this refers to the S3 credentials - could you try re-running after setting use_S3: False in the config_XXX.yml ?

sure in that case I run locally?

I think that'd help to run locally.

msleckman · 2022-09-21T02:09:03Z

README.md

+The re-gridding process for the Delaware River Basin has been run on USGS's Tallgrass via Singularity. It has been run for the National Hydrologic Model (NHM) fabric and the National Hydrographic Database (NHD) fabric. It should be able to be run via Docker as well as Singularity, but it has not been tried yet. Instructions for running and modifying the pipeline are below.
+
+_Parallelization in Snakemake_
+The Snakemake workflow parallelizes the re-gridding of the 8 Gridmet variables. As long as you provide at least 8 cores, these tasks will run in parallel.


I like this set up and the Snakefile is clear! I did not get a chance to say this earlier, but thanks for setting this up like this.

README.md

jzwart · 2022-09-22T10:33:27Z

README.md

+salloc -N 1 -n 8 -t 10:00:00 -p cpu -A iidd
+module load singularity
+singularity exec-agg_v0.3.sif /opt/conda/bin/snakemake -j --configfile config_nhd.yml 


There isn't write access to all the files / directories (see below) so everyone but Jeff will get this error. We'll have to ask HPC folks to give group permissions in this folder because I'm pretty sure Jeff doesn't have access to USGS HPC anymore

[jzwart@tg-login1 drb_gridmet_tools] ls -la total 995356 drwxr-sr-x 11 jsadler pump 4096 Jun 17 12:24 . drwxr-sr-x 3 jsadler pump 4096 Jan 28 2022 .. drwxr-sr-x 3 jsadler pump 4096 Jun 17 12:22 archive -rw-r--r-- 1 jsadler pump 509 Jun 15 13:43 config_nhd.yml -rw-r--r-- 1 jsadler pump 536 Jun 15 16:04 config_nhm.yml drwxr-sr-x 4 jsadler pump 4096 Jun 14 11:09 drb-gridmet -rw-r--r-- 1 jsadler pump 536 Apr 8 11:42 environment.yml drwxr-sr-x 8 jsadler pump 4096 Jun 17 12:24 .git -rw-r--r-- 1 jsadler pump 123 Jan 28 2022 .gitignore -rw-r--r-- 1 jsadler pump 3079 Jun 15 12:34 gridmet_aggregation_PRMS.py -rwxrwxr-x 1 jsadler pump 1019154432 Jun 15 13:40 gridmet-agg_v0.3.sif -rw-r--r-- 1 jsadler pump 7329 Jun 14 15:51 gridmet_split_script.py drwxr-sr-x 2 jsadler pump 4096 Apr 21 15:57 .ipynb_checkpoints drwxr-sr-x 2 jsadler pump 4096 Jun 15 12:34 __pycache__ -rw-r--r-- 1 jsadler pump 4081 Jun 15 16:06 README.md drwxr-sr-x 5 jsadler pump 4096 Jun 15 13:46 scratch drwxr-sr-x 2 jsadler pump 4096 Jun 17 12:23 slurm drwxr-sr-x 2 jsadler pump 4096 Jun 17 12:23 slurm_out -rw-r--r-- 1 jsadler pump 6400 Jun 15 16:02 Snakefile drwxr-sr-x 11 jsadler pump 4096 Jun 9 16:57 .snakemake

jzwart · 2022-09-22T10:34:09Z

README.md

+salloc -N 1 -n 8 -t 10:00:00 -p cpu -A iidd
+module load singularity
+singularity exec-agg_v0.3.sif /opt/conda/bin/snakemake -j --configfile config_nhd.yml 


I'll ask in #tallgrass-help channel

jzwart · 2022-09-22T10:40:13Z

README.md

+git clone [email protected]:USGS-R/drb_gridmet_tools.git
+cd drb_gridmet_tools
+```
+2. Pull down Docker image into a Singularity file
+```
+singularity pull docker://jsadler2/gridmet-agg:v0.3
+```
+3. Do Steps 2-4 from Option A.
+


I think this refers to the S3 credentials - could you try re-running after setting use_S3: False in the config_XXX.yml ?

Co-authored-by: Jake Zwart <[email protected]>

jsadler2 added 2 commits May 19, 2022 13:11

[USGS-R#13] first draft of snakefile

fa5cb8e

[USGS-R#13] updates to snakefile

d2a4daf

jsadler2 marked this pull request as draft May 19, 2022 18:13

jsadler2 added 7 commits June 14, 2022 15:51

[USGS-R#13] Snakefile parallel vars

a2224a5

[USGS-R#13] remove uneeded data files

c7b23cd

[USGS-R#13] S3 with Snakemake

d36a767

[USGS-R#13] rm fxn calls from .py files

c38d1fd

[USGS-R#13] config files for snakemake

c1faf55

[USGS-R#13] instructions on running snakemake workflow

bffb26c

[USGS-R#13] which config opts probs shouldn't change

1cdf7b4

jsadler2 marked this pull request as ready for review June 15, 2022 14:48

jsadler2 added 2 commits June 15, 2022 13:14

[USGS-R#13] more meaningful ids in output nc files

f6f3626

[USGS-R#13] write and push zarr files

be5ca80

msleckman reviewed Sep 21, 2022

View reviewed changes

jzwart reviewed Sep 22, 2022

View reviewed changes

Update README.md

87d3d85

Co-authored-by: Jake Zwart <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

snakemake #14

snakemake #14

jsadler2 commented May 19, 2022

msleckman left a comment

msleckman Sep 21, 2022

msleckman Sep 21, 2022

msleckman Sep 21, 2022

jzwart Sep 22, 2022

jzwart Sep 22, 2022

msleckman Sep 22, 2022

msleckman Oct 4, 2022

jzwart Oct 5, 2022

msleckman Sep 21, 2022

jzwart Sep 22, 2022

msleckman Sep 22, 2022

jzwart Sep 22, 2022

msleckman Sep 21, 2022

jzwart Sep 22, 2022

jzwart Sep 22, 2022

jzwart Sep 22, 2022

	sbatch -A iidd slurm/launch_snakemake.slum config_nhd.yml
	sbatch -A iidd slurm/launch_snakemake.slurm config_nhd.yml

	singularity exec-agg_v0.3.sif /opt/conda/bin/snakemake -j --configfile config_nhd.yml
	singularity exec gridmet-agg_v0.3.sif /opt/conda/bin/snakemake -j --configfile config_nhd.yml

snakemake #14

Are you sure you want to change the base?

snakemake #14

Conversation

jsadler2 commented May 19, 2022

msleckman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment