From 6112a3593f8b361a5594493a5a18c4ca18a0a52d Mon Sep 17 00:00:00 2001 From: Jeanette Clark Date: Thu, 21 Mar 2024 14:46:15 -0700 Subject: [PATCH] combine group projects --- sections/group-project-1.qmd | 13 +++++++++-- sections/group-project-2.qmd | 32 ++++++++++++++++++------- sections/group-project-3.qmd | 45 ------------------------------------ 3 files changed, 35 insertions(+), 55 deletions(-) delete mode 100644 sections/group-project-3.qmd diff --git a/sections/group-project-1.qmd b/sections/group-project-1.qmd index 05dbd5f..02d4517 100644 --- a/sections/group-project-1.qmd +++ b/sections/group-project-1.qmd @@ -33,9 +33,10 @@ In the final step of the workflow, we will take the raster files and resample th ![](../images/gp-step3.png) -## Staging and Tiling -Today we will undertake the first step of the workflow, staging and tiling the data. +Today we will undertake the first two steps of the workflow, staging and tiling the data, and then rasterizing it. + +## Staging and Tiling ![](../images/gp-step1.png) @@ -45,12 +46,20 @@ In your small groups, work together to write the solution, but everyone should a Only one person in the group should run the parallel code. +## Rasterizing GeoPackages + +In the last step, we staged the input files into GeoPackages. Now we will import those `.gpkg` files and write each as a raster (`.tif`). The resulting rasters will have 2 bands, one for each statistic we calculate based on the vector geometries in each pixel. + +We need to create the highest zoom level rasters before we create the lower zoom levels in the next lesson. This brings us one step closer to visualizing the ice wedge polygons on a basemap, with the ability to zoom in and out! + ### Packages, Libraries, and Modules - [os 3.11.2](https://docs.python.org/3/library/os.html) - [parsl 2023.3.20](https://parsl.readthedocs.io/en/stable/) - [pdgstaging](https://github.com/PermafrostDiscoveryGateway/viz-staging) - package developed by Permafrost Discovery Gateway software developer and designer Robyn Thiessen-Bock +- [pdgraster](https://github.com/PermafrostDiscoveryGateway/viz-raster/tree/main) + - package developed by Permafrost Discovery Gateway software developer and designer Robyn Thiessen-Bock - [geopandas0.11](https://geopandas.org/en/stable/docs.html) - [random](https://docs.python.org/3/library/random.html) - [matplotlib 3.5](https://matplotlib.org/3.5.3/api/_as_gen/matplotlib.pyplot.html) \ No newline at end of file diff --git a/sections/group-project-2.qmd b/sections/group-project-2.qmd index baba9a9..53c9a43 100644 --- a/sections/group-project-2.qmd +++ b/sections/group-project-2.qmd @@ -1,29 +1,45 @@ --- -title: 'Group Project: Data Processing' +title: 'Group Project: Visualization' --- + ## Setup -In your fork of the [`scalable-computing-examples`](https://github.com/NCEAS/scalable-computing-examples) repository, open the Jupyter notebook in the `group-project` directory called `session-11.ipynb`. This workbook will serve as a skeleton for you to work in. It will load in all the libraries you need, including a few helper functions we wrote for the course, show an example for how to use the method on one file, and then lays out blocks for you and your group to fill in with code that will run that method in parallel. +In your fork of the [`scalable-computing-examples`](https://github.com/NCEAS/scalable-computing-examples) repository, open the Jupyter notebook in the `group-project` directory called `session-15.ipynb`. This workbook will serve as a skeleton for you to work in. It will load in all the libraries you need, including a few helper functions we wrote for the course, show an example for how to use the method on one file, and then lays out blocks for you and your group to fill in with code that will run that method in parallel. In your small groups, work together to write the solution, but everyone should aim to have a working solution on their own fork of the repository. In other words, everyone should type out the solution themselves as part of the group effort. Writing the code out yourself (even if others are contributing to the content) is a great way to get "mileage" as you develop these skills. Only one person in the group should run the parallel code. -## Rasterizing GeoPackages +## Resampling rasters + +In this portion of the group project, we will further process the raster files by resampling them to lower zoom levels. This means we will take the average of clusters of pixels, and that value will be assigned to the single pixel that encompasses the entire area that is represented by the original cluster of pixels. Here is an example of creating a zoom level 10 pixel from 4 zoom level 11 pixels: + +![Drawing of downsampling a raster, created by Robyn Thiessen-Bock.](../images/resampling.png) + +As we aggregate cells to produce rasters at lower zoom levels, the _cell size increases_, but the _extent of the raster remains the same_. This is called "downsampling". + +After we do this for each zoom level, you can imagine they are organzied as a pyramid, with the lowest zoom levels (and lowest resolution rasters) at the top, like so: + +![A pyramid of rasters at different resolutions (zoom levels). Souce: [maptiler](https://www.maptiler.com/google-maps-coordinates-tile-bounds-projection/#3/15.00/50.00)](../images/raster_pyramid.png) + +The highest zoom level has the most rasters. As we create the lower zoom levels, we aggregate the rasters _only_ from the zoom level directly above. As the zoom level decreases, our computation is faster. + +Check out the [Permafrost Discovery Gateway Imagery Viewer]() and zoom in and out. You can see for yourself how the resolution increases and the extent that you are trying to view decreases. -In the last lession, we staged the input files into GeoPackages. Now we will import those `.gpkg` files and write each as a raster (`.tif`). The resulting rasters will have 2 bands, one for each statistic we calculate based on the vector geometries in each pixel. +![Ice wedge polygons at high resolution, visible when zoomed-in to the Permafrost Discovery Gateway Imagery Viewer.](../images/zoom_in_IWP.png) -We need to create the highest zoom level rasters before we create the lower zoom levels in the next lesson. This brings us one step closer to visualizing the ice wedge polygons on a basemap, with the ability to zoom in and out! +The higher resolution tiles are the "child" tiles, and the lower resolution tiles are the "parent" tiles. ### Packages, Libraries, and Modules - [os 3.11.2](https://docs.python.org/3/library/os.html) - [parsl 2023.3.20](https://parsl.readthedocs.io/en/stable/) - [pdgstaging](https://github.com/PermafrostDiscoveryGateway/viz-staging) - - package developed by Permafrost Discovery Gateway software developer and designer Robyn Thiessen-Bock + - developed by Permafrost Discovery Gateway software developer and designer Robyn Thiessen-Bock - [pdgraster](https://github.com/PermafrostDiscoveryGateway/viz-raster/tree/main) - - package developed by Permafrost Discovery Gateway software developer and designer Robyn Thiessen-Bock + - developed by Permafrost Discovery Gateway software developer and designer Robyn Thiessen-Bock - [geopandas0.11](https://geopandas.org/en/stable/docs.html) - [random](https://docs.python.org/3/library/random.html) -- [matplotlib 3.5](https://matplotlib.org/3.5.3/api/_as_gen/matplotlib.pyplot.html) \ No newline at end of file +- [matplotlib 3.5](https://matplotlib.org/3.5.3/api/_as_gen/matplotlib.pyplot.html) +- [ipyleaflet 0.17](https://ipyleaflet.readthedocs.io/en/latest/) \ No newline at end of file diff --git a/sections/group-project-3.qmd b/sections/group-project-3.qmd deleted file mode 100644 index 53c9a43..0000000 --- a/sections/group-project-3.qmd +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: 'Group Project: Visualization' ---- - - -## Setup - -In your fork of the [`scalable-computing-examples`](https://github.com/NCEAS/scalable-computing-examples) repository, open the Jupyter notebook in the `group-project` directory called `session-15.ipynb`. This workbook will serve as a skeleton for you to work in. It will load in all the libraries you need, including a few helper functions we wrote for the course, show an example for how to use the method on one file, and then lays out blocks for you and your group to fill in with code that will run that method in parallel. - -In your small groups, work together to write the solution, but everyone should aim to have a working solution on their own fork of the repository. In other words, everyone should type out the solution themselves as part of the group effort. Writing the code out yourself (even if others are contributing to the content) is a great way to get "mileage" as you develop these skills. - -Only one person in the group should run the parallel code. - -## Resampling rasters - -In this portion of the group project, we will further process the raster files by resampling them to lower zoom levels. This means we will take the average of clusters of pixels, and that value will be assigned to the single pixel that encompasses the entire area that is represented by the original cluster of pixels. Here is an example of creating a zoom level 10 pixel from 4 zoom level 11 pixels: - -![Drawing of downsampling a raster, created by Robyn Thiessen-Bock.](../images/resampling.png) - -As we aggregate cells to produce rasters at lower zoom levels, the _cell size increases_, but the _extent of the raster remains the same_. This is called "downsampling". - -After we do this for each zoom level, you can imagine they are organzied as a pyramid, with the lowest zoom levels (and lowest resolution rasters) at the top, like so: - -![A pyramid of rasters at different resolutions (zoom levels). Souce: [maptiler](https://www.maptiler.com/google-maps-coordinates-tile-bounds-projection/#3/15.00/50.00)](../images/raster_pyramid.png) - -The highest zoom level has the most rasters. As we create the lower zoom levels, we aggregate the rasters _only_ from the zoom level directly above. As the zoom level decreases, our computation is faster. - -Check out the [Permafrost Discovery Gateway Imagery Viewer]() and zoom in and out. You can see for yourself how the resolution increases and the extent that you are trying to view decreases. - -![Ice wedge polygons at high resolution, visible when zoomed-in to the Permafrost Discovery Gateway Imagery Viewer.](../images/zoom_in_IWP.png) - -The higher resolution tiles are the "child" tiles, and the lower resolution tiles are the "parent" tiles. - -### Packages, Libraries, and Modules - -- [os 3.11.2](https://docs.python.org/3/library/os.html) -- [parsl 2023.3.20](https://parsl.readthedocs.io/en/stable/) -- [pdgstaging](https://github.com/PermafrostDiscoveryGateway/viz-staging) - - developed by Permafrost Discovery Gateway software developer and designer Robyn Thiessen-Bock -- [pdgraster](https://github.com/PermafrostDiscoveryGateway/viz-raster/tree/main) - - developed by Permafrost Discovery Gateway software developer and designer Robyn Thiessen-Bock -- [geopandas0.11](https://geopandas.org/en/stable/docs.html) -- [random](https://docs.python.org/3/library/random.html) -- [matplotlib 3.5](https://matplotlib.org/3.5.3/api/_as_gen/matplotlib.pyplot.html) -- [ipyleaflet 0.17](https://ipyleaflet.readthedocs.io/en/latest/) \ No newline at end of file