ICESat-2_DAAC_DataAccess_Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing ICESat-2 Data\n",
    "### Data Query and Basic Download Example Notebook\n",
    "This notebook illustrates the use of icepyx for ICESat-2 data access and download from the NASA NSIDC DAAC (NASA National Snow and Ice Data Center Distributed Active Archive Center).\n",
    "A complimentary notebook demonstrates in greater detail the subsetting options available when ordering data.\n",
    "\n",
    "#### Credits\n",
    "* notebook by: Jessica Scheick\n",
    "* source material: [NSIDC Data Access Notebook](https://github.com/ICESAT-2HackWeek/ICESat2_hackweek_tutorials/tree/master/03_NSIDCDataAccess_Steiker) by Amy Steiker and Bruce Wallin\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import packages, including icepyx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import icepyx as ipx\n",
    "import os\n",
    "import shutil\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick-Start\n",
    "\n",
    "The entire process of getting ICESat-2 data (from query to download) can ultimately be accomplished in three minimal lines of code:\n",
    "\n",
    "`region_a = ipx.Query(short_name, spatial_extent, date_range)`\n",
    "\n",
    "`region_a.earthdata_login(earthdata_uid, email)`\n",
    "\n",
    "`region_a.download_granules(path)`\n",
    "\n",
    "where the function inputs are described in more detail below.\n",
    "\n",
    "**The rest of this notebook explains the required inputs used above, optional inputs not available in the minimal example, and the other data search and visualization tools built in to icepyx that make it easier for the user to find, explore, and download ICESat-2 data programmatically from NSIDC.** The detailed steps outlined and the methods showcased below are meant to give the user more control over the data they find and download (including options to order/download only the relevant portions of a data granule), some of which are called using default values behind the scenes if the user simply skips to the `download_granules` step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an ICESat-2 data object with the desired search parameters\n",
    "\n",
    "There are three required inputs:\n",
    "- `short_name` = the dataset of interest, known as its \"short name\".\n",
    "See https://nsidc.org/data/icesat-2/data-sets for a list of the available datasets.\n",
    "- `spatial extent` = a region of interest to search within. This can be entered as a bounding box, polygon vertex coordinate pairs, or a polygon geospatial file (currently shp, kml, and gpkg are supported).\n",
    "    - bounding box: Given in decimal degrees for the lower left longitude, lower left latitude, upper right longitude, and upper right latitude\n",
    "    - polygon vertices: Given as longitude, latitude coordinate pairs of decimal degrees with the last entry a repeat of the first.\n",
    "    - polygon file: A string containing the full file path and name.\n",
    "- `date_range` = the date range for which you would like to search for results. Must be formatted as a set of 'YYYY-MM-DD' strings.\n",
    "\n",
    "Below are examples of each type of spatial extent input. Please choose and run only one of the next three cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#bounding box\n",
    "short_name = 'ATL06'\n",
    "spatial_extent = [-55, 68, -48, 71]\n",
    "date_range = ['2019-02-20','2019-02-28']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#polygon vertices (here equivalent to the bounding box, above)\n",
    "short_name = 'ATL06'\n",
    "spatial_extent = [(-55, 68), (-55, 71), (-48, 71), (-48, 68), (-55, 68)]\n",
    "date_range = ['2019-02-20','2019-02-28']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#polygon geospatial file (metadata match but no subset match)\n",
    "# short_name = 'ATL06'\n",
    "# spatial_extent = './supporting_files/data-access_PineIsland/glims_polygons.kml'\n",
    "# date_range = ['2019-02-22','2019-02-28']\n",
    "\n",
    "#polygon geospatial file (subset and metadata match)\n",
    "short_name = 'ATL06'\n",
    "spatial_extent = './supporting_files/data-access_PineIsland/glims_polygons.shp'\n",
    "date_range = ['2019-10-01','2019-10-05']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the data object using our inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a = ipx.Query(short_name, spatial_extent, date_range)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Formatted parameters and function calls allow us to see the the properties of the data object we have created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(region_a.dataset)\n",
    "print(region_a.dates)\n",
    "print(region_a.start_time)\n",
    "print(region_a.end_time)\n",
    "print(region_a.dataset_version)\n",
    "print(region_a.cycles)\n",
    "print(region_a.tracks)\n",
    "# print(region_a.spatial_extent)\n",
    "region_a.visualize_spatial_extent()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are also several optional inputs to allow the user finer control over their search.\n",
    "- `start_time` = start time to search for data on the start date. If no input is given, this defaults to 00:00:00.\n",
    "- `end_time` = end time for the end date of the temporal search parameter. If no input is given, this defaults to 23:59:59. Times must be input as 'HH:mm:ss' strings.\n",
    "- `version` = What version of the dataset to use, input as a numerical string. If no input is given, this value defaults to the most recent version of the dataset specified in `short_name`.\n",
    "\n",
    "*NOTE* Version 001 is used as an example in the below cell. However, using it will cause 'no results' errors in granule ordering for some search parameters. These issues have been resolved in later versions of the datasets, so it is best to use the most recent version where possible. Thus, you will need to change the version associated with `region_a` and rerun the next cell for the rest of this notebook to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a = ipx.Query(short_name, spatial_extent, date_range, \\\n",
    "   start_time='03:30:00', end_time='21:30:00', version='002')\n",
    "\n",
    "print(region_a.dataset)\n",
    "print(region_a.dates)\n",
    "print(region_a.start_time)\n",
    "print(region_a.end_time)\n",
    "print(region_a.dataset_version)\n",
    "print(region_a.cycles)\n",
    "print(region_a.tracks)\n",
    "# print(region_a.spatial_extent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, there are also two optional inputs to allow the user to search based on orbital parameters. Note that your search will return a no-data error if your cycles or tracks are not withint your spatial extent.\n",
    "- `cycles` = Which orbital cycle of the dataset to use, input as a numerical string or a list of strings. If no input is given, this value defaults to all available cycles within the search parameters.  An orbital cycle refers to the 91-day repeat period of the ICESat-2 orbit.\n",
    "- `tracks` = Which [Reference Ground Track (RGT)](https://icesat-2.gsfc.nasa.gov/science/specs) of the dataset to use, input as a numerical string or a list of strings. If no input is given, this value defaults to all available RGTs within the spatial and temporal search parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a = ipx.Query(short_name, spatial_extent, date_range, \\\n",
    "   cycles='02', tracks=['0849','0902'])\n",
    "\n",
    "print(region_a.dataset)\n",
    "print(region_a.dates)\n",
    "print(region_a.start_time)\n",
    "print(region_a.end_time)\n",
    "print(region_a.dataset_version)\n",
    "print(region_a.cycles)\n",
    "print(region_a.tracks)\n",
    "print(region_a.orbit_number)\n",
    "# print(region_a.spatial_extent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can also just create the data object without creating named variables first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# region_a = ipx.Query('ATL06',[-55, 68, -48, 71],['2019-02-01','2019-02-28'], \n",
    "#                            start_time='00:00:00', end_time='23:59:59', version='002')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Built in methods allow us to get more information about our dataset\n",
    "In addition to viewing the stored object information shown above (e.g. dataset, start and end date and time, version, etc.), we can also request summary information about the dataset itself or confirm that we have manually specified the latest version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.dataset_summary_info()\n",
    "print(region_a.latest_version())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the summary does not provide all of the information you are looking for, or you would like to see information for previous versions of the dataset, all available metadata for the collection dataset is available in a readable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "region_a.dataset_all_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Querying a dataset\n",
    "In order to search the dataset collection for available data granules, we need to build our search parameters. This is done automatically behind the scenes when you run `region_a.avail_granules()`, but you can also build and view them by calling `region_a.CMRparams`. These are formatted as a dictionary of key:value pairs according to the CMR documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#build and view the parameters that will be submitted in our query\n",
    "region_a.CMRparams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that our parameter dictionary is constructed, we can search the CMR database for the available granules. Granules returned by the CMR metadata search are automatically stored within the data object. The search completed at this level relies completely on the granules' metadata. As a result, some (and in rare cases all) of the granules returned may not actually contain data in your specified region, particularly if the region is small or located near the boundaries of a given granule. If this is the case, the subsetter will not return any data when you actually place the order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#search for available granules and provide basic summary info about them\n",
    "region_a.avail_granules()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get a list of granule IDs for the available granules\n",
    "region_a.avail_granules(ids=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#print detailed information about the returned search results\n",
    "region_a.granules.avail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading the found granules\n",
    "In order to download any data from NSIDC, we must first authenticate ourselves using a valid Earthdata login. This will create a valid token to interface with the DAAC as well as start an active logged-in session to enable data download. Once you have successfully logged in for a given query instance, the token and session will be passed behind the scenes as needed for you to order and download data. Passwords are entered but not shown or stored in plain text by the system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "earthdata_uid = 'icepyx_devteam'\n",
    "email = 'icepyx.dev@gmail.com'\n",
    "region_a.earthdata_login(earthdata_uid, email)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have generated our session, we must build the required configuration parameters needed to actually download data. These will tell the system how we want to download the data. As with the CMR search parameters, these will be built automatically when you run `region_a.order_granules()`, but you can also create and view them with `region_a.reqparams`. The default parameters, given below, should work for most users.\n",
    "- `page_size` = 10. This is the number of granules we will request per order.\n",
    "- `page_num` = 1. Determine the number of pages based on page size and the number of granules available. If no page_num is specified, this calculation is done automatically to set page_num, which then provides the number of individual orders we will request given the number of granules.\n",
    "- `request_mode` = 'async'\n",
    "- `agent` = 'NO'\n",
    "- `include_meta` = 'Y'\n",
    "\n",
    "#### More details about the configuration parameters\n",
    "`request_mode` is \"synchronous\" by default, meaning that the request relies on a direct, continous connection between you and the API endpoint. Outputs are directly downloaded, or \"streamed\", to your working directory. For this tutorial, we will set the request mode to asynchronous, which will allow concurrent requests to be queued and processed without the need for a continuous connection.\n",
    "\n",
    "**Use the streaming `request_mode` with caution: While it can be beneficial to stream outputs directly to your local directory, note that timeout errors can result depending on the size of the request, and your request will not be queued in the system if NSIDC is experiencing high request volume. For best performance, NSIDC recommends setting `page_size=1` to download individual outputs, which will eliminate extra time needed to zip outputs and will ensure faster processing times per request.**\n",
    "\n",
    "Recall that we queried the total number and volume of granules prior to applying customization services. `page_size` and `page_num` can be used to adjust the number of granules per request up to a limit of 2000 granules for asynchronous, and 100 granules for synchronous (streaming). For now, let's select 9 granules to be processed in each zipped request. For ATL06, the granule size can exceed 100 MB so we want to choose a granule count that provides us with a reasonable zipped download size. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(region_a.reqparams)\n",
    "# region_a.reqparams['page_size'] = 9\n",
    "# print(region_a.reqparams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Additional Parameters: Subsetting\n",
    "\n",
    "In addition to the required parameters (CMRparams and reqparams) that are submitted with our order, for ICESat-2 datasets we can also submit subsetting parameters to NSIDC. This utilizes the NSIDC's built in subsetter to extract only the data you are interested (spatially, temporally, variables of interest, etc.). The advantages of using the NSIDC's subsetter include:\n",
    "* easily reproducible downloads, particularly when coupled with and icepyx data object\n",
    "* smaller file size, meaning faster downloads, less storage required, and no need to subset the data on your own\n",
    "* still easy to go back and order more data/variables with the same or similar search parameters\n",
    "* no extraneous data means you can move directly to analysis and easily navigate your dataset\n",
    "\n",
    "Certain subset parameters are specified by default unless `subset=False` is included as an input to `order_granules()`. A separate, companion notebook tutorial covers subsetting in more detail, including how to get a list of subsetting options, how to build your list of subsetting parameters, and how to generate a list of desired variables (most datasets have more than 200 variable fields!), including using pre-built default lists (these lists are still in progress and we welcome contributions!).\n",
    "\n",
    "As for the CMR and required parameters, default subset parameters can be built and viewed using `subsetparams`. Where an input spatial file is used, rather than a bounding box or manually entered polygon, the spatial file will be used for subsetting (unless subset is set to False) but not show up in the `subsetparams` dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.subsetparams()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a._geom_filepath"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Place the order\n",
    "Then, we can send the order to NSIDC using the order_granules function. Information about the granules ordered and their status will be printed automatically as well as emailed to the address provided. Additional information on the order, including request URLs, can be viewed by setting the optional keyword input 'verbose' to True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.order_granules()\n",
    "# region_a.order_granules(verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#view a short list of order IDs\n",
    "region_a.granules.orderIDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download the order\n",
    "Finally, we can download our order to a specified directory (which needs to have a full path but doesn't have to point to an existing directory) and the download status will be printed as the program runs. Additional information is again available by using the optional boolean keyword `verbose`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = './download'\n",
    "region_a.download_granules(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}