02.1-Datasets.geospatial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">Copyright 2021, by the California Institute of Technology. ALL RIGHTS RESERVED. United States Government sponsorship acknowledged. Any commercial use must be negotiated with the Office of Technology Transfer at the California Institute of Technology.</font>\n",
    "    \n",
    "<font size=\"1\">This software may be subject to U.S. export control laws and regulations. By accepting this document, the user agrees to comply with all applicable U.S. export laws and regulations. User has the responsibility to obtain export licenses, or other export authority as may be required, before exporting such information to foreign countries or providing access to foreign persons.<font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datasets - geospatial and time filtered queries\n",
    "This notebook will use the Pele API to issue a geospatial query to find available datasets and download one from S3 based on the metadata returned by the query.\n",
    "\n",
    "#### Kernel: isce, plant or mintpy\n",
    "\n",
    "## Setup\n",
    "\n",
    "This notebook assumes you've already gone through the first notebook and registered a user and password, as well sa populated your .netrc file. Let's go ahead and set things up so that we can utilize the Pele client library to query our datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import requests, json, getpass\n",
    "from requests.auth import HTTPBasicAuth\n",
    "import urllib3\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "urllib3.disable_warnings()\n",
    "\n",
    "# this block makes sure the directory set-up/change is only done once and relative to the notebook's directory\n",
    "try:\n",
    "    start_dir\n",
    "except NameError:\n",
    "    start_dir = os.getcwd()\n",
    "    !mkdir -p ./notebook_output/02-Datasets-geospatial\n",
    "    os.chdir('notebook_output/02-Datasets-geospatial')\n",
    "    \n",
    "# set the base url to interact with the goddess, Pele\n",
    "base_url = input(\"Enter Pele REST API base url (e.g. https://<mozart_ip>/pele/api/v0.1) then press <Enter>: \")\n",
    "print(\"Using base url {}.\".format(base_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's validate that we can interact with Pele:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pele_client.client import PeleRequests\n",
    "\n",
    "# instantiate PeleRequests object\n",
    "pr = PeleRequests(base_url, verify=False)\n",
    "\n",
    "# now use like requests module (`request()`, `get()`, `head()`, `post()`, `put()`, `delete()`, `patch()`)\n",
    "r = pr.get(base_url + '/test/echo', params={'echo_str': 'hello world'})\n",
    "\n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "print(json.dumps(r.json(), indent=2))\n",
    "assert r.status_code == 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying the dataset types\n",
    "\n",
    "Let's see what datasets we have. Note that the page_size parameter is passed to increase the size of the result set beyond the default of 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get datasets\n",
    "r = pr.get(base_url + '/pele/datasets', params={'page_size' : 50})\n",
    "\n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "res = r.json()\n",
    "print(json.dumps(res, indent=2))\n",
    "assert r.status_code == 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Here we list the available L2_L_GSLC datasets, just for reference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# query for all dataset IDs of the `L2_L_GSLC` dataset\n",
    "r = pr.get(base_url + '/pele/dataset/L2_L_GSLC/dataset_ids')\n",
    "           \n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "res = r.json()\n",
    "print(json.dumps(res, indent=2))\n",
    "assert r.status_code == 200\n",
    "\n",
    "datasets = res['dataset_ids']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Refine the search\n",
    "\n",
    "FYI, the target datasets cover a lat/lon polygon of:\n",
    "\n",
    "```[[[-118,34],[-117,34],[-117,35.5],[-118,35.5],[-118,34]]]```\n",
    "\n",
    "The search polygon defined below partially overlaps this. \n",
    "\n",
    "We will also add start and end time parameters to further refine the results. These 'time' parameters can have any of the following formats:\n",
    "\n",
    "```YYYY-MM-DDTHH:MI:SSZ```\n",
    "\n",
    "```YYYY-MM-DD``` (the time portion is considered all zero, i.e. midnight)\n",
    "\n",
    "```nnnnnnnnnnnnn``` (milliseconds since epoch)\n",
    "\n",
    "For start_time, the dataset start_time must be greater than or equal to the search value. For end_time, the dataset end_time must be less than the search value.\n",
    "\n",
    "*polygon*, *start_time* and *end_time* are all optional."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_poly=[[[-118,34],[-117,34],[-117,35.5],[-118,35.5],[-118,34]]]\n",
    "search_start_time = '2008-02-18'\n",
    "search_end_time = '2008-02-18T23:59:59Z'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submit the query\n",
    "\n",
    "The polygon and time parameters are passed in the json of an HTTP post - we should end up with one result dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# query for the dataset ID of the qualifying `L2_L_GSLC` dataset(s)\n",
    "r = pr.post(base_url + '/pele/dataset/L2_L_GSLC/dataset_ids', json = { 'polygon' : search_poly, 'start_time' : search_start_time, 'end_time' : search_end_time })\n",
    "           \n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "res = r.json()\n",
    "print(json.dumps(res, indent=2))\n",
    "assert r.status_code == 200\n",
    "\n",
    "datasets = res['dataset_ids']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the metadata for the datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = pr.get(base_url + '/pele/dataset/{}'.format(datasets[0]))\n",
    "\n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "res = r.json()\n",
    "print(json.dumps(res, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This pulls the granule's entire JSON metadata. Let's focus on the URLs so that we can download the granule:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pull the urls\n",
    "urls = res['result']['urls']\n",
    "print(\"urls: {}\".format(urls))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to utilize the S3 URL so that we can utilize the S3 API for faster downloads:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_url = None\n",
    "for i in urls:\n",
    "    if i.startswith('s3://'): s3_url = i\n",
    "assert s3_url is not None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now let's download that dataset from S3, but before we can we need to populate the .aws/credentials file with the access key information. \n",
    "\n",
    "Use a terminal to execute aws-login:\n",
    "\n",
    "    aws-login -pub -p default -r us-west-2\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# get the S3 url format that awscli requires\n",
    "url = 's3://{}'.format(urlparse(s3_url).path[1:])\n",
    "print(url)\n",
    "local_dir = os.path.basename(url)\n",
    "print (local_dir)\n",
    "!aws s3 sync {url} {local_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd\n",
    "!ls -al {local_dir}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">This notebook is compatible with NISAR Jupyter Server Stack v1.7.1 and above</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "isce",
   "language": "python",
   "name": "isce"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}