L1_RSLC_to_L2_GSLC.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">Copyright 2021, by the California Institute of Technology. ALL RIGHTS RESERVED. United States Government sponsorship acknowledged. Any commercial use must be negotiated with the Office of Technology Transfer at the California Institute of Technology.</font>\n",
    "    \n",
    "<font size=\"1\">This software may be subject to U.S. export control laws and regulations. By accepting this document, the user agrees to comply with all applicable U.S. export laws and regulations. User has the responsibility to obtain export licenses, or other export authority as may be required, before exporting such information to foreign countries or providing access to foreign persons.<font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# The NISAR GSLC processor\n",
    "#### Notebook kernel: plant\n",
    "\n",
    "This notebook submits a query to identify an L1_L_RSLC product to be used to invoke the NISAR GSLC processor to produce an L2_L_GSLC."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup configuration for interaction with on-demand NISAR SDS\n",
    "In this section, we setup the necessary python libraries and API endpoints needed to interact with the **on-demand NISAR SDS**.\n",
    "\n",
    "The `otello` python library provides high-level access to operations on the on-demand SDS. In particular for this demo, it provides us with access to and information about the job types registered on the SDS along with the capability to submit jobs, check for job statuses, and to query for products generated by the job.\n",
    "\n",
    "The `pele` python library provides high-level read-only access to the dataset catalog managed by the on-demand SDS. In particular for this demo, it provides us with the capability to list the dataset types contained in the catalog, query for individual dataset granules of a particular dataset type, and to return metadata pertaining to an individual dataset granule."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# this block makes sure the directory set-up/change is only done once and relative to the notebook's directory\n",
    "try:\n",
    "    start_dir\n",
    "except NameError:\n",
    "    start_dir = os.getcwd()\n",
    "    !mkdir -p ./notebook_output/05a-Tunable_Parameters_GSLC\n",
    "    os.chdir('notebook_output/05a-Tunable_Parameters_GSLC')\n",
    "    \n",
    "import json\n",
    "import time\n",
    "import re\n",
    "import shutil\n",
    "from glob import glob\n",
    "from pprint import pprint\n",
    "from datetime import datetime\n",
    "from pip._internal.commands.show import search_packages_info\n",
    "import logging\n",
    "import matplotlib\n",
    "logging.getLogger('matplotlib').setLevel(logging.WARNING)\n",
    "\n",
    "import otello\n",
    "from pele_client.client import PeleRequests\n",
    "\n",
    "import plant\n",
    "%matplotlib inline\n",
    "\n",
    "print(f\"PLAnT version: {plant.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure for interaction with the on-demand NISAR SDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the mozart IP\n",
    "mozart_ip = \"100.64.122.181\"\n",
    "#mozart_ip = input(\"Enter IP address of your mozart instance then press <Enter>: \")\n",
    "print(f\"Using mozart IP address {mozart_ip}.\")\n",
    "\n",
    "# instantiate otello and get mozart object to interact with SDS - you will be prompted for input values\n",
    "# which will be defaulted on subsequent calls.\n",
    "otello.client.initialize()\n",
    "m = otello.mozart.Mozart()\n",
    "\n",
    "# set pele URL\n",
    "pele_url = f\"https://{mozart_ip}/pele/api/v0.1\"\n",
    "print(f\"Using pele URL {pele_url}.\")\n",
    "\n",
    "# instantiate PeleRequests object\n",
    "pr = PeleRequests(pele_url, verify=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting a job for the NISAR GSLC processor\n",
    "### Next let's get more information on the NISAR GSLC processor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(m.get_job_types()[\"job-SCIFLO_GSLC:jdyoung-test\"])\n",
    "\n",
    "# get GSLC job type and initialize\n",
    "job_type = \"job-SCIFLO_GSLC:jdyoung-test\"\n",
    "gslc = m.get_job_types()[job_type]\n",
    "gslc.initialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query NISAR SDS for an RSLC dataset\n",
    "The input dataset for the NISAR GSLC processor is the NISAR RSLC product. Here we submit a geospatial query to the on-demand NISAR SDS for an input dataset. For an additional example on pele queries, see the _02-Datasets_ and _02-Datasets-geospatial_ notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# query for all dataset IDs of the `L1_L_RSLC` dataset\n",
    "r = pr.get(pele_url + '/pele/dataset/L1_L_RSLC/dataset_ids')\n",
    "           \n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "res = r.json()\n",
    "print(json.dumps(res, indent=2))\n",
    "assert r.status_code == 200\n",
    "\n",
    "datasets = res['dataset_ids']\n",
    "\n",
    "search_poly=[[[-118,34],[-118,35],[-117,35],[-117,34],[-118,34]]]\n",
    "search_start_time = '2008-02-18'\n",
    "search_end_time = '2008-02-18T23:59:59Z'\n",
    "\n",
    "# query for the dataset ID of the qualifying `L1_L_RSLC` dataset(s)\n",
    "r = pr.post(pele_url + '/pele/dataset/L1_L_RSLC/dataset_ids', json = { 'polygon' : search_poly, 'start_time' : search_start_time, 'end_time' : search_end_time })\n",
    "           \n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "res = r.json()\n",
    "print(json.dumps(res, indent=2))\n",
    "assert r.status_code == 200\n",
    "\n",
    "datasets = res['dataset_ids']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the first identified RSLC dataset from the above\n",
    "r = pr.get(f\"{pele_url}/pele/dataset/{datasets[0]}\")\n",
    "rslc_dataset = r.json()[\"result\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare input for the on-demand GSLC processor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# set dataset params\n",
    "gslc.set_input_dataset(rslc_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit the GSLC job to run in the on-demand NISAR SDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submit job\n",
    "job1 = gslc.submit_job(tag=\"gslc-test1-default-params\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's wait for the job to complete generation of the standard GSLC product\n",
    "\n",
    "Processing will take a few minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wait for job\n",
    "job1.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get information about the generated GSLC product from the job\n",
    "#### The generated GSLC product is stored in the cloud next to the on-demand NISAR SDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "std_prod = job1.get_generated_products()\n",
    "print(json.dumps(std_prod, indent=2, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the generated standard GSLC product from the cloud into this notebook\n",
    "#### Here we use the AWS CLI to download the generated dataset. Be sure that your AWS credentials are valid - refresh with aws-login in a terminal window if necessary.\n",
    "\n",
    "Run the following cell if the job above succeeded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "std_prod_url = re.sub(r'^s3://.+?/(.+)$', r's3://\\1', std_prod[0][\"urls\"][-1]) # get s3 url\n",
    "std_local_dir = os.path.basename(std_prod_url)\n",
    "\n",
    "if os.path.isdir(std_local_dir): shutil.rmtree(std_local_dir)\n",
    "print(std_local_dir)\n",
    "!aws s3 sync $std_prod_url $std_local_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Use PLAnT to visualize the output GSLC product\n",
    "Here we list the contents of the generated dataset. The actual GSLC product is the HD5 file (*.h5)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -l $std_local_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Identify the GSLC product file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "gslc_file_names = glob.glob(os.path.join(std_local_dir, \"*.h5\"))\n",
    "if(len(gslc_file_names) > 0):\n",
    "    gslc_file_name = gslc_file_names[0]\n",
    "    print(gslc_file_name)\n",
    "else:\n",
    "    print(\"No GSLC H5 file is present in {}\".format(std_local_dir))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read in the GSLC product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py\n",
    "\n",
    "gslc_file = h5py.File(gslc_file_name, 'r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot the HH polarization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read in HH polarization from the GSLC\n",
    "hh_polarization = gslc_file['science']['LSAR']['GSLC']['grids']['frequencyA']['HH']\n",
    "\n",
    "# options\n",
    "options = {}\n",
    "# options['mute'] = True\n",
    "# options['force'] = True\n",
    "\n",
    "# display options\n",
    "display_options = options.copy()\n",
    "\n",
    "# display composition of HH polaraization.\n",
    "plant.display(hh_polarization, title=\"test\", **display_options)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">This notebook is compatible with NISAR Jupyter Server Stack v1.7.1 and above</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "plant [conda env:.local-plant]",
   "language": "python",
   "name": "conda-env-.local-plant-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}