deploy_boulia/01_Define_AoI_Grid.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "41295a98",
   "metadata": {},
   "source": [
    "# Split Chosen AoI Shapes into Grid Patches\n",
    "\n",
    "The FloodMapper system requires area-of-interest (AoI) shapes that have been resampled by the fixed grid of processing 'patches' defined in the database. This notebook demonstrates how transform arbitrary AoI shapes into a GeoJSON file suitable for use with FloodMapper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef00587a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Necessary Python modules\n",
    "import sys\n",
    "import os\n",
    "os.environ['USE_PYGEOS'] = '0'\n",
    "\n",
    "import pandas as pd\n",
    "import geopandas as gpd\n",
    "import folium\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "from db_utils import DB\n",
    "from ml4floods.data import utils\n",
    "\n",
    "# Set bucket will not be requester pays\n",
    "utils.REQUESTER_PAYS_DEFAULT = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe28b382",
   "metadata": {},
   "source": [
    "## Load environment and project details\n",
    "\n",
    "As with the other notebooks, we load credentials and project details from a hidden ```.env``` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f0e02bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load environment variables (including path to credentials) from '.env' file\n",
    "env_file_path = \"../.env\" \n",
    "\n",
    "# Uncomment for alternative version for Windows (r\"\" indicates raw string)\n",
    "#env_file_path = r\"C:/Users/User/floodmapper/.env\"\n",
    "\n",
    "assert load_dotenv(dotenv_path=env_file_path) == True, \"[ERR] Failed to load environment!\"\n",
    "assert \"GOOGLE_APPLICATION_CREDENTIALS\" in os.environ, \"[ERR] Missing $GOOGLE_APPLICATION_CREDENTIAL!\"\n",
    "assert \"GS_USER_PROJECT\" in os.environ, \"[ERR] Missing $GS_USER_PROJECT!\"\n",
    "key_file_path = os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"]\n",
    "assert os.path.exists(key_file_path), f\"[ERR] Google credential key file does not exist: \\n{key_file_path} \"\n",
    "assert \"ML4FLOODS_BASE_DIR\" in os.environ, \"[ERR] Missing $ML4FLOODS_BASE_DIR!\"\n",
    "base_path = os.environ[\"ML4FLOODS_BASE_DIR\"]\n",
    "assert os.path.exists(base_path), f\"[ERR] Base path does not exist: \\n{base_path} \"\n",
    "bucket_name = os.environ[\"BUCKET_URI\"]\n",
    "assert bucket_name is not None and bucket_name != \"\", f\"Bucket name not defined {bucket_name}\"\n",
    "print(\"[INFO] Successfully loaded FloodMapper environment.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66eae6d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to the database (point to the .env file for credentials)\n",
    "db_conn = DB(env_file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f314c5e",
   "metadata": {},
   "source": [
    "## Select LGA shapes from the database\n",
    "\n",
    "Define a list of LGA names to map and execute the cells below to load their shapes from the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "518f1780",
   "metadata": {},
   "outputs": [],
   "source": [
    "# EDIT THIS CELL\n",
    "\n",
    "# Set the list of LGA names to query\n",
    "lga_names_lst = [\"Boulia\"]\n",
    "\n",
    "# Set a session name\n",
    "session_name = \"boulia_test\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04f55d47",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define and execute the query\n",
    "query = (f\"SELECT lga_name22, ST_AsText(geometry_col) \"\n",
    "         f\"FROM lgas_info \"\n",
    "         f\"WHERE lga_name22 IN %s;\")\n",
    "data = (tuple(lga_names_lst),)\n",
    "aoi_df = db_conn.run_query(query, data, fetch=True)\n",
    "print(f\"[INFO] Returned {len(aoi_df)} rows.\")\n",
    "\n",
    "# Format the results into a correct GeoDataFrame\n",
    "aoi_df['geometry'] = gpd.GeoSeries.from_wkt(aoi_df['st_astext'])\n",
    "aoi_df.drop(['st_astext'], axis=1, inplace = True)\n",
    "aoi_gdf = gpd.GeoDataFrame(aoi_df, geometry='geometry', crs=\"EPSG:4326\")\n",
    "aoi_gdf.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "509f2054",
   "metadata": {},
   "source": [
    "## Visualise the AoIs to be mapped\n",
    "\n",
    "The ```aoi_gdf``` dataframe contains a 'geometry' column with the boundaries of the LGAs, which we can visualise to confirm that they correspond to our previous selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58a3826a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the boundaries over OpenStreetMap layer\n",
    "aoi_gdf.explore()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "614cf7b5",
   "metadata": {},
   "source": [
    "## Load the pre-defined grid of processing patches\n",
    "\n",
    "Next we load the pre-defined grid of square processing patches from the FloodMapper database. To do this we define a Strutured Query Language (SQL - 'sequel') query to fetch the information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "702f342d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define and execute the query\n",
    "query = (f\"SELECT patch_name, ST_AsText(geometry) \"\n",
    "         f\"FROM grid_loc\")\n",
    "grid_df = db_conn.run_query(query, fetch= True)\n",
    "print(f\"[INFO] Returned {len(grid_df)} rows.\")\n",
    "\n",
    "# Format the results into a correct GeoDataFrame\n",
    "grid_df['geometry'] = gpd.GeoSeries.from_wkt(grid_df['st_astext'])\n",
    "grid_df.drop(['st_astext'], axis=1, inplace = True)\n",
    "grid_gdf = gpd.GeoDataFrame(grid_df, geometry='geometry', crs=\"EPSG:4326\")\n",
    "grid_gdf.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "290fdc42",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the grid as a sanity check\n",
    "grid_gdf.plot(facecolor=\"none\", lw=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c1a3551",
   "metadata": {},
   "source": [
    "## Use the AoIs to choose the grid patches to process\n",
    "\n",
    "The next step is to filter the grid patches to be processed by choosing only those that overlap with the selected AoIs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43f62caf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do a spatial join to choose grid patches that overlap with AoIs\n",
    "valid_names = grid_gdf.sjoin(aoi_gdf, how=\"inner\")[\"patch_name\"]\n",
    "grid_selected_gdf = grid_gdf[grid_gdf.patch_name.isin(valid_names)]\n",
    "grid_selected_gdf.drop_duplicates()\n",
    "grid_selected_gdf.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de575993",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Overplot the grid on the selected AOIs\n",
    "m = aoi_gdf.explore(style_kwds={\"fillOpacity\": 0.1}, name=\"Selected AoIs\")\n",
    "grid_selected_gdf.explore(m=m, style_kwds={\"fillOpacity\": 0.0}, color=\"red\", name=\"Grid\")\n",
    "folium.LayerControl(collapsed=False).add_to(m)\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07d81ca6",
   "metadata": {},
   "source": [
    "## Save the gridded AoIs to the GCP bucket\n",
    "\n",
    "The Floodmapper system (including the three main scripts) is driven by a saved version of these selected grid patches. The final step is to save the the selection to a GeoJSON format file on the GCP bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffa217a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output filename\n",
    "outfile = \"patches_to_map.geojson\"\n",
    "\n",
    "# Form the session path and output path on the GCP bucket\n",
    "session_path = os.path.join(bucket_name, \"0_DEV/1_Staging/operational\", \n",
    "                            session_name).replace(\"\\\\\", \"/\")\n",
    "outfile_path = os.path.join(session_path, outfile).replace(\"\\\\\", \"/\")\n",
    "print(f\"[INFO] Will save selected grid patches to:\\n\\t{outfile_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb604ded",
   "metadata": {},
   "source": [
    "**Note down this path - the FloodMapper scripts will need to read this file on the command line.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f00036c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ML4Floods provides a utility to write to the GCP bucket\n",
    "utils.write_geojson_to_gcp(outfile_path, grid_selected_gdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b07f285",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can also check if the file was written successfully\n",
    "fs = utils.get_filesystem(outfile_path)\n",
    "success = fs.exists(outfile_path)\n",
    "print(f\"[INFO] Was file written successfully to GCP? -> {success}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed60688",
   "metadata": {},
   "source": [
    "Finally, we can also write the selection to a GeoJSON file on our local disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ec3a1cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the selected grid patches to local disk\n",
    "folder_path = os.path.join(base_path, \"flood-activations\", session_name).replace(\"\\\\\", \"/\")\n",
    "outfile_path = os.path.join(folder_path, outfile).replace(\"\\\\\", \"/\")\n",
    "os.makedirs(folder_path, exist_ok=True)\n",
    "grid_selected_gdf.to_file(outfile_path, driver=\"GeoJSON\")\n",
    "print(f\"[INFO] Selected grid patches writen to:\\n\\t{outfile_path}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2c0da4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check that the file is on our local disk\n",
    "!ls -l {folder_path}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc17eaf8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (floodmapper)",
   "language": "python",
   "name": "floodmapper"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}