diff --git a/_toc.yml b/_toc.yml index 2ead8d90..baee52f4 100644 --- a/_toc.yml +++ b/_toc.yml @@ -13,6 +13,7 @@ parts: - file: notebooks/foundations/pyart-corrections - file: notebooks/foundations/pyart-gridding - file: notebooks/foundations/interactive-radar-visualization + - file: notebooks/foundations/interactive-radar-visualization-jmaradar-cfradial1 - caption: Example Workflows chapters: - file: notebooks/example-workflows/moore-oklahoma-tornado diff --git a/notebooks/foundations/interactive-radar-visualization-jmaradar-cfradial1.ipynb b/notebooks/foundations/interactive-radar-visualization-jmaradar-cfradial1.ipynb new file mode 100644 index 00000000..9ddab069 --- /dev/null +++ b/notebooks/foundations/interactive-radar-visualization-jmaradar-cfradial1.ipynb @@ -0,0 +1,497 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "19317554-925a-424b-86c5-43ce7b9df5a0", + "metadata": {}, + "source": [ + "\"Multipanel\n", + "# Interactive JMA Radar Visualization" + ] + }, + { + "cell_type": "markdown", + "id": "f43f16ad-81bd-4281-8b69-c330147451fe", + "metadata": {}, + "source": [ + "## Overview\n", + "Within this cookbook, we will detail how to create interactive plots of radar data!\n", + "\n", + "1. Reading data with Xradar\n", + "1. Creating your first interactive figure with Xradar + hvPlot\n", + "1. Combining your plots into a single dashboard\n", + "1. Filtering and Checking Data Quality\n", + "1. Create a Dashboard to Analyze ZDR Bias" + ] + }, + { + "cell_type": "markdown", + "id": "b85cbf6c-bfab-4f57-8735-63bf6a5014fc", + "metadata": {}, + "source": [ + "## Imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d0759d-7a1e-4071-bd23-0e2fd2df52bd", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "import xarray as xr\n", + "import xradar as xd\n", + "import glob\n", + "import pyart\n", + "import hvplot.xarray\n", + "import holoviews as hv\n", + "import panel as pn\n", + "\n", + "hv.extension(\"bokeh\")" + ] + }, + { + "cell_type": "markdown", + "id": "6bc80ec6-2aae-466f-ab6b-2f194f01adf8", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "\n", + "It is recommended that you are familiar with working with weather radar data, the core data structures, and the basics of reading in different radar datasets.\n", + "\n", + "| Concepts | Importance | Notes |\n", + "| --- | --- | --- |\n", + "| [Intro to Cartopy](https://foundations.projectpythia.org/core/cartopy/cartopy.html) | Necessary | |\n", + "| [Xradar User Guide: Plot a PPI](https://docs.openradarscience.org/projects/xradar/en/stable/notebooks/plot-ppi.html) | Necessary | |\n", + "| [Understanding of NetCDF](https://foundations.projectpythia.org/core/data-formats/netcdf-cf.html) | Helpful | Familiarity with metadata structure |\n", + "\n", + "- **Time to learn**: 30 minutes" + ] + }, + { + "cell_type": "markdown", + "id": "6d2218e8-5606-402b-a5eb-d456fabb13ee", + "metadata": {}, + "source": [ + "## Reading Data with Xradar\n", + "\n", + "While we have focused much of the content around the Python ARM Radar Toolkit (Py-ART), [Xradar](https://docs.openradarscience.org/projects/xradar/en/stable) is another helpful package we can use to work with this in xarray!\n", + "\n", + "Here, we use data from the Japanese weather radar, using sample data [from here](https://github.com/openradar/asean2024-pyrad-course/tree/main/data/JMA)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "27133c8d-ddd2-4557-acda-1b78dd8e3064", + "metadata": {}, + "outputs": [], + "source": [ + "!wget -nc https://github.com/openradar/asean2024-pyrad-course/raw/main/data/JMA/Z__C_RJTD_20230801200000_RDR_JMAGPV_RS47937_Gar0p250km0p35deg_PRref_N11_ANAL_cfrad.nc\n", + "!wget -nc https://github.com/openradar/asean2024-pyrad-course/raw/main/data/JMA/Z__C_RJTD_20230801200000_RDR_JMAGPV_RS47937_Gar0p250km0p35deg_PRzdr_N11_ANAL_cfrad.nc\n", + "!wget -nc https://github.com/openradar/asean2024-pyrad-course/raw/main/data/JMA/Z__C_RJTD_20230801200000_RDR_JMAGPV_RS47937_Gar0p250km0p35deg_PRkdp_N11_ANAL_cfrad.nc" + ] + }, + { + "cell_type": "markdown", + "id": "c1615d14-50a8-4786-adb8-7c6e80f0ce94", + "metadata": {}, + "source": [ + "Sample data will be saved locally, and then the variables of reflectivity and differential reflectivity are used in this notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1949ed05-f36c-4468-9ad4-af22c075fe23", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "files = glob.glob('./Z__C_RJTD_20230801200000_RDR_JMAGPV_RS47937_*')\n", + "files" + ] + }, + { + "cell_type": "markdown", + "id": "2db15915-afae-4b1d-ad53-99dff955459b", + "metadata": {}, + "source": [ + "### Read the data in xradar\n", + "We use [xradar](https://docs.openradarscience.org/projects/xradar/en/stable/), an open-source toolkit to read weather radar data and load into the Xarray data structure. The data format here is a CF/Radial1 file, so we use the [`open_cfradial1_datatree`](https://docs.openradarscience.org/projects/xradar/en/stable/notebooks/CfRadial1.html) reader." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "59a2ff5c-11e2-4b4a-b7d0-6325dbb72c2b", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "radar = xd.io.open_cfradial1_datatree(files[0]).xradar.georeference()\n", + "radar" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "393e3d2a-d2e0-43f2-b824-f1185016c5a7", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "radar_zdr = xd.io.open_cfradial1_datatree(files[1]).xradar.georeference()\n", + "radar_zdr" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "efd44959-159b-4053-a915-c1252a2456d1", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "radar_kdp = xd.io.open_cfradial1_datatree(files[2]).xradar.georeference()\n", + "radar_kdp" + ] + }, + { + "cell_type": "markdown", + "id": "857914ae-28ae-4aac-9937-66475132c090", + "metadata": {}, + "source": [ + "## Creating Your First Interactive Figure with Xradar and hvPlot\n", + "hvPlot is helpful tool when working with interactive visualizions! It is a tool built on top of several other packages, that we can use with Xarray.\n", + "\n", + "By default, this visualization plots azimuth along the y-axis and range along the y-axis. While this is desired in certain cases, we cannot gather much spatial information from this." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "999baa2f-73c0-493f-8928-c20d6fa701fd", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "ref = radar[\"sweep_0\"].DBZH.hvplot.quadmesh(cmap='pyart_ChaseSpectral',\n", + " title='Horizontal Reflectivity (dBZ)',\n", + " clim=(-20,60))\n", + "ref" + ] + }, + { + "cell_type": "markdown", + "id": "dd70b7e7-c6c7-45a6-b170-dd91783fc5da", + "metadata": {}, + "source": [ + "### Refining Our Plot - Recreating a Plan Position Indicator (PPI)\n", + "We instead would like to create a Plan Position Indicator (PPI) plot. Since we already georeferenced the dataset, we set x/y to be `x` and `y`, or the distance away from the radar, as well as tuning some additional parameters. We set `rasterize=True` to lazily load in the data, which renders the plot more quickly and increases resolution as we zoom in." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3d2502bb-d4dc-45e3-8cf9-61a53fc0dd2e", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "ref = radar[\"sweep_0\"].DBZH.hvplot.quadmesh(x='x',\n", + " y='y',\n", + " cmap='pyart_ChaseSpectral',\n", + " clabel='Horizontal Reflectivity (dBZ)',\n", + " title=f'Horizontal Reflectivity \\n {radar.attrs[\"site_name\"]} Radar',\n", + " clim=(-20, 60),\n", + " height=400,\n", + " rasterize=True,\n", + " width=500,)\n", + "\n", + "#ref" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e61f8c13-732e-4d7d-9649-04bd5a503037", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "zdr = radar_zdr[\"sweep_0\"].ZDR.hvplot.quadmesh(x='x',\n", + " y='y',\n", + " cmap='pyart_ChaseSpectral',\n", + " clabel='Differential Reflectivity (dB)',\n", + " title=f'Differential Reflectivity \\n {radar.attrs[\"site_name\"]} Radar',\n", + " clim=(-1, 6),\n", + " height=400,\n", + " rasterize=True,\n", + " width=500,)\n", + "#zdr" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cb765aab-0f61-4d2c-877e-375203c8cf55", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "kdp = radar_kdp[\"sweep_0\"].KDP.hvplot.quadmesh(x='x',\n", + " y='y',\n", + " cmap='pyart_ChaseSpectral',\n", + " clabel='Specific differential phase (degree/km)',\n", + " title=f'Specific differential phase \\n {radar.attrs[\"site_name\"]} Radar',\n", + " clim=(-1, 6),\n", + " height=400,\n", + " rasterize=True,\n", + " width=500,)\n", + "#kdp" + ] + }, + { + "cell_type": "markdown", + "id": "70cc3022-0ca5-4190-8af2-270e83ec9e9c", + "metadata": {}, + "source": [ + "### Combining your plots into a single dashboard\n", + "You can combine plots using the `+` syntax to add plots side-by-side, or `*` to add them to the same plot. For example, let's combine our reflectivity and velocity plot." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ca5a1bc4-9931-429d-bd81-50f4f4496deb", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "(ref + zdr + kdp).cols(3)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e916fa08-fd62-4c5a-9e12-2c9b46b1f74d", + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "id": "65594dca-31c6-4b80-b5d7-92c101e07c97", + "metadata": {}, + "source": [ + "## Filtering and Checking Data Quality\n", + "We can also filter our data - notice the low values in both differential reflectivity and reflectivity. We can mask these out using `Xarray`!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "adf08965-7498-440f-8626-c2a4bb5f6927", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# Subset our first sweep\n", + "ds0 = radar[\"sweep_0\"].to_dataset()\n", + "ds1 = radar_zdr[\"sweep_0\"].to_dataset()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d4bc93b9-09d0-41ba-9d20-3c6c90682c68", + "metadata": {}, + "outputs": [], + "source": [ + "ds = xr.merge([ds0, ds1])\n", + "ds" + ] + }, + { + "cell_type": "markdown", + "id": "e5805719-60be-4f12-8a81-d1aa1becfeb6", + "metadata": {}, + "source": [ + "### Determine Mask Thresholds\n", + "Let's determine our thresholds for filtering the data, using histograms! These are available through `hvPlot`, using the `.hvplot.hist()` extension." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e772bb99-0ce2-45be-a0fd-82642fd8b871", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "zdr_hist = ds.ZDR.hvplot.hist()\n", + "ref_hist = ds.DBZH.hvplot.hist()\n", + "(zdr_hist + ref_hist).cols(1)" + ] + }, + { + "cell_type": "markdown", + "id": "10fa470b-0b9b-4224-81ec-5491fba7687c", + "metadata": {}, + "source": [ + "Notice how we have very low values for both fields, which we can threshold using:\n", + "- Differential Reflectivity < -5\n", + "- Horizontal Reflectivity < -32" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "797e6ab3-471b-4d03-a16f-4a1074aa6d21", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "ds = ds.where((ds.ZDR >= -5) & (ds.DBZH != -32))\n", + "ds" + ] + }, + { + "cell_type": "markdown", + "id": "0e3a2df6-fa74-4472-bb82-3e3c038acb03", + "metadata": {}, + "source": [ + "### Double Check our Filtered Data\n", + "Let's double check that our filtering worked - notice the new, more representative distributions!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7ac0c4a1-d5e4-4130-9536-3120b7069e77", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "zdr_hist = ds.ZDR.hvplot.hist()\n", + "ref_hist = ds.DBZH.hvplot.hist()\n", + "(zdr_hist + ref_hist).cols(1)" + ] + }, + { + "cell_type": "markdown", + "id": "29efd090-2943-4ba6-a574-4d9d521f7de9", + "metadata": {}, + "source": [ + "## Create a Dashboard to Analyze ZDR Bias\n", + "A common data quality check is differential reflectivity bias. This value should be around 0 for low values of horizontal reflectivity. We use a few steps here to create this visualization\n", + "- Unstack the dataset so we are left with a single dimension - the single range gate (single points)\n", + "- Create histograms (`.hist`) and a 2-dimensional histogram (`.hexbin`) to visualize the data\n", + "- Stack these into single view using `gridspec`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ee29f1cc-d916-4771-888b-f5f634d271a1", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "ds = ds.stack({\"gate\": {\"azimuth\", \"range\"}}).reset_index(\"gate\")\n", + "\n", + "hist_dbz = ds.hvplot.hist(\"DBZH\",\n", + " width=500,\n", + " height=200,\n", + " title=\"Horizontal Reflectivity Distribution\",)\n", + "hist_zdr = ds.hvplot.hist(\"ZDR\",\n", + " height=400,\n", + " title=\"Differential Reflectivity Distribution\",\n", + " ).opts(invert_axes=True)\n", + "hexbin = ds.hvplot.hexbin(x='DBZH',\n", + " y='ZDR',\n", + " title='Reflectivity vs. Differential Reflectivity Distribution',\n", + " width=500,\n", + " height=400) * hv.HLine(0,\n", + " label='Differential Reflectivity = 0 Line').opts(color='red',\n", + " line_width=1)\n", + "\n", + "gspec = pn.GridSpec(width=800, height=400)\n", + "\n", + "gspec[0, 0:2 ] = hist_dbz\n", + "gspec[1:3, 0:2 ] = hexbin\n", + "gspec[1:3, 2 ] = hist_zdr\n", + "\n", + "gspec" + ] + }, + { + "cell_type": "markdown", + "id": "127774f0-0edb-4f51-95e1-d24fec6eef3d", + "metadata": {}, + "source": [ + "## Summary\n", + "Within this notebook, we covered how to use interactive visualizations with your weather radar data, including applications to checking data quality.\n", + "\n", + "### What's Next?\n", + "Next, we will continue to explore methods of cleaning and visualizing data!" + ] + }, + { + "cell_type": "markdown", + "id": "e743814a-044e-470f-92eb-dc5fdfdb115d", + "metadata": {}, + "source": [ + "## Resources and References\n", + "- [Xradar documentation](https://docs.openradarscience.org/projects/xradar/en/stable/index.html)\n", + "- [IDEAM radar data](https://registry.opendata.aws/ideam-radares/)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2ada93cd-cfb8-4687-b7c3-6e626d20619c", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "radar-cookbook-dev", + "language": "python", + "name": "radar-cookbook-dev" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/notebooks/images/bokeh_plot_jmaradar.png b/notebooks/images/bokeh_plot_jmaradar.png new file mode 100644 index 00000000..b0b4cad8 Binary files /dev/null and b/notebooks/images/bokeh_plot_jmaradar.png differ