01-rbr.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "89320b07",
   "metadata": {},
   "source": [
    "# RBR pressure sensor\n",
    "\n",
    "## Convert from instrument data to raw .cdf\n",
    "\n",
    "In this notebook we will process data from an RBR pressure sensor to a \"raw\" .cdf file, then apply atmospheric pressure corrections and write to a final, \"clean\" .nc file. In this case we have already used the RBR Ruskin software to export from the binary data format to text (.csv) format by using (File -> Export -> Text (\\*.zip)) in Ruskin.\n",
    "\n",
    "Open up another terminal if you need to. Change to the `data/rbrCSF20SC201` directory.\n",
    "\n",
    "Make sure you have the `stglib` environment activated by typing `mamba activate stglib`.\n",
    "\n",
    "The \"run script\" we are using here is `runrskcsv2cdf.py`. Broken up into its component parts: \n",
    "- `run` (run) \n",
    "- `rsk` (file extension of RBR instruments, .rsk), \n",
    "- `csv2cdf` (converting from exported ASCII csv files to a raw .cdf file), \n",
    "-`.py` (python file extension).\n",
    "\n",
    "If you are on macOS, you can simply type:\n",
    "`runrskcsv2cdf.py gatts_CSF20SC2.txt csf20sc201_config.yaml`\n",
    "\n",
    "If you are on Windows, you will need to specify the full path to the run script. It will probably be something like\n",
    "\n",
    "`python c:/users/dan/mambaforge/envs/stglib/scripts/runrskcsv2cdf.py gatts_CSF20SC2.txt csf20sc201_config.yaml`\n",
    "\n",
    "This should generate the file\n",
    "`CSF20SC201pt-raw.cdf`\n",
    "\n",
    "Take a look at the output of the run script. Are there any errors? Warnings? \n",
    "\n",
    "Check out the contents of the netCDF file by running `ncdump -h CSF20SC201pt-raw.cdf` from the commandline.\n",
    "\n",
    "Note we aren't importing stglib in this notebook. The run scripts do import it, however."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fde6473",
   "metadata": {},
   "outputs": [],
   "source": [
    "import xarray as xr\n",
    "import hvplot.xarray\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import requests\n",
    "ds = xr.load_dataset('data/rbrCSF20SC201/CSF20SC201pt-raw.cdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aefa600",
   "metadata": {},
   "source": [
    "Type ds to see the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "226b1a76",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "416dea93",
   "metadata": {},
   "source": [
    "You can explore the data by clicking the triangles, document icons, and disk icons. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae2db72e",
   "metadata": {},
   "source": [
    "Now plot the data. Note that this is plotting the raw data with time on the y axis and burst number on the x axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cf2d8e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.P_1.hvplot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a48df68",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.T_28.hvplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f21bc5a",
   "metadata": {},
   "source": [
    "Note that we have lots of out-of water data we need to clip, but this raw file preserves all the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71892a2d",
   "metadata": {},
   "source": [
    "## Get atmos pressure data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51acc613",
   "metadata": {},
   "source": [
    "This downloads data using the NOAA Tides & Currents API using the reqeusts module and writes to a text file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a155dc75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use plus signs to concatenate a long string\n",
    "# this station is in Oakland and is the closest easily available NOAA met data\n",
    "# https://tidesandcurrents.noaa.gov/stationhome.html?id=9414763\n",
    "r = requests.get(\"https://api.tidesandcurrents.noaa.gov/api/prod/datagetter?\" + \n",
    "                 \"product=air_pressure&application=NOS.COOPS.TAC.WL&\" +  # product type\n",
    "                 \"begin_date=20200701&end_date=20200901&station=9414763\" +  # date range and station id\n",
    "                 \"&time_zone=GMT&units=metric&interval=h&format=csv\")  # time zone, interval, units, format\n",
    "\n",
    "# write data to file\n",
    "with open('data/rbrCSF20SC201/atmos_pressure.txt', 'w') as f:\n",
    "    f.write(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "625dea59",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/rbrCSF20SC201/atmos_pressure.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08c81783",
   "metadata": {},
   "source": [
    "Take a look at the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba27e752",
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c611129",
   "metadata": {},
   "source": [
    "Now do a bunch of data massaging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9405c9d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e39ae47",
   "metadata": {},
   "outputs": [],
   "source": [
    "# need to strip the column names because there are spaces after the commas\n",
    "df.columns = df.columns.str.strip()\n",
    "# rename Date Time to time and Pressure to atmpres to match stglib expectations\n",
    "df = df.rename(columns={'Date Time': 'time', 'Pressure': 'atmpres'})\n",
    "# pressure is in mbar, so convert to dbar\n",
    "df['atmpres'] = df['atmpres'] * 0.01\n",
    "# set time to be the index\n",
    "df = df.set_index('time')\n",
    "\n",
    "# and convert to an xarray Dataset\n",
    "atm = df.to_xarray()\n",
    "# ensure time is stored as a datetime\n",
    "atm['time'] = pd.DatetimeIndex(atm['time'])\n",
    "# drop unneeded variables\n",
    "atm = atm.drop(['X', 'N', 'R'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "736b1569",
   "metadata": {},
   "source": [
    "Look at our new xarray Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea92e885",
   "metadata": {},
   "outputs": [],
   "source": [
    "atm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49514230",
   "metadata": {},
   "source": [
    "Because our atmospheric pressure data was collected ourly, but our water-level data was collected every 5 minutes, we need to reindex our atmos data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26f2ec9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "atm = atm.reindex_like(ds,  method='nearest', tolerance='60min')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "280b585a",
   "metadata": {},
   "source": [
    "Look at our data and figure out our offset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47f51198",
   "metadata": {},
   "outputs": [],
   "source": [
    "P_1mean = ds.P_1.mean(dim='sample')\n",
    "diff = P_1mean-atm.atmpres\n",
    "diff.name = 'diff'\n",
    "P_1mean.hvplot() * diff.hvplot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2b11a5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# looks like we need an offset\n",
    "offset = 0.13\n",
    "(diff-offset).hvplot()\n",
    "\n",
    "# zoom in to see if the offset looks right and change it if not"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7c20164",
   "metadata": {},
   "source": [
    "Attach our offset value as an attribute on the xarray Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94993dbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "atm\n",
    "atm.atmpres.attrs['offset'] = offset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cf12aad",
   "metadata": {},
   "outputs": [],
   "source": [
    "atm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fbe6394",
   "metadata": {},
   "source": [
    "And write to netcdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "185c6af8",
   "metadata": {},
   "outputs": [],
   "source": [
    "atm.to_netcdf('data/rbrCSF20SC201/atmpres.cdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71b1d2cd",
   "metadata": {},
   "source": [
    "## Convert data to final, clean netCDF file\n",
    "Now we have an atmospheric pressure compensation file and the raw data file, so we are ready to process to final .nc with atmospheric compensation applied\n",
    "\n",
    "`runrskcdf2nc.py CSF20SC201pt-raw.cdf --atmpres=atmpres.cdf`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f81d87e",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = xr.load_dataset('data/rbrCSF20SC201/CSF20SC201ptb-cal.nc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fbd70c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db41331a",
   "metadata": {},
   "source": [
    "Looks great!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "980cc4fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.P_1ac.mean(dim='sample').hvplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "996036dd",
   "metadata": {},
   "source": [
    "## During the meeting: computing waves statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72de329e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}