06-LibHDF5-to-HSDS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accessing HSDS from the HDF5 library\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\">\n",
    "This notebook will work only with a running HSDS instance. If you don't have one running, you can start one using the <a href=\"./05-HSDS.ipynb\">HSDS notebook</a>.\n",
    "</div>\n",
    "\n",
    "## Introducing the HDF5 REST VOL connector\n",
    "\n",
    "In our discussion of the Highly Scalable Data Service (HSDS), we showed that by merely swapping Python modules (`import h5pyd as h5py`), without further modifications, we were able to run a Python script against HSDS rather than a file system. This begs the question if something similar can be done with the HDF5 library. The answer is, yes, and what makes this possible is an extension layer of the HDF5 library, the so-called __[Virtual Object Layer (VOL)](https://github.com/HDFGroup/arch-doc)__. Such extensions or *VOL connectors*, are implemented as dynamically loadable plugins, and the connector implementing \"HSDS connectivity\" is the __[REST VOL connector](https://github.com/HDFGroup/vol-rest)__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing the HDF5 REST VOL connector\n",
    "\n",
    "As a plugin, the REST VOL connector is not installed by default. It also is only available with HDF5 library versions 1.12 and newer. The library version pre-installed in this Docker container is pre-1.12, and we must install a newer library version and build the REST VOL connector from source (__[GitHub](https://github.com/HDFGroup/vol-rest)__).\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "The cloning and compilation takes, depending on the machine type, about three minutes.\n",
    "</div>\n",
    "\n",
    "The script shown below clones the HDF5 library and REST VOL repositories and build the binaries using CMake. The headers and binaries are installed in `$HOME/.local`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "git clone https://github.com/HDFGroup/hdf5.git build/hdf5\n",
    "mkdir -p build/hdf5/build\n",
    "cd build/hdf5/build\n",
    "cmake -DCMAKE_INSTALL_PREFIX=/home/vscode/.local -DBUILD_STATIC_LIBS=OFF -DBUILD_TESTING=OFF -DHDF5_BUILD_EXAMPLES=OFF -DHDF5_BUILD_TOOLS=OFF -DHDF5_BUILD_UTILS=OFF ../ 2>&1 > /dev/null\n",
    "make -j 4 2>&1 > /dev/null\n",
    "make install 2>&1 > /dev/null\n",
    "cd ../../..\n",
    "git clone https://github.com/HDFGroup/vol-rest.git build/rest-vol\n",
    "cd build/rest-vol\n",
    "./build_vol_cmake.sh -P /home/vscode/.local -H /home/vscode/.local -B ./build 2>&1 > /dev/null\n",
    "cd build && make install 2>&1 > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying the C++ HDF5 example to use the REST VOL connector\n",
    "\n",
    "By adding five lines and changing a single line of code, we can make our original C++ example \"talk\" to HSDS instead of the file system.\n",
    "\n",
    "- We must initialize the REST VOL connector (line 27)\n",
    "- We must add a file access property (lines 28,29) and pass it to `H5Fcreate` (line 30) and release the file access property list (line 31)\n",
    "- We terminate the REST VOL connector (line 70)\n",
    "\n",
    "And that's it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/ou_restvol.cpp\n",
    "#include \"docstring.hpp\"\n",
    "#include \"ou_sampler.hpp\"\n",
    "#include \"rest_vol_public.h\"\n",
    "#include \"hdf5.h\"\n",
    "#include <iostream>\n",
    "#include <vector>\n",
    "\n",
    "using namespace std;\n",
    "\n",
    "int main()\n",
    "{\n",
    "    const size_t path_count = 100, step_count = 1000;\n",
    "    const double dt = 0.01, theta = 1.0, mu = 0.0, sigma = 0.1;\n",
    "\n",
    "    cout << \"Running with parameters:\"\n",
    "         << \" paths=\" << path_count << \" steps=\" << step_count\n",
    "         << \" dt=\" << dt << \" theta=\" << theta << \" mu=\" << mu << \" sigma=\" << sigma << endl;\n",
    "\n",
    "    vector<double> ou_process;\n",
    "    ou_sampler(ou_process, path_count, step_count, dt, theta, mu, sigma);\n",
    "    \n",
    "    //\n",
    "    // Write the sample paths to an HDF5 file using the HDF5 REST VOL!\n",
    "    //\n",
    "\n",
    "    H5rest_init();\n",
    "    auto fapl = H5Pcreate(H5P_FILE_ACCESS);\n",
    "    H5Pset_fapl_rest_vol(fapl);\n",
    "    auto file = H5Fcreate(\"/home/vscode/ou_restvol.h5\", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);\n",
    "    H5Pclose(fapl);\n",
    "\n",
    "    add_docstring(file, \".\", \"source\", \"https://github.com/HDFGroup/hdf5-tutorial\");\n",
    "\n",
    "    { // create & write the dataset\n",
    "        hsize_t dimsf[] = {(hsize_t)path_count, (hsize_t)step_count};\n",
    "        auto space = H5Screate_simple(2, dimsf, NULL);\n",
    "        auto dataset = H5Dcreate(file, \"/dataset\", H5T_NATIVE_DOUBLE, space, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);\n",
    "        H5Dwrite(dataset, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL, H5P_DEFAULT, ou_process.data());\n",
    "        H5Dclose(dataset);\n",
    "        H5Sclose(space);\n",
    "    }\n",
    "\n",
    "    { // make the file self-describing by adding a few attributes to `dataset`\n",
    "        add_docstring(file, \"dataset\", \"comment\", \"This dataset contains sample paths of an Ornstein-Uhlenbeck process.\");\n",
    "        add_docstring(file, \"dataset\", \"Wikipedia\", \"https://en.wikipedia.org/wiki/Ornstein%E2%80%93Uhlenbeck_process\");\n",
    "        add_docstring(file, \"dataset\", \"rows\", \"path\");\n",
    "        add_docstring(file, \"dataset\", \"columns\", \"time\");\n",
    "        \n",
    "        auto scalar = H5Screate(H5S_SCALAR);\n",
    "        auto acpl = H5Pcreate(H5P_ATTRIBUTE_CREATE);\n",
    "        H5Pset_char_encoding(acpl, H5T_CSET_UTF8);\n",
    "        \n",
    "        auto set_attribute = [&](const string& name, const double& value) {\n",
    "            auto attr = H5Acreate_by_name(file, \"dataset\", name.c_str(), H5T_NATIVE_DOUBLE, scalar, acpl, H5P_DEFAULT, H5P_DEFAULT);\n",
    "            H5Awrite(attr, H5T_NATIVE_DOUBLE, &value);\n",
    "            H5Aclose(attr);\n",
    "        };\n",
    "        set_attribute(\"dt\", dt);\n",
    "        set_attribute(\"θ\", theta);\n",
    "        set_attribute(\"μ\", mu);\n",
    "        set_attribute(\"σ\", sigma);\n",
    "\n",
    "        H5Pclose(acpl);\n",
    "        H5Sclose(scalar);\n",
    "    }\n",
    "\n",
    "    H5Fclose(file);\n",
    "\n",
    "    H5rest_term();\n",
    "\n",
    "    return 0;\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "g++ -I/home/vscode/.local/include -L/home/vscode/.local/lib ./src/ou_restvol.cpp ./src/docstring.cpp ./src/ou_sampler.cpp -o ./build/ou_restvol -lhdf5 -lhdf5_vol_rest -lcurl -ldl -Wl,-rpath=/home/vscode/.local/lib\n",
    "export HSDS_USERNAME=vscode\n",
    "export HSDS_PASSWORD=vscode\n",
    "export HSDS_ENDPOINT=http://localhost:5101\n",
    "./build/ou_restvol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "hsls --showattrs /home/vscode/ou_restvol.h5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improving performance with multi-dataset I/O\n",
    "\n",
    "Sometimes we perform I/O (read or write) against multiple datasets. In the past, there was no API for \"vectorizing\" such access patterns. Since HDF5 library version 1.14.0, the VOL layer has `H5D[read,write]_multi` calls that support this particular use case. The REST VOL connector implements both calls, by taking advantage of __[`libcurl`'s multi interface](https://curl.se/libcurl/c/libcurl-multi.html)__. The example below, shows how to use the `H5Dwrite_multi` API by creating request vectors whose length is the number of datasets to be written in one call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/multi_dataset.cpp\n",
    "#include \"hdf5.h\"\n",
    "#include \"rest_vol_public.h\"\n",
    "\n",
    "#include <iostream>\n",
    "#include <sstream>\n",
    "#include <vector>\n",
    "\n",
    "using namespace std;\n",
    "\n",
    "const size_t NUM_DATASETS = 2;\n",
    "const size_t NUM_INTEGERS = (1000 * 1000 * 100) / NUM_DATASETS;\n",
    "\n",
    "int main()\n",
    "{\n",
    "    H5rest_init();\n",
    "\n",
    "    hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);\n",
    "    H5Pset_fapl_rest_vol(fapl);\n",
    "    hid_t file = H5Fcreate(\"/home/vscode/multi_dataset.h5\", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);\n",
    "    H5Pclose(fapl);\n",
    "    \n",
    "    vector<int> write_buf(NUM_DATASETS*NUM_INTEGERS), read_buf(NUM_DATASETS*NUM_INTEGERS);\n",
    "    int *write_data[NUM_DATASETS], *read_data[NUM_DATASETS];\n",
    "    for (size_t i = 0; i < NUM_DATASETS; ++i)\n",
    "    {\n",
    "        write_data[i] = &write_buf[i*NUM_INTEGERS];\n",
    "        read_data[i] = &read_buf[i*NUM_INTEGERS];\n",
    "    }\n",
    "\n",
    "    hid_t dspace_ids[NUM_DATASETS], dset_ids[NUM_DATASETS], type_ids[NUM_DATASETS], sel_space_ids[NUM_DATASETS];\n",
    "    \n",
    "    ostringstream dset_name;\n",
    "    hsize_t dims[] = { NUM_INTEGERS };\n",
    "\n",
    "    for (size_t i = 0; i < NUM_DATASETS; ++i)\n",
    "    {\n",
    "        type_ids[i] = H5T_NATIVE_INT;\n",
    "        dspace_ids[i] = H5Screate_simple(1, dims, NULL);\n",
    "        sel_space_ids[i] = H5S_ALL;\n",
    "\n",
    "        dset_name.str(\"\");\n",
    "        dset_name << \"/dset_\" << i;\n",
    "        dset_ids[i] = H5Dcreate(file, dset_name.str().c_str(), type_ids[i], dspace_ids[i], H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);\n",
    "\n",
    "        for (size_t j = 0; j < NUM_INTEGERS; ++j)\n",
    "            write_data[i][j] = i * 10000 + j;\n",
    "    }\n",
    "\n",
    "    H5Dwrite_multi(NUM_DATASETS, dset_ids, type_ids, sel_space_ids, sel_space_ids, H5P_DEFAULT, (const void**)write_data);\n",
    "\n",
    "    H5Dread_multi(NUM_DATASETS, dset_ids, type_ids, sel_space_ids, sel_space_ids, H5P_DEFAULT, (void**) read_data);\n",
    "\n",
    "    for (size_t i = 0; i < read_buf.size(); ++i)\n",
    "    {\n",
    "        if (read_buf[i] != write_buf[i])\n",
    "        {\n",
    "            cerr << \"At read_buf[\" << i << \"], \" << read_buf[i] << \" != expected \" << write_buf[i] << endl;\n",
    "            exit(1);\n",
    "        }\n",
    "    }\n",
    "    \n",
    "    cout << \"SUCCESS!\" << endl;\n",
    "\n",
    "    H5Fclose(file);\n",
    "    H5rest_term();\n",
    "\n",
    "    return 0;\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "g++ -std=c++17 -Wall -pedantic -I/home/vscode/.local/include -L/home/vscode/.local/lib ./src/multi_dataset.cpp -o ./build/restvol -lhdf5 -lhdf5_vol_rest -lcurl -ldl -Wl,-rpath=/home/vscode/.local/lib\n",
    "export HSDS_USERNAME=vscode\n",
    "export HSDS_PASSWORD=vscode\n",
    "export HSDS_ENDPOINT=http://localhost:5101\n",
    "./build/restvol\n",
    "hsls /home/vscode/multi_dataset.h5"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hdf5-tutorial",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}