aiot_dataset_creation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "feb87f6e",
   "metadata": {},
   "source": [
    "# MongoDB Handling\n",
    "\n",
    "After installing the MongoDB server in your machine, you can use this notebook for handling the initial processes with the database.\n",
    "\n",
    "Specifically, in this step, we utilize Python's `pymongo` library to exploit its capabilities for MongoDB server interaction.\n",
    "\n",
    "**Important Note: Be sure that the MongoDB server is up and running as a service in the background.**\n",
    "\n",
    "For example, in macOS, to run MongoDB (i.e. the mongod process) as a service, run:\n",
    "\n",
    "* `brew services start mongodb-community`\n",
    "\n",
    "To stop a mongod running as a macOS service, use the following command as needed:\n",
    "\n",
    "* `brew services stop mongodb-community`\n",
    "\n",
    "To install MongoDB in your system, follow the instructions here:\n",
    "\n",
    "* https://www.mongodb.com/docs/manual/administration/install-community/\n",
    "\n",
    "\n",
    "**Note:** You can modify any of the processes below, however, you have to explain your thoughts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a192af2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import library for various processes with the OS\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1a34708",
   "metadata": {},
   "source": [
    "## Load configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e970c28",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import library for yaml handling\n",
    "import yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbb4f44b",
   "metadata": {},
   "outputs": [],
   "source": [
    "config_path = os.path.join(os.getcwd(), \"config.yml\")\n",
    "\n",
    "with open(config_path) as file:\n",
    "    config = yaml.load(file, Loader=yaml.FullLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65bac888",
   "metadata": {},
   "source": [
    "## MongoDB database instantiation\n",
    "\n",
    "The relevant information for the MongoDB client connection, the database name, and collection name is located in the configuration file.\n",
    "\n",
    "```\n",
    "# DB Connection with the uri (host)\n",
    "client: \"mongodb://localhost:27017/\"\n",
    "\n",
    "# db name\n",
    "db: \"aiot_course\"\n",
    "\n",
    "# db collection\n",
    "col: \"NAME YOUR COLLECTION\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3cc2add",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import library for hanlding the MongoDB client\n",
    "import pymongo\n",
    "# import library for retrieving datetime\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dbbbe9f",
   "metadata": {},
   "source": [
    "### Create the database\n",
    "\n",
    "To create a database in MongoDB, start by creating a MongoClient object, then specify a connection URL with the correct ip address and the name of the database you want to create.\n",
    "\n",
    "MongoDB will create the database if it does not exist, and make a connection to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdbb888f",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = pymongo.MongoClient(config[\"client\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "203c2bc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "db = client[config[\"db\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0772e744",
   "metadata": {},
   "source": [
    "### Instantiate the collection\n",
    "\n",
    "To create a collection in MongoDB, use database object and specify the name of the collection you want to create.\n",
    "\n",
    "MongoDB will create the collection if it does not exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9937a0c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "col = db[config[\"col\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bcde222",
   "metadata": {},
   "source": [
    "Initially, no collection will be shown in MongoDB before you enter the first document!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b5cd593",
   "metadata": {},
   "source": [
    "## Create the data collection\n",
    "\n",
    "Uploading the gathered data to MongoDB collection. The data directory structure should be as follows:\n",
    "\n",
    "```\n",
    ".\n",
    "└── data/\n",
    "    ├── class_A/\n",
    "    │   ├── data_A_01.csv\n",
    "    │   ├── data_A_02.csv\n",
    "    │   └── ..\n",
    "    ├── class_B/\n",
    "    │   ├── data_B_01.csv\n",
    "    │   ├── data_B_02.csv\n",
    "    │   └── .\n",
    "    └── class ...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "622636fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import library for hanlding the csv data and transformations\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9532ede2",
   "metadata": {},
   "source": [
    "Get data path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "912f079f",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = os.path.join(os.getcwd(), \"data\")\n",
    "print(data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a981a220",
   "metadata": {},
   "source": [
    "List all files in a path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd476759",
   "metadata": {},
   "outputs": [],
   "source": [
    "classes_folders_list = [f for f in os.listdir(data_path) if os.path.isdir(os.path.join(data_path, f))]\n",
    "print(classes_folders_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e8075f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print files in folder\n",
    "folder_path = os.path.join(data_path, classes_folders_list[0])\n",
    "files_in_folder = [f for f in os.listdir(folder_path) if os.path.isfile(os.path.join(folder_path, f))]\n",
    "print(files_in_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c65d2961",
   "metadata": {},
   "source": [
    "Each document in the MongoDB database should have the following schema:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"data\": {\n",
    "    \"acc_x\": [\"array\", \"of\", \"values\"],\n",
    "    \"acc_y\": [\"array\", \"of\", \"values\"],\n",
    "    \"acc_z\": [\"array\", \"of\", \"values\"],\n",
    "  },\n",
    "  \"label\": \"The label of the instance\",\n",
    "  \"datetime\": \"MongoDB datetime object (it can be generated with the datetime.datetime.now() function\"\n",
    "}\n",
    "```\n",
    "\n",
    "Accordingly, if you are using gyroscope or both accelerometer and gyroscope, the following order and naming of the sensor keys should be defined:\n",
    "\n",
    "* for gyroscope: `gyr_x`, `gyr_y`, `gyr_z` for the three axes\n",
    "* for accelerometer and gyroscope: `acc_x`, `acc_y`, `acc_z`, `gyr_x`, `gyr_y`, `gyr_z` for the six axes\n",
    "\n",
    "**Note: Be careful, the document is mandatory to have the aforementioned schema, in order to argue and proceed with the rest of the processes later on, in data engineering, plotting, etc.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bc16310",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import df_rebase"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "115f2ae5",
   "metadata": {},
   "source": [
    "## Provide the code to upload the data to MongoDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56c8d4e2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}