10_Logging.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logging in Data Science Projects\n",
    "\n",
    "In this notebook, we'll explore the importance of logging in data science projects, how to implement logging effectively using Python's built-in logging module, and best practices for maintaining comprehensive and useful logs. Proper logging helps in debugging, monitoring, and understanding the behavior of your code over time, which is crucial for data science workflows that often involve long-running processes and complex data transformations.\n",
    "\n",
    "**Table of Contents**\n",
    "1. [Why Logging is Important](#1)\n",
    "2. [Setting Up Logging in Python](#2)\n",
    "3. [Logging Best Practices](#3)\n",
    "4. [Step-by-Step Example](#4)\n",
    "5. [Exercise](#5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 1. Why Logging is Important <a id=\"1\"></a>\n",
    "\n",
    "Logging is a critical component of any robust data science project for several reasons:\n",
    "\n",
    "- **Debugging**: Logs provide insights into the code's execution flow and help identify where errors occur.\n",
    "- **Monitoring**: Logs allow you to monitor the performance and behavior of your code over time.\n",
    "- **Audit Trail**: Logs create a historical record of what your code did, which is essential for reproducibility and accountability.\n",
    "- **Communication**: Logs can help communicate the state and progress of your code to team members and stakeholders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 2. Setting Up Logging in Python <a id=\"2\"></a>\n",
    "\n",
    "Python's built-in logging module provides a flexible framework for emitting log messages from Python programs. It is part of the standard library, so you don't need to install anything extra to use it.\n",
    "\n",
    "\n",
    "### Basic Logging Configuration\n",
    "\n",
    "Here is a basic example of how to set up logging in Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "# Set up basic configuration\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
    "\n",
    "# Example log messages\n",
    "logging.debug(\"This is a debug message\")\n",
    "logging.info(\"This is an info message\")\n",
    "logging.warning(\"This is a warning message\")\n",
    "logging.error(\"This is an error message\")\n",
    "logging.critical(\"This is a critical message\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Logging to a File\n",
    "\n",
    "To log messages to a file instead of the console, you can modify the configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "# Set up logging to a file\n",
    "logging.basicConfig(filename='example.log', level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
    "\n",
    "# Example log messages\n",
    "logging.info(\"This is an info message logged to a file\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 3. Logging Best Practices <a id=\"3\"></a>\n",
    "\n",
    "- **Use Appropriate Log Levels**: Use the appropriate log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL) to categorize the importance and type of log messages.\n",
    "- **Log with Context**: Include context in your log messages to make them more informative. For example, log variable values and the state of your application.\n",
    "- **Avoid Logging Sensitive Information**: Be mindful of logging sensitive information such as passwords, personal data, or proprietary information.\n",
    "- **Configure Rotating Logs**: For long-running applications, configure log rotation to prevent log files from growing indefinitely.\n",
    "- **Use Structured Logging**: Use structured logging (e.g., JSON format) for better log parsing and analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 4. Step-by-Step Example <a id=\"4\"></a>\n",
    "\n",
    "Let's create a more detailed example of logging in a data science project. We'll log the steps of data loading, preprocessing, and model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Set up logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
    "\n",
    "# Step 1: Load Data\n",
    "logging.info(\"Loading data...\")\n",
    "data = pd.read_csv('data/raw/sample_data.csv')\n",
    "logging.info(f\"Data loaded with shape {data.shape}\")\n",
    "\n",
    "# Step 2: Preprocess Data\n",
    "logging.info(\"Preprocessing data...\")\n",
    "data.dropna(inplace=True)\n",
    "X = data.drop('target', axis=1)\n",
    "y = data['target']\n",
    "logging.info(f\"Data preprocessed. Features shape: {X.shape}, Target shape: {y.shape}\")\n",
    "\n",
    "# Step 3: Split Data\n",
    "logging.info(\"Splitting data into training and test sets...\")\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "logging.info(f\"Data split. Training set shape: {X_train.shape}, Test set shape: {X_test.shape}\")\n",
    "\n",
    "# Step 4: Train Model\n",
    "logging.info(\"Training model...\")\n",
    "model = RandomForestClassifier(random_state=42)\n",
    "model.fit(X_train, y_train)\n",
    "logging.info(\"Model training completed\")\n",
    "\n",
    "# Step 5: Evaluate Model\n",
    "logging.info(\"Evaluating model...\")\n",
    "y_pred = model.predict(X_test)\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "logging.info(f\"Model accuracy: {accuracy}\")\n",
    "\n",
    "# Step 6: Save Model\n",
    "logging.info(\"Saving model...\")\n",
    "import joblib\n",
    "joblib.dump(model, 'models/random_forest_model.pkl')\n",
    "logging.info(\"Model saved successfully\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 5. Exercise <a id=\"5\"></a>\n",
    "\n",
    "In this exercise, you'll implement a logging system for a data science workflow. Here is a sample code without any logging:\n",
    "\n",
    "**Sample Code (Without Logging)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "import joblib\n",
    "\n",
    "# Step 1: Load Data\n",
    "data = pd.read_csv('data/raw/sample_data.csv')\n",
    "\n",
    "# Step 2: Preprocess Data\n",
    "data.dropna(inplace=True)\n",
    "X = data.drop('target', axis=1)\n",
    "y = data['target']\n",
    "\n",
    "# Step 3: Split Data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Step 4: Train Model\n",
    "model = RandomForestClassifier(random_state=42)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Step 5: Evaluate Model\n",
    "y_pred = model.predict(X_test)\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(f\"Model accuracy: {accuracy}\")\n",
    "\n",
    "# Step 6: Save Model\n",
    "joblib.dump(model, 'models/random_forest_model.pkl')\n",
    "print(\"Model saved successfully\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task**\n",
    "1. Set Up Logging: Configure a logging system that logs messages to both the console and a file.\n",
    "2. Implement Logging: Add logging to the sample code to log each step of the data loading, preprocessing, splitting, training, evaluating, and saving processes.\n",
    "3. Review Logs: Run your code and review the generated logs to ensure they provide a clear and comprehensive overview of the code execution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your Code (With Logging)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "import joblib\n",
    "\n",
    "# Step 1: Load Data\n",
    "data = pd.read_csv('data/raw/sample_data.csv')\n",
    "\n",
    "# Step 2: Preprocess Data\n",
    "data.dropna(inplace=True)\n",
    "X = data.drop('target', axis=1)\n",
    "y = data['target']\n",
    "\n",
    "# Step 3: Split Data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# Step 4: Train Model\n",
    "model = RandomForestClassifier(random_state=42)\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Step 5: Evaluate Model\n",
    "y_pred = model.predict(X_test)\n",
    "accuracy = accuracy_score(y_test, y_pred)\n",
    "print(f\"Model accuracy: {accuracy}\")\n",
    "\n",
    "# Step 6: Save Model\n",
    "joblib.dump(model, 'models/random_forest_model.pkl')\n",
    "print(\"Model saved successfully\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "><details>\n",
    "><summary>Do you need some help?</summary>\n",
    ">\n",
    "> Here is a working solution:\n",
    ">\n",
    ">\n",
    ">```python\n",
    ">\n",
    ">import logging\n",
    ">import pandas as pd\n",
    ">from sklearn.model_selection import train_test_split\n",
    ">from sklearn.ensemble import RandomForestClassifier\n",
    ">from sklearn.metrics import accuracy_score\n",
    ">import joblib\n",
    ">\n",
    "># Set up logging\n",
    ">logging.basicConfig(filename='project.log', level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')\n",
    ">\n",
    "># Step 1: Load Data\n",
    ">logging.info(\"Loading data...\")\n",
    ">data = pd.read_csv('data/raw/sample_data.csv')\n",
    ">logging.info(f\"Data loaded with shape {data.shape}\")\n",
    ">\n",
    "># Step 2: Preprocess Data\n",
    ">logging.info(\"Preprocessing data...\")\n",
    ">data.dropna(inplace=True)\n",
    ">X = data.drop('target', axis=1)\n",
    ">y = data['target']\n",
    ">logging.info(f\"Data preprocessed. Features shape: {X.shape}, Target shape: {y.shape}\")\n",
    ">\n",
    "># Step 3: Split Data\n",
    ">logging.info(\"Splitting data into training and test sets...\")\n",
    ">X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    ">logging.info(f\"Data split. Training set shape: {X_train.shape}, Test set shape: {X_test.shape}\")\n",
    ">\n",
    "># Step 4: Train Model\n",
    ">logging.info(\"Training model...\")\n",
    ">model = RandomForestClassifier(random_state=42)\n",
    ">model.fit(X_train, y_train)\n",
    ">logging.info(\"Model training completed\")\n",
    ">\n",
    "># Step 5: Evaluate Model\n",
    ">logging.info(\"Evaluating model...\")\n",
    ">y_pred = model.predict(X_test)\n",
    ">accuracy = accuracy_score(y_test, y_pred)\n",
    ">logging.info(f\"Model accuracy: {accuracy}\")\n",
    ">\n",
    "># Step 6: Save Model\n",
    ">logging.info(\"Saving model...\")\n",
    ">joblib.dump(model, 'models/random_forest_model.pkl')\n",
    ">logging.info(\"Model saved successfully\")\n",
    ">```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}