Day_1_Lab_1_Data_Preprocessing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DrKenReid/Introductory-Data-Science/blob/main/Day_1_Lab_1_Data_Preprocessing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Hands-on Exercise: Introduction to ML data preprocessing"
      ],
      "metadata": {
        "id": "a7XqSj2HZrNu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Problem Description:\n",
        "\n",
        "For this hands-on exercise, we will work on a real-world classification problem using the \"Adult Income\" dataset. The goal is to predict whether an individual's annual income exceeds \\$50,000 based on various demographic and employment-related features. This is a binary classification problem, where the target variable has two classes: \">50K\" (representing an annual income greater than \\$50,000) and \"<=50K\" (representing an annual income less than or equal to \\$50,000).\n",
        "\n",
        "## Dataset:\n",
        "\n",
        "The \"Adult Income\" dataset is publicly available and can be obtained from the UCI Machine Learning Repository. The dataset contains information about individuals, including their age, education, occupation, marital status, and other relevant features.\n"
      ],
      "metadata": {
        "id": "8PhRrMaFZRFU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Key Features:\n",
        "\n",
        "*   age: The age of the individual (continuous)\n",
        "*   workclass: The type of employer (categorical)\n",
        "*   fnlwgt: The number of people the census takers believe that observation represents (continuous)\n",
        "*   education: The highest level of education achieved (categorical)\n",
        "*   education-num: The numeric representation of the   education level (continuous)\n",
        "*   marital-status: The marital status of the individual (categorical)\n",
        "*   occupation: The occupation of the individual (categorical)\n",
        "*   relationship: The family relationship of the individual (categorical)\n",
        "*   race: The race of the individual (categorical)\n",
        "*   sex: The gender of the individual (categorical)\n",
        "*   capital-gain: The capital gains of the individual (continuous)\n",
        "*   capital-loss: The capital losses of the individual (continuous)\n",
        "*   hours-per-week: The number of hours the individual works per week (continuous)\n",
        "*   native-country: The native country of the individual (categorical)\n",
        "*   income: The target variable, indicating whether the individual's annual income exceeds $50,000 (binary)\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "MUrafjFSZa8b"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Load dataset"
      ],
      "metadata": {
        "id": "9b_9E2wpdxUH"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "_LwcqDLtj6wO"
      },
      "outputs": [],
      "source": [
        "# Import necessary libraries\n",
        "import pandas as pd\n",
        "from sklearn.preprocessing import LabelEncoder, StandardScaler\n",
        "from sklearn.model_selection import train_test_split"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Load the dataset\n",
        "url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data\"\n",
        "column_names = [\n",
        "    \"age\", \"workclass\", \"fnlwgt\", \"education\", \"education-num\", \"marital-status\",\n",
        "    \"occupation\", \"relationship\", \"race\", \"sex\", \"capital-gain\", \"capital-loss\",\n",
        "    \"hours-per-week\", \"native-country\", \"income\"\n",
        "]\n",
        "data = pd.read_csv(url, header=None, names=column_names)"
      ],
      "metadata": {
        "id": "gtvzSqInkL9N"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display the first few rows of the dataset\n",
        "print(\"First few rows of the dataset:\")\n",
        "print(data.head())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JYGuxq-SkOgI",
        "outputId": "9a0cefd6-35cb-4ed0-c9ce-4a225627a052"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "First few rows of the dataset:\n",
            "   age          workclass  fnlwgt   education  education-num  \\\n",
            "0   39          State-gov   77516   Bachelors             13   \n",
            "1   50   Self-emp-not-inc   83311   Bachelors             13   \n",
            "2   38            Private  215646     HS-grad              9   \n",
            "3   53            Private  234721        11th              7   \n",
            "4   28            Private  338409   Bachelors             13   \n",
            "\n",
            "        marital-status          occupation    relationship    race      sex  \\\n",
            "0        Never-married        Adm-clerical   Not-in-family   White     Male   \n",
            "1   Married-civ-spouse     Exec-managerial         Husband   White     Male   \n",
            "2             Divorced   Handlers-cleaners   Not-in-family   White     Male   \n",
            "3   Married-civ-spouse   Handlers-cleaners         Husband   Black     Male   \n",
            "4   Married-civ-spouse      Prof-specialty            Wife   Black   Female   \n",
            "\n",
            "   capital-gain  capital-loss  hours-per-week  native-country  income  \n",
            "0          2174             0              40   United-States   <=50K  \n",
            "1             0             0              13   United-States   <=50K  \n",
            "2             0             0              40   United-States   <=50K  \n",
            "3             0             0              40   United-States   <=50K  \n",
            "4             0             0              40            Cuba   <=50K  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data cleaning"
      ],
      "metadata": {
        "id": "vU0l--nSdvbC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Check for missing values\n",
        "print(\"\\nMissing values in each column:\")\n",
        "print(data.isnull().sum())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "71uzTIUhkR3D",
        "outputId": "79957085-267c-44db-c395-3bc4b1b15d5b"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Missing values in each column:\n",
            "age               0\n",
            "workclass         0\n",
            "fnlwgt            0\n",
            "education         0\n",
            "education-num     0\n",
            "marital-status    0\n",
            "occupation        0\n",
            "relationship      0\n",
            "race              0\n",
            "sex               0\n",
            "capital-gain      0\n",
            "capital-loss      0\n",
            "hours-per-week    0\n",
            "native-country    0\n",
            "income            0\n",
            "dtype: int64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The above table shows that there is no missing values in columns.\n",
        "The next step is to check if there are unexpected, or irregular values in the dataset.\n",
        "\n",
        "For categorical columns, list unique values to spot unexpected entries:"
      ],
      "metadata": {
        "id": "31FBo2V71JOI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data['age'].unique()\n",
        "data['workclass'].unique()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w7YD8Tvx-zJu",
        "outputId": "7987be57-2594-4b22-e2fd-5267f08c22c5"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([' State-gov', ' Self-emp-not-inc', ' Private', ' Federal-gov',\n",
              "       ' Local-gov', ' ?', ' Self-emp-inc', ' Without-pay',\n",
              "       ' Never-worked'], dtype=object)"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "For numeric columns, generate descriptive statistics to get a quick overview of potential issues in numeric columns.\n",
        "\n",
        "Potentially, plot the numeric columns to understand the distributions."
      ],
      "metadata": {
        "id": "T-YzTlxP1EXM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "statistics_summary = data.describe()\n",
        "statistics_summary"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "lRfb5FpH_afg",
        "outputId": "8a748329-e870-4efa-b25f-85d9c6afd71f"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                age        fnlwgt  education-num  capital-gain  capital-loss  \\\n",
              "count  32561.000000  3.256100e+04   32561.000000  32561.000000  32561.000000   \n",
              "mean      38.581647  1.897784e+05      10.080679   1077.648844     87.303830   \n",
              "std       13.640433  1.055500e+05       2.572720   7385.292085    402.960219   \n",
              "min       17.000000  1.228500e+04       1.000000      0.000000      0.000000   \n",
              "25%       28.000000  1.178270e+05       9.000000      0.000000      0.000000   \n",
              "50%       37.000000  1.783560e+05      10.000000      0.000000      0.000000   \n",
              "75%       48.000000  2.370510e+05      12.000000      0.000000      0.000000   \n",
              "max       90.000000  1.484705e+06      16.000000  99999.000000   4356.000000   \n",
              "\n",
              "       hours-per-week  \n",
              "count    32561.000000  \n",
              "mean        40.437456  \n",
              "std         12.347429  \n",
              "min          1.000000  \n",
              "25%         40.000000  \n",
              "50%         40.000000  \n",
              "75%         45.000000  \n",
              "max         99.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4e7d52f4-461c-45b8-a7a7-b73ae9a8cacb\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>age</th>\n",
              "      <th>fnlwgt</th>\n",
              "      <th>education-num</th>\n",
              "      <th>capital-gain</th>\n",
              "      <th>capital-loss</th>\n",
              "      <th>hours-per-week</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>32561.000000</td>\n",
              "      <td>3.256100e+04</td>\n",
              "      <td>32561.000000</td>\n",
              "      <td>32561.000000</td>\n",
              "      <td>32561.000000</td>\n",
              "      <td>32561.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>38.581647</td>\n",
              "      <td>1.897784e+05</td>\n",
              "      <td>10.080679</td>\n",
              "      <td>1077.648844</td>\n",
              "      <td>87.303830</td>\n",
              "      <td>40.437456</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>13.640433</td>\n",
              "      <td>1.055500e+05</td>\n",
              "      <td>2.572720</td>\n",
              "      <td>7385.292085</td>\n",
              "      <td>402.960219</td>\n",
              "      <td>12.347429</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>17.000000</td>\n",
              "      <td>1.228500e+04</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>28.000000</td>\n",
              "      <td>1.178270e+05</td>\n",
              "      <td>9.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>40.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>37.000000</td>\n",
              "      <td>1.783560e+05</td>\n",
              "      <td>10.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>40.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>48.000000</td>\n",
              "      <td>2.370510e+05</td>\n",
              "      <td>12.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>45.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>90.000000</td>\n",
              "      <td>1.484705e+06</td>\n",
              "      <td>16.000000</td>\n",
              "      <td>99999.000000</td>\n",
              "      <td>4356.000000</td>\n",
              "      <td>99.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4e7d52f4-461c-45b8-a7a7-b73ae9a8cacb')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-4e7d52f4-461c-45b8-a7a7-b73ae9a8cacb button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-4e7d52f4-461c-45b8-a7a7-b73ae9a8cacb');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-10c8f3d3-2263-45d9-a143-5c98fef020ab\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-10c8f3d3-2263-45d9-a143-5c98fef020ab')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-10c8f3d3-2263-45d9-a143-5c98fef020ab button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_0d7b3e69-6186-4103-9f3f-583f4a2bd03b\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('statistics_summary')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_0d7b3e69-6186-4103-9f3f-583f4a2bd03b button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('statistics_summary');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "statistics_summary",
              "summary": "{\n  \"name\": \"statistics_summary\",\n  \"rows\": 8,\n  \"fields\": [\n    {\n      \"column\": \"age\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 11498.327099385337,\n        \"min\": 13.640432553581341,\n        \"max\": 32561.0,\n        \"num_unique_values\": 8,\n        \"samples\": [\n          38.58164675532078,\n          37.0,\n          32561.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"fnlwgt\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 486902.252790567,\n        \"min\": 12285.0,\n        \"max\": 1484705.0,\n        \"num_unique_values\": 8,\n        \"samples\": [\n          189778.36651208502,\n          178356.0,\n          32561.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"education-num\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 11508.98951393426,\n        \"min\": 1.0,\n        \"max\": 32561.0,\n        \"num_unique_values\": 8,\n        \"samples\": [\n          10.0806793403151,\n          10.0,\n          32561.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"capital-gain\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35111.63595222616,\n        \"min\": 0.0,\n        \"max\": 99999.0,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          1077.6488437087312,\n          99999.0,\n          7385.292084840338\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"capital-loss\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 11366.935322921165,\n        \"min\": 0.0,\n        \"max\": 32561.0,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          87.303829734959,\n          4356.0,\n          402.9602186489998\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"hours-per-week\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 11498.057671750134,\n        \"min\": 1.0,\n        \"max\": 32561.0,\n        \"num_unique_values\": 7,\n        \"samples\": [\n          32561.0,\n          40.437455852092995,\n          45.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We found that there are irregular entries like ' ?' in categorical columns. Next, we need to clean up the irregular values."
      ],
      "metadata": {
        "id": "5f1PL7Z_1QXI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Handle irregular values\n",
        "data[\"workclass\"] = data[\"workclass\"].replace(\" ?\", \"Unknown\")\n",
        "data[\"occupation\"] = data[\"occupation\"].replace(\" ?\", \"Unknown\")\n",
        "data[\"native-country\"] = data[\"native-country\"].replace(\" ?\", \"Unknown\")"
      ],
      "metadata": {
        "id": "JXaSB-IPkTg3"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Preparing data for machine learning models that require numerical input:\n",
        "\n",
        "(1) Transform categorical variables into numeric format:"
      ],
      "metadata": {
        "id": "V46ISmQZ1Vrp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Encode categorical variables using LabelEncoder\n",
        "categorical_features = [\n",
        "    \"workclass\", \"education\", \"marital-status\", \"occupation\",\n",
        "    \"relationship\", \"race\", \"sex\", \"native-country\", \"income\"\n",
        "]\n",
        "label_encoders = {}\n",
        "for feature in categorical_features:\n",
        "    label_encoders[feature] = LabelEncoder()\n",
        "    data[feature] = label_encoders[feature].fit_transform(data[feature])"
      ],
      "metadata": {
        "id": "5jcamphskVFH"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "(2) Scale numerical features using StandardScaler:"
      ],
      "metadata": {
        "id": "yjM48VDq1aLH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "numerical_features = [\n",
        "    \"age\", \"fnlwgt\", \"education-num\", \"capital-gain\", \"capital-loss\", \"hours-per-week\"\n",
        "]\n",
        "scaler = StandardScaler()\n",
        "\n",
        "# Separate features and target variable\n",
        "X = data.drop(\"income\", axis=1)\n",
        "y = data[\"income\"]\n",
        "\n",
        "X[numerical_features] = scaler.fit_transform(X[numerical_features])"
      ],
      "metadata": {
        "id": "wqJUnZOlkYmK"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Prepare training and testing sets\n",
        "\n",
        "Splitting data into training and testing sets is essential for building a reliable machine learning model. Here's why:\n",
        "\n",
        "Training the model: The training set is used to teach the model by allowing it to find patterns in the data. The model learns to predict the target variable (like \"income\") based on the features (like age, hours worked, etc.).\n",
        "\n",
        "Testing the model: The testing set is kept separate to check how well the model performs on new, unseen data. This gives a realistic idea of how the model will work in real-world situations.\n",
        "\n",
        "Preventing overfitting: If the model memorizes the training data too well, it might become too specific to that data, a problem called overfitting. Overfitting means the model performs well on the training set but poorly on new data because it doesn't generalize well. By using a testing set, we can check if the model overfits.\n",
        "\n",
        "This way, the model is evaluated on both known and new data, ensuring it can generalize its predictions."
      ],
      "metadata": {
        "id": "16jLUa3kd1Q-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Separate features and target variable\n",
        "X = data.drop(\"income\", axis=1)\n",
        "y = data[\"income\"]"
      ],
      "metadata": {
        "id": "myWVg56hkXOm"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Split the data into training and testing sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
      ],
      "metadata": {
        "id": "OjC8_3zgkaAu"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Print the shapes of the training and testing sets\n",
        "print(\"\\nShape of training set:\")\n",
        "print(\"Features:\", X_train.shape)\n",
        "print(\"Target:\", y_train.shape)\n",
        "print(\"\\nShape of testing set:\")\n",
        "print(\"Features:\", X_test.shape)\n",
        "print(\"Target:\", y_test.shape)\n",
        "print(\"\\nExample of the data: \")\n",
        "print(X_train.head())"
      ],
      "metadata": {
        "id": "lpDzFpOpkbTu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6e225c3d-d146-4dbe-809d-1e14eecf3d18"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Shape of training set:\n",
            "Features: (26048, 14)\n",
            "Target: (26048,)\n",
            "\n",
            "Shape of testing set:\n",
            "Features: (6513, 14)\n",
            "Target: (6513,)\n",
            "\n",
            "Example of the data: \n",
            "       age  workclass  fnlwgt  education  education-num  marital-status  \\\n",
            "5514    33          1  198183          9             13               4   \n",
            "19777   36          3   86459          8             11               2   \n",
            "10781   58          5  203039          6              5               5   \n",
            "32240   21          3  180190          8             11               2   \n",
            "9876    27          3  279872         15             10               0   \n",
            "\n",
            "       occupation  relationship  race  sex  capital-gain  capital-loss  \\\n",
            "5514            9             1     4    0             0             0   \n",
            "19777           3             0     4    1             0          1887   \n",
            "10781           2             1     4    1             0             0   \n",
            "32240           4             0     4    1             0             0   \n",
            "9876            7             1     4    1             0             0   \n",
            "\n",
            "       hours-per-week  native-country  \n",
            "5514               50              38  \n",
            "19777              50              38  \n",
            "10781              40              38  \n",
            "32240              46              38  \n",
            "9876               40              38  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Conclusion\n",
        "\n",
        "Congratulations! You reached the end of the first lab. You have successfully:\n",
        "\n",
        "\n",
        "*   Retrieved a dataset\n",
        "*   Inspected for oddities\n",
        "*   Cleaned the data\n",
        "*   Split into training & test\n"
      ],
      "metadata": {
        "id": "bgJm7UY4ljjK"
      }
    }
  ]
}