t81_558_class_02_2_pandas_cat.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_02_2_pandas_cat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 2: Python for Machine Learning**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 2 Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* Part 2.1: Introduction to Pandas [[Video]](https://www.youtube.com/watch?v=bN4UuCBdpZc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_1_python_pandas.ipynb)\n",
    "* **Part 2.2: Categorical Values** [[Video]](https://www.youtube.com/watch?v=4a1odDpG0Ho&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_2_pandas_cat.ipynb)\n",
    "* Part 2.3: Grouping, Sorting, and Shuffling in Python Pandas [[Video]](https://www.youtube.com/watch?v=YS4wm5gD8DM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_3_pandas_grouping.ipynb)\n",
    "* Part 2.4: Using Apply and Map in Pandas for Keras [[Video]](https://www.youtube.com/watch?v=XNCEZ4WaPBY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_4_pandas_functional.ipynb)\n",
    "* Part 2.5: Feature Engineering in Pandas for Deep Learning in Keras [[Video]](https://www.youtube.com/watch?v=BWPTj4_Mi9E&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_02_5_pandas_features.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2.2: Categorical and Continuous Values\n",
    "\n",
    "Neural networks require their input to be a fixed number of columns. This input format is very similar to spreadsheet data; it must be entirely numeric. It is essential to represent the data so that the neural network can train from it. Before we look at specific ways to preprocess data, it is important to consider four basic types of data, as defined by [[Cite:stevens1946theory]](http://psychology.okstate.edu/faculty/jgrice/psyc3214/Stevens_FourScales_1946.pdf). Statisticians commonly refer to as the [levels of measure](https://en.wikipedia.org/wiki/Level_of_measurement):\n",
    "\n",
    "* Character Data (strings)\n",
    "    * **Nominal** - Individual discrete items, no order. For example, color, zip code, and shape.\n",
    "    * **Ordinal** - Individual distinct items have an implied order. For example, grade level, job title, Starbucks(tm) coffee size (tall, vente, grande) \n",
    "* Numeric Data\n",
    "    * **Interval** - Numeric values, no defined start.  For example, temperature. You would never say, \"yesterday was twice as hot as today.\"\n",
    "    * **Ratio** - Numeric values, clearly defined start.  For example, speed. You could say, \"The first car is going twice as fast as the second.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding Continuous Values\n",
    "\n",
    "One common transformation is to normalize the inputs.  It is sometimes valuable to normalize numeric inputs in a standard form so that the program can easily compare these two values.  Consider if a friend told you that he received a 10-dollar discount.  Is this a good deal?  Maybe.  But the cost is not normalized.  If your friend purchased a car, the discount is not that good.  If your friend bought lunch, this is an excellent discount!\n",
    "\n",
    "Percentages are a prevalent form of normalization.  If your friend tells you they got 10% off, we know that this is a better discount than 5%.  It does not matter how much the purchase price was.  One widespread machine learning normalization is the Z-Score:\n",
    "\n",
    "$$ z = \\frac{x - \\mu}{\\sigma} $$\n",
    "\n",
    "To calculate the Z-Score, you also need to calculate the mean(&mu; or $\\bar{x}$) and the standard deviation (&sigma;).  You can calculate the mean with this equation:\n",
    "\n",
    "$$ \\mu = \\bar{x} = \\frac{x_1+x_2+\\cdots +x_n}{n} $$\n",
    "\n",
    "The standard deviation is calculated as follows:\n",
    "\n",
    "$$ \\sigma = \\sqrt{\\frac{1}{N} \\sum_{i=1}^N (x_i - \\mu)^2} $$\n",
    "\n",
    "The following Python code replaces the mpg with a z-score.  Cars with average MPG will be near zero, above zero is above average, and below zero is below average.  Z-Scores more that 3 above or below are very rare; these are outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mpg</th>\n",
       "      <th>cylinders</th>\n",
       "      <th>displacement</th>\n",
       "      <th>...</th>\n",
       "      <th>year</th>\n",
       "      <th>origin</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.706439</td>\n",
       "      <td>8</td>\n",
       "      <td>307.0</td>\n",
       "      <td>...</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>chevrolet chevelle malibu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.090751</td>\n",
       "      <td>8</td>\n",
       "      <td>350.0</td>\n",
       "      <td>...</td>\n",
       "      <td>70</td>\n",
       "      <td>1</td>\n",
       "      <td>buick skylark 320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>396</th>\n",
       "      <td>0.574601</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>...</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>ford ranger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>397</th>\n",
       "      <td>0.958913</td>\n",
       "      <td>4</td>\n",
       "      <td>119.0</td>\n",
       "      <td>...</td>\n",
       "      <td>82</td>\n",
       "      <td>1</td>\n",
       "      <td>chevy s-10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>398 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          mpg  cylinders  displacement  ...  year  origin  \\\n",
       "0   -0.706439          8         307.0  ...    70       1   \n",
       "1   -1.090751          8         350.0  ...    70       1   \n",
       "..        ...        ...           ...  ...   ...     ...   \n",
       "396  0.574601          4         120.0  ...    82       1   \n",
       "397  0.958913          4         119.0  ...    82       1   \n",
       "\n",
       "                          name  \n",
       "0    chevrolet chevelle malibu  \n",
       "1            buick skylark 320  \n",
       "..                         ...  \n",
       "396                ford ranger  \n",
       "397                 chevy s-10  \n",
       "\n",
       "[398 rows x 9 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "pd.set_option('display.max_columns', 7)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "df['mpg'] = zscore(df['mpg'])\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Categorical Values as Dummies\n",
    "The traditional means of encoding categorical values is to make them dummy variables.  This technique is also called one-hot-encoding.  Consider the following data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>area</th>\n",
       "      <th>...</th>\n",
       "      <th>retail_dense</th>\n",
       "      <th>crime</th>\n",
       "      <th>product</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>...</td>\n",
       "      <td>0.492126</td>\n",
       "      <td>0.071100</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>c</td>\n",
       "      <td>...</td>\n",
       "      <td>0.342520</td>\n",
       "      <td>0.400809</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>qp</td>\n",
       "      <td>c</td>\n",
       "      <td>...</td>\n",
       "      <td>0.598425</td>\n",
       "      <td>0.117803</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>...</td>\n",
       "      <td>0.539370</td>\n",
       "      <td>0.451973</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id job area  ...  retail_dense     crime  product\n",
       "0        1  vv    c  ...      0.492126  0.071100        b\n",
       "1        2  kd    c  ...      0.342520  0.400809        c\n",
       "...    ...  ..  ...  ...           ...       ...      ...\n",
       "1998  1999  qp    c  ...      0.598425  0.117803        c\n",
       "1999  2000  pe    c  ...      0.539370  0.451973        c\n",
       "\n",
       "[2000 rows x 14 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "pd.set_option('display.max_columns', 7)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *area* column is not numeric, so you must encode it with one-hot encoding. We display the number of areas and individual values. There are just four values in the *area* categorical variable in this case.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of areas: 4\n",
      "Areas: ['c', 'd', 'a', 'b']\n"
     ]
    }
   ],
   "source": [
    "areas = list(df['area'].unique())\n",
    "print(f'Number of areas: {len(areas)}')\n",
    "print(f'Areas: {areas}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are four unique values in the *area* column.  To encode these dummy variables, we would use four columns, each representing one of the areas.  For each row, one column would have a value of one, the rest zeros.  For this reason, this type of encoding is sometimes called one-hot encoding.  The following code shows how you might encode the values \"a\" through \"d.\"  The value A becomes [1,0,0,0] and the value B becomes [0,1,0,0]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   area_a  area_b  area_c  area_d\n",
      "0       1       0       0       0\n",
      "1       0       1       0       0\n",
      "2       0       0       1       0\n",
      "3       0       0       0       1\n"
     ]
    }
   ],
   "source": [
    "dummies = pd.get_dummies(['a','b','c','d'],prefix='area')\n",
    "print(dummies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now encode the actual column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    area_a  area_b  area_c  area_d\n",
      "0        0       0       1       0\n",
      "1        0       0       1       0\n",
      "..     ...     ...     ...     ...\n",
      "8        0       0       1       0\n",
      "9        1       0       0       0\n",
      "\n",
      "[10 rows x 4 columns]\n"
     ]
    }
   ],
   "source": [
    "dummies = pd.get_dummies(df['area'],prefix='area')\n",
    "print(dummies[0:10]) # Just show the first 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the new dummy/one hot encoded values to be of any use, they must be merged back into the data set.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([df,dummies],axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To encode the *area* column, we use the following code. Note that it is necessary to merge these dummies back into the data frame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>area</th>\n",
       "      <th>income</th>\n",
       "      <th>area_a</th>\n",
       "      <th>area_b</th>\n",
       "      <th>area_c</th>\n",
       "      <th>area_d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>c</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>c</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>kl</td>\n",
       "      <td>d</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>1996</td>\n",
       "      <td>vv</td>\n",
       "      <td>c</td>\n",
       "      <td>51017.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>1997</td>\n",
       "      <td>kl</td>\n",
       "      <td>d</td>\n",
       "      <td>26576.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>1998</td>\n",
       "      <td>kl</td>\n",
       "      <td>d</td>\n",
       "      <td>28595.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>qp</td>\n",
       "      <td>c</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>pe</td>\n",
       "      <td>c</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id job area   income  area_a  area_b  area_c  area_d\n",
       "0        1  vv    c  50876.0       0       0       1       0\n",
       "1        2  kd    c  60369.0       0       0       1       0\n",
       "2        3  pe    c  55126.0       0       0       1       0\n",
       "3        4  11    c  51690.0       0       0       1       0\n",
       "4        5  kl    d  28347.0       0       0       0       1\n",
       "...    ...  ..  ...      ...     ...     ...     ...     ...\n",
       "1995  1996  vv    c  51017.0       0       0       1       0\n",
       "1996  1997  kl    d  26576.0       0       0       0       1\n",
       "1997  1998  kl    d  28595.0       0       0       0       1\n",
       "1998  1999  qp    c  67949.0       0       0       1       0\n",
       "1999  2000  pe    c  61467.0       0       0       1       0\n",
       "\n",
       "[2000 rows x 8 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.set_option('display.max_columns', 0)\n",
    "pd.set_option('display.max_rows', 10)\n",
    "\n",
    "display(df[['id','job','area','income','area_a',\n",
    "                  'area_b','area_c','area_d']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, you will remove the original column *area* because the goal is to get the data frame to be entirely numeric for the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>income</th>\n",
       "      <th>area_a</th>\n",
       "      <th>area_b</th>\n",
       "      <th>area_c</th>\n",
       "      <th>area_d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>qp</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>pe</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id job   income  area_a  area_b  area_c  area_d\n",
       "0        1  vv  50876.0       0       0       1       0\n",
       "1        2  kd  60369.0       0       0       1       0\n",
       "...    ...  ..      ...     ...     ...     ...     ...\n",
       "1998  1999  qp  67949.0       0       0       1       0\n",
       "1999  2000  pe  61467.0       0       0       1       0\n",
       "\n",
       "[2000 rows x 7 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.set_option('display.max_columns', 0)\n",
    "pd.set_option('display.max_rows', 5)\n",
    "\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "display(df[['id','job','income','area_a',\n",
    "                  'area_b','area_c','area_d']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing the First Level\n",
    "\n",
    "The **pd.concat** function also includes a parameter named *drop_first*, which specifies whether to get k-1 dummies out of k categorical levels by removing the first level. Why would you want to remove the first level, in this case, *area_a*? This technique provides a more efficient encoding by using the ordinarily unused encoding of [0,0,0]. We encode the *area* to just three columns and map the categorical value of *a* to [0,0,0]. The following code demonstrates this technique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   area_b  area_c  area_d\n",
      "0       0       0       0\n",
      "1       1       0       0\n",
      "2       0       1       0\n",
      "3       0       0       1\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd \n",
    "\n",
    "dummies = pd.get_dummies(['a','b','c','d'],prefix='area', drop_first=True)\n",
    "print(dummies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from the above data, the *area_a* column is missing, as it **get_dummies** replaced it by the encoding of [0,0,0]. The following code shows how to apply this technique to a dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>job</th>\n",
       "      <th>income</th>\n",
       "      <th>area_b</th>\n",
       "      <th>area_c</th>\n",
       "      <th>area_d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>vv</td>\n",
       "      <td>50876.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>kd</td>\n",
       "      <td>60369.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>pe</td>\n",
       "      <td>55126.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>51690.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>kl</td>\n",
       "      <td>28347.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>1996</td>\n",
       "      <td>vv</td>\n",
       "      <td>51017.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>1997</td>\n",
       "      <td>kl</td>\n",
       "      <td>26576.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>1998</td>\n",
       "      <td>kl</td>\n",
       "      <td>28595.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>1999</td>\n",
       "      <td>qp</td>\n",
       "      <td>67949.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>2000</td>\n",
       "      <td>pe</td>\n",
       "      <td>61467.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2000 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id job   income  area_b  area_c  area_d\n",
       "0        1  vv  50876.0       0       1       0\n",
       "1        2  kd  60369.0       0       1       0\n",
       "2        3  pe  55126.0       0       1       0\n",
       "3        4  11  51690.0       0       1       0\n",
       "4        5  kl  28347.0       0       0       1\n",
       "...    ...  ..      ...     ...     ...     ...\n",
       "1995  1996  vv  51017.0       0       1       0\n",
       "1996  1997  kl  26576.0       0       0       1\n",
       "1997  1998  kl  28595.0       0       0       1\n",
       "1998  1999  qp  67949.0       0       1       0\n",
       "1999  2000  pe  61467.0       0       1       0\n",
       "\n",
       "[2000 rows x 6 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd \n",
    "\n",
    "# Read the dataset\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
    "    na_values=['NA','?'])\n",
    "\n",
    "# encode the area column as dummy variables\n",
    "dummies = pd.get_dummies(df['area'], drop_first=True, prefix='area')\n",
    "df = pd.concat([df,dummies],axis=1)\n",
    "df.drop('area', axis=1, inplace=True)\n",
    "\n",
    "# display the encoded dataframe\n",
    "pd.set_option('display.max_columns', 0)\n",
    "pd.set_option('display.max_rows', 10)\n",
    "\n",
    "display(df[['id','job','income',\n",
    "                  'area_b','area_c','area_d']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Target Encoding for Categoricals\n",
    "\n",
    "Target encoding is a popular technique for Kaggle competitions. Target encoding can sometimes increase the predictive power of a machine learning model. However, it also dramatically increases the risk of overfitting. Because of this risk, you must take care of using this method.  \n",
    "\n",
    "Generally, target encoding can only be used on a categorical feature when the output of the machine learning model is numeric (regression).\n",
    "\n",
    "The concept of target encoding is straightforward. For each category, we calculate the average target value for that category. Then to encode, we substitute the percent corresponding to the category that the categorical value has. Unlike dummy variables, where you have a column for each category with target encoding, the program only needs a single column. In this way, target coding is more efficient than dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cont_9</th>\n",
       "      <th>cat_0</th>\n",
       "      <th>cat_1</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.505457</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60.906654</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.339096</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24.058962</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32.713906</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>85.913749</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>66.609021</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>54.116221</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.901382</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>73.374830</td>\n",
       "      <td>cat</td>\n",
       "      <td>tiger</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      cont_9 cat_0  cat_1  y\n",
       "0  11.505457   dog   wolf  1\n",
       "1  60.906654   dog   wolf  0\n",
       "2  13.339096   dog   wolf  1\n",
       "3  24.058962   dog   wolf  1\n",
       "4  32.713906   dog   wolf  1\n",
       "5  85.913749   cat   wolf  1\n",
       "6  66.609021   cat   wolf  0\n",
       "7  54.116221   cat   wolf  0\n",
       "8   2.901382   cat   wolf  0\n",
       "9  73.374830   cat  tiger  0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a small sample dataset\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "np.random.seed(43)\n",
    "df = pd.DataFrame({\n",
    "    'cont_9': np.random.rand(10)*100,\n",
    "    'cat_0': ['dog'] * 5 + ['cat'] * 5,\n",
    "    'cat_1': ['wolf'] * 9 + ['tiger'] * 1,\n",
    "    'y': [1, 0, 1, 1, 1, 1, 0, 0, 0, 0]\n",
    "})\n",
    "\n",
    "pd.set_option('display.max_columns', 0)\n",
    "pd.set_option('display.max_rows', 0)\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to change them to a number rather than creating dummy variables for \"dog\" and \"cat,\" we would like to change them to a number. We could use 0 for a cat and 1 for a dog. However, we can encode more information than just that. The simple 0 or 1 would also only work for one animal. Consider what the mean target value is for cat and dog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'cat': 0.2, 'dog': 0.8}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "means0 = df.groupby('cat_0')['y'].mean().to_dict()\n",
    "means0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The danger is that we are now using the target value ($y$) for training. This technique will potentially lead to overfitting. The possibility of overfitting is even greater if a small number of a particular category. To prevent this from happening, we use a weighting factor. The stronger the weight, the more categories with fewer values will tend towards the overall average of $y$. You can perform this calculation as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['y'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can implement target encoding as follows.  For more information on Target Encoding, refer to the article [\"Target Encoding Done the Right Way\"](https://maxhalford.github.io/blog/target-encoding/), that I based this code upon."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_smooth_mean(df1, df2, cat_name, target, weight):\n",
    "    # Compute the global mean\n",
    "    mean = df[target].mean()\n",
    "\n",
    "    # Compute the number of values and the mean of each group\n",
    "    agg = df.groupby(cat_name)[target].agg(['count', 'mean'])\n",
    "    counts = agg['count']\n",
    "    means = agg['mean']\n",
    "\n",
    "    # Compute the \"smoothed\" means\n",
    "    smooth = (counts * means + weight * mean) / (counts + weight)\n",
    "\n",
    "    # Replace each value by the according smoothed mean\n",
    "    if df2 is None:\n",
    "        return df1[cat_name].map(smooth)\n",
    "    else:\n",
    "        return df1[cat_name].map(smooth),df2[cat_name].map(smooth.to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code encodes these two categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cont_9</th>\n",
       "      <th>cat_0</th>\n",
       "      <th>cat_1</th>\n",
       "      <th>y</th>\n",
       "      <th>cat_0_enc</th>\n",
       "      <th>cat_1_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11.505457</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60.906654</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.339096</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24.058962</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32.713906</td>\n",
       "      <td>dog</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>85.913749</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>1</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>66.609021</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>54.116221</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.901382</td>\n",
       "      <td>cat</td>\n",
       "      <td>wolf</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.535714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>73.374830</td>\n",
       "      <td>cat</td>\n",
       "      <td>tiger</td>\n",
       "      <td>0</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.416667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      cont_9 cat_0  cat_1  y  cat_0_enc  cat_1_enc\n",
       "0  11.505457   dog   wolf  1       0.65   0.535714\n",
       "1  60.906654   dog   wolf  0       0.65   0.535714\n",
       "2  13.339096   dog   wolf  1       0.65   0.535714\n",
       "3  24.058962   dog   wolf  1       0.65   0.535714\n",
       "4  32.713906   dog   wolf  1       0.65   0.535714\n",
       "5  85.913749   cat   wolf  1       0.35   0.535714\n",
       "6  66.609021   cat   wolf  0       0.35   0.535714\n",
       "7  54.116221   cat   wolf  0       0.35   0.535714\n",
       "8   2.901382   cat   wolf  0       0.35   0.535714\n",
       "9  73.374830   cat  tiger  0       0.35   0.416667"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "WEIGHT = 5\n",
    "df['cat_0_enc'] = calc_smooth_mean(df1=df, df2=None, \n",
    "    cat_name='cat_0', target='y', weight=WEIGHT)\n",
    "df['cat_1_enc'] = calc_smooth_mean(df1=df, df2=None, \n",
    "    cat_name='cat_1', target='y', weight=WEIGHT)\n",
    "\n",
    "pd.set_option('display.max_columns', 0)\n",
    "pd.set_option('display.max_rows', 0)\n",
    "\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding Categorical Values as Ordinal\n",
    "\n",
    "Typically categoricals will be encoded as dummy variables. However, there might be other techniques to convert categoricals to numeric. Any time there is an order to the categoricals, a number should be used. Consider if you had a categorical that described the current education level of an individual.   \n",
    "\n",
    "* Kindergarten (0)\n",
    "* First Grade (1)\n",
    "* Second Grade (2)\n",
    "* Third Grade (3)\n",
    "* Fourth Grade (4)\n",
    "* Fifth Grade (5)\n",
    "* Sixth Grade (6)\n",
    "* Seventh Grade (7)\n",
    "* Eighth Grade (8)\n",
    "* High School Freshman (9)\n",
    "* High School Sophomore (10)\n",
    "* High School Junior (11)\n",
    "* High School Senior (12)\n",
    "* College Freshman (13)\n",
    "* College Sophomore (14)\n",
    "* College Junior (15)\n",
    "* College Senior (16)\n",
    "* Graduate Student (17)\n",
    "* PhD Candidate (18)\n",
    "* Doctorate (19)\n",
    "* Post Doctorate (20)\n",
    "\n",
    "The above list has 21 levels and would take 21 dummy variables to encode. However, simply encoding this to dummies would lose the order information. Perhaps the most straightforward approach would be to simply number them and assign the category a single number equal to the value in the parenthesis above. However, we might be able to do even better. A graduate student is likely more than a year so you might increase one value.  \n",
    "\n",
    "## High Cardinality Categorical\n",
    "\n",
    "If there were many, perhaps thousands or tens of thousands, then one-hot encoding is no longer a good choice. We call these cases high cardinality categorical. We generally encode such values with an embedding layer, which we will discuss later when introducing natural language processing (NLP)."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}