NN_overfitting_is_good.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Neural Networks Overfitting is good",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nadirbelarouci/NN-overfitting-isgood/blob/master/NN_overfitting_is_good.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "RHrZkSgt4jwb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "#Neural Networks Overfitting is GOOD \n",
        "\n",
        "Overffiting may caused a lot of problems to many machine learning developers, and mostly all articles discuess how to avoid it, however in this notebook we will see what if overfitting is actually good in some senarios.\n",
        "\n",
        "The idea is to use overfetting to memorize bytes of data into a neural network, the ordinary way that we do this is to map each index to one byte.\n",
        "Our apprach here is to map multipe indices to the same byte.\n",
        "\n",
        "what do we get from this ? \n",
        "- a lossless compression algorithm\n",
        "- a read only data: any changes to the weights of the neural network will impact the data, thus assuring its integrity\n",
        "- an abstraction to the type of data\n",
        "\n",
        "The following is an example demonstrating the goal of the idea.\n",
        "We will be using a neural network of 3 layers:\n",
        "- the intput layer is an index mapping a byte in a 32 bit format \n",
        "- a hidden layer of size 64\n",
        "- the output layer which represents the byte itself."
      ]
    },
    {
      "metadata": {
        "id": "ge1Ge7fBnRj5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DzITXZfl9miA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Preparing the data\n",
        "- The data is represented as long enough string.\n",
        "- The string will be represented by `x_train` and `y_train` vectors\n",
        "- Each training data features are  represented as a 32 bit format of the index \n",
        "- Each training example label is the rank of the letter in the alphabet: A is 0, B is 1 etc and SPACE is 27.\n",
        "- In a typical senario, the size of the possible labels is 255.\n",
        "\n",
        " "
      ]
    },
    {
      "metadata": {
        "id": "D1Pi5grLRyBz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#CONSTANTS \n",
        "INPUT_LAYER_SIZE = 32\n",
        "LABELS_SIZE = 27\n",
        "\n",
        "def prepare_data():\n",
        "  random_text = \"BASED ON YOUR INPUT GET A RANDOM ALPHA NUHE RANDOM STRING GENERATOR CREATES A SERIES OF AND LETTERS ALRBRFNKPZNJEBPCZ FHGDQPZLHE HELLO THERE HOW ARE YOU DOING TODAY AND BLA BLA BLA\"\n",
        "  \n",
        "  length = len(random_text)\n",
        "  \n",
        "  x_train = np.zeros((INPUT_LAYER_SIZE,length)) \n",
        "  y_train = np.zeros(length)                   \n",
        "  \n",
        "  for i in range(length):  \n",
        "    x = [int(x) for x in bin(i)[2:]] # convert the index to a list of bits\n",
        "    x = np.pad(x,(INPUT_LAYER_SIZE-len(x),0),'constant') # add 0 to the lists, size = INPUT_LAYER_SIZE\n",
        "    \n",
        "    x_train[:,i] = x\n",
        "    y_train[i] = LABELS_SIZE-1 if random_text[i]==' ' else ord(random_text[i]) - 65 # map each char to its rank in the alphabet except the space\n",
        "   \n",
        "    \n",
        "  x_train = x_train.T\n",
        "  y_train = y_train.T \n",
        "  \n",
        "  return x_train, y_train"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "colab_type": "code",
        "id": "BWO1ARPnn8RZ",
        "outputId": "cbcdc340-0213-40d1-92fa-498936a01c45",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "x_train, y_train = prepare_data()\n",
        "print(x_train.shape,y_train.shape)\n"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(180, 32) (180,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "tO2c4hmaSfPE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Creating the neural network as described above \n",
        "model = tf.keras.models.Sequential([\n",
        "  tf.keras.layers.Flatten(),\n",
        "  tf.keras.layers.Dense(64, activation=tf.nn.relu),\n",
        "  tf.keras.layers.Dropout(0.2),\n",
        "  tf.keras.layers.Dense(LABELS_SIZE, activation=tf.nn.softmax)\n",
        "])\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OrrBht4qT6h2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Train neural network until we overfitte the model\n",
        "score = 0.0\n",
        "while score < 1.0:\n",
        "  history = model.fit(x_train, y_train, epochs=5,verbose=False)\n",
        "  l = len(history.history['acc'])-1\n",
        "  score = history.history['acc'][l];"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pt-WPlxQUTwN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "81756c49-21de-4e75-ce81-d52ce1071b68"
      },
      "cell_type": "code",
      "source": [
        "score = model.evaluate(x_train, y_train)\n",
        "print(score)\n"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "180/180 [==============================] - 0s 94us/step\n",
            "[0.2428318397866355, 1.0]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EJ8ODKZswW4h",
        "colab_type": "code",
        "outputId": "8afad4c3-8427-4c74-cb0f-74e9c4dbd18a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "cell_type": "code",
      "source": [
        "\n",
        "# get the predictions     \n",
        "predictions = np.argmax(model.predict(x_train),axis=1) \n",
        "# convert them to the appropriate string format \n",
        "to_char = lambda x: \" \" if x == LABELS_SIZE-1 else chr(x+65)\n",
        "text = \"\".join(list(map(to_char, predictions)))\n",
        "\n",
        "print (text)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "BASED ON YOUR INPUT GET A RANDOM ALPHA NUHE RANDOM STRING GENERATOR CREATES A SERIES OF AND LETTERS ALRBRFNKPZNJEBPCZ FHGDQPZLHE HELLO THERE HOW ARE YOU DOING TODAY AND BLA BLA BLA\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "vETvIDmK_64e",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Eventough the proof of concept work, the neural network can memorize the data, this example needs optimisation and don't answer to the real problem of compressing data since the actual size of the neural network weights is way bigger than the actual size of the string which is (128 bytes).\n",
        "\n",
        "Thank you for reading. \n",
        "\n",
        "```\n",
        "Author: Nadir Belarouci\n",
        "Twitter: @nadirbelarouci\n",
        "Github: nadirbelarouci\n",
        "```\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "s1cQFDZfBITE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}