Linear and Logistic Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Credit to [Rohan Varma](https://github.com/rohan-varma) for these great explanations.**\n",
    "\n",
    "Linear regression is probably the first machine learning algorithm that most people learn when starting off in this field. Learning this model is a great way to get introduced to the idea of supervised learning. \n",
    "\n",
    "We have some (input, output) pairs which we denote as $ (x_i, y_i) $ and we have $n$ of these, so $i \\in [1...n]$. We want to learn a function $f: x \\rightarrow{} y$ that maps inputs to outputs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inputs and Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to learn a function $ f: x \\rightarrow{} y$ that maps information about a house to the house's price prediction. With linear regression, our function $f$ is just a ** linear combination ** of our inputs. That means our output is just the sum of our inputs, but each of our inputs are weighted by some value: \n",
    "\n",
    "$$f(x) = w_1 x_1 + w_2 x_2 + ... w_{13}x_{13} + b = \\sum_{j=1}^{13} w_j x_j + b$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(dtype=tf.float32, shape=[None, 13])\n",
    "y = tf.placeholder(dtype=tf.float32, shape=[None, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss Functions and Optimizations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will initialize this linear model with initially random weights. As a result, our model won't be able to predict house prices very well at all. Learning is the process of adjusting these parameters so that our model's accuracy increases. In order to do this, we need to mathematically quantify how \"bad\" our model is currently. We can do this by calculating how off each prediction is from the actual value: \n",
    "\n",
    "$$ L = \\frac{1}{N} \\sum_{i=1}^{N} (y_i - f(x_i))^2 $$\n",
    "\n",
    "If we take the derivative of this function with respect to each of the weights $w$, we will know how much to \"adjust\" each weight $w$ by in order to make our function more accurate. This is an algorithm called ** gradient descent **. \n",
    "\n",
    "If you know some multivariable calculus, you can determine that the derivative with respect to the $i$th weight is $$ \\frac{dL}{dw_i} = \\frac{-2}{N} \\sum_{i=1}^{N} (y_i - f(x_i))x_i $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.random_normal([13, 1]))\n",
    "b = tf.Variable(tf.random_normal([1]))\n",
    "y_pred = tf.matmul(x, W) + b\n",
    "loss = tf.reduce_mean(tf.square(y - y_pred))\n",
    "opt = tf.train.GradientDescentOptimizer(learning_rate = .5).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "from sklearn.preprocessing import normalize # to standardize our data\n",
    "from sklearn.model_selection import train_test_split\n",
    "data, targets = load_boston(True)\n",
    "data = normalize(data)\n",
    "targets = targets.reshape((targets.shape[0],1)) # reshape targets to follow our variables\n",
    "X_train, X_test, y_train, y_test = train_test_split(data, targets, \n",
    "                                                    test_size = 0.3, random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the part where we start training our model. We'll feeding our training set and labels into our two placeholders and then evaluate our optimizer object which minimizes the MSE loss function. We repeat this for a set number of iterations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 492.783\n",
      "Loss: 63.8958\n",
      "Loss: 61.8711\n",
      "Loss: 60.3902\n",
      "Loss: 59.1143\n",
      "Loss: 57.9496\n",
      "Loss: 56.8632\n",
      "Loss: 55.8401\n",
      "Loss: 54.872\n",
      "Loss: 53.9535\n",
      "Loss: 53.0807\n",
      "Loss: 52.2507\n",
      "Loss: 51.461\n",
      "Loss: 50.7093\n",
      "Loss: 49.9936\n",
      "Loss: 49.3122\n",
      "Loss: 48.6632\n",
      "Loss: 48.0451\n",
      "Loss: 47.4562\n",
      "Loss: 46.8952\n"
     ]
    }
   ],
   "source": [
    "numEpochs = 5000\n",
    "sess = tf.InteractiveSession()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "for i in range(numEpochs):\n",
    "    sess.run(opt, feed_dict={x: X_train, y: y_train})\n",
    "    if (i % 250 == 0):\n",
    "        print ('Loss:', loss.eval(feed_dict={x: X_train, y: y_train}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "House prices are off by an average of 4.40974366037 thousand dollars.\n"
     ]
    }
   ],
   "source": [
    "predictions = sess.run(y_pred, feed_dict={x: X_test})\n",
    "differences = predictions.flatten() - y_test.flatten()\n",
    "differences = [abs(x) for x in differences]\n",
    "print (\"House prices are off by an average of\", np.mean(differences), \"thousand dollars.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Visualization "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADrBJREFUeJzt3WGIZWd9x/Hvr5sEJUqza6bDYkzHwJIioW7aIVgMYt1E\nYlPctS+WBFqmJbB9YSVCod36otVCYVtasS+KsFXbkcbo1pjuYsSyTiNWkJjZGDXJJl0bdjFhd2dM\nDBoLlei/L+YEp+vM3nvn3tk798n3A8s55znnzvk/PMxvzjxzztlUFZKkyfcL4y5AkjQaBrokNcJA\nl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEZddypNdffXVNTMzcylPKUkT78SJE9+rqqle\nx/UM9CTXA59Z1XQd8OfAJ7v2GeA0sL+qvn+xrzUzM8Pi4mKvU0qSVklypp/jek65VNVTVbW7qnYD\nvw78D3A/cBBYqKpdwEK3LUkak0Hn0PcA/11VZ4C9wHzXPg/sG2VhkqTBDBrodwD3duvTVXW2Wz8H\nTI+sKknSwPoO9CRXAO8G/vXCfbXyDt4138Ob5ECSxSSLy8vLGy5UknRxg1yhvwt4pKrOd9vnk+wE\n6JZLa32oqg5X1WxVzU5N9fwjrSRpgwYJ9Dv52XQLwDFgrlufA46OqihJ0uD6CvQkVwK3Ap9b1XwI\nuDXJKeCWbluSNCZ9PVhUVT8CXndB23Os3PUiSdoCfPRfkhpxSR/9H8bMwQfGdu7Th24f27klqV9e\noUtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6\nJDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmN6CvQk1yV5LNJnkxyMslvJNmR5HiSU91y+2YX\nK0laX79X6H8PfLGqfgV4M3ASOAgsVNUuYKHbliSNSc9AT/KLwNuAjwNU1Y+r6gVgLzDfHTYP7Nus\nIiVJvfVzhf5GYBn4pyTfSPKxJFcC01V1tjvmHDC9WUVKknrrJ9AvA34N+GhV3Qj8iAumV6qqgFrr\nw0kOJFlMsri8vDxsvZKkdfQT6M8Az1TVQ932Z1kJ+PNJdgJ0y6W1PlxVh6tqtqpmp6amRlGzJGkN\nPQO9qs4B301yfde0B3gCOAbMdW1zwNFNqVCS1JfL+jzufcA9Sa4Angb+gJUfBkeS3AWcAfZvTomS\npH70FehV9Sgwu8auPaMtR5K0UT4pKkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqE\ngS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjo\nktQIA12SGnFZPwclOQ38EPgJ8FJVzSbZAXwGmAFOA/ur6vubU6YkqZdBrtB/s6p2V9Vst30QWKiq\nXcBCty1JGpNhplz2AvPd+jywb/hyJEkb1W+gF/ClJCeSHOjapqvqbLd+DpgeeXWSpL71NYcO3FxV\nzyb5JeB4kidX76yqSlJrfbD7AXAA4Nprrx2qWEnS+vq6Qq+qZ7vlEnA/cBNwPslOgG65tM5nD1fV\nbFXNTk1NjaZqSdLP6RnoSa5M8tqX14F3Ao8Bx4C57rA54OhmFSlJ6q2fKZdp4P4kLx//qar6YpKH\ngSNJ7gLOAPs3r0xJUi89A72qngbevEb7c8CezShKkjQ4nxSVpEYY6JLUCANdkhphoEtSIwx0SWqE\ngS5JjTDQJakRBrokNcJAl6RGGOiS1Ih+X5/7ijZz8IGxnPf0odvHcl5Jk8krdElqhIEuSY0w0CWp\nEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmN6DvQk2xL8o0kn++2dyQ5nuRUt9y+eWVK\nknoZ5Ar9buDkqu2DwEJV7QIWum1J0pj0FehJrgFuBz62qnkvMN+tzwP7RluaJGkQ/V6hfwT4E+Cn\nq9qmq+pst34OmB5lYZKkwfQM9CS/DSxV1Yn1jqmqAmqdzx9IsphkcXl5eeOVSpIuqp8r9LcC705y\nGvg08I4k/wKcT7IToFsurfXhqjpcVbNVNTs1NTWisiVJF+oZ6FX1Z1V1TVXNAHcA/1FVvwscA+a6\nw+aAo5tWpSSpp2HuQz8E3JrkFHBLty1JGpOB/gu6qvoy8OVu/Tlgz+hLkiRthE+KSlIjDHRJaoSB\nLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS\n1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRvQM9CSvSvL1JN9M8niSD3XtO5IcT3Kq\nW27f/HIlSevp5wr9f4F3VNWbgd3AbUneAhwEFqpqF7DQbUuSxqRnoNeKF7vNy7t/BewF5rv2eWDf\nplQoSepLX3PoSbYleRRYAo5X1UPAdFWd7Q45B0yv89kDSRaTLC4vL4+kaEnSz+sr0KvqJ1W1G7gG\nuCnJDRfsL1au2tf67OGqmq2q2ampqaELliStbaC7XKrqBeBB4DbgfJKdAN1yafTlSZL61c9dLlNJ\nrurWXw3cCjwJHAPmusPmgKObVaQkqbfL+jhmJzCfZBsrPwCOVNXnk3wNOJLkLuAMsH8T65Qk9dAz\n0KvqW8CNa7Q/B+zZjKIkSYPzSVFJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0\nSWqEgS5JjTDQJakRBrokNcJAl6RG9PP6XI3JzMEHxnLe04duH8t5JQ3HK3RJaoSBLkmNMNAlqREG\nuiQ1wkCXpEYY6JLUCANdkhrRM9CTvCHJg0meSPJ4kru79h1Jjic51S23b365kqT19HOF/hLwx1X1\nJuAtwHuTvAk4CCxU1S5goduWJI1Jz0CvqrNV9Ui3/kPgJPB6YC8w3x02D+zbrCIlSb0NNIeeZAa4\nEXgImK6qs92uc8D0SCuTJA2k70BP8hrgPuD9VfWD1fuqqoBa53MHkiwmWVxeXh6qWEnS+voK9CSX\nsxLm91TV57rm80l2dvt3AktrfbaqDlfVbFXNTk1NjaJmSdIa+rnLJcDHgZNV9eFVu44Bc936HHB0\n9OVJkvrVz+tz3wr8HvDtJI92bR8ADgFHktwFnAH2b06JkqR+9Az0qvoqkHV27xltOZKkjfJJUUlq\nhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY\n6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1IiegZ7kE0mWkjy2qm1HkuNJ\nTnXL7ZtbpiSpl36u0P8ZuO2CtoPAQlXtAha6bUnSGPUM9Kr6CvD8Bc17gflufR7YN+K6JEkD2ugc\n+nRVne3WzwHT6x2Y5ECSxSSLy8vLGzydJKmXof8oWlUF1EX2H66q2aqanZqaGvZ0kqR1bDTQzyfZ\nCdAtl0ZXkiRpIzYa6MeAuW59Djg6mnIkSRvVz22L9wJfA65P8kySu4BDwK1JTgG3dNuSpDG6rNcB\nVXXnOrv2jLgWSdIQfFJUkhphoEtSIwx0SWpEzzl0vfLMHHxgbOc+fej2sZ1bmnReoUtSIwx0SWqE\ngS5JjTDQJakRBrokNcJAl6RGeNuitpRx3TLp7ZJqgVfoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAl\nqREGuiQ1wkCXpEYY6JLUCANdkhoxVKAnuS3JU0m+k+TgqIqSJA1uw+9ySbIN+AfgVuAZ4OEkx6rq\niVEVJ10qr8T/ds8+X1qXos/DXKHfBHynqp6uqh8Dnwb2jqYsSdKghgn01wPfXbX9TNcmSRqDTX99\nbpIDwIFu88UkT23wS10NfG80VW0JLfWnpb7AJe5P/npTv/yWHJsh+rwl+9OPNfo8SF9+uZ+Dhgn0\nZ4E3rNq+pmv7f6rqMHB4iPMAkGSxqmaH/TpbRUv9aakv0FZ/WuoLtNWfzejLMFMuDwO7krwxyRXA\nHcCx0ZQlSRrUhq/Qq+qlJH8E/DuwDfhEVT0+ssokSQMZag69qr4AfGFEtfQy9LTNFtNSf1rqC7TV\nn5b6Am31Z+R9SVWN+mtKksbAR/8lqRETEegtvWIgyekk307yaJLFcdczqCSfSLKU5LFVbTuSHE9y\nqltuH2eNg1inPx9M8mw3Ro8m+a1x1tivJG9I8mCSJ5I8nuTurn3ixucifZnUsXlVkq8n+WbXnw91\n7SMdmy0/5dK9YuC/WPWKAeDOSX3FQJLTwGxVTea9tMnbgBeBT1bVDV3b3wDPV9Wh7gfu9qr603HW\n2a91+vNB4MWq+ttx1jaoJDuBnVX1SJLXAieAfcDvM2Hjc5G+7GcyxybAlVX1YpLLga8CdwO/wwjH\nZhKu0H3FwBZSVV8Bnr+geS8w363Ps/KNNxHW6c9EqqqzVfVIt/5D4CQrT29P3PhcpC8TqVa82G1e\n3v0rRjw2kxDorb1ioIAvJTnRPUXbgumqOtutnwOmx1nMiLwvybe6KZktP0VxoSQzwI3AQ0z4+FzQ\nF5jQsUmyLcmjwBJwvKpGPjaTEOitubmqdgPvAt7b/crfjFqZw9va83i9fRS4DtgNnAX+brzlDCbJ\na4D7gPdX1Q9W75u08VmjLxM7NlX1k+57/xrgpiQ3XLB/6LGZhEDv6xUDk6Kqnu2WS8D9rEwpTbrz\n3Zzny3OfS2OuZyhVdb775vsp8I9M0Bh187P3AfdU1ee65okcn7X6Mslj87KqegF4ELiNEY/NJAR6\nM68YSHJl9wceklwJvBN47OKfmgjHgLlufQ44OsZahvbyN1jnPUzIGHV/ePs4cLKqPrxq18SNz3p9\nmeCxmUpyVbf+alZu8niSEY/Nlr/LBaC7Nekj/OwVA3815pI2JMl1rFyVw8pTup+atL4kuRd4Oytv\nijsP/AXwb8AR4FrgDLC/qibiD43r9OftrPxKX8Bp4A9XzXNuWUluBv4T+Dbw0675A6zMPU/U+Fyk\nL3cymWPzq6z80XMbKxfSR6rqL5O8jhGOzUQEuiSpt0mYcpEk9cFAl6RGGOiS1AgDXZIaYaBLUiMM\ndElqhIEuSY0w0CWpEf8HjUNA9B7tL+sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11a588828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.hist(differences)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Logistic Regression with MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll see how this regression model can perform on the MNIST dataset. Now, as you know, linear regression is a model that produces continuous values. Like we saw in the last example, the model predicted house prices which can take any real value. However, there are prediction tasks where we'd like the model to output a category or class. In the case of binary classification, you'd want the network to output the probabilities for the input being part of either class 1 or class 2.\n",
    "\n",
    "This leads us to another type of regression called logistic regression. You can think of logistic regression as being the same thing as linear regression, except it outputs probabilities instead of real values. The way the model acheives this is by placing the output of (wx + b) through a softmax layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n",
      "step 0, training accuracy 0.09375\n",
      "step 1000, training accuracy 0.53125\n",
      "step 2000, training accuracy 0.640625\n",
      "step 3000, training accuracy 0.78125\n",
      "step 4000, training accuracy 0.75\n",
      "step 5000, training accuracy 0.75\n",
      "step 6000, training accuracy 0.890625\n",
      "step 7000, training accuracy 0.9375\n",
      "step 8000, training accuracy 0.8125\n",
      "step 9000, training accuracy 0.9375\n",
      "step 10000, training accuracy 0.859375\n",
      "step 11000, training accuracy 0.921875\n",
      "step 12000, training accuracy 0.8125\n",
      "step 13000, training accuracy 0.90625\n",
      "step 14000, training accuracy 0.828125\n",
      "step 15000, training accuracy 0.859375\n",
      "step 16000, training accuracy 0.875\n",
      "step 17000, training accuracy 0.921875\n",
      "step 18000, training accuracy 0.875\n",
      "step 19000, training accuracy 0.953125\n",
      "test accuracy 0.9375\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "\n",
    "numClasses = 10\n",
    "inputSize = 784  \n",
    "trainingIterations = 20000\n",
    "batchSize = 64\n",
    "tf.reset_default_graph() \n",
    "\n",
    "X = tf.placeholder(tf.float32, shape = [None, inputSize])\n",
    "y = tf.placeholder(tf.float32, shape = [None, numClasses])\n",
    "\n",
    "W1 = tf.Variable(tf.random_normal([inputSize, numClasses], stddev=0.1))\n",
    "B1 = tf.Variable(tf.constant(0.1), [numClasses])\n",
    "\n",
    "y_pred = tf.nn.softmax(tf.matmul(X, W1) + B1)\n",
    "\n",
    "loss = tf.reduce_mean(tf.square(y - y_pred))\n",
    "opt = tf.train.GradientDescentOptimizer(learning_rate = .05).minimize(loss)\n",
    "\n",
    "correct_prediction = tf.equal(tf.argmax(y_pred,1), tf.argmax(y,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
    "\n",
    "sess = tf.Session()\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)\n",
    "\n",
    "for i in range(trainingIterations):\n",
    "    batch = mnist.train.next_batch(batchSize)\n",
    "    batchInput = batch[0]\n",
    "    batchLabels = batch[1]\n",
    "    _, trainingLoss = sess.run([opt, loss], feed_dict={X: batchInput, y: batchLabels})\n",
    "    if i%1000 == 0:\n",
    "        train_accuracy = accuracy.eval(session=sess, feed_dict={X: batchInput, y: batchLabels})\n",
    "        print (\"step %d, training accuracy %g\"%(i, train_accuracy))\n",
    "\n",
    "batch = mnist.test.next_batch(batchSize)\n",
    "testAccuracy = sess.run(accuracy, feed_dict={X: batch[0], y: batch[1]})\n",
    "print (\"test accuracy %g\"%(testAccuracy))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}