deep_learning_tutorial_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Express Deep Learning in Python: Advanced Layers\n",
    "\n",
    "The `Dense` layer is only one of the possible core layers of Keras. `Dense` is a *forward* layer, this are the ones that take an input and do some transformation on it (in this case a matrix multiplication).\n",
    "\n",
    "Other important layers to consider are: activation layers, regularization layers, dropout layers, convolutional layers, pooling layers, recurrent layers, normalization layers, embedding layers, noise layers, etc.\n",
    "\n",
    "For this tutorial we will focus on some layers to aid in the tuning of the network: activations, regularizers and dropout; as well as the layers needed to design convolutional neural networks: convolutional and pooling layers.\n",
    "\n",
    "We will point out other tutorials and examples to learn about the other kind of layers at the end of this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy\n",
    "import keras\n",
    "\n",
    "from keras import backend as K\n",
    "from keras import losses, optimizers, regularizers\n",
    "from keras.datasets import mnist\n",
    "from keras.layers import Activation, ActivityRegularization, Conv2D, Dense, Dropout, Flatten, MaxPooling2D\n",
    "from keras.models import Sequential\n",
    "from keras.utils.np_utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation Functions\n",
    "\n",
    "A neural network classifier with linear activations has no more *representation* power than a logistic regression classifier. In order to express non-linearity with a neural network model a non-linear function is needed as activation function for each neuron.\n",
    "\n",
    "One simple activation function to use is the **sigmoid (or logistic) function**, the same one used in the logistic regression algorithm, which restricts the output value to be between zero and one. This was one of the most common nonlinearities used as activation function in some of the *first versions* of neural networks. There are however other possibilities (all the following available in Keras, but there are more which can be adapted):\n",
    "\n",
    "* rectified linear unit (ReLU)\n",
    "* tanh\n",
    "* hard sigmoid\n",
    "* softsign\n",
    "* softplus\n",
    "* exponential linear unit (elu)\n",
    "* scaled exponential linear unit (selu)\n",
    "* leaky rectifier linear unit (Leaky ReLU)\n",
    "* parametric rectified linear unit (PReLU)\n",
    "\n",
    "#### Activation Functions Examples\n",
    "\n",
    "![Activation Functions](files/activation_functions.png \"Activation Functions\")\n",
    "<div style=\"text-align: right;\">Source: https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/</div>\n",
    "\n",
    "Of these, the one most used in the present state-of-the-art neural networks classifiers is the **ReLU**, because tipically learns much faster in networks with many layers [1].\n",
    "\n",
    "There is another activation layer which is the **SoftMax** activation. This is generally used as the last activation layer, i.e. as the output of the network. This function, also known as *normalized exponential function* is a generalization of the logistic function that \"squashes\" a K-dimensional vector ${\\displaystyle \\mathbf {z}}$ of arbitrary real values to a K-dimensional vector ${\\displaystyle \\sigma (\\mathbf {z} )}$ of real values in the range [0, 1] that add up to 1.\n",
    "\n",
    "### Activation Functions in Keras\n",
    "\n",
    "Keras provides two ways to define an activation function. Any method is equally valid.\n",
    "\n",
    "#### Activation as a parameter of a forward layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(64, input_shape=(784,), activation='relu'))\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Activation as a layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(64, input_shape=(784,)))\n",
    "model.add(Activation('tanh'))\n",
    "model.add(Dense(10))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Activation from a TensorFlow function\n",
    "\n",
    "In the previous examples we used some of the available functions in the Keras library.\n",
    "\n",
    "We can also use an element-wise TensorFlow function as activation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(64, input_shape=(784,),\n",
    "                activation=K.sigmoid))\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regularizers\n",
    "\n",
    "Regularizers allow to apply penalties on layer parameters or layer activity during optimization. These penalties are incorporated in the loss function that the network optimizes. The penalties are applied on a per-layer basis.\n",
    "\n",
    "The regularizers can be applied to three parameters:\n",
    "\n",
    "* Weight/kernel matrix regularization: Applies the regularizer function to the weight matrix (called kernel matrix in Keras documentation).\n",
    "* Bias regularization: Applies the regularizer to the bias vector.\n",
    "* Activity regularizer: Applies the regularizer to the output (i.e. the activation function).\n",
    "\n",
    "There are three possible penalties to apply as regularizers already present in Keras (but the API permits the definition of a custom regularizer) [2]: l1, l2 and elasticnet.\n",
    "\n",
    "### Regularizers in Keras\n",
    "\n",
    "As with activation functions, there are two ways to use a regularizer in keras. Although not for all the parameters.\n",
    "\n",
    "#### Regularization as parameter of a layer\n",
    "\n",
    "This is the most practical way and the only one which allows the individual regularization of each available parameter.\n",
    "\n",
    "The regularizer is given as a parameter of the layers (e.g. `Dense`):\n",
    "\n",
    "* `kernel_regularizer`: Regularization of the weight matrix.\n",
    "* `bias_regularizer`: Regularization of the bias vector.\n",
    "* `activity_regularizer`: Regularization of the total output.\n",
    "\n",
    "The available penalties for this case are:\n",
    "\n",
    "* `keras.regularizers.l1`: L1 norm or \"sum of weights\".\n",
    "* `keras.regularizers.l2`: L2 norm or \"sum of weights squared\".\n",
    "* `keras.regularizers.l1_l2`: Linear combination of L1 and L2 penalties or \"elastic net regularization\".\n",
    "\n",
    "For more information on the difference between L1 and L2 see [5]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(64, input_shape=(784,),\n",
    "                activation='relu',\n",
    "                kernel_regularizer=regularizers.l2(0.01),\n",
    "                activity_regularizer=regularizers.l1(0.01)))\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regularization as a layer\n",
    "\n",
    "The core layer `ActivityRegularization` is another way to apply regularization, in this case (as the name indicates), only for the activation function (not for the weight matrix or the bias vector)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(64, input_shape=(784,), activation='relu'))\n",
    "model.add(ActivityRegularization(l1=0.01, l2=0.1))\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout\n",
    "\n",
    "This are special layers useful for regularization which randomly drop (i.e. set to zero) units of the neural network during training. This prevents units from co-adapting too much to the input [3].\n",
    "\n",
    "Keras has a special layer which can be added to a sequential model which takes a value `rate`, between 0 and 1, and sets the fraction given by the value to 0 during training of the input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(64, input_shape=(784,), activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional Neural Networks\n",
    "\n",
    "CNNs were responsible for major breakthroughs in Image Classification and are the core of most Computer Vision systems today, from Facebook's automated photo tagging to self-driving cars [6].\n",
    "\n",
    "### What is convolution?\n",
    "\n",
    "A simple way to think about it is as a sliding window function applied to a matrix:\n",
    "\n",
    "![Convolution](files/convolution.gif \"Convolution\")\n",
    "<div style=\"text-align: right;\">Source: http://deeplearning.stanford.edu/wiki/index.php/Feature_extraction_using_convolution</div>\n",
    "\n",
    "Imagine that the matrix on the left represents an black and white image. Each entry corresponds to one pixel, 0 for black and 1 for white (typically it's between 0 and 255 for grayscale images). The sliding window is called a kernel, filter, or feature detector. Here we use a 3×3 filter, multiply its values element-wise with the original matrix, then sum them up. To get the full convolution we do this for each element by sliding the filter over the whole matrix.\n",
    "\n",
    "There are different uses for a convolution, particularly in images: averaging each pixel with its neighboring values blurs an image; taking the difference between a pixel and its neighbors detects edges; etc. For a better understanding of how a convolution work we recommend [Chris Olah's post](http://colah.github.io/posts/2014-07-Understanding-Convolutions/).\n",
    "\n",
    "### What are convolutional neural networks?\n",
    "\n",
    "CNNs are basically just several layers of convolutions with nonlinear activation functions (e.g. *ReLU* or *tanh*) applied to the results. \n",
    "\n",
    "In a traditional feedforward neural network we connect each input neuron to each output neuron in the next layer. These are fully connected layer (or `Dense` layers). In CNNs we don't do that. Instead, we use convolutions over the input layer to compute the output. This results in local connections, where each region of the input is connected to a neuron in the output. Each layer applies different filters, typically hundreds or thousands like the ones showed above, and combines their results.\n",
    "\n",
    "During the training phase, **a CNN automatically learns the values of its filters** based on the task you want to perform. For example, in Image Classification a CNN may learn to detect edges from raw pixels in the first layer, then use the edges to detect simple shapes in the second layer, and then use these shapes to deter higher-level features, such as facial shapes in higher layers. The last layer is then a classifier that uses these high-level features.\n",
    "\n",
    "![CNN](files/cnn.png \"Convolutional Neural Network\")\n",
    "<div style=\"text-align: right;\">Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/</div>\n",
    "\n",
    "### CNN Hyperparameters\n",
    "\n",
    "#### Narrow vs. wide convolution\n",
    "\n",
    "Applying a 3x3 filter at the center of the matrix works fine, but what about the edges? How would you apply the filter to the first element of a matrix that doesn't have any neighboring elements to the top and left? You can use zero-padding. All elements that would fall outside of the matrix are taken to be zero. By doing this you can apply the filter to every element of your input matrix, and get a larger or equally sized output. Adding zero-padding is also called wide convolution, and not using zero-padding would be a narrow convolution.\n",
    "\n",
    "![NvW](files/narrowvswide.png \"Narrow vs Wide convolution\")\n",
    "<div style=\"text-align: right;\">Source: http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/</div>\n",
    "\n",
    "The previous example shows the difference between narrow and wide convolution for 1 dimension for an input size of 7 and a filter size of 5.\n",
    "\n",
    "#### Stride size\n",
    "\n",
    "Another hyperparameter for your convolutions is the stride size, defining by how much you want to shift your filter at each step. A larger stride size leads to fewer applications of the filter and a smaller output size. The typical stride size is 1. The following example shows the differente outputs of a convolution for different size of strides (stride size of 1 vs stride size of 2).\n",
    "\n",
    "![Stride](files/stride.png \"Stride size\")\n",
    "<div style=\"text-align: right;\">Source: http://cs231n.github.io/convolutional-networks/</div>\n",
    "\n",
    "#### Channels\n",
    "\n",
    "Channels are different \"views\" of your input data. For example, in image recognition you typically have RGB (red, green, blue) channels. You can apply convolutions across channels, either with different or equal weights.\n",
    "\n",
    "### Pooling layers\n",
    "\n",
    "A key aspect of Convolutional Neural Networks are **pooling layers**, typically applied after the convolutional layers. Pooling layers subsample their input. The most common way to do pooling it to apply a *max* operation to the result of each filter. You don't necessarily need to pool over the complete matrix, you could also pool over a window. For example, the following shows max pooling for a 2x2 window:\n",
    "\n",
    "![Pooling](files/pooling.png \"Pooling Layer\")\n",
    "<div style=\"text-align: right;\">Source: http://cs231n.github.io/convolutional-networks/#pool</div>\n",
    "\n",
    "One property of pooling is that it provides a fixed size output matrix, which typically is required for classification. For example, if you have 1,000 filters and you apply max pooling to each, you will get a 1000-dimensional output, regardless of the size of your filters, or the size of your input. This allows you to use variable size sentences, and variable size filters, but always get the same output dimensions to feed into a classifier. Pooling also reduces the output dimensionality but (hopefully) keeps the most salient information. You can think of each filter as detecting a specific feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CNNs in Keras\n",
    "\n",
    "Keras has many different kinds of convolutional layers. The most commonly used for doing spatial convolution over images is `keras.layers.convolutional.Conv2D`. The layer takes as arguments the number output of filters in the convolution, the size of the 2D convolution window, the strides of the convolution and the padding.\n",
    "\n",
    "Keras also is shipped with many different pooling layers. For spatial data the layer is `keras.layers.pooling.MaxPooling2D`. This layer takes the pool size and the data format. The data format corresponds to whether the channels are the *first* (i.e. the input has shape `(batch, channels, height, width)`) or the *last* (i.e. the input has shape `(batch, height, width, channels)`) dimension (this one is the default for Keras with a TensorFlow backend).\n",
    "\n",
    "Finally, there is a layer, which doesn't take any parameters and serves as the connection between the convolutional layers and the dense layers, which is `keras.layers.core.Flatten()` which basically flattens the input to one dimension (without affecting the batch size, i.e. the number of examples to use for training/classifying)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "# input: 100x100 images with 3 channels -> (100, 100, 3) tensors.\n",
    "# this applies 32 convolution filters of size 3x3 each.\n",
    "\n",
    "model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(100, 100, 3)))\n",
    "model.add(Conv2D(32, kernel_size=(3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(256, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compiling the model: loss functions and optimizers\n",
    "\n",
    "When compiling a model there are two important parameters: the loss function and the optimizer algorithm. Both of them depend on the problem and can change the performance of the model.\n",
    "\n",
    "### Loss function\n",
    "\n",
    "Also know as the *objective function*, is the function we want to optimize when training the algorithm (that is find the minimum). Depending on the task (whether it is classification or regression), and some other parameters, the objective function can change. Two of the most popular objective functions are the **mean squared error** for regression and **categorical crossentropy** for classification. Keras bring a number of different loss functions already available [4], but for this course we will be using only the *categorical crossentropy* (since we have a classification task to work with).\n",
    "\n",
    "### Optimizer\n",
    "\n",
    "The optimizer algorithm is the way to find the minimum values to the loss function. As with loss functions, there are many available optimizers already packaged with Keras. One of the most popular algorithms is **stochastic gradient descent** (or SGD) optimizer, which is also one of the simplest to understand. However, in this tutorial we will be exploring other optimizers (e.g. RMSProp, Adam, Adadelta, etc.) which give better results.\n",
    "\n",
    "### Loss function and optimizer in Keras\n",
    "\n",
    "In Keras, is the `.compile()` method of a model which takes as parameters the loss function and the optimizer. The parameters can either be instances of a loss function (e.g. `keras.losses.hinge_loss`) or an optimizer (e.g. `keras.optimizers.RMSprop`), or a string calling the loss function/optimizer by the name. \n",
    "\n",
    "In the case of loss functions, the advantage of using an instance of a function is to have a custom defined loss function besides the ones given by Keras. E.g. you can pass a TensorFlow symbolic function that returns a scalar for each data-point and takes two arguments: the true labels and the predicted labels.\n",
    "\n",
    "For optimizers, the main difference between an instance and a string is that in the latter case the optimizer will have default parameter values. Besides, there is a wrapper class (`keras.optimizers.TFOptimizer`) for native TensorFlow optimizers.\n",
    "\n",
    "#### Loss function/optimizer as a string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(64, input_shape=(784,), activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(10, activation='softmax'))\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loss function/optimizer as an instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Simple 1 layer denoising autoencoder\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(200, input_shape=(784,), activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(784))\n",
    "\n",
    "sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)\n",
    "model.compile(loss=losses.mean_squared_error, optimizer=sgd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Categorical format\n",
    "\n",
    "In case of using a loss function for classification (e.g. the categorical crossentropy) having more than 2 classes, **Keras requires the targets to be in categorical format** (e.g. if you have 10 classes, the target for each sample should be a 10-dimensional vector that is all-zeros expect for a 1 at the index corresponding to the class of the sample). In order to convert integer targets into categorical targets, you can use the Keras utility `keras.utils.np_utils.to_categorical` to transform an input vector of integers into a matrix of one-hot encoding representations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping up\n",
    "\n",
    "Finally, to end the tutorial, we use what we have learned so far and use this to create a new classifier for the MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train samples: 20000\n",
      "Test samples: 5000\n",
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 11s - loss: 13.4005 - acc: 0.7447 - val_loss: 0.8993 - val_acc: 0.8636\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 11s - loss: 0.7521 - acc: 0.8427 - val_loss: 0.5580 - val_acc: 0.8974\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 11s - loss: 0.5090 - acc: 0.8902 - val_loss: 0.3500 - val_acc: 0.9392\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 11s - loss: 0.3863 - acc: 0.9179 - val_loss: 0.2513 - val_acc: 0.9524\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 11s - loss: 0.3118 - acc: 0.9325 - val_loss: 0.2782 - val_acc: 0.9454\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 11s - loss: 0.2664 - acc: 0.9420 - val_loss: 0.1942 - val_acc: 0.9560\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 11s - loss: 0.2383 - acc: 0.9466 - val_loss: 0.2401 - val_acc: 0.9348\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 11s - loss: 0.2206 - acc: 0.9492 - val_loss: 0.1691 - val_acc: 0.9648\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 11s - loss: 0.2022 - acc: 0.9539 - val_loss: 0.1709 - val_acc: 0.9684\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 11s - loss: 0.1888 - acc: 0.9555 - val_loss: 0.1318 - val_acc: 0.9748\n",
      "Test loss: 0.10\n",
      "Test accuracy: 97.48\n"
     ]
    }
   ],
   "source": [
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 10\n",
    "TRAIN_EXAMPLES = 20000\n",
    "TEST_EXAMPLES = 5000\n",
    "\n",
    "# image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "# load the data (already shuffled and splitted)\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "# reshape the data to add the \"channels\" dimension\n",
    "x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "# normalize the input in the range [0, 1]\n",
    "# to make quick runs, select a smaller set of images.\n",
    "train_mask = numpy.random.choice(x_train.shape[0], TRAIN_EXAMPLES, replace=False)\n",
    "x_train = x_train[train_mask, :].astype('float32')\n",
    "y_train = y_train[train_mask]\n",
    "test_mask = numpy.random.choice(x_test.shape[0], TEST_EXAMPLES, replace=False)\n",
    "x_test = x_test[test_mask, :].astype('float32')\n",
    "y_test = y_test[test_mask]\n",
    "\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "print('Train samples: %d' % x_train.shape[0])\n",
    "print('Test samples: %d' % x_test.shape[0])\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = to_categorical(y_train, num_classes)\n",
    "y_test = to_categorical(y_test, num_classes)\n",
    "\n",
    "# define the network architecture\n",
    "model = Sequential()\n",
    "model.add(Conv2D(filters=16,\n",
    "                 kernel_size=(3, 3),\n",
    "                 strides=(1,1),\n",
    "                 padding='valid',\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape,\n",
    "                 activity_regularizer='l2'))\n",
    "model.add(Conv2D(32, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "# compile the model\n",
    "model.compile(loss=losses.categorical_crossentropy,\n",
    "              optimizer=optimizers.RMSprop(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# train the model\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n",
    "\n",
    "# evaluate the model\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss: %.2f' % score[0])\n",
    "print('Test accuracy: %.2f' % (100. * score[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- [1] LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. \"Deep learning.\" Nature 521, no. 7553 (2015): 436-444.\n",
    "- [2] \"Developing new regularizers\". Keras Documentation. https://keras.io/regularizers/\n",
    "- [3] Srivastava, Nitish, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. \"Dropout: a simple way to prevent neural networks from overfitting.\" Journal of machine learning research 15, no. 1 (2014): 1929-1958. Harvard\t\n",
    "- [4] \"Available loss functions\". Keras documentation. https://keras.io/losses/\n",
    "- [5] \"Differences between L1 and L2 as Loss Function and Regularization\". http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/\n",
    "- [6] Britz, Denny. \"Understanding Convolutional Neural Networks for NLP\". http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/#more-348"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:keras]",
   "language": "python",
   "name": "conda-env-keras-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}