02A_TensorFlow-Slim.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow-Slim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[TensorFlow-Slim](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim) is a high-level API for building TensorFlow models. TF-Slim makes defining models in TensorFlow easier, cutting down on the number of lines required to define models and reducing overall clutter. In particular, TF-Slim shines in image domain problems, and weights pre-trained on the [ImageNet dataset](http://www.image-net.org/) for many famous CNN architectures are provided for [download](https://github.com/tensorflow/models/tree/master/slim#pre-trained-models).\n",
    "\n",
    "*Note: Unlike previous notebooks, not every cell here is necessarily meant to run. Some are just for illustration.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VGG-16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To show these benefits, this tutorial will focus on [VGG-16](https://arxiv.org/abs/1409.1556). This style of architecture came in 2nd during the 2014 ImageNet Large Scale Visual Recognition Challenge and is famous for its simplicity and depth. The model looks like this:\n",
    "\n",
    "![vgg16](Figures/vgg16.png)\n",
    "\n",
    "The architecture is pretty straight-forward: simply stack multiple 3x3 convolutional filters one after another, interleave with 2x2 maxpools, double the number of convolutional filters after each maxpool, flatten, and finish with fully connected layers. A couple ideas behind this model:\n",
    "\n",
    "- Instead of using larger filters, VGG notes that the receptive field of two stacked layers of 3x3 filters is 5x5, and with 3 layers, 7x7. Using 3x3's allows VGG to insert additional non-linearities and requires fewer weight parameters to learn.\n",
    "\n",
    "- Doubling the width of the network every time the features are spatially downsampled (maxpooled) gives the model more representational capacity while achieving spatial compression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorFlow Core"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In code, setting up the computation graph for prediction with just TensorFlow Core API is kind of a lot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Set up the data loading:\n",
    "images, labels = ...\n",
    "\n",
    "# Define the model\n",
    "with tf.name_scope('conv1_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 3, 64], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(images, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[64], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv1 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv1_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 64, 64], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv1, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[64], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv1 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool1')\n",
    "    \n",
    "with tf.name_scope('conv2_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 64, 128], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(pool1, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[128], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv2 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv2_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 128, 128], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv2, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[128], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv2 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool2')\n",
    "    \n",
    "with tf.name_scope('conv3_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 128, 256], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(pool2, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv3 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv3_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 256, 256], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv3, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv3 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv3_3') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 256, 256], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv3, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv3 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool3 = tf.nn.max_pool(conv3, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool3')\n",
    "    \n",
    "with tf.name_scope('conv4_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 256, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(pool3, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv4 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv4_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv4, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv4 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv4_3') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv4, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv4 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool4 = tf.nn.max_pool(conv4, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool4')\n",
    "\n",
    "with tf.name_scope('conv5_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(pool4, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv5 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv5_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv5, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv5 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv5_3') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv5, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv5 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool5 = tf.nn.max_pool(conv5, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool5')\n",
    "    \n",
    "with tf.name_scope('fc_6') as scope:\n",
    "    flat = tf.reshape(pool5, [-1, 7*7*512])\n",
    "    weights = tf.Variable(tf.truncated_normal([7*7*512, 4096], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    mat = tf.matmul(flat, weights)\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[4096], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(mat, biases)\n",
    "    fc6 = tf.nn.relu(bias, name=scope)\n",
    "    fc6_drop = tf.nn.dropout(fc6, keep_prob=0.5, name='dropout')\n",
    "\n",
    "with tf.name_scope('fc_7') as scope:\n",
    "    weights = tf.Variable(tf.truncated_normal([4096, 4096], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    mat = tf.matmul(fc6, weights)\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[4096], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(mat, biases)\n",
    "    fc7 = tf.nn.relu(bias, name=scope)\n",
    "    fc7_drop = tf.nn.dropout(fc7, keep_prob=0.5, name='dropout')\n",
    "    \n",
    "with tf.name_scope('fc_8') as scope:\n",
    "    weights = tf.Variable(tf.truncated_normal([4096, 1000], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    mat = tf.matmul(fc7, weights)\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[1000], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(mat, biases)\n",
    "\n",
    "predictions = bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understanding every line of this model isn't important. The main point to notice is how much space this takes up. Several of the above lines (conv2d, bias_add, relu, maxpool) can obviously be combined to cut down on the size a bit, and you could also try to compress the code with some clever `for` looping, but all at the cost of sacrificing readability. With this much code, there is high potential for bugs or typos (to be honest, there are probably a few up there^), and modifying or refactoring the code becomes a huge pain.\n",
    "\n",
    "By the way, although VGG-16's paper was titled \"Very Deep Convolutional Networks for Large-Scale Image Recognition\", it isn't even considered a particularly deep network by today's standards. [Residual Networks](https://arxiv.org/abs/1512.03385) (2015) started beating state-of-the-art results with 50, 101, and 152 layers in their first incarnation, before really going off the deep end and getting up to 1001 layers and beyond. I'll spare you from me typing out the uncompressed TensorFlow Core code for that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-Slim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enter TF-Slim. The same VGG-16 model can be expressed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "slim = tf.contrib.slim\n",
    "\n",
    "# Set up the data loading:\n",
    "images, labels = ...\n",
    "\n",
    "# Define the model:\n",
    "with slim.arg_scope([slim.conv2d, slim.fully_connected],\n",
    "                    activation_fn=tf.nn.relu,\n",
    "                    weights_initializer=tf.truncated_normal_initializer(0.0, 0.01),\n",
    "                    weights_regularizer=slim.l2_regularizer(0.0005)):\n",
    "    net = slim.repeat(images, 2, slim.conv2d, 64, [3, 3], scope='conv1')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool1')\n",
    "    net = slim.repeat(net, 2, slim.conv2d, 128, [3, 3], scope='conv2')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool2')\n",
    "    net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool3')\n",
    "    net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv4')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool4')\n",
    "    net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv5')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool5')\n",
    "    net = slim.fully_connected(net, 4096, scope='fc6')\n",
    "    net = slim.dropout(net, 0.5, scope='dropout6')\n",
    "    net = slim.fully_connected(net, 4096, scope='fc7')\n",
    "    net = slim.dropout(net, 0.5, scope='dropout7')\n",
    "    net = slim.fully_connected(net, 1000, activation_fn=None, scope='fc8')\n",
    "\n",
    "predictions = net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much cleaner. For the TF-Slim version, it's much more obvious what the network is doing, writing it is faster, and typos and bugs are much less likely.\n",
    "\n",
    "Things to notice:\n",
    "\n",
    "- Weight and bias variables for every layer are automatically generated and tracked. Also, the \"in_channel\" parameter for determining weight dimension is automatically inferred from the input. This allows you to focus on what layers you want to add to the model, without worrying as much about boilerplate code. \n",
    "\n",
    "- The repeat() function allows you to add the same layer multiple times. In terms of variable scoping, repeat() will add \"_#\" to the scope to distinguish the layers, so we'll still have layers of scope \"`conv1_1`, `conv1_2`, `conv2_1`, etc...\".\n",
    "\n",
    "- The non-linear activation function (here: ReLU) is wrapped directly into the layer. In more advanced architectures with batch normalization, that's included as well.\n",
    "\n",
    "- With slim.argscope(), we're able to specify defaults for common parameter arguments, such as the type of activation function or weights_initializer. Of course, these defaults can still be overridden in any individual layer, as demonstrated in the finally fully connected layer (fc8).\n",
    "\n",
    "If you're reusing one of the famous architectures (like VGG-16), TF-Slim already has them defined, so it becomes even easier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "slim = tf.contrib.slim\n",
    "vgg = tf.contrib.slim.nets.vgg\n",
    "\n",
    "# Set up the data loading:\n",
    "images, labels = ...\n",
    "\n",
    "# Define the model:\n",
    "predictions = vgg.vgg16(images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-Trained Weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TF-Slim provides weights pre-trained on the ImageNet dataset available for [download](https://github.com/tensorflow/models/tree/master/slim#pre-trained-models). First a quick tutorial on saving and restoring models:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving and Restoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the nice features of modern machine learning frameworks is the ability to save model parameters in a clean way. While this may not have been a big deal for the MNIST logistic regression model because training only took a few seconds, it's easy to see why you wouldn't want to have to re-train a model from scratch every time you wanted to do inference or make a small change if training takes days or weeks.\n",
    "\n",
    "TensorFlow provides this functionality with its [Saver()](https://www.tensorflow.org/programmers_guide/variables#saving_and_restoring) class. While I just said that saving the weights for the MNIST logistic regression model isn't necessary because of how it is easy to train, let's do it anyway for illustrative purposes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████| 1000/1000 [00:02<00:00, 393.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model saved in file: ./log_reg_model.ckpt\n",
      "Test accuracy: 0.916700005531311\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tqdm import trange\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Import data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "\n",
    "# Create the model\n",
    "x = tf.placeholder(tf.float32, [None, 784], name='x')\n",
    "W = tf.Variable(tf.zeros([784, 10]), name='W')\n",
    "b = tf.Variable(tf.zeros([10]), name='b')\n",
    "y = tf.nn.bias_add(tf.matmul(x, W), b, name='y')\n",
    "\n",
    "# Define loss and optimizer\n",
    "y_ = tf.placeholder(tf.float32, [None, 10], name='y_')\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))\n",
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)\n",
    "\n",
    "# Variable Initializer\n",
    "init_op = tf.global_variables_initializer()\n",
    "\n",
    "# Create a Saver object for saving weights\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "# Create a Session object, initialize all variables\n",
    "sess = tf.Session()\n",
    "sess.run(init_op)\n",
    "\n",
    "# Train\n",
    "for _ in trange(1000):\n",
    "    batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})\n",
    "    \n",
    "# Save model\n",
    "save_path = saver.save(sess, \"./log_reg_model.ckpt\")\n",
    "print(\"Model saved in file: %s\" % save_path)\n",
    "    \n",
    "# Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print('Test accuracy: {0}'.format(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})))\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, the differences from what we worked with yesterday:\n",
    "\n",
    "- In lines 9-12, 15, there are now 'names' properties attached to certain ops and variables of the graph. There are many reasons to do this, but here, it will help us identify which variables are which when restoring. \n",
    "- In line 23, we create a Saver() object, and in line 35, we save the variables of the model to a checkpoint file. This will create a series of files containing our saved model.\n",
    "\n",
    "Otherwise, the code is more or less the same.\n",
    "\n",
    "To restore the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n",
      "INFO:tensorflow:Restoring parameters from ./log_reg_model.ckpt\n",
      "Model restored.\n",
      "Test accuracy: 0.916700005531311\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tqdm import trange\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Import data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "\n",
    "# Create a Session object, initialize all variables\n",
    "sess = tf.Session()\n",
    "\n",
    "# Restore weights\n",
    "saver = tf.train.import_meta_graph('./log_reg_model.ckpt.meta')\n",
    "saver.restore(sess, tf.train.latest_checkpoint('./'))\n",
    "print(\"Model restored.\")\n",
    "\n",
    "graph = tf.get_default_graph()\n",
    "x = graph.get_tensor_by_name(\"x:0\")\n",
    "y = graph.get_tensor_by_name(\"y:0\")\n",
    "y_ = graph.get_tensor_by_name(\"y_:0\")\n",
    "    \n",
    "# Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print('Test accuracy: {0}'.format(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})))\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importantly, notice that we didn't have to retrain the model. Instead, the graph and all variable values were loaded directly from our checkpoint files. In this example, this probably takes just as long, but for more complex models, the utility of saving/restoring is immense."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-Slim Model Zoo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the biggest and most surprising unintended benefits of the ImageNet competition was deep networks' transfer learning properties: CNNs trained on ImageNet classification could be re-used as general purpose feature extractors for other tasks, such as object detection. Training on ImageNet is very intensive and expensive in both time and computation, and requires a good deal of set-up. As such, the availability of weights already pre-trained on ImageNet has significantly accelerated and democratized deep learning research.\n",
    "\n",
    "Pre-trained models of several famous architectures are listed in the TF Slim portion of the [TensorFlow repository](https://github.com/tensorflow/models/tree/master/slim#pre-trained-models). Also included are the papers that proposed them and their respective performances on ImageNet. Side note: remember though that accuracy is not the only consideration when picking a network; memory and speed are important to keep in mind as well.\n",
    "\n",
    "Each entry has a link that allows you to download the checkpoint file of the pre-trained network. Alternatively, you can download the weights as part of your program. A tutorial can be found [here](https://github.com/tensorflow/models/blob/master/slim/slim_walkthrough.ipynb), but the general idea:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from datasets import dataset_utils\n",
    "import tensorflow as tf\n",
    "\n",
    "url = \"http://download.tensorflow.org/models/vgg_16_2016_08_28.tar.gz\"\n",
    "checkpoints_dir = './checkpoints'\n",
    "\n",
    "if not tf.gfile.Exists(checkpoints_dir):\n",
    "    tf.gfile.MakeDirs(checkpoints_dir)\n",
    "\n",
    "dataset_utils.download_and_uncompress_tarball(url, checkpoints_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import tensorflow as tf\n",
    "from nets import vgg\n",
    "\n",
    "slim = tf.contrib.slim\n",
    "\n",
    "# Load images\n",
    "images = ...\n",
    "\n",
    "# Pre-process\n",
    "processed_images = ...\n",
    "\n",
    "# Create the model, use the default arg scope to configure the batch norm parameters.\n",
    "with slim.arg_scope(vgg.vgg_arg_scope()):\n",
    "    logits, _ = vgg.vgg_16(processed_images, num_classes=1000, is_training=False)\n",
    "    \n",
    "probabilities = tf.nn.softmax(logits)\n",
    "\n",
    "# Load checkpoint values\n",
    "init_fn = slim.assign_from_checkpoint_fn(\n",
    "    os.path.join(checkpoints_dir, 'vgg_16.ckpt'),\n",
    "    slim.get_model_variables('vgg_16'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}