Document classification with word embeddings tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word Embeddings for Fun and Profit\n",
    "### Document classification with Gensim\n",
    "\n",
    "In this tutorial we'll classify movie plots by genre using word embeddings techniques in [gensim](http://radimrehurek.com/gensim/) . \n",
    "\n",
    "See accompanying slides in this repo.\n",
    "\n",
    "We will show how to get a __'hello-world'__ first untuned run using 7 techniques:\n",
    "\n",
    "- Bag of words\n",
    "\n",
    "- Character n-grams\n",
    "\n",
    "- TF-IDF \n",
    "\n",
    "- Averaging word2vec vectors\n",
    "\n",
    "- doc2vec\n",
    "\n",
    "- Deep IR \n",
    "\n",
    "- Word Mover's Distance\n",
    "\n",
    "The goal of this tutorial is to show the API so you can start tuning them yourself. Model tuning of the models is out of scope of this tutorial.\n",
    "\n",
    "We will also compare the accuracy of this first 'no tuning'/out of the box run of these techniques. It is in no way an indication of their best peformance that can be achieved with proper tuning. The benefit of the comparison is to manage the expectations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Requirements\n",
    "- Python 3\n",
    "- [Google News pre-trained word2vec (1.5 GB)](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing)\n",
    "- gensim\n",
    "- sklearn\n",
    "- pandas\n",
    "- matplotlib\n",
    "- nltk with English stopwords\n",
    "- pyemd\n",
    "- 4 GB RAM\n",
    "- 8 GB disk space for WMD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dataset\n",
    "We will use MovieLens dataset linked with plots from OMDB. Thanks to [Sujit Pal](http://sujitpal.blogspot.de/2016/04/predicting-movie-tags-from-plots-using.html) for this linking idea. The prepared csv is in this repository. If you wish to link the datasets yourself - see the code in the [blog]((http://sujitpal.blogspot.de/2016/04/predicting-movie-tags-from-plots-using.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-01-30 21:15:38,249 : INFO : 'pattern' package found; tag filters are available for English\n",
      "/usr/lib64/python2.7/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "logging.root.handlers = []  # Jupyter messes up logging so needs a reset\n",
    "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
    "from smart_open import smart_open\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from numpy import random\n",
    "import gensim\n",
    "import nltk\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn import linear_model\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "from gensim.models import Word2Vec\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn import linear_model\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exploring the data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "171156"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/tagged_plots_movielens.csv')\n",
    "df = df.dropna()\n",
    "df['plot'].apply(lambda x: len(x.split(' '))).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The dataset is only __170k__ words. It is quite small but makes sure we don't have to wait a long time for the code to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f75e8cbc5d0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFtVJREFUeJzt3X+QZWV95/H3B0ZE0TD86J3CmTHDxomW2ZRIWoPxR1hQ\nA5h1SEURypKRndREi/hzs5FY7sbdTbZAs0tCahdrIurgEhVRwkQoDRlAk80C9iAOIBo6CM7MDtAi\njkHWH8B3/zhP62W2Z/r29O1p5uz7VdV1z3nOc855nnvO/fRzz/2VqkKS1F8HLXYDJEkLy6CXpJ4z\n6CWp5wx6Seo5g16Ses6gl6SeM+glqecMeknqOYNeknpuyWI3AODoo4+uVatWLXYzJOmAsmXLlm9X\n1dhs9Z4UQb9q1SomJiYWuxmSdEBJcu8w9Ya6dJPkXUnuSHJ7kk8kOTTJsUluSjKZ5FNJDml1n9rm\nJ9vyVfveDUnSfM0a9EmWA28HxqvqXwAHA2cCFwAXVtVzgIeAdW2VdcBDrfzCVk+StEiGfTF2CfC0\nJEuApwM7gZOAK9ryjcDpbXpNm6ctPzlJRtNcSdJczRr0VbUD+GPgW3QBvwvYAny3qh5t1bYDy9v0\ncmBbW/fRVv+o3bebZH2SiSQTU1NT8+2HJGkPhrl0cwTdKP1Y4FnAYcAp891xVW2oqvGqGh8bm/VF\nY0nSPhrm0s0rgW9W1VRV/Rj4LPBSYGm7lAOwAtjRpncAKwHa8sOBB0faaknS0IYJ+m8BJyR5ervW\nfjLwNeB64HWtzlrgqja9qc3Tll9X/oyVJC2aYa7R30T3ouotwG1tnQ3Ae4B3J5mkuwZ/SVvlEuCo\nVv5u4LwFaLckaUh5Mgy2x8fHyw9MSdLcJNlSVeOz1XtSfDJ2X6067+r9ur97zn/Nft2fJI2CX2om\nST1n0EtSzxn0ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9JPWcQS9JPWfQS1LPGfSS1HMGvST1nEEv\nST1n0EtSzxn0ktRzBr0k9dysQZ/kuUluHfj7XpJ3JjkyybVJ7mq3R7T6SXJRkskkW5Mcv/DdkCTt\nyTC/GfuNqjquqo4Dfgl4BLiS7rdgN1fVamAzP/1t2FOB1e1vPXDxQjRckjScuV66ORn4x6q6F1gD\nbGzlG4HT2/Qa4NLq3AgsTXLMSForSZqzuQb9mcAn2vSyqtrZpu8DlrXp5cC2gXW2tzJJ0iIYOuiT\nHAK8Fvj07suqqoCay46TrE8ykWRiampqLqtKkuZgLiP6U4Fbqur+Nn//9CWZdvtAK98BrBxYb0Ur\ne4Kq2lBV41U1PjY2NveWS5KGMpegP4ufXrYB2ASsbdNrgasGys9u7745Adg1cIlHkrSfLRmmUpLD\ngFcBvz1QfD5weZJ1wL3AGa38GuA0YJLuHTrnjKy1kqQ5Gyroq+r7wFG7lT1I9y6c3esWcO5IWidJ\nmjc/GStJPWfQS1LPGfSS1HMGvST1nEEvST1n0EtSzxn0ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9\nJPWcQS9JPWfQS1LPGfSS1HMGvST1nEEvST1n0EtSzw0V9EmWJrkiydeT3JnkJUmOTHJtkrva7RGt\nbpJclGQyydYkxy9sFyRJezPsiP5Pgc9X1fOAFwB3AucBm6tqNbC5zQOcCqxuf+uBi0faYknSnMwa\n9EkOB14BXAJQVT+qqu8Ca4CNrdpG4PQ2vQa4tDo3AkuTHDPylkuShjLMiP5YYAr4aJKvJPlwksOA\nZVW1s9W5D1jWppcD2wbW397KJEmLYJigXwIcD1xcVS8Evs9PL9MAUFUF1Fx2nGR9kokkE1NTU3NZ\nVZI0B8ME/XZge1Xd1OavoAv++6cvybTbB9ryHcDKgfVXtLInqKoNVTVeVeNjY2P72n5J0ixmDfqq\nug/YluS5rehk4GvAJmBtK1sLXNWmNwFnt3ffnADsGrjEI0naz5YMWe9twGVJDgHuBs6h+ydxeZJ1\nwL3AGa3uNcBpwCTwSKsrSVokQwV9Vd0KjM+w6OQZ6hZw7jzbJUkaET8ZK0k9Z9BLUs8Z9JLUcwa9\nJPWcQS9JPWfQS1LPGfSS1HMGvST1nEEvST1n0EtSzxn0ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9\nJPWcQS9JPTdU0Ce5J8ltSW5NMtHKjkxybZK72u0RrTxJLkoymWRrkuMXsgOSpL2by4j+X1bVcVU1\n/ZOC5wGbq2o1sLnNA5wKrG5/64GLR9VYSdLczefSzRpgY5veCJw+UH5pdW4EliY5Zh77kSTNw7BB\nX8BfJ9mSZH0rW1ZVO9v0fcCyNr0c2Daw7vZWJklaBEuGrPeyqtqR5J8B1yb5+uDCqqokNZcdt38Y\n6wGe/exnz2VVSdIcDDWir6od7fYB4ErgxcD905dk2u0DrfoOYOXA6ita2e7b3FBV41U1PjY2tu89\nkCTt1axBn+SwJM+cngZeDdwObALWtmprgava9Cbg7PbumxOAXQOXeCRJ+9kwl26WAVcmma7/F1X1\n+SRfBi5Psg64Fzij1b8GOA2YBB4Bzhl5qyVJQ5s16KvqbuAFM5Q/CJw8Q3kB546kdZKkefOTsZLU\ncwa9JPWcQS9JPWfQS1LPGfSS1HMGvST1nEEvST1n0EtSzxn0ktRzBr0k9ZxBL0k9Z9BLUs8N+8Mj\nWgSrzrt6v+7vnvNfs1/3J2n/cEQvST1n0EtSzxn0ktRzBr0k9ZxBL0k9N3TQJzk4yVeSfK7NH5vk\npiSTST6V5JBW/tQ2P9mWr1qYpkuShjGXEf07gDsH5i8ALqyq5wAPAeta+TrgoVZ+YasnSVokQwV9\nkhXAa4APt/kAJwFXtCobgdPb9Jo2T1t+cqsvSVoEw47o/wT4PeDxNn8U8N2qerTNbweWt+nlwDaA\ntnxXqy9JWgSzBn2SXwceqKoto9xxkvVJJpJMTE1NjXLTkqQBw4zoXwq8Nsk9wCfpLtn8KbA0yfRX\nKKwAdrTpHcBKgLb8cODB3TdaVRuqaryqxsfGxubVCUnSns0a9FX1+1W1oqpWAWcC11XVG4Hrgde1\namuBq9r0pjZPW35dVdVIWy1JGtp83kf/HuDdSSbprsFf0sovAY5q5e8GzptfEyVJ8zGnb6+sqhuA\nG9r03cCLZ6jzA+D1I2ibJGkE/JpiLRq/hlnaP/wKBEnqOYNeknrOoJeknjPoJannDHpJ6jmDXpJ6\nzrdXSgvAt47qycQRvST1nEEvST1n0EtSzxn0ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9JPXcrEGf\n5NAkNyf5apI7kvyHVn5skpuSTCb5VJJDWvlT2/xkW75qYbsgSdqbYUb0PwROqqoXAMcBpyQ5AbgA\nuLCqngM8BKxr9dcBD7XyC1s9SdIimTXoq/Nwm31K+yvgJOCKVr4ROL1Nr2nztOUnJ8nIWixJmpOh\nrtEnOTjJrcADwLXAPwLfrapHW5XtwPI2vRzYBtCW7wKOGmWjJUnDGyroq+qxqjoOWAG8GHjefHec\nZH2SiSQTU1NT892cJGkP5vSum6r6LnA98BJgaZLprzleAexo0zuAlQBt+eHAgzNsa0NVjVfV+NjY\n2D42X5I0m2HedTOWZGmbfhrwKuBOusB/Xau2FriqTW9q87Tl11VVjbLRkqThDfPDI8cAG5McTPeP\n4fKq+lySrwGfTPKHwFeAS1r9S4CPJ5kEvgOcuQDtliQNadagr6qtwAtnKL+b7nr97uU/AF4/ktZJ\nkubNnxKUNGf+VOKBxa9AkKSeM+glqecMeknqOYNeknrOoJeknjPoJannDHpJ6jmDXpJ6zqCXpJ4z\n6CWp5wx6Seo5g16Ses6gl6SeM+glqecMeknqOYNeknrOoJeknhvmx8FXJrk+ydeS3JHkHa38yCTX\nJrmr3R7RypPkoiSTSbYmOX6hOyFJ2rNhRvSPAv+mqp4PnACcm+T5wHnA5qpaDWxu8wCnAqvb33rg\n4pG3WpI0tFmDvqp2VtUtbfqfgDuB5cAaYGOrthE4vU2vAS6tzo3A0iTHjLzlkqShzOkafZJVwAuB\nm4BlVbWzLboPWNamlwPbBlbb3sokSYtg6KBP8gzgM8A7q+p7g8uqqoCay46TrE8ykWRiampqLqtK\nkuZgyTCVkjyFLuQvq6rPtuL7kxxTVTvbpZkHWvkOYOXA6ita2RNU1QZgA8D4+Pic/klI0kJadd7V\n+3V/95z/mgXd/jDvuglwCXBnVf3XgUWbgLVtei1w1UD52e3dNycAuwYu8UiS9rNhRvQvBd4E3Jbk\n1lb2XuB84PIk64B7gTPasmuA04BJ4BHgnJG2WJI0J7MGfVX9HZA9LD55hvoFnDvPdkmSRsRPxkpS\nzxn0ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9JPWcQS9JPWfQS1LPGfSS1HMGvST1nEEvST1n0EtS\nzxn0ktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUc8P8OPhHkjyQ5PaBsiOTXJvkrnZ7RCtPkouSTCbZmuT4\nhWy8JGl2w4zoPwacslvZecDmqloNbG7zAKcCq9vfeuDi0TRTkrSvZg36qvoS8J3ditcAG9v0RuD0\ngfJLq3MjsDTJMaNqrCRp7vb1Gv2yqtrZpu8DlrXp5cC2gXrbW9n/I8n6JBNJJqampvaxGZKk2cz7\nxdiqKqD2Yb0NVTVeVeNjY2PzbYYkaQ/2Nejvn74k024faOU7gJUD9Va0MknSItnXoN8ErG3Ta4Gr\nBsrPbu++OQHYNXCJR5K0CJbMViHJJ4ATgaOTbAf+ADgfuDzJOuBe4IxW/RrgNGASeAQ4ZwHaLEma\ng1mDvqrO2sOik2eoW8C5822UJGl0/GSsJPWcQS9JPWfQS1LPGfSS1HMGvST1nEEvST1n0EtSzxn0\nktRzBr0k9ZxBL0k9Z9BLUs8Z9JLUcwa9JPWcQS9JPWfQS1LPGfSS1HMGvST13IIEfZJTknwjyWSS\n8xZiH5Kk4Yw86JMcDPw34FTg+cBZSZ4/6v1IkoazECP6FwOTVXV3Vf0I+CSwZgH2I0kawkIE/XJg\n28D89lYmSVoEqarRbjB5HXBKVf1Wm38T8MtV9Tu71VsPrG+zzwW+MdKG7N3RwLf34/72N/t34Opz\n38D+jdrPVtXYbJWWLMCOdwArB+ZXtLInqKoNwIYF2P+skkxU1fhi7Ht/sH8Hrj73DezfYlmISzdf\nBlYnOTbJIcCZwKYF2I8kaQgjH9FX1aNJfgf4AnAw8JGqumPU+5EkDWchLt1QVdcA1yzEtkdkUS4Z\n7Uf278DV576B/VsUI38xVpL05OJXIEhSzxn0s0hyYpLPLXY79qa18VcG5t+S5OzFbJPmLsl4kov2\nsOzlSe5IcmuS5Umu2N/tm8koz7Uk791t/u9Hsd057P/tSe5Mctk+rPve2WstHi/dzCLJicDvVtWv\nL3Zb9iTJ+4GHq+qPF7ste5MkdOfc44vdlgNNkg8Bf1dV/2Ox27JQkjxcVc9YxP1/HXhlVW3fh3UX\nte2zqqoD5g84G9gKfBX4OLAKuK6VbQae3ep9DLgYuBG4GzgR+AhwJ/Cxge29GvhfwC3Ap4FntPJT\ngK+38ouAz9E9+7kLGGt1DgImp+cXqL9/CWwB7gDWD7TtlnYfbG73wX10n1W4FXg58H66f04Ax7X7\nYStwJXBEK78BuAC4GfgH4OUL1IdVdB+Gu7T1Yy1wG3A7cMFAvYeBD7Y6f0P3VRo3tOP32oFt/W3r\n/y3Ar7TyE1vdK9pxu4yfDmJeBPx9u79uBp5J926wD9K9FXgr8NsLfN4eBlzd2nA78IY9tOtE4HMz\nrP9bwHeAb7a+rQJu38/n3cPAH7X23ggsa+WD59oNwIXABN1j7UXAZ+keN384y/bPBx5r5/Bl0/ts\nt2nH6/Z27rxhtuO+D33+EPCjtv330OXCV9oxem6r8+bWn8+3Pn1gL22fqY8H02XTdD/eBfwccMtA\nO1YPzo/smC7kCT7ik+8X6ALp6DZ/JPBXwNo2/6+Bv2zTH6P7jp3Qfc/O94BfpAvnLXThdzTwJeCw\nts57gH8PHEr3FQ6r2/qX0x58wB8A72zTrwY+s8B9PrLdPq2dHMta247dbfn7aQ+2GR58W4FfbdP/\nEfiTgQflf2nTpwF/s0B9WAU8DpwAPAv4FjBG946v64DTW70CTm3TVwJ/DTwFeAFwayt/OnDowANi\nok2fCOyi+3DeQXQP0pcBh9D9o3hRq/czbb/rgfe1sqfSBdOxC3gcfxP484H5w/fQrhOZIegHzunX\nDdynCxn0u593R7Xj869a+QcG7r/Bc+0G2j9v4B3A/waOaffxduCoPW2/zT+8Wzumg/43gWvpgnJZ\nO4eO2dNxn0e/76HLhZ8BlrSyV9Ie53RBf3c7focC9wIr99D2me7DXwKuHaiztN1eDxzXpv8z8LZR\nH9MD6Rr9ScCnq+rbAFX1HeAlwF+05R+ne3BP+6vq7rnbgPur6rbqLhncQfdAOYHu2zX/Z5Jb6Uaa\nPws8D/hmVd3V1h98qvwRumcV0P1j+ejIe/lEb08yPYJaSRdQX6qqb8JP7oM9SnI43cn0xVa0EXjF\nQJXPttstdPfJQrm3qm6kG+HdUFVTVfUo3Qhsuj0/ohspQXfMvlhVP27T0217CvDnSW6jewY2+K2o\nN1fV9naMb23rPBfYWVVfBqiq77X9vho4ux33m+gehKtH3+2fuA14VZILkrwcePYe2vVksft5t5ru\n+Ey/VrW382X6w5G3AXdU1c6q+iFdQE5/Yn6m7e/Ny4BPVNVjVXU/8EW6cwlmPu7zdTjw6SS30z1D\n+YWBZZuraldV/QD4Gl1mzGSmPt4N/PMkf5bkFLoBKMCHgXPaN/++gZ9m2sgsyPvonyR+2G4fH5ie\nnl9C91Tr2qo6a3ClJMftaYNVtS3J/UlOoru08MbRNvkJ7TiRbjTxkqp6JMkNdCfy80a4m+n75TEW\n9lz4/hB1ftz+scLAMauqx5NMt+1dwP10o/yDgB8MrD94jGfrT+hGTV8Yol3zVlX/kOR4umdOf0j3\nTGavknyBbvQ6Ue17o/aHPZx3h/LE47O3+3evj7u9bH9fzeW4D+s/AddX1W8kWUX3TGXo/e2pj1X1\nUJIXAL8GvAU4g27A+Bm6qwXXAVuq6sER9OEJDqQR/XXA65McBZDkSLrrZ2e25W+ku347rBuBlyZ5\nTtveYUl+nu5a36okP9fqnbXbeh+mG+V/uqoe26eeDOdw4KF2ojyP7hnIocArkhzb2nxkq/tPdNd4\nn6CqdgEPtVEkwJvoRkOL5WbgV5Mc3UYvZ82xPYfTjYQfp+vLwbPU/wZwTJIXASR5Zvun8QXgrUme\n0sp/Pslhc+zL0JI8C3ikuhdSPwj88h7a9RNV9WtVddz+DPlmpvNuf23/x9PHZDd/C7whycFJxuie\nBd484nbt3sbp7+d685DrDLZ9xj4mORo4qKo+A7wPOB6gPTv4At3rigtyleCAGdFX1R1J/gj4YpLH\n6F4oeRvw0ST/FpgCzpnD9qaSvBn4RJKntuL3tdHXeuDqJI/QnWSDIbqJ7mAs9GWbzwNvSXInXWDd\nSNfH9cBnkxwEPAC8iu61iiuSrKG7TwatBT6U5Ol0Tx2Hvo9Grap2tl8cu55uVH11VV01h038d+Az\n7e18n2eWZwpV9aMkbwD+LMnTgP9DN9L6MN1T/FvaO4GmgNPn2p85+EXgg0keB34MvJWu/7u368lg\npvNuf21/A7A1yS1VNfhs+Uq6y7RfpXut4Peq6r4WogvhA8DGJO+jexF9GD9pO90ofaY+LqfLq+kB\n9u8PrH8Z8Bt0r02NnG+vnKMk48CFVfXyWStL0hCS/C5weFX9u4XY/gEzon8yaKPRt7KA1+Yl/f8l\nyZV0b7M8acH24YhekvrtQHoxVpK0Dwx6Seo5g16Ses6gl6SeM+glqecMeknquf8LOzZr63G7KQkA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f76213f5750>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "my_tags = ['sci-fi' , 'action', 'comedy', 'fantasy', 'animation', 'romance']\n",
    "df.tag.value_counts().plot(kind=\"bar\", rot=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The data is very unbalanced. We have Comedy as majority class. \n",
    "\n",
    "A naive classifier that predicts everything to be comedy already achieves __40%__ accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The language in sci-fi plots differs a lot from action plots, so there should be some signal here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>movieId</th>\n",
       "      <th>plot</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>A little boy named Andy loves to be in his roo...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>When two kids find and play a magical board ga...</td>\n",
       "      <td>fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>Things don't seem to change much in Wabasha Co...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>Hunters and their prey--Neil and his professio...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>An ugly duckling having undergone a remarkable...</td>\n",
       "      <td>romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>Some terrorists kidnap the Vice President of t...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>James Bond teams up with the lone survivor of ...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>15</td>\n",
       "      <td>Morgan Adams and her slave, William Shaw, are ...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>17</td>\n",
       "      <td>When Mr. Dashwood dies, he must leave the bulk...</td>\n",
       "      <td>romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>18</td>\n",
       "      <td>This movie features the collaborative director...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10</td>\n",
       "      <td>19</td>\n",
       "      <td>Ace Ventura, emerging from self-imposed exile ...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>11</td>\n",
       "      <td>29</td>\n",
       "      <td>Krank (Daniel Emilfork), who cannot dream, kid...</td>\n",
       "      <td>sci-fi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>12</td>\n",
       "      <td>32</td>\n",
       "      <td>In a future world devastated by disease, a con...</td>\n",
       "      <td>sci-fi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>13</td>\n",
       "      <td>34</td>\n",
       "      <td>Farmer Hoggett wins a runt piglet at a local f...</td>\n",
       "      <td>fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>14</td>\n",
       "      <td>39</td>\n",
       "      <td>A rich high school student tries to boost a ne...</td>\n",
       "      <td>romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>15</td>\n",
       "      <td>44</td>\n",
       "      <td>Based on the popular video game of the same na...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>16</td>\n",
       "      <td>48</td>\n",
       "      <td>Capt. John Smith leads a rag-tag band of Engli...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>17</td>\n",
       "      <td>50</td>\n",
       "      <td>Following a truck hijack in New York, five con...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>18</td>\n",
       "      <td>57</td>\n",
       "      <td>After losing her job, making out with her soon...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>19</td>\n",
       "      <td>69</td>\n",
       "      <td>Two homies, Smokey and Craig, smoke a dope dea...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>20</td>\n",
       "      <td>70</td>\n",
       "      <td>Two criminals and their hostages unknowingly s...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>21</td>\n",
       "      <td>76</td>\n",
       "      <td>(SIRIUS 6B, Year 2078) On a distant mining pla...</td>\n",
       "      <td>sci-fi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>22</td>\n",
       "      <td>82</td>\n",
       "      <td>In an anonymous Dutch village, a sturdy, stron...</td>\n",
       "      <td>fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>23</td>\n",
       "      <td>88</td>\n",
       "      <td>Comedy about the prospective Washington State ...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>24</td>\n",
       "      <td>95</td>\n",
       "      <td>\"Broken Arrow\" is the term used to describe a ...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>25</td>\n",
       "      <td>104</td>\n",
       "      <td>A rejected hockey player puts his skills to th...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>26</td>\n",
       "      <td>105</td>\n",
       "      <td>Photographer Robert Kincaid wanders into the l...</td>\n",
       "      <td>romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>27</td>\n",
       "      <td>110</td>\n",
       "      <td>When his secret bride is executed for assaulti...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>28</td>\n",
       "      <td>141</td>\n",
       "      <td>Armand Goldman owns a popular drag nightclub i...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>29</td>\n",
       "      <td>145</td>\n",
       "      <td>Marcus Burnett is a hen-pecked family man. Mik...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2415</th>\n",
       "      <td>2415</td>\n",
       "      <td>139863</td>\n",
       "      <td>The anime's story is set in 2027, one year aft...</td>\n",
       "      <td>sci-fi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2416</th>\n",
       "      <td>2416</td>\n",
       "      <td>139895</td>\n",
       "      <td>Agasyta, an urban bengali who seamlessly shutt...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2417</th>\n",
       "      <td>2417</td>\n",
       "      <td>139909</td>\n",
       "      <td>When a high-ranking government official's daug...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2418</th>\n",
       "      <td>2418</td>\n",
       "      <td>140345</td>\n",
       "      <td>In this feature-length film based on the \"Flin...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2419</th>\n",
       "      <td>2419</td>\n",
       "      <td>140539</td>\n",
       "      <td>One night, Arlequin come to see his lover Colo...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2420</th>\n",
       "      <td>2420</td>\n",
       "      <td>140545</td>\n",
       "      <td>The first all-animated film in history, a seri...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2421</th>\n",
       "      <td>2421</td>\n",
       "      <td>140553</td>\n",
       "      <td>Cartoon figures announce, via comic strip ball...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2423</th>\n",
       "      <td>2423</td>\n",
       "      <td>141741</td>\n",
       "      <td>Tale of vengeance -- outlaw style -- as Red Pi...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2424</th>\n",
       "      <td>2424</td>\n",
       "      <td>142450</td>\n",
       "      <td>Madoka Kaname used to be a normal girl living ...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2425</th>\n",
       "      <td>2425</td>\n",
       "      <td>142663</td>\n",
       "      <td>Everyone's favorite nursery rhyme characters c...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2426</th>\n",
       "      <td>2426</td>\n",
       "      <td>142975</td>\n",
       "      <td>A team of highly skilled Navy SEALS find thems...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2427</th>\n",
       "      <td>2427</td>\n",
       "      <td>142997</td>\n",
       "      <td>The Drac pack is back for an all-new monster c...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2428</th>\n",
       "      <td>2428</td>\n",
       "      <td>143245</td>\n",
       "      <td>From Mark Osborne comes the first-ever animate...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2429</th>\n",
       "      <td>2429</td>\n",
       "      <td>143695</td>\n",
       "      <td>The Falls is a feature film about two missiona...</td>\n",
       "      <td>romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2430</th>\n",
       "      <td>2430</td>\n",
       "      <td>143874</td>\n",
       "      <td>Three people in Tokyo take a surreal voyage of...</td>\n",
       "      <td>fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2431</th>\n",
       "      <td>2431</td>\n",
       "      <td>143880</td>\n",
       "      <td>Steve Austin stars as U.S. Border Patrol agent...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2432</th>\n",
       "      <td>2432</td>\n",
       "      <td>143969</td>\n",
       "      <td>Adolf Hitler wakes up to find himself in the 2...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2433</th>\n",
       "      <td>2433</td>\n",
       "      <td>145066</td>\n",
       "      <td>With her bold, straightforward style of comedy...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2434</th>\n",
       "      <td>2434</td>\n",
       "      <td>145759</td>\n",
       "      <td>When a US plane is taken down over seas by an ...</td>\n",
       "      <td>sci-fi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2435</th>\n",
       "      <td>2435</td>\n",
       "      <td>145935</td>\n",
       "      <td>Charlie Brown, Lucy, Snoopy, and the whole gan...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2436</th>\n",
       "      <td>2436</td>\n",
       "      <td>146114</td>\n",
       "      <td>Raju is a joker, a clown. It is what he is and...</td>\n",
       "      <td>romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2437</th>\n",
       "      <td>2437</td>\n",
       "      <td>146427</td>\n",
       "      <td>On their way to find Paris, Sture the dog (Has...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2438</th>\n",
       "      <td>2438</td>\n",
       "      <td>146684</td>\n",
       "      <td>Scrat's epic pursuit of the elusive acorn cata...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2440</th>\n",
       "      <td>2440</td>\n",
       "      <td>147781</td>\n",
       "      <td>Mourning the loss of her brother/collaborator,...</td>\n",
       "      <td>action</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2441</th>\n",
       "      <td>2441</td>\n",
       "      <td>148382</td>\n",
       "      <td>A modern Cinderella in Paris: While the plain ...</td>\n",
       "      <td>romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2443</th>\n",
       "      <td>2443</td>\n",
       "      <td>148618</td>\n",
       "      <td>Three kids who travel back in time to 65 milli...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2444</th>\n",
       "      <td>2444</td>\n",
       "      <td>148624</td>\n",
       "      <td>Top Cat and the gang face a new police chief, ...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2445</th>\n",
       "      <td>2445</td>\n",
       "      <td>149088</td>\n",
       "      <td>Raggedy Ann and the rest of the toys in Marcel...</td>\n",
       "      <td>animation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2446</th>\n",
       "      <td>2446</td>\n",
       "      <td>149406</td>\n",
       "      <td>Continuing his \"legendary adventures of awesom...</td>\n",
       "      <td>comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2447</th>\n",
       "      <td>2447</td>\n",
       "      <td>151451</td>\n",
       "      <td>A romance fantasy humorous situations cleverly...</td>\n",
       "      <td>romance</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2427 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0  movieId                                               plot  \\\n",
       "0              0        1  A little boy named Andy loves to be in his roo...   \n",
       "1              1        2  When two kids find and play a magical board ga...   \n",
       "2              2        3  Things don't seem to change much in Wabasha Co...   \n",
       "3              3        6  Hunters and their prey--Neil and his professio...   \n",
       "4              4        7  An ugly duckling having undergone a remarkable...   \n",
       "5              5        9  Some terrorists kidnap the Vice President of t...   \n",
       "6              6       10  James Bond teams up with the lone survivor of ...   \n",
       "7              7       15  Morgan Adams and her slave, William Shaw, are ...   \n",
       "8              8       17  When Mr. Dashwood dies, he must leave the bulk...   \n",
       "9              9       18  This movie features the collaborative director...   \n",
       "10            10       19  Ace Ventura, emerging from self-imposed exile ...   \n",
       "11            11       29  Krank (Daniel Emilfork), who cannot dream, kid...   \n",
       "12            12       32  In a future world devastated by disease, a con...   \n",
       "13            13       34  Farmer Hoggett wins a runt piglet at a local f...   \n",
       "14            14       39  A rich high school student tries to boost a ne...   \n",
       "15            15       44  Based on the popular video game of the same na...   \n",
       "16            16       48  Capt. John Smith leads a rag-tag band of Engli...   \n",
       "17            17       50  Following a truck hijack in New York, five con...   \n",
       "18            18       57  After losing her job, making out with her soon...   \n",
       "19            19       69  Two homies, Smokey and Craig, smoke a dope dea...   \n",
       "20            20       70  Two criminals and their hostages unknowingly s...   \n",
       "21            21       76  (SIRIUS 6B, Year 2078) On a distant mining pla...   \n",
       "22            22       82  In an anonymous Dutch village, a sturdy, stron...   \n",
       "23            23       88  Comedy about the prospective Washington State ...   \n",
       "24            24       95  \"Broken Arrow\" is the term used to describe a ...   \n",
       "25            25      104  A rejected hockey player puts his skills to th...   \n",
       "26            26      105  Photographer Robert Kincaid wanders into the l...   \n",
       "27            27      110  When his secret bride is executed for assaulti...   \n",
       "28            28      141  Armand Goldman owns a popular drag nightclub i...   \n",
       "29            29      145  Marcus Burnett is a hen-pecked family man. Mik...   \n",
       "...          ...      ...                                                ...   \n",
       "2415        2415   139863  The anime's story is set in 2027, one year aft...   \n",
       "2416        2416   139895  Agasyta, an urban bengali who seamlessly shutt...   \n",
       "2417        2417   139909  When a high-ranking government official's daug...   \n",
       "2418        2418   140345  In this feature-length film based on the \"Flin...   \n",
       "2419        2419   140539  One night, Arlequin come to see his lover Colo...   \n",
       "2420        2420   140545  The first all-animated film in history, a seri...   \n",
       "2421        2421   140553  Cartoon figures announce, via comic strip ball...   \n",
       "2423        2423   141741  Tale of vengeance -- outlaw style -- as Red Pi...   \n",
       "2424        2424   142450  Madoka Kaname used to be a normal girl living ...   \n",
       "2425        2425   142663  Everyone's favorite nursery rhyme characters c...   \n",
       "2426        2426   142975  A team of highly skilled Navy SEALS find thems...   \n",
       "2427        2427   142997  The Drac pack is back for an all-new monster c...   \n",
       "2428        2428   143245  From Mark Osborne comes the first-ever animate...   \n",
       "2429        2429   143695  The Falls is a feature film about two missiona...   \n",
       "2430        2430   143874  Three people in Tokyo take a surreal voyage of...   \n",
       "2431        2431   143880  Steve Austin stars as U.S. Border Patrol agent...   \n",
       "2432        2432   143969  Adolf Hitler wakes up to find himself in the 2...   \n",
       "2433        2433   145066  With her bold, straightforward style of comedy...   \n",
       "2434        2434   145759  When a US plane is taken down over seas by an ...   \n",
       "2435        2435   145935  Charlie Brown, Lucy, Snoopy, and the whole gan...   \n",
       "2436        2436   146114  Raju is a joker, a clown. It is what he is and...   \n",
       "2437        2437   146427  On their way to find Paris, Sture the dog (Has...   \n",
       "2438        2438   146684  Scrat's epic pursuit of the elusive acorn cata...   \n",
       "2440        2440   147781  Mourning the loss of her brother/collaborator,...   \n",
       "2441        2441   148382  A modern Cinderella in Paris: While the plain ...   \n",
       "2443        2443   148618  Three kids who travel back in time to 65 milli...   \n",
       "2444        2444   148624  Top Cat and the gang face a new police chief, ...   \n",
       "2445        2445   149088  Raggedy Ann and the rest of the toys in Marcel...   \n",
       "2446        2446   149406  Continuing his \"legendary adventures of awesom...   \n",
       "2447        2447   151451  A romance fantasy humorous situations cleverly...   \n",
       "\n",
       "            tag  \n",
       "0     animation  \n",
       "1       fantasy  \n",
       "2        comedy  \n",
       "3        action  \n",
       "4       romance  \n",
       "5        action  \n",
       "6        action  \n",
       "7        action  \n",
       "8       romance  \n",
       "9        comedy  \n",
       "10       comedy  \n",
       "11       sci-fi  \n",
       "12       sci-fi  \n",
       "13      fantasy  \n",
       "14      romance  \n",
       "15       action  \n",
       "16    animation  \n",
       "17       comedy  \n",
       "18       comedy  \n",
       "19       comedy  \n",
       "20       action  \n",
       "21       sci-fi  \n",
       "22      fantasy  \n",
       "23       comedy  \n",
       "24       action  \n",
       "25       comedy  \n",
       "26      romance  \n",
       "27       action  \n",
       "28       comedy  \n",
       "29       action  \n",
       "...         ...  \n",
       "2415     sci-fi  \n",
       "2416     comedy  \n",
       "2417     action  \n",
       "2418  animation  \n",
       "2419  animation  \n",
       "2420  animation  \n",
       "2421  animation  \n",
       "2423  animation  \n",
       "2424  animation  \n",
       "2425  animation  \n",
       "2426     action  \n",
       "2427  animation  \n",
       "2428  animation  \n",
       "2429    romance  \n",
       "2430    fantasy  \n",
       "2431     action  \n",
       "2432     comedy  \n",
       "2433     comedy  \n",
       "2434     sci-fi  \n",
       "2435  animation  \n",
       "2436    romance  \n",
       "2437  animation  \n",
       "2438  animation  \n",
       "2440     action  \n",
       "2441    romance  \n",
       "2443  animation  \n",
       "2444  animation  \n",
       "2445  animation  \n",
       "2446     comedy  \n",
       "2447    romance  \n",
       "\n",
       "[2427 rows x 4 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def print_plot(index):\n",
    "    example = df[df.index == index][['plot', 'tag']].values[0]\n",
    "    if len(example) > 0:\n",
    "        print(example[0])\n",
    "        print('Genre:', example[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In a future world devastated by disease, a convict is sent back in time to gather information about the man-made virus that wiped out most of the human population on the planet.\n",
      "('Genre:', 'sci-fi')\n"
     ]
    }
   ],
   "source": [
    "print_plot(12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Based on the popular video game of the same name \"Mortal Kombat\" tells the story of an ancient tournament where the best of the best of different Realms fight each other. The goal - ten wins to be able to legally invade the losing Realm. Outworld has so far collected nine wins against Earthrealm, so it's up to Lord Rayden and his fighters to stop Outworld from reaching the final victory...\n",
      "('Genre:', 'action')\n"
     ]
    }
   ],
   "source": [
    "print_plot(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Train/test split of 90/10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data, test_data = train_test_split(df, test_size=0.1, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "243"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f75e8c06dd0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEUJJREFUeJzt3X2QXXV9x/H3lwSMgIaH7GQgoJtqhME6ol0QRJQBRBQr\nOFKBcTRanFSnPtcKMrQ6rXZAbPFhWp3IU2opVQIIggPFQFBrATcxEkJUMgEkNMBSEUWqPOTbP85v\n9RJ2c+/u3nt38/P9mtm595x7Hr6/c8793N8592EjM5Ekbf92mO4CJEndYaBLUiUMdEmqhIEuSZUw\n0CWpEga6JFXCQJekShjoklQJA12SKjG7nyubN29eDg4O9nOVkrTdW7Vq1UOZOdBuur4G+uDgIMPD\nw/1cpSRt9yLink6m85KLJFXCQJekShjoklQJA12SKmGgS1IlDHRJqoSBLkmVMNAlqRIGuiRVoq/f\nFJ2swdOv6ev67j7ruL6uT5K6wR66JFXCQJekShjoklQJA12SKmGgS1IlDHRJqoSBLkmVMNAlqRIG\nuiRVwkCXpEp0FOgR8eGIWBcRt0fEJRExJyIWRsQtEbEhIr4WETv1ulhJ0vjaBnpELAA+AAxl5h8D\ns4CTgbOBczPzhcDDwKm9LFSStG2dXnKZDTw7ImYDOwObgSOB5eXxZcAJ3S9PktSptoGemfcBnwV+\nRhPkjwCrgF9k5pNlsk3AgrHmj4glETEcEcMjIyPdqVqS9AydXHLZHTgeWAjsDewCHNvpCjJzaWYO\nZebQwMDApAuVJG1bJ5dcjgbuysyRzHwCuBw4DNitXIIB2Ae4r0c1SpI60Emg/ww4JCJ2jogAjgLu\nAG4ETizTLAau7E2JkqROdHIN/RaaNz9XA2vLPEuB04CPRMQGYE/g/B7WKUlqo6N/QZeZnwA+sdXo\njcDBXa9IkjQpflNUkiphoEtSJQx0SaqEgS5JlTDQJakSBrokVcJAl6RKGOiSVAkDXZIqYaBLUiUM\ndEmqhIEuSZUw0CWpEga6JFXCQJekShjoklQJA12SKmGgS1IlDHRJqoSBLkmVMNAlqRIGuiRVwkCX\npEoY6JJUCQNdkiphoEtSJQx0SaqEgS5JlTDQJakSBrokVcJAl6RKGOiSVAkDXZIqYaBLUiUMdEmq\nhIEuSZUw0CWpEga6JFXCQJekSnQU6BGxW0Qsj4gfR8T6iDg0IvaIiOsj4s5yu3uvi5Ukja/THvrn\ngWszc3/gpcB64HRgRWYuAlaUYUnSNGkb6BExF3g1cD5AZj6emb8AjgeWlcmWASf0qkhJUnud9NAX\nAiPAhRHxw4g4LyJ2AeZn5uYyzf3A/LFmjoglETEcEcMjIyPdqVqS9AydBPps4OXAlzLzZcCv2ery\nSmYmkGPNnJlLM3MoM4cGBgamWq8kaRydBPomYFNm3lKGl9ME/AMRsRdAuX2wNyVKkjrRNtAz837g\n3ojYr4w6CrgDuApYXMYtBq7sSYWSpI7M7nC69wMXR8ROwEbgXTQvBl+PiFOBe4C39qZESVInOgr0\nzFwDDI3x0FHdLUeSNFl+U1SSKmGgS1IlDHRJqoSBLkmVMNAlqRIGuiRVwkCXpEoY6JJUCQNdkiph\noEtSJQx0SaqEgS5JlTDQJakSBrokVcJAl6RKGOiSVAkDXZIqYaBLUiUMdEmqhIEuSZUw0CWpEga6\nJFXCQJekShjoklQJA12SKmGgS1IlDHRJqoSBLkmVMNAlqRIGuiRVwkCXpEoY6JJUCQNdkiphoEtS\nJQx0SaqEgS5JlTDQJakSBrokVWL2dBcgGDz9mr6t6+6zjuvbuiT1V8c99IiYFRE/jIiry/DCiLgl\nIjZExNciYqfelSlJamcil1w+CKxvGT4bODczXwg8DJzazcIkSRPTUaBHxD7AccB5ZTiAI4HlZZJl\nwAm9KFCS1JlOe+ifAz4GbCnDewK/yMwny/AmYMFYM0bEkogYjojhkZGRKRUrSRpf20CPiDcCD2bm\nqsmsIDOXZuZQZg4NDAxMZhGSpA508imXw4A3RcQbgDnAc4HPA7tFxOzSS98HuK93ZUqS2mnbQ8/M\nj2fmPpk5CJwM3JCZbwNuBE4sky0GruxZlZKktqbyxaLTgI9ExAaaa+rnd6ckSdJkTOiLRZm5ElhZ\n7m8EDu5+SZKkyfCr/5JUCQNdkiphoEtSJQx0SaqEgS5JlTDQJakSBrokVcJ/cKGe6uc/7wD/gYf+\nsNlDl6RKGOiSVAkDXZIq4TV0aQp8j0AziT10SaqEgS5JlTDQJakSBrokVcI3RSWNyzd9ty/20CWp\nEga6JFXCQJekShjoklQJA12SKmGgS1IlDHRJqoSBLkmVMNAlqRIGuiRVwkCXpEoY6JJUCQNdkiph\noEtSJQx0SaqEgS5JlTDQJakSBrokVcJAl6RKGOiSVAkDXZIqYaBLUiXaBnpE7BsRN0bEHRGxLiI+\nWMbvERHXR8Sd5Xb33pcrSRpPJz30J4G/yswDgEOAv4yIA4DTgRWZuQhYUYYlSdOkbaBn5ubMXF3u\n/wpYDywAjgeWlcmWASf0qkhJUnsTuoYeEYPAy4BbgPmZubk8dD8wv6uVSZImpONAj4hdgcuAD2Xm\nL1sfy8wEcpz5lkTEcEQMj4yMTKlYSdL4Ogr0iNiRJswvzszLy+gHImKv8vhewINjzZuZSzNzKDOH\nBgYGulGzJGkMnXzKJYDzgfWZ+U8tD10FLC73FwNXdr88SVKnZncwzWHA24G1EbGmjDsDOAv4ekSc\nCtwDvLU3JUqSOtE20DPze0CM8/BR3S1HkjRZflNUkiphoEtSJQx0SaqEgS5JlejkUy6SVKXB06/p\n6/ruPuu4ni7fHrokVcJAl6RKGOiSVAkDXZIqYaBLUiUMdEmqhIEuSZUw0CWpEga6JFXCQJekShjo\nklQJA12SKmGgS1IlDHRJqoSBLkmVMNAlqRIGuiRVwkCXpEoY6JJUCQNdkiphoEtSJQx0SaqEgS5J\nlTDQJakSBrokVcJAl6RKGOiSVAkDXZIqYaBLUiUMdEmqhIEuSZUw0CWpEga6JFXCQJekSkwp0CPi\n2Ij4SURsiIjTu1WUJGniJh3oETEL+Gfg9cABwCkRcUC3CpMkTcxUeugHAxsyc2NmPg78B3B8d8qS\nJE3UVAJ9AXBvy/CmMk6SNA0iMyc3Y8SJwLGZ+e4y/HbgFZn5vq2mWwIsKYP7AT+ZfLkTNg94qI/r\n67ea21dz28D2be/63b7nZ+ZAu4lmT2EF9wH7tgzvU8Y9TWYuBZZOYT2TFhHDmTk0Hevuh5rbV3Pb\nwPZt72Zq+6ZyyeUHwKKIWBgROwEnA1d1pyxJ0kRNuoeemU9GxPuA64BZwAWZua5rlUmSJmQql1zI\nzG8B3+pSLb0wLZd6+qjm9tXcNrB927sZ2b5JvykqSZpZ/Oq/JFXCQAci4oiIuHq662in1PnKluH3\nRMQ7prMmTVxEDEXEF8Z57PCIWBcRayJiQUQs73d9Y9TUteMsIs7Yavj73VjuBNb/gYhYHxEXT2Le\nM9pPNb285EITlMBHM/ON013LtkTEJ4FHM/Oz013LeCIiaI6rLdNdy/YoIr4MfC8z/226a+mFiHg0\nM3edxvX/GDg6MzdNYt5prb0jmTmj/oB3ALcBPwK+CgwCN5RxK4DnlekuAr4E3AxsBI4ALgDWAxe1\nLO8Y4L+B1cClwK5l/LHAj8v4LwBX05yx3AkMlGl2ADaMDvewzd8AVgHrgCUt9a0u22FF2Q7303zW\nfw1wOPBJmhcigAPLtrgNuALYvYxfCZwN3Ar8FDi8B/UP0nxh7F9LGxYDa4HbgbNbpnsUOKdM822a\nn49YWfbfm1qW9d3S9tXAK8v4I8q0y8t+u5jfd0gOAr5fttWtwHNoPnl1Ds3Ha28D/qIPx+4uwDWl\njtuBk8ap7Qjg6jHmfzfwc+Cu0r5B4PY+HnOPAp8utd4MzC/jW4+zlcC5wDDNc+0g4HKa582n2iz/\nLOCpcvxePLrOchtlf91ejp2T2u33SbT5y8DjZfmn0eTCD8v+2a9M887SnmtLmz6zjdrHauMsmmwa\nbceHgRcAq1vqWNQ63NX92uuDfIIb/MU0oTOvDO8BfBNYXIb/HPhGuX8Rze/HBM1vyPwSeAlNCK+i\nCbh5wHeAXco8pwF/C8yh+dmCRWX+r1OeYMAngA+V+8cAl/Wh3XuU22eXA2F+qW/hVo9/kvLEGuOJ\ndhvwmnL/74DPtTwB/7HcfwPw7R7UPwhsAQ4B9gZ+BgzQfIrqBuCEMl0Cry/3rwD+E9gReCmwpozf\nGZjTcuAPl/tHAI/QfIFtB5on46uAnWheEA4q0z23rHcJcGYZ9yyaAFrY4/34FuArLcNzx6ntCMYI\n9Jbj+sSW7dqrQN/6mNuz7J8/LeM/07L9Wo+zlZQXaeCDwP8Ae5VtvAnYc7zll+FHt6pjNNDfAlxP\nE4jzyzG013j7fQrtvpsmF54LzC7jjqY8z2kCfWPZd3OAe4B9x6l9rG34J8D1LdPsVm5vBA4s9/8B\neH8v9utMu4Z+JHBpZj4EkJk/Bw4F/r08/lWaJ/Gob2azhdYCD2Tm2mxO9dfRPBkOofklyP+KiDU0\nPcfnA/sDd2XmnWX+1tPbC2jOEqB5Abmw6618pg9ExGivaF+aMPpOZt4Fv9sO44qIuTQHzk1l1DLg\n1S2TXF5uV9Fsl164JzNvpumxrczMkcx8kqZHNVrL4zQ9H2j22U2Z+US5P1rXjsBXImItzRlV6y94\n3pqZm8o+XlPm2Q/YnJk/AMjMX5b1HgO8o+z3W2iebIu63+ynWQu8NiLOjojDgeeNU9tMsPUxt4hm\n/4y+l7StY2X0C4RrgXWZuTkzf0sThKPfHh9r+dvyKuCSzHwqMx8AbqI5lmDs/T5Vc4FLI+J2mjOO\nF7c8tiIzH8nM3wB30GTGWMZq40bgjyLiixFxLE1HE+A84F3lV2pP4veZ1lVT+hz6DPDbcrul5f7o\n8GyaU6TrM/OU1pki4sDxFpiZ90bEAxFxJM0lgbd1t+SnK9fvjwYOzczHImIlzUG7fxdXM7ptnqJ3\n+/zXHUzzRHkBhZZ9lplbImK0rg8DD9D02ncAftMyf+s+bteWoOkFXddBXV2RmT+NiJfTnAl9iubs\nZJsi4jqaHulwlt9F6rVxjrk5PH3/bGv7bvN5t43lT9ZE9nun/h64MTPfHBGDNGceHa9vvDZm5sMR\n8VLgdcB7gLfSdAwvozn7vwFYlZn/24U2PMNM66HfAPxZROwJEBF70FzfOrk8/jaa66uduhk4LCJe\nWJa3S0S8iOZa3GBEvKBMd8pW851H02u/NDOfmlRLOjcXeLgcFPvTnFXMAV4dEQtL3XuUaX9Fcw32\naTLzEeDh0isEeDtND2c63Aq8JiLmld7IKROsZS5Nr3YLTTtmtZn+J8BeEXEQQEQ8p7w4XAe8NyJ2\nLONfFBG7TLAtExIRewOPZfOG5jnAK8ap7Xcy83WZeWC/wrwY65jr1/KfGN0nW/kucFJEzIqIAZqz\nulu7XNfWNY7+9tQ7O5yntfYx2xgR84AdMvMy4Ezg5QClt38dzft+PTvrn1E99MxcFxGfBm6KiKdo\n3rB4P3BhRPw1MAK8awLLG4mIdwKXRMSzyugzS09qCXBNRDxGczC1BuVVNBu9H5dbrgXeExHracLp\nZpp2LgEuj4gdgAeB19K8n7A8Io6n2S6tFgNfjoidaU77Ot5O3ZSZm8t/r7qRppd8TWZeOYFF/Atw\nWfmY3LW06fln5uMRcRLwxYh4NvB/ND2n82hOzVeXT96MACdMtD0T9BLgnIjYAjwBvJdmG2xd23Qb\n65jr1/KXArdFxOrMbD37vYLm8uqPaK7lfywz7y9h2QufAZZFxJk0b2R34ne10/S6x2rjApq8Gu0s\nf7xl/ouBN9O8d9QTfmxxDBExBJybmYe3nViSOhARHwXmZubf9GodM6qHPhOU3uV76fG1c0l/OCLi\nCpqPLx7Z0/XYQ5ekOsy0N0UlSZNkoEtSJQx0SaqEgS5JlTDQJakSBrokVeL/AYxSJULOKjFWAAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f76213e9f10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_data.tag.value_counts().plot(kind=\"bar\", rot=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model evaluation approach\n",
    "We will use confusion matrices to evaluate all classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.Blues):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(my_tags))\n",
    "    target_names = my_tags\n",
    "    plt.xticks(tick_marks, target_names, rotation=45)\n",
    "    plt.yticks(tick_marks, target_names)\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def evaluate_prediction(predictions, target, title=\"Confusion matrix\"):\n",
    "    print('accuracy %s' % accuracy_score(target, predictions))\n",
    "    cm = confusion_matrix(target, predictions)\n",
    "    print('confusion matrix\\n %s' % cm)\n",
    "    print('(row=expected, col=predicted)')\n",
    "    \n",
    "    cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "    plot_confusion_matrix(cm_normalized, title + ' Normalized')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "def predict(vectorizer, classifier, data):\n",
    "    data_features = vectorizer.transform(data['plot'])\n",
    "    predictions = classifier.predict(data_features)\n",
    "    target = data['tag']\n",
    "    evaluate_prediction(predictions, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Baseline: bag of words, n-grams, tf-idf\n",
    "Let's start with some simple baselines before diving into more advanced methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag of words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest document feature is just a count of each word occurrence in a document.\n",
    "\n",
    "We remove stop-words and use NLTK tokenizer then limit our vocabulary to 3k most frequent words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tokenize_text(text):\n",
    "    tokens = []\n",
    "    for sent in nltk.sent_tokenize(text):\n",
    "        for word in nltk.word_tokenize(sent):\n",
    "            if len(word) < 2:\n",
    "                continue\n",
    "            tokens.append(word.lower())\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.45 s, sys: 9.1 ms, total: 1.46 s\n",
      "Wall time: 1.46 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# training\n",
    "count_vectorizer = CountVectorizer(\n",
    "    analyzer=\"word\", tokenizer=nltk.word_tokenize,\n",
    "    preprocessor=None, stop_words='english', max_features=3000) \n",
    "train_data_features = count_vectorizer.fit_transform(train_data['plot'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Multi-modal logistic regression is a simple white-box classifier. We will use either logistic regression or KNN throughout this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 359 ms, sys: 1.58 ms, total: 361 ms\n",
      "Wall time: 367 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "logreg = linear_model.LogisticRegression(n_jobs=1, C=1e5)\n",
    "logreg = logreg.fit(train_data_features, train_data['tag'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'adventurous',\n",
       " u'advice',\n",
       " u'affair',\n",
       " u'affairs',\n",
       " u'afraid',\n",
       " u'africa',\n",
       " u'african',\n",
       " u'aftermath',\n",
       " u'age',\n",
       " u'aged']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vectorizer.get_feature_names()[80:90]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Nothing impressive - only 2% better better than the classifier that thinks that everything is a comedy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy 0.423868312757\n",
      "confusion matrix\n",
      " [[21  2 10  1  4  4]\n",
      " [ 4 10  8  0  3  6]\n",
      " [10 11 45  4 15  1]\n",
      " [ 2  5  3  4  2  0]\n",
      " [ 1  3 15  2 11  3]\n",
      " [ 8  5  6  2  0 12]]\n",
      "(row=expected, col=predicted)\n",
      "CPU times: user 293 ms, sys: 32.4 ms, total: 325 ms\n",
      "Wall time: 293 ms\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAAEmCAYAAADfpHMGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8VVX9//HXm0FEUBywckocM+cUTU2NDP06a85m+sUh\n0iKntEz9mWNqlmVmGZqaQ5aa9cUhsTLMVBQ0Ucl5yoEUxHlAhPfvj7WObG6Xew7cfSb4PHmcB+fs\nvc/en7PPvZ+79lprryXbhBBC6J4ezQ4ghBDmB5FMQwihBJFMQwihBJFMQwihBJFMQwihBJFMQwih\nBJFM5yOS+kq6QdIbkq7txn72k3RrmbE1i6QtJD3W7DjqQdLJkq7Mzz8p6W1JPUs+xrOShpa5z/lV\nJNMmkPRlSePzD/8kSX+StHkJu94D+DiwlO0953Untq+yvU0J8dSVJEtatattbN9h+1PzuP9nJb0i\nqV9h2SGSxszL/urJ9r9t97c9o9mxLKgimTaYpKOBnwDfJyW+TwI/B3YpYfcrAo/b/rCEfbU9Sb1K\n2E1P4IgSYpGk+H2bj8WX20CSBgCnAt+wfb3td2xPt32D7WPzNn0k/UTSS/nxE0l98rohkl6Q9K1c\nYpok6cC87hTgJGDvXOI9uHgZmLcZlEtzvfLrYZKelvSWpGck7VdY/o/C+zaTNC5XH4yTtFlh3RhJ\np0m6M+/nVkkD5/D5K/F/uxD/rpK2l/S4pKmSji9sv7GkuyW9nrf9maSF8rq/580m5M+7d2H/35H0\nH+DSyrL8nlXyMTbIr5eVNFnSkC6+tnOAYyQtPofPVO3cnCHpTuBdYOW87HRJd+W4b5C0lKSrJL2Z\n9zGosI/zJD2f190naYs5xPHRdytp07zvyuN9Sc/m7XpIOk7SU5JelXSNpCUL+9lf0nN53QldnJfQ\nke14NOgBbAt8CPTqYptTgbHAx4ClgbuA0/K6Ifn9pwK9ge1Jv6RL5PUnA1cW9tXx9SDAQC+gH/Am\n8Km8bhlgrfx8GPCP/HxJ4DVg//y+ffPrpfL6McBTwOpA3/z6rDl8tkr8J+X4vwpMBn4DLAqsBbwH\nrJS33xDYJB93EPAIcGRhfwZW7WT/ZwN9cjxDgBcK23wV+BewCDAa+GEX38WzwFDgeuD0vOwQYMxc\nnJt/58/VK3/mMcCTwCrAgBzL4/k4vYDLgUsLMXwFWCqv+xbwH2Dhjt9v8bvt8Bl6A7cDZ+bXR5B+\nvpbP5+iXwNV53ZrA28CWed25+XwObfbvTjs8omTaWEsBU9z1Zfh+wKm2X7E9GTiF9MtaMT2vn277\nZtIP/zzVCQIzgbUl9bU9yfbETrbZAXjC9hW2P7R9NfAosFNhm0ttP277PeAaYP0ujjkdOMP2dOC3\nwEDgPNtv5eP/C1gPwPZ9tsfm4z5L+sX/fA2f6Xu2p+V4ZmP7IlIyu4f0B6SW0tdJwDclLd1heS3n\n5jLbE/P66XnZpbafsv0G8CfgKdt/yT8X1wKfKcR7pe1X8/t/REpyc/N9/xR4q/A5DwVOsP2C7Wmk\nhLxHvlrZA7jR9t/zuv9HOp+hBpFMG+tVYGCVurxlgecKr5/Lyz7aR4dk/C7Qf24Dsf0OsDfpl2uS\npJskrVFDPJWYliu8/s9cxPOqZzWSVJLdy4X171XeL2l1STdK+o+kN0n1zJ1WIRRMtv1+lW0uAtYG\nzs9Jo0u2HwZuBI7rsKqWc/N8J7vs+Hk7/fwAko6R9EiuRnidVJqtdg4q7/0aqWT+ZduVpLgi8Idc\ndfI6qbQ/g1R/v2wx3vwz8motxwqRTBvtbmAasGsX27xE+oGv+GReNi/eIV3OVnyiuNL2aNtbk0po\nj5KSTLV4KjG9OI8xzY1fkOJazfZiwPGAqryny2HQJPUnNQD+Cji5WF9YxfdIVQTFRFnLuZnnYdly\n/ei3gb1IVTmLA29Q/RxU3nsasIvtNwurnge2s7144bGw7ReBScAKhX0sQrqaCjWIZNpA+bLuJOCC\n3PCyiKTekraT9IO82dXAiZKWzg05JwFXzmmfVTwAbKnUB3EA8N3KCkkfl7SLUrefaaTqgs4u6W4G\nVlfqztVL0t6kurUb5zGmubEoqV737VxqPqzD+peBledyn+cB420fAtwEXFjLm2w/CfwOOLywuN7n\nZlFSneVkoJekk4DFqr1J0gqk6pYDbD/eYfWFwBmSVszbLi2p0pPkOmBHSZvnhr5TiRxRszhRDZbr\nvY4GTiT9kjwPjAD+mDc5HRgPPAg8BNyfl83Lsf5MSgAPAvcx+y95jxzHS8BUUl1kx2SF7VeBHUmN\nH6+SSko72p4yLzHNpWOAL5Pq/C4ifZaik4Ff50vWvartLCeNbZn1OY8GNlDuxVCDU0kNd0BDzs1o\n4BZSA9VzwPt0Xm3Q0RdJl+3XFVr0K/Xh5wGjgFslvUVqjPps/jwTgW+QGgQnkRrTXijps8z3ZMfg\n0CGE0F1RMg0hhBJEMg0hhBJEMg0hhBJEMg0hhBKUMRDEAkm9F7H6DGhqDOustmz1jRpg8jtV+73X\n3cf7L9zsEAD44MO4YajiXw/9c4rtjneNzbOei61of/hfN7X9F783ebTtbcs6bq0imc4j9RlAn/UP\nbmoMN994UlOPX/HLe55tdggctcUqzQ4BgBenVv9lX1Css8KiHe8O6xZ/+B59PlW1BxzvP3BBTXeI\nlS2SaQihPUjQo9Sxr0sVyTSE0D5aeEjYSKYhhPahqsMSNE0k0xBCm1BLl0xbN7IQQigSqc602qOW\nXUnbSnpM0pOSOg6tWJltYrKkB/LjkGr7jJJpCKFNqJTLfKUZXC8AtiYN5DJO0ijb/+qw6e9sj6h1\nv1EyDSG0D/Wo/qhuY+BJ20/b/oA040O3J7SMZBpCaB9S9UeazWJ84TG8w16WY/ahDF9g9kG/K3aX\n9KCk6/IYsV2Ky/wQQnuovZ/pFNuDu3m0G0gTDU7L07/8GtiqqzdEyTSE0D7Kucx/kcL0LKSZWmeb\nhidPYli5T/pi0ky5XYpkGkJoEyormY4DVpO0Up6eZR/S7AOzjiQtU3i5M2niwS7FZX4IoT0I6Nn9\n20ltfyhpBGlamJ7AJbYnSjqVND/YKOBwSTuT5uCaCgyrtt8FJplKGkyaYOzwTtZtQZpobDppLvTz\nbO/R4BBDCNWUdAeU7ZtJEyIWl51UeP5dChNQ1mKBSaa2x5MmquvMfsCZtiuzgEYiDaHlxB1QdSWp\nn6SbJE2Q9LCkvSVtJOmuvOxeSYtKGiLpv6bgzXc27AWcJukqSYMkPdz4TxJCqKq2rlFNMT+UTLcF\nXrK9A0CeH/6fwN62x0laDJjjIJO2L5a0OXCj7eskDZrTtrm/Wuqz1qfq9OUhhDK1+BB8bV8yJc0t\nv7Wks3Pd5yeBSbbHAdh+0/aHZRzI9kjbg20PVq9FythlCGFulNOaXxdtn0xtPw5sQEqqpwO7VXuP\npNF58IKL6x1fCKFEcZlfP5KWBabavlLS68DXgWUkbZQv8xelw2W+7f9pRqwhhO5o7Qaotk+mwDrA\nOZJmkro2HUbqkXa+pL6kRDq0ifGFEMpQGYKvRbV9MrU9mtT5tqNNOrwekx+d7WNY4fmzwNqlBBdC\nKFGUTEMIoRwxbUkIIZQgLvNDCKGbFJf5IYRQjrjMDyGE7lMk0xBC6J50lR/JNIQQuklRMg0hhDJE\nMg0hhBL06BGt+SGE0D3KjxYVyXQefWqlT3DJZcc0NYYDr7q/qcevWGO5Ac0Ogb4LtUZnbjc7AOD+\nSVObHUJdKOpMQwihHJFMQwihBFFnGkII3RV1piGEUI64zA8hhG6KBqgQQihJ3E4aQgjdpbjMDyGE\nUkQyDSGEbhKKrlEhhFCK1i2Y0rppPoQQinKdabVHTbuStpX0mKQnJR3XxXa7S7KkwdX2Gck0hNA2\nykimknoCFwDbAWsC+0pas5PtFgWOAO6pJbZIpiGEtqEeqvqowcbAk7aftv0B8Ftgl062Ow04G3i/\nlp3Od8lU0hBJmxVeHyrpgGbGFEIoR0mX+csBzxdev5CXFY+zAbCC7ZtqjW1+bIAaArwN3AVg+8Km\nRhNCKMVcJMuBksYXXo+0PXIujtMDOBcYNjfxtU0ylfRHYAVgYeA82yMlbQt8H+gJTAEOBg4FZkj6\nCvBN4IvA27Z/KGl94EJgEeAp4CDbr0kaQ6oX+QKwOHCw7Tsa+gFDCFXVmEyn2O6qwehFUi6pWD4v\nq1gUWBsYk4/3CWCUpJ1tF5P0bNommZIS31RJfYFxkv4PuAjY0vYzkpbM6y8kJ08ASV8s7ONy4Ju2\nb5d0KvA94Mi8rpftjSVtn5cPbdgnCyHUpKTbSccBq0laiZRE9wG+XFlp+w1g4EfHTIWtY7pKpNBe\ndaaHS5oAjCX9VRkO/N32MwC2uxxeXNIAYHHbt+dFvwa2LGxyff7/PmDQHPYxXNJ4SeNfmzplnj9I\nCGHelFFnavtDYAQwGngEuMb2REmnStp5XmNri5KppCGkkuKmtt/NfykeANYo8TDT8v8zmMN5yfUu\nIwE+vc5nWmGGihAWHCXem2/7ZuDmDstOmsO2Q2rZZ7uUTAcAr+VEugawCanudMtcVEfSknnbt0h1\nHrPJRffXJG2RF+0P3N5xuxBCa0q3k1Z/NEtblEyBW4BDJT0CPEa61J9MutS/Pre+vQJsDdwAXCdp\nF1IDVNH/AhdKWgR4GjiwQfGHEErQwuOctEcytT2NdLdCZ/7UYdvHgXULi+4orHuAVKrtuP8hhedT\nmEOdaQihuWLUqBBC6C5FyTSEELpNQM+erZtNI5mGENpGXOaHEEJ3xWV+CCF0n4iSaQghlKC5/Uir\niWQaQmgbUTINIYTuijrTEELoPkFc5ocQQhniMj+EEErQwrk0kum8EtC7yXdjnPo/ZY5AOO++sOeJ\nzQ6B44ac2+wQAHhsypvNDoHd112+2SEAcznnRy1KHIKvHiKZhhDagqJrVAghlKOFC6aRTEMI7SMu\n80MIoZuk6BoVQgiliJJpCCGUoIVzaSTTEEL7iJJpCCF0kxRdo0IIoRQtXDCNZBpCaB89WjibzjGZ\nSlqsqzfabv59cyGEBUoL59IuS6YTAZNuQ6+ovDbwyTrGFUIIs5GgZzvWmdpeoZGBhBBCNa3cmt+j\nlo0k7SPp+Px8eUkb1jes+pM0RNKNzY4jhFA7qfqjWaomU0k/A74A7J8XvQtcWM+gQgihIwE9paqP\nZqmlZLqZ7a8B7wPYngosVGYQkg6Q9KCkCZKukDRI0m152V8lfTJvd5mkX0gaK+npXLq8RNIjki4r\n7G8bSXdLul/StZL65+XbSnpU0v3AbnlZD0lPSFq68PrJyusQQouQUA2P2nalbSU9ln/Xj+tk/aGS\nHpL0gKR/SFqz2j5rSabTJfUgNTohaSlgZk0R10DSWsCJwFa21wOOAM4Hfm17XeAq4KeFtywBbAoc\nBYwCfgysBawjaX1JA/P+htreABgPHC1pYeAiYCdgQ+ATALZnAlcC++X9DwUm2J7cSazDJY2XNP61\nqa+WdQpCCDUq4zJfUk/gAmA7YE1g306S5W9sr2N7feAHQNXRx2tJphcAvweWlnQK8A/g7BreV6ut\ngGttT4GPSr6bAr/J668ANi9sf4NtAw8BL9t+KCfEicAgYBPSCbpT0gPA/wIrAmsAz9h+Ir//ysI+\nLwEOyM8PAi7tLFDbI20Ptj14iSWX6ubHDiHMDZH6mVZ71GBj4EnbT9v+APgtsEtxgw5dP/uRC5Nd\nqdpp3/blku4jldgA9rT9cC0R18m0/P/MwvPK617ADODPtvctvknS+nPaoe3nJb0saSvSid5vTtuG\nEJqnxttJB0oaX3g90vbIwuvlgOcLr18APttxJ5K+ARxNqtbcqmpstUQG9ASmAx/MxXtqdRuwZ64+\nQNKSwF3APnn9fsAdc7G/scDnJK2a99dP0urAo8AgSavk7fbt8L6LSaXVa23PmKdPEkKom1ou8XPB\ndErlCjI/RlbZdadsX2B7FeA7pKrDLtXSmn8CcDWwLLA88BtJ352X4DpjeyJwBnC7pAmkuolvAgdK\nepDUi+CIudjfZNJcXlfn998NrGH7fWA4cFNugHqlw1tHAf2ZwyV+CKH5SrrMfxEo9qNfPi+bk98C\nu1bbaS335h8AfMb2uwCSzgD+CZxZw3trYvvXwK87LP6vYrXtYYXnzwJrz2HdbcBGnbz/FlLdaWfW\nIzU8PVp75CGERiqp49M4YDVJK5GS6D7Al2c7jrSa7Sfyyx2AJ6iilmQ6qcN2vfKy+UbuGnEYUVca\nQssS5dxOavtDSSOA0aQqzEtsT5R0KjDe9ihghKShpOrN10gN2V3qaqCTH5NasKYCEyWNzq+3IWX2\n+Ybts4Czmh1HCKELc9GPtBrbNwM3d1h2UuF5zVWLFV2VTCst9hOBmwrLx87tQUIIoQwtfGt+lwOd\n/KqRgYQQQlfKusyvl6p1prkr0RmkjvALV5bbXr2OcYUQwn9p91GjLiN1FxLp9qtrgN/VMaYQQuiU\nang0Sy3JdBHbowFsP2X7RFJSDSGEhpFK62daF7V0jZqWBzp5StKhpH5Zi9Y3rBBC+G/tPjvpUaQb\n/Q8n1Z0OIA0GEkIIDdXCVaY1DXRyT376FrMGiA4hhIYSzb2Mr6arTvt/oIthp2zvVpeI2oSBGTOr\njspVV6f/9fGmHr/ioov/a2zdhjv3jqebHQIAe675iWaHQK+eZY9F1CKaPC1JNV2VTH/WsChCCKEG\nzZyWpJquOu3/tZGBhBBCV0Rr9zOtpQEqhBBaQgs35kcyDSG0B6nNbyetkNTH9rTqW4YQQn20cC6t\naaT9jSU9RB4cVdJ6ks6ve2QhhNBBGbOT1kstfSh+CuwIvApgewLwhXoGFUIIHZU4O2ld1HKZ38P2\ncx1a0WLCuRBCw/Vs4cv8WpLp85I2BiypJ2myu9boLR5CWGCoySXPampJpoeRLvU/CbwM/CUvCyGE\nhmrhXFrTvfmvMGsO+xBCaAoBvVq4Ob+WkfYvopN79G0Pr0tEIYQwB21dMiVd1lcsDHwJeL4+4YQQ\nwhyotfuZ1nKZP9sUJZKuAP5Rj2AkHU6qj73f9lzNYS/peNvfr0dcIYTWoKZOTNK1eRmrayXg42UH\nkn0d2HpuE2l2fNnBhBBaR6ozrf5ollrugHpN0tT8eB34M/DdsgORdCGwMvAnSd+RdLekf0q6S9Kn\n8jbDJF0v6RZJT0j6QV5+FtBX0gOSrsrL/ijpPkkTJQ3Py3pKukzSw5IeknSUpFUk3V+IY7Xi6xBC\n65BU9dEsXV7mK0W2HmneJ4CZtusyIrLtQyVtS7q76gPgR7Y/lDQU+D6we950feAzwDTgMUnn2z5O\n0gjb6xd2eZDtqZL6AuMk/R4YBCxne+38+Ra3/bqkNyStb/sB4EDSbKwhhBaS7oBqdhRz1mXJNCfO\nm23PyI9GDS0/ALhW0sPAj4G1Cuv+avsN2+8D/wJWnMM+Dpc0ARgLrACsBjwNrCzp/Jy438zbXgwc\nmG9K2Bv4TWc7lDRc0nhJ41+fOqWbHzGEMFdquC+/1e/Nf0DSZ+oeyexOA/6WS5A7kXoRVBRHrppB\nJ6VrSUOAocCmttcD/gksbPs1Ukl7DHAoKYkC/J40ffWOwH22X+0sKNsjbQ+2PXjxJQfO+6cLIcy1\nSj/Tao9m6WoOqF62PyRdUo+T9BTwDukz2fYGdYxrALOqFobV+J7pknrbnp7f/5rtdyWtAWwCIGkg\n8IHt30t6DLgSwPb7kkYDvwAOLvFzhBBK1K79TO8FNgB2blAsRT8Afi3pROCmGt8zEngwNx4dBBwq\n6RHgMdKlPsBywKWSKiXyYkPaVaQ+tLd2N/gQQj2IHiV1jcrVfOcBPYGLbZ/VYf3RwCHAh8BkUhvM\nc13ts6tkKgDbT3Un6Llhe1B+OgVYvbDqxLz+MuCywvY7Fp5/B/hO4T3bzeEwcypRbw5cajtGxAqh\nBaWR9svYj3oCFwBbAy+QrrxH2f5XYbN/AoPz1e1hpALe3l3tt6tkunTOzp2yfW7N0be4PK31KsBW\nzY4lhDBnJY0atTHwpO2nAST9FtiF1KANgO2/FbYfC3yl2k67SqY9gf7QwrcclMT2l5odQwiha2l2\n0po2HShpfOH1SNsjC6+XY/Zb4l8APtvF/g4G/lTtoF0l00m2T622gxBCaJQaS6ZTbA8u43iSvgIM\nBj5fbduqdaYhhNAKRGkj7b9I6ntesTyzeg/NOl66YegE4PO1TCbaVXXuF+c2whBCqBuVdjvpOGA1\nSStJWog0XvOo2Q6V+tb/Etg5j+lc1RyTqe2ptewghBAaRTU8qsn950cAo4FHgGtsT5R0qqRKV9Bz\nSG1G1+YxP0bNYXcfqWU80xBCaLrK7KRlsH0zcHOHZScVng+d231GMg0htI1WHugkkmkIoU00d4i9\naiKZhhDagpi30ewbJZJpCKFtlFVnWg+RTOdRn949WPUT/Zsaww93Xqv6Rg2w+XdvaHYI3H9ua9zE\ndtg1E5odAj/apTV+LkqXu0a1qkimIYS2EJf5IYRQkiiZhhBCCaJrVAghdFO6zG/dbBrJNITQNlr4\nKj+SaQihXQhFyTSEELonDcEXyTSEELpHcZkfQgiliGQaQgjdFJf5IYRQkmiACiGEErRwwTSSaQih\nfbRyybRlxg2QdKikA0ra1/EdXt9Vxn5DCM0jRE9VfzRLyyRT2xfavryk3c2WTG1vVtJ+QwjNkrtG\nVXs0S12TqaQ/SrpP0kRJw/OytyWdIWmCpLGSPp6XnyzpmPx8jKQfSxov6RFJG0m6XtITkk6vsv+z\ngL55RsGrKsfM/0vSOZIelvSQpL3z8iH5mNdJelTSVWrl4WlCWECVMTtpvdS7ZHqQ7Q2BwcDhkpYC\n+gFjba8H/B346hze+4HtwcCFwP8B3wDWBobl/XS6f9vHAe/ZXt/2fh32uRuwPrAeMBQ4R9Iyed1n\ngCOBNYGVgc91DEjS8Jzgx786Zcrcn40QwjyrdI1aUC/zD5c0ARgLrACsBnwA3JjX3wcMmsN7K/NU\nPwRMtD3J9jTg6byvOe2/K5sDV9ueYftl4HZgo7zuXtsv2J4JPNBZXLZH2h5se/BSAwdWOVQIoXQt\nXDStW2u+pCGk0t+mtt+VNAZYGJhu23mzGV3EMC3/P7PwvPK6Vxf7n1fFY3QVVwihSRbU1vwBwGs5\n0a0BbNLA/U+X1LuT99wB7C2pp6SlgS2Be0uOK4RQJwtqA9QtpBLkI8BZpEvxRu1/JPBgpQGq4A/A\ng8AE4Dbg27b/U3JcIYQ6aeVkWrdL2Vy/uV0nq/oXtrkOuC4/P7mwfEjh+RhgTGfr5rB/bH8H+E7h\ndf/8v4Fj86O4fcdjjOj8U4UQmiVVibbuZX7UC4YQ2kMMwRdCCOVo4VwayTSE0C7U0lM9t8ztpCGE\nUE1ZDVCStpX0mKQnJR3XyfotJd0v6UNJe9Syz0imIYS2UEt//VpyqaSewAWkBuw1gX0lrdlhs38D\nw4Df1BpfXOaHENpGSZf5GwNP2n467/O3wC7Avyob2H42r5tZ606jZBpCaBs1XuYPrIyhkR/DO+xm\nOeD5wusX8rJuiZJpCKFt1FgunZIHSWqoSKYhhPZQ3kAmLzJrsCSA5fOybolkGkJoCwJ6lFNnOg5Y\nTdJKpCS6D/Dl7u40kuk8evmtaZz796ebGsPnV1yiqcev+NkRWzQ7BGYNRNZcvz9k42aHwCan/7XZ\nIdRNGanU9oeSRgCjgZ7AJbYnSjoVGG97lKSNSGN5LAHsJOkU22t1td9IpiGE9lFSn33bNwM3d1h2\nUuH5ONLlf80imYYQ2kYMdBJCCCXo0bq5NJJpCKGNRDINIYTuifFMQwihDIrL/BBCKEck0xBC6C7F\nZX4IIZShhceGjmQaQmgPIpJpCCGUIi7zQwihBFEyDSGEErRwLo1kGkJoEypt2pK6aGoyVTozsl3z\nPCshhAVTqzdANXwOKEmD8hSrlwMPA/tLekjSw5LOLmz3tqRzJE2U9BdJG0saI+lpSTsX9nVHnpL1\nfkmb5eVD8rbXSXpU0lU5cSNpI0l3SZog6V5Ji0rqmY81TtKDkr7W6PMSQqiujNlJ66VZJdPVgP8l\nTac6FtgQeA24VdKutv8I9ANus32spD8ApwNbk6Zm/TUwCngF2Nr2+5JWA64GKnO/fAZYC3gJuBP4\nnKR7gd8Be9seJ2kx4D3gYOAN2xtJ6gPcKelW288Ug84Tcw0HWOxjy9blxIQQ5qykkfbrolnJ9Dnb\nYyXtAoyxPRlA0lXAlsAfgQ+AW/L2DwHTbE+X9BAwKC/vDfxM0vrADGD1wjHutf1C3u8D+T1vAJPy\nwK/YfjOv3wZYV9Ie+b0DSAl/tmRqeyQwEmCZ1dZujaHdQ1iQtG4ubVoyfaeGbaZ71lwUM4FpALZn\nSqrEfRTwMrAeqcri/cL7pxWez6Drzyrgm7ZH1xBXCKFJWjiXNr7OtIN7gc9LGiipJ7AvcPtcvH8A\nqaQ5E9ifNJ9LVx4Dlsnzu5DrS3uR5oI5TFLvvHx1Sf3m8rOEEOpIqu3RLE1tzbc9SdJxwN9If3Ru\nsv1/c7GLnwO/l3QAqUqgyxKv7Q8k7Q2cL6kvqb50KHAxqRrg/txQNRnYdW4/TwihvqJrVIHtZ4G1\nC6+vJjUcddyuf+H5yZ2ts/0EsG5h1Xfy8jHAmML2IwrPxwGbdBLa8fkRQmhRrZtKo9N+CKGNtHDB\nNJJpCKE9CLV016hmN0CFEMJ8IUqmIYS20cIF00imIYT2EeOZhhBCNylmJw0hhJJEMg0hhO5r5cv8\naM0PIbSNsm4nlbRtHgr0yXwXZsf1fST9Lq+/R9KgavuMZBpCaBtlJNM8DsgFwHakIT33lbRmh80O\nBl6zvSrwY+BsqohkGkJoG6rhXw02Bp60/bTtD4DfArt02GYX0rjJANcBX1SVgQE0a5S7MDckTQae\n6+ZuBgJTSginu1ohjlaIAVojjlaIAbofx4q2ly4rGEm3kGKqZmFmH45zZB6LuLKfPYBtbR+SX+8P\nfLY4hoekh/M2lTGRn8rbzPF8RAPUPCrjh0TSeNuDq29ZX60QRyvE0CpxtEIMrRRHhe1tmx1DV+Iy\nP4SwoHmvxREMAAAULklEQVQRWKHwevm8rNNt8pjHA4BXu9ppJNMQwoJmHLCapJUkLQTsQ5pTrmgU\naZ46gD1I89F1WScal/nNNbL6Jg3RCnG0QgzQGnG0QgzQOnGUyvaHkkaQZtjoCVxie6KkU4HxtkcB\nvwKukPQkMJWUcLsUDVAhhFCCuMwPIYQSRDINIYQSRDINIYQSRDJtEZI2lPTtwuvWHdFhARXfSehK\nJNMWIGlDYBJwuaR1AGy7Wb+8kjr9uWiFZCJpuWYdO38nm0s6pFkxdNQK30lRq8XTSJFMm0zS54G/\nAdOBN4CfSzofmpdQbc/MsR0o6ZuS9mtmPDmWHpIGALdLauadMAsBn5PUt9mJQ5IqfR8lbZcfSzQp\nlqUkLVStL+b8LJJp830C+AHwRVIn4QOANSSdC81LYJJ2B44j/YwclfvgNTyefPcJtmfafgM4C9hU\n0kJzKkHX2Quk72xQM/+4QPouACQdBJwDHA+cJGmLRsYh6VvA+cBvJe0oqW8jj98qIpk232PA7sCZ\nwL22nwG+Cqwn6RyY9UvTKJJ2AbYCvmL7PGBvYGdJJzcqHkkD87E+lLReLpn2Au4E1gd6257ZiGQm\naR1Jv5H0MduPA9cDp0lapNklMUk7AF8C1gGGAu8CO0navEHH3x3Y3vaXSbdlbm/7vUYcu9VEMm2+\n/wCPAxOAlSUta/tZ4CBgK0ln1DuASkIqJKY1SMl03Xzp9hQp4Q+TdEID4ukDnCLp53nRacAlpBL8\nS6T65Y9KynWKoXJOliPdAfMOcLqki4C3gZdJ92vPsY65nnHl54sAGwCfBza0PQ34KanKaF9JmzYg\njoGkqqkRpPN0ZF5f2mhRbcN2PJr4AHrk/7cBfkNKoh/Lyz5JGsasnsdX4fkawCL5+X7ALcCmQK+8\nbCVg5Qack17AusDlwFF52cqkBHoz6Va/24GFOn6GMs8JsBNwbeU7AFYBvgzcAbwOnN/gn5Xid9WX\nVBjqA5wM/AFYJ69bBvhe5eeoTrGsl//fLZ+Pm4Geedm3gZ9XXi8oj6YHsCA9gMHADoXXvYs/cMAQ\n0oC0XweWbnBsXwfGAlcBvyOVuoYBNwFbVhJqA+Ko/HHZilTKehA4obB+U9LAEy8CR9cxjs2AhwtJ\nozewcH6+HLBFPk91/+OSj1lMpEfmP7xj8nlaDziKNIjx+nmbuiUy4BvA/flnZCXgjzmBbgXsDzwA\nrNWon91WecRlfoPkS8H+wH2SVgCwPd32DEmbSfqJ7TGkv/DrAR/UOZ7FC883Aw4F9iI1YjxAKulc\nTSp1jKBBg+I41YNuQSqV/gW4ktQg94O8/m7b1wH/A6ypNAVFPWwI3AZMzpew1wA3SFrU9ou27yAl\n2I/V6fizcSWjSjuREta3SD8rewKrk87To8AxeSSkmfWIQ9J2pKunL9l+w6mO//8B/UgNqNuR6ton\n1uP4La3Z2XxBeDDrsrEH6RLsduDAvGxdUiPU9oXt+9c5ntVJCbJymbwhcFF+3js/fgVsk5ct1eDz\ntSfwvfy8H7AJ8FfgtMI2+5GGUlu45O+oUqWxCnAr6Q/LEcDnSJeug/P6VUn13KvW+VwMBQ4uvD4S\n+Enh9VdIpcQlgKXq/V0BOwMn5ud9mXVpXzl/fRv5s9JKjyiZ1lmxLyCwvO1JwHmkFtd9gGdJvyw3\nV0pZtt+uc1jTSZeJq+RS6bOkOW6+6lRanp63WTFvP7WewXTSIv8acKCkT9l+x/ZY4BVgI0lr5G3e\nAPaz/T7dVPmOJG0DHJc75T8D7Ah80alHw7vAF/L/2H4SGJr/r6dHgds0a8K3B4AlCzd3XAk8RfrZ\netV2lwMYl+Ad0nezru33nK6shpPqkvEC2pIPMZ5p3VUSqaRvAnvly6RbSZdhXwVm2L42b16XS7OK\n3Pr7nu1nJC0LHAKI1EfwS8A1klYkJy7gh8XPUKeYKolsK1Ip7HHgbtJskOdKOhEwsDQwwvajOaYb\ny4ohH3870ucdQZpgbb38epKkLUm9CY60/S9JPW3PsD25rBg6qpwX2y9I6g28KekM26dL2g3YU9Jg\nUnJfF6hnLMNI9f3Xk+rVzwTOl/RTYEngMFIJeYEWybQBlCbwGgbsWSl1Sqok1G9J6mH7d3VOWv1I\nDUnvSVqXVPq7nNTl6WvApaT6ruGkTukHNKDUVUlk2wOnkzqeDyMlhzOBRYBfkOqPf1pJpB1K+92W\nO5l/DdiXVAf6Eqlh5Xs5rimkUvA9OeYZZR17DvEsBewK/ErS3qSJG9cG7pA0FTgGOJDU4NMb2N32\nf+oUy56kaY//TqruuIlUl/wWqY79XRbUOtKOml3PMD8+mL3ltR+pfu+Q/HqRwro+wA7ACg2IqRfp\nh3886bJwlbx8LVLCOBNYtwnnqgdwCqkb2NAc3zKF9X2BRTue124esx/p8h1S74A1SY2DqwF3kUZf\n70/qT3oa0K8J5+USUn/aO4El87JVSVcNhxe2W6SOMWwB3Aesll/vShp9/1BmdaErtVtaOz+izrRk\nxVKTpG+QLuU/DhwuaQnb7+Z1BwIb2L7J9vP1jAfSnUSk0sWbpBb6NfIdPBNJJdQ+pLucGnoroNM4\nAAsBV5D6S37J9iRJO+S7az6w/VbetqzS6AfA0ZJuI9Vf93a6YuiR1y1B6v50G3CD7XdKOm5Vhfrj\nX5CmK17I9lT4qJ72c8APJX03L3u3TnFUekn0I5WEsf1H4EZS3fE+knqV+J20vUimJSsk0q+Ruopc\na/tc0gRdl0haS9LBwNGkS+266SSxf4nUGjuG1LVo17zpO6S+gr90nRsQCncWrZsbvyD1bTVwk+3n\n862QPwamuuRL6nxOpgNnAJ8BXrY9Ia+eAtxDapy7BbjA9r1lHr+ruOCjao9epP61qwAvSLqrsOlL\npNs2r6tjLJ8AFnPq/rU/sLhm3Uo8CrgM+FP+Ax2ymAOqDnLp7mpS6WI8qavP8qT6wDGkS8gT3KB6\nJkmHk+oDD7U9Icd3EOmydhng08BWtqc0KJ6tSR3y3yF1ebqB1MD0DeA9Ui+C411iI1M+rgrJaglS\n1cLlwN22D8nbfBqYASzeyERa+KP3deBTpD8kp+RlfyQ19PyK1Gi4nevU40PSUcC2pHMw2vZ5kj5L\n6pL1gu1j63Hc+UKz6xnm1wepIeefpERxNqme6WxSo0rvBsaxECmxr5Rf98n/9yU1SB1PA+5WodCP\nk9TJe50cwwWkhp718rlZuRBr6fVxwNb5mMPy6/7AROBnOaYbqHM/3y5i+zqpKmYVUin5EnL9Mamk\nfgWwdh2P/1VSH+i+pLrRmcAped3mpEbKgc04N+3waHoA8+sDWJjUvajSePBl0rildWswyMeZ7f7t\n/P+dleRRWLdRE87JrqRL6HvINykAK5C6Zp1HvnWzjsffjNSgciSpv+aJpNbwRUgl5DHArk04Lz1I\nJc/LST0pvgn8mXS76g2kPqQf/SGsYxw7ker3jyCNSbAGqbX+pLy+lBsk5tdH0wOY3x/5F+Vg4KF6\nlirysYqJdEQugR1Bqrv9JXlcAFLvgvuo40AYncS2Vi517UXqOfAgaaSjSkK9EPhUHY+/Mqneelgh\nnt/nhLpw/p4GdjyPDfquKuMR9CPdjTam8Pp1Usm9blcz+Y9J5c6vpfN5qgyacilpHIQlG/Wz0q6P\n6GdafwuTLpf2sv1IPQ/kym9pqnfbk5Q0xwP3khoNzs0dvjcF9rD9Sj3jqZC0OunS/mHb15BuDpgE\nXChphO17JB3hNIRcvSxL+h72lnSr7Yn5hoAfkxLpmc51xpXzWE+F72ofYG1J1zrVZ78L9JS0Mqku\nezRwqVOjWekkHQtsnJ8f7tST4mXSANxDSedsQ+ceBWHOogGqAcruZF7lWIsB55KS116kjviTSXWV\n1wFPAJOdbmutZxzFRpX+wAmkOsnvA2OdBjQ5llRq3gx4xyW23FfuUpK0PPCq7fdyUj+U1P3pvJw4\n1iRVh9xX1rGrxFU8L3uRzstNwC7Ad0l3GA0ndT9aFtitXn+E87m5nFTt8RXSz8sGpOS6Del7GW77\nwXocf34TyXQ+pDS48hqkATG+kLvdvAr8hFQCq0spp3D8Sqv5pqQ7it6yfZukU0it6L8B7snbDHIa\nDLusY69I+rl+Nt9ZdQapPnQx28MlrUXqVbEQcFa9/6h0iK2YSFckXdI/lUuk++e4ziEN4LIYMN32\nC3WKZXfSwCgDbX8/L/sRsD2wpe3Jkvq5gX1s2130M50P5cvld4FeSgNi7EBq0Lis3ok0H99Kk96N\nJHUyP13Sxba/R0rqB5FGgqLkRDqIdFm8TO7D+n1gH9KdRLtIGuXUHe1KUr/WAWUdu4bYion0cNKN\nEyeR7v7C9hWk+skzgE1tP1PHRLob6dxsT5rNYYccw7dIjXB/y53263JDwPwqSqbzqVw6PZJ0i+ay\npHEB/tWgY/cgdcf6ne3r87K7SXcUnUKaFO+iMi9fc+l7d1KS/gmp98QtpNbx00mXsJcCb9reRVJ/\n1390rkpsfZ1vhlAaNGUY6TwsReoO9Zbto/L6vUil9ufqFMtwUpe4o0iNW0eTGp3+ZvumvM3HGlWf\nPj+Jkul8KpdOzyWVArepdyIt3Nk0hJTUXmH2ks1BwLK2PwCOLbseMJf6/kJqdHsY+HOu69sNODuX\ngP8OrChpgwYm0k+Tbr1cSNLHSd3APkYaQvBBUv/WRZXmlsL2NfVKpFk/0h+awfkq5SpSnfpOkv4n\nb1O3EajmZ5FM52NOY5M+b/vFBhzLSqPA/xj4N2kczguVJqSDdK/7IEkDSJfY9fAm6UaJN0kjTwFM\nI43bugepymFP2/fX6fid6UfqajSIdHfXIXnZTgC2HyB1C5uWk21dSFpRUh/bPyY1cP1W0nq5KuFq\n0gDl/8wxxeXqPIiuUaEUucX+IOAbTkPV3aM0XfNoSaNJ9XPH2H6jXjE4DZqyvaRPAn/NpeUrSAls\na+BC20/U6/hFlTpS2+OVZuo8FniBNEbqiaQ6S0m6Om/zUL26hkk6klRvPkPS7aTS8QzgVknb2b5f\n0k/L7E2xIIo601AKpfFSRwOn2r610KL/v6QbBPrYvq9R3cQkfYbUa+BHti+u1Fs24vidHSM3iO1J\nuoQ+j3T77M9JVRBX1zGWDUmTNH6eWcMNLmP7KKWRpw7Jy6bnP0ZhHkUyDaVRmk1gKVLD0yO5a9RJ\npLuOXm5CPBuS+tZ+ocxeA1WOWWy1P4zURW08aYLCVUml95dIdaVrAf+pY2PTYaSBbFaxvV9etiHp\nOznB9sOSlnL9pzpZIESdaSjT9aSfqZGSziR1QfpZMxIpQO6Iv26jEmk+ZiWRDiENXzeJ1An+dOBp\n0shPq5PqLe+tYyLdldTx/ybgY0oj9lfOyduk0ijUeRjIBUmUTEOp8uX+RqQBM57N9afNjKdhd58V\njrk/qevRgblD/mBgD9IA3CeThv57pV5/ZHKj31jgdttfyVUtQ0mJ/SHgONJAM/XsNbDAiWQaQjd1\nTNiSViB1e7rC9uF52Yak/qXvAN9tQL3tbqReAofYHiVpA9JoVFOAy20/VM/jL4iiNT+EbuhQRzqC\nNPHdQ6QxEW6W9KLts3Pj2wzgpUaUlG1fL+kD4Pu5S9S1pCmae0RDU31EMg2hGwqJtDhS14OkUuHB\nwM/yPe4n5T6ljYztxpzAR0qaafv3kUjrJxqgQugmpZG6NiCNA7AbaaCSQcCOpFt695O0VOUusUay\n/SdSD4J/NvrYC5qoMw2hBOp8pK7XSY09VzrPsBrmX3GZH0IJbE9TGti5MlLXiqSBVm6ORLpgiJJp\nCCVp5khdofkimYZQIkm9ScP+zWzEADOhdUQyDSGEEkRrfgghlCCSaQghlCCSaQghlCCSaQghlCCS\naQghlCCSaZgnkmZIekDSw5KulbRIN/Y1RNKN+fnOko7rYtvF833wc3uMkyUdU+vyDttclueQqvVY\ngyQ9PLcxhvYWyTTMq/dsr297beAD4NDiSiVz/fNle5Tts7rYZHHS9MghtJRIpqEMdwCr5hLZY5Iu\nJ023vIKkbSTdLen+XILtDyBpW0mPSrqfNDgIefkwST/Lzz8u6Q+SJuTHZsBZpNlGH5B0Tt7uWEnj\nJD0o6ZTCvk6Q9LikfwCfqvYhJH0172eCpN93KG0PlTQ+72/HvH1PSecUjv217p7I0L4imYZukdSL\nNHZnZbDh1YCf216LNBDyicBQ2xuQ5kI6WtLCwEWk6Y43JN0x1JmfkkaLX480KtNE0sAhT+VS8bGS\ntsnH3BhYH9hQ0pZ5MOZ98rLtSaP/V3O97Y3y8R4hDaFXMSgfYwfSFNYL5/Vv2N4o7/+rklaq4Thh\nPhQDnYR51VdSZXzOO0hzGy0LPGd7bF6+CWmuoTvz6HMLAXeTRld6pjLtsqQrSXMidbQVcABAnob4\nDUlLdNhmm/yoDDHXn5RcFwX+YPvdfIxRNXymtSWdTqpK6E+abbXimjwW6BOSns6fYRtg3UJ96oB8\n7MdrOFaYz0QyDfPqPdvrFxfkhPlOcRHwZ9v7dthutvd1k4Azbf+ywzGOnId9XQbsmudtGgYMKazr\neN+187G/abuYdJE0aB6OHdpcXOaHehoLfE7SqpAm25O0OvAoMEjSKnm7fefw/r8Ch+X39pQ0AHiL\nVOqsGA0cVKiLXU7Sx4C/A7tK6itpUVKVQjWLApPyYCX7dVi3p6QeOeaVgcfysQ/L2yNpdaUJBcMC\nKEqmoW5sT84lvKvz8HQAJ9p+XNJw4KY8BugdzJ4gK44gTblxMDADOMz23ZLuzF2P/pTrTT8N3J1L\nxm8DX7F9v6TfAROAV0ij31fz/4B7gMn5/2JM/wbuBRYDDrX9vqSLSXWp9+fBoCcDu9Z2dsL8JkaN\nCiGEEsRlfgghlCCSaQghlCCSaQghlCCSaQghlCCSaQghlCCSaQghlCCSaQghlOD/A3DjEWwQoN0M\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f75e6a63210>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "predict(count_vectorizer, logreg, test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "White box vectorizer and classifier are great! We can see what are the most important words for sci-fi. This makes it very easy to tune and debug."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def most_influential_words(vectorizer, genre_index=0, num_words=10):\n",
    "    features = vectorizer.get_feature_names()\n",
    "    max_coef = sorted(enumerate(logreg.coef_[genre_index]), key=lambda x:x[1], reverse=True)\n",
    "    return [features[x[0]] for x in max_coef[:num_words]]    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "action\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[u'fight',\n",
       " u'animation',\n",
       " u'ending',\n",
       " u'adventurous',\n",
       " u'tale',\n",
       " u'forest',\n",
       " u'halloween',\n",
       " u'adventure',\n",
       " u'toys',\n",
       " u'snow']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# words for the fantasy genre\n",
    "genre_tag_id = 1\n",
    "print(my_tags[genre_tag_id])\n",
    "most_influential_words(count_vectorizer, genre_tag_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x3000 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 43 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_features[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Character N-grams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A character _n-gram_ is a chunk of a document of length _n_. It is a poor man's tokenizer but sometimes works well. The parameter _n_ depends on language and the corpus. We choose length between 3 and 6 characters and to only focus on 3k most popular ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 12.5 s, sys: 31.1 ms, total: 12.6 s\n",
      "Wall time: 12.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "n_gram_vectorizer = CountVectorizer(\n",
    "    analyzer=\"char\",\n",
    "    ngram_range=([2,5]),\n",
    "    tokenizer=None,    \n",
    "    preprocessor=None,                               \n",
    "    max_features=3000) \n",
    "\n",
    "logreg = linear_model.LogisticRegression(n_jobs=1, C=1e5)\n",
    "\n",
    "train_data_features = n_gram_vectorizer.fit_transform(train_data['plot'])\n",
    "\n",
    "logreg = logreg.fit(train_data_features, train_data['tag'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u' are ',\n",
       " u' as',\n",
       " u' as ',\n",
       " u' as a',\n",
       " u' as t',\n",
       " u' ass',\n",
       " u' at',\n",
       " u' at ',\n",
       " u' at t',\n",
       " u' att']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_gram_vectorizer.get_feature_names()[50:60]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The results are worse than using a tokenizer and bag of words. Probably due to not removing the stop words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy 0.399176954733\n",
      "confusion matrix\n",
      " [[17  3 12  1  5  4]\n",
      " [ 4  9 10  3  1  4]\n",
      " [12 10 41  6 15  2]\n",
      " [ 2  1  2  5  1  5]\n",
      " [ 5  1 15  1 11  2]\n",
      " [ 8  1  7  3  0 14]]\n",
      "(row=expected, col=predicted)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAAEmCAYAAADfpHMGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe8HGXZ//HPNwklJBAgAaSHKgISSkBBwYiBhypFOuIT\nWgRFmqAI/JAqIM+DIIIYVJDeRB6aBBWDCAQIEEqkF6VECL0nIfn+/rjvleFwztnNObMtud557evs\nzszOXDt7cp27zT2yTQghhN7p0+wAQghhdhDJNIQQShDJNIQQShDJNIQQShDJNIQQShDJNIQQShDJ\ndDYiqb+k6yW9JemqXuxnd0m3lBlbs0jaUNLjzY6jHiQdK+ni/HwZSe9K6lvyMZ6TNLLMfc6uIpk2\ngaTdJE3Iv/yTJf1R0pdL2PUOwGLAYNs79nQnti+xvWkJ8dSVJEtasbttbN9u+7M93P9zkl6RNKCw\nbB9J43qyv3qy/S/bA23PaHYsc6pIpg0m6VDgDOAnpMS3DHAOsE0Ju18WeML2RyXsq+1J6lfCbvoC\nB5UQiyTF/7fZWHy5DSRpEHA88F3b19h+z/Z029fbPjxvM4+kMyS9lB9nSJonrxsh6QVJ388lpsmS\n9szrjgOOAXbOJd69i9XAvM3QXJrrl1+PkvSMpHckPStp98Lyvxfet4Gke3Pzwb2SNiisGyfpBEl3\n5P3cImlIF5+/Ev8PCvFvK2kLSU9Iel3SkYXt15N0l6Q387a/kDR3Xve3vNmD+fPuXNj/DyX9Gzi/\nsiy/Z4V8jLXz6yUkTZE0opuv7TTgMEkLdvGZqp2bkyTdAbwPLJ+XnSjpzhz39ZIGS7pE0tt5H0ML\n+zhT0vN53X2SNuwijv98t5LWz/uuPD6U9Fzero+kIyQ9Lek1SVdKWriwnz0k/TOvO6qb8xI6sh2P\nBj2AzYCPgH7dbHM8MB5YFFgEuBM4Ia8bkd9/PDAXsAXpP+lCef2xwMWFfXV8PRQw0A8YALwNfDav\nWxxYLT8fBfw9P18YeAPYI79v1/x6cF4/DngaWBnon1+f0sVnq8R/TI5/X2AKcCkwP7Aa8AGwXN5+\nHeCL+bhDgUeBgwv7M7BiJ/s/FZgnxzMCeKGwzb7AP4D5gLHA/3TzXTwHjASuAU7My/YBxs3CuflX\n/lz98mceBzwFrAAMyrE8kY/TD7gQOL8QwzeBwXnd94F/A/N2/H6L322HzzAXcBtwcn59EOn3a6l8\njn4FXJbXrQq8C2yU152ez+fIZv/faYdHlEwbazDwqruvhu8OHG/7FdtTgONI/1krpuf1023fRPrl\n71GbIDATWF1Sf9uTbU/qZJstgSdtX2T7I9uXAY8BWxe2Od/2E7Y/AK4E1uzmmNOBk2xPBy4HhgBn\n2n4nH/8fwDAA2/fZHp+P+xzpP/5XavhMP7Y9NcfzCbbPIyWzu0l/QGopfR0DfE/SIh2W13JuLrA9\nKa+fnpedb/tp228BfwSetv3n/HtxFbBWId6Lbb+W3/+/pCQ3K9/3z4F3Cp9zP+Ao2y/YnkpKyDvk\n2soOwA22/5bX/T/S+Qw1iGTaWK8BQ6q05S0B/LPw+p952X/20SEZvw8MnNVAbL8H7Ez6zzVZ0o2S\nVqkhnkpMSxZe/3sW4nnNH3eSVJLdy4X1H1TeL2llSTdI+rekt0ntzJ02IRRMsf1hlW3OA1YHzspJ\no1u2HwFuAI7osKqWc/N8J7vs+Hk7/fwAkg6T9GhuRniTVJqtdg4q7/02qWS+m+1KUlwW+ENuOnmT\nVNqfQWq/X6IYb/4dea2WY4VIpo12FzAV2LabbV4i/cJXLJOX9cR7pOpsxWeKK22Ptb0JqYT2GCnJ\nVIunEtOLPYxpVvySFNdKthcAjgRU5T3dToMmaSCpA/A3wLHF9sIqfkxqIigmylrOTY+nZcvtoz8A\ndiI15SwIvEX1c1B57wnANrbfLqx6Htjc9oKFx7y2XwQmA0sX9jEfqTYVahDJtIFyte4Y4Ozc8TKf\npLkkbS7pp3mzy4CjJS2SO3KOAS7uap9VTAQ2UhqDOAj4UWWFpMUkbaM07GcqqbmgsyrdTcDKSsO5\n+knamdS2dkMPY5oV85Padd/Npeb9O6x/GVh+Fvd5JjDB9j7AjcC5tbzJ9lPAFcCBhcX1Pjfzk9os\npwD9JB0DLFDtTZKWJjW3fMv2Ex1WnwucJGnZvO0ikiojSa4GtpL05dzRdzyRI2oWJ6rBcrvXocDR\npP8kzwMHANfmTU4EJgAPAQ8D9+dlPTnWn0gJ4CHgPj75n7xPjuMl4HVSW2THZIXt14CtSJ0fr5FK\nSlvZfrUnMc2iw4DdSG1+55E+S9GxwO9ylXWnajvLSWMzPv6chwJrK49iqMHxpI47oCHnZixwM6mD\n6p/Ah3TebNDR10jV9qsLPfqV9vAzgeuAWyS9Q+qM+kL+PJOA75I6BCeTOtNeKOmzzPZkx+TQIYTQ\nW1EyDSGEEkQyDSGEEkQyDSGEEkQyDSGEEpQxEcQcSfPM7z4Daho7XTefW3qhph6/4vk3P3WhUcMt\nN3i+6hs1wNTpzb9gSKo6DLUhHn34gVdtd7xqrMf6LrCs/VH13zV/MGWs7c3KOm6tIpn2UJ8BQ5hv\n02ObGsM1Z/R4lr1SHXLtI80OgYv2WLvZIQDw+OR3mx0C/ecudUrTHltr2QU6Xh3WK/7oA+b5bNUR\ncHw48eymlHIimYYQ2oMEfVrjD0VnIpmGENpHC08JG8k0hNA+WqQ9uDORTEMIbUJRMg0hhF4T0WYa\nQgi9p6jmhxBCKaKaH0IIJYiSaQgh9FKMMw0hhJJENT+EEHorhkaFEELvCejbutX81k3zJZM0XNLP\nu1i3oaRJkiZKWlLS1Y2OL4RQA6n6o0nmmJKp7QmkG9V1ZnfgZNuVu4Du0JioQgi1a+1qfutGViNJ\nAyTdKOlBSY9I2lnSupLuzMvukTS/pBGSPnULXkn7kO5LfoKkSyQNldT8OeVCCJ8WJdO62gx4yfaW\nAPn+8A8AO9u+V9ICQJczytr+taQvAzfYvlrS0K62lTQaGA2g+QaX9wlCCNW1+NCoti+Zku4tv4mk\nUyVtCCwDTLZ9L4Dtt21/VMaBbI+xPdz2cM0zfxm7DCHMCvWp/miStk+mtp8A1iYl1ROB7au9R9LY\n3Nn063rHF0IoUVTz60fSEsDrti+W9CbwHWBxSevmav78dKjm2/6vZsQaQuiN1u6AavtkCnweOE3S\nTGA6sD9pRNpZkvqTEunIJsYXQihDTMFXX7bHAmM7WfXFDq/H5Udn+xhVeP4csHopwYUQShQl0xBC\nKEfMGhVCCCWIan4IIfSSopofQgjliGp+CCH0niKZhhBC76RafiTTEELoJbV0ybR1W3NDCKEDSVUf\nNe5nM0mPS3pK0hHdbPcNSZY0vNo+o2QaQmgbffr0vvwnqS9wNrAJ8AJwr6TrbP+jw3bzAwcBd9cU\nW68jCyGERlCNj+rWA56y/YztacDlwDadbHcCcCrwYS07jZJpD624xCB+ddxWTY3hwGseburxK159\n/f1mh8C/XutyytqGWmGxAc0OgX+89HazQ6gL1d5mOkRS8a4aY2yPKbxeEni+8PoF4AufOJa0NrC0\n7RslHV7LQSOZhhDaRo3J9FXbVds4uzlGH+B0YNSsvC+SaQihbZTRZgq8CCxdeL1UXlYxP2myo3E5\neX8GuE7S1/O95DoVyTSE0B5qbxOt5l5gJUnLkZLoLsBulZW23wKG/Oew0jjgsO4SKUQHVAihjZQx\nNCrfxugA0tSdjwJX2p4k6XhJX+9pbFEyDSG0hVnogKrK9k3ATR2WHdPFtiNq2Wck0xBC24jLSUMI\nobcUE52EEEIpIpmGEEIvCZU1NKouIpmGENpH6xZMI5mGENpEtJmGEEI5IpmGEEIJYmhUA0kaAUyz\nfWd+vR/wvu0LmxpYCKHXomTaWCOAd4E7AWyf29RoQgilmJWZ9JuhdccZdCDpWkn3SZokaXRetpmk\n+yU9KOkvkoYC+wGHSJooaUNJx0o6LG+/pqTxkh6S9AdJC+Xl4ySdKukeSU9I2rBZnzOE0LWybltS\nD+1UMt3L9uuS+pNuM/B/wHnARraflbRwXn8u8K7t/wGQ9LXCPi4Evmf7NknHAz8GDs7r+tleT9IW\nefnIhn2yEEJNos20HAdK2i4/XxoYDfzN9rMAtl/v7s2SBgEL2r4tL/odcFVhk2vyz/uAoV3sY3Q+\nLostsVQPPkIIoTeimt9LuVNpJLC+7WHAA8DEkg8zNf+cQRd/ZGyPsT3c9vBBCw0u+fAhhG6ptav5\nbZFMgUHAG7bfl7QK8EVgXmCjPMErkhbO275Dmin7E/KEr28U2kP3AG7ruF0IoTWly0mrP5qlXar5\nNwP7SXoUeBwYD0whVbmvyfdseYV069brgaslbQN8r8N+/hs4V9J8wDPAng2KP4RQghau5bdHMrU9\nFdi8i9V/7LDtE8AahUW3F9ZNJJVqO+5/ROH5q3TRZhpCaK5WbjNti2QaQgipzbTZQXQtkmkIoS0I\n6Nu3dbNpJNMQQtuIan4IIfRWVPNDCKH3RJRMQwihBM0dR1pNJNMQQtuIkmkIIfRWtJmGEELvCaKa\nH0IIZYhqfgghlKCFc2kk054yMG3GzKbGcOymn23q8Su+uuPRzQ6BAft+odkhAHDz4/9udghs9bnF\nmx1CfcStnkMIofcUQ6NCCKEcLVwwjWQaQmgfUc0PIYReklp7aFS73LYkhBBKuwdUvk3845KeknRE\nJ+v3k/RwvmX83yWtWm2fkUxDCG1Dqv6ovg/1Bc4m3b1jVWDXTpLlpbY/b3tN4KfA6dX2G8k0hNA2\nSiqZrgc8ZfsZ29OAy4FtihvYfrvwcgBpNGS3os00hNAWpJqHRg2RNKHweoztMYXXSwLPF16/AHxq\noLKk7wKHAnMDG1c7aCTTEELbqLFJ9FXbw3t7LNtnA2dL2g04mnR34y5FMg0htI0+5QyNehFYuvB6\nqbysK5cDv6y20y6TqaQFuntjhzaFEEKou5KGmd4LrCRpOVIS3QXY7ZPH0Uq2n8wvtwSepIruSqaT\nSI2uxfArrw0sU3PoIYTQSxL0LWGcqe2PJB0AjAX6Ar+1PUnS8cAE29cBB0gaCUwH3qBKFR+6Saa2\nl+5qXQghNENZV0DZvgm4qcOyYwrPD5rVfdY0NErSLpKOzM+XkrTOrB6o1UgaIemGZscRQqhdGeNM\n66VqMpX0C+CrwB550fvAufUMKoQQOhLQV6r6aJZaSqYb2P428CGA7ddJ465KI+lbkh6S9KCkiyQN\nlXRrXvYXScvk7S6Q9EtJ4yU9k0uXv5X0qKQLCvvbVNJdku6XdJWkgXn5ZpIek3Q/sH1e1kfSk5IW\nKbx+qvI6hNAiahiw38yJUGpJptMl9SFfASBpMFDarMiSViON4drY9jDgIOAs4He21wAuAX5eeMtC\nwPrAIcB1wM+A1YDPS1pT0pC8v5G21wYmAIdKmhc4D9gaWAf4DIDtmcDFwO55/yOBB21P6STW0ZIm\nSJrw1uuvlXUKQgg1autqPuka1t8Di0g6Dvg7cGqJMWwMXGX7VfhPyXd94NK8/iLgy4Xtr7dt4GHg\nZdsP54Q4CRgKfJF0ve0dkiaSeuGWBVYBnrX9ZH7/xYV9/hb4Vn6+F3B+Z4HaHmN7uO3hgxYe3MuP\nHUKYFSKNM632aJaqg/ZtXyjpPlKJDWBH24/UN6xuTc0/ZxaeV173A2YAf7K9a/FNktbsaoe2n5f0\nsqSNSdft7t7VtiGE5pkdpuDrSxpvNW0W3lOrW4Edc/MBkhYG7iQNpIWU2G6fhf2NB74kacW8vwGS\nVgYeA4ZKWiFvt2uH9/2aVFq9yvaMHn2SEELd1FLFb+lqvqSjgMuAJUiXXV0q6UdlBWB7EnAScJuk\nB0lTXX0P2FPSQ6RRBDWP+cptnaOAy/L77wJWsf0hMBq4MXdAvdLhrdcBA+miih9CaL62ruaT2hLX\nsv0+gKSTgAeAk8sKwvbvgN91WPypWVpsjyo8fw5YvYt1twLrdvL+m0ltp50ZRup4eqz2yEMIjdS6\nlfzakunkDtv1y8tmG3mm7f2JttIQWpYo53LSeuluopOfkYZDvQ5MkjQ2v96UNFHAbMP2KcApzY4j\nhNCNJo8jraa7kmmlx34ScGNh+fj6hRNCCF1r4Vza7UQnv2lkICGE0J22reZX5KFEJ5EGws9bWW57\n5TrGFUIIn9LK1fxaxoxeQBouJNLd/K4ErqhjTCGE0CnV8GiWWpLpfLbHAth+2vbRpKQaQggNI7X/\nONOpeaKTpyXtR5rmf/76hhVCCJ/WypeT1pJMDyHdN/pAUtvpINJkICGE0FAt3GRa00Qnd+en7/Dx\nBNEhhNBQornV+Gq6G7T/B/Icpp2xvX1dImoTfYB5+/VtagyLDpqnqcevGHvF8c0OgVEXTmh2CABc\nNOpTVzE33Nbn3NnsEOqjyROZVNNdyfQXDYsihBBq0MzbklTT3aD9vzQykBBC6I5o7XGmtXRAhRBC\nS2jhzvxIpiGE9iC1+eWkFZLmsT21+pYhhFAfLZxLa5ppfz1JDwNP5tfDJJ1V98hCCKGDtr5tCek2\ny1sBrwHYfhD4aj2DCiGEjtr+7qRAH9v/7NCLFjecCyE0XN8WrubXkkyfl7QeYEl9STe7e6K+YYUQ\nwiepySXPampJpvuTqvrLAC8Df87LQgihoVo4l9Z0bf4rfHwP+xBCaAoB/Vq4O7+WmfbPo5Nr9G2P\nrktEIYTQhbYumZKq9RXzAtsBz9cnnBBC6ILafJyp7SsKj98B2wPr1CMYSQdKelTSJT1475H1iCmE\n0DpUw7+a9iNtJulxSU9JOqKT9YdK+oekhyT9RdKy1fZZyzjTjpYDFuvB+2rxHWAT27v34L2RTEOY\njaU20+qPqvtJo5LOJt1+aVVgV0mrdtjsAWC47TWAq4GfVttvLVdAvSHp9fx4E/gT8KPqIc8aSecC\nywN/lPRDSXdJekDSnZI+m7cZJekaSTdLelLST/PyU4D+kiZWSrWSrpV0n6RJkkbnZX0lXSDpEUkP\nSzpE0gqS7i/EsVLxdQihdUiq+qjBesBTtp+xPQ24HNimuIHtv9p+P78cDyxVbafdtpkqRTaMdN8n\ngJm2u5wwujds7ydpM9LVVdOA/7X9kaSRwE+Ab+RN1wTWAqYCj0s6y/YRkg6wvWZhl3vZfl1Sf+Be\nSb8HhgJL2l49f74Fbb8p6S1Ja9qeCOxJuhtrCKGFpCugatp0iKTibOFjbI8pvF6ST/b7vAB8oZv9\n7Q38sdpBu02mti3ppkryaaBBwO8krUQaSTBXYd1fbL8FIOkfwLJ03iF2oKTt8vOlgZWAx4Hl89wC\nNwK35PW/BvaUdCiwM+kv16fkEu5ogMWWqPqHKoRQptqvvX/V9vBSDil9ExgOfKXatrW0mU6UtFav\no5o1JwB/zUl8a9IogorizFUz6OQPgqQRwEhgfdvDSO0f89p+g1TSHgfsR0qiAL8ntZ9sBdxn+7XO\ngrI9xvZw28MXXGhwzz9dCGGWVcaZVnvU4EVSAatiKT6ufX98vFQrPgr4ei0z5nV3D6h+tj8iVanv\nlfQ08F7+TLa9di1R99AgPv5wo2p8z3RJc9ment//hu33Ja0CfBFA0hBgmu3fS3ocuBjA9oeSxgK/\nJBXpQwgtqKRxpvcCK0lajpRndgF2++RxtBbwK2CzfOFSVd1V8+8B1ga+3qNwe+enpGr+0aTqeC3G\nAA/lzqO9gP0kPUqq2o/P2ywJnC+pUiIvdqRdQhpDewshhBYk+tQ49Kk7uS/mAGAs0Bf4re1Jko4H\nJti+DjgNGAhclTu1/mW721zYXTJVPvDTvY6+RraH5qevAisXVh2d118AXFDYfqvC8x8CPyy8Z/Mu\nDtNVifrLwPm2Y0asEFpQmmm/nH3Zvgm4qcOyYwrPR87qPrtLpovkDpmugjl9Vg/WqvJtrVcANm52\nLCGErrXrrFF9ScXc1o2+JLa3q75VCKGZ0t1Jmx1F17pLppNtH9+wSEIIoYp2LZm2btQhhDmOaN+Z\n9r/WsChCCKEaUevlok3RZTK1/XojAwkhhGpaN5XWNp9pCCE0XeXupK0qkmkIoW208uTQkUxDCG2i\n5in2miKSaQihLYiezWbfKJFMQwhtI9pMZ0MzDO9N/6ipMSzQf67qGzXAf/33ec0Ogeev/E6zQwDg\n6+fc2ewQuHSvTqfjbbjlflDyDtt1aFQIIbSSqOaHEEJJomQaQggliKFRIYTQS6ma37rZNJJpCKFt\ntHAtP5JpCKFdCEXJNIQQeidNwRfJNIQQekdRzQ8hhFJEMg0hhF6Kan4IIZQkOqBCCKEELVwwjWQa\nQmgfrVwybZl5AyTtJ+lbJe3ryA6vmz+VTwihV4Toq+qPZmmZZGr7XNsXlrS7TyRT2xuUtN8QQrPk\noVHVHs1S12Qq6VpJ90maJGl0XvaupJMkPShpvKTF8vJjJR2Wn4+T9DNJEyQ9KmldSddIelLSiVX2\nfwrQX9JESZdUjpl/StJpkh6R9LCknfPyEfmYV0t6TNIlauXpaUKYQ6mGR7PUu2S6l+11gOHAgZIG\nAwOA8baHAX8D9u3ivdNsDwfOBf4P+C6wOjAq76fT/ds+AvjA9pq2d++wz+2BNYFhwEjgNEmL53Vr\nAQcDqwLLA1/qGJCk0TnBT3j7jddm/WyEEHqsMjRqTq3mHyjpQWA8sDSwEjANuCGvvw8Y2sV7r8s/\nHwYm2Z5seyrwTN5XV/vvzpeBy2zPsP0ycBuwbl53j+0XbM8EJnYWl+0xtofbHr7AQoM7rg4h1FsL\nF03r1psvaQSp9Le+7fcljQPmBabbdt5sRjcxTM0/ZxaeV17362b/PVU8RndxhRCaZE7tzR8EvJET\n3SrAFxu4/+mSOrtB0u3AzpL6SloE2Ai4p+S4Qgh1Mqd2QN1MKkE+CpxCqoo3av9jgIcqHVAFfwAe\nAh4EbgV+YPvfJccVQqiTVk6mdavK5vbNzTtZNbCwzdXA1fn5sYXlIwrPxwHjOlvXxf6x/UPgh4XX\nA/NPA4fnR3H7jsc4oPNPFUJoltQkWk62lLQZcCbQF/i17VM6rN8IOANYA9gl56putcw40xBC6FZJ\n40wl9QXOJhXGVgV2lbRqh83+BYwCLq01vOhkCSG0jZJq8esBT9l+BkDS5cA2wD8qG9h+Lq+bWetO\nI5mGENqEar3V8xBJEwqvx9geU3i9JPB84fULwBd6G10k0xBC26ixg+nVfMFPQ0UyDSG0hRLH5L/I\nxxf+ACyVl/VKdECFENqGpKqPGtwLrCRpOUlzA7vw8RWXPRbJNITQNsrozbf9EXAAMBZ4FLjS9iRJ\nx0v6ejqO1pX0ArAj8CtJk6rtN6r5IYS2UdaYfNs3ATd1WHZM4fm9pOp/zSKZhhDaQ7Pn2KsikmkI\noS0I6NPC0wxHMu2hNz+czv89OqWpMQxfZqGmHr/iyEO2aHYIvPTGB80OAYBbD92o2SGw0Ha/bHYI\nddO6qTSSaQihnbRwNo1kGkJoG608n2kk0xBC2+jTurk0kmkIoY1EMg0hhN4pcz7TeohkGkJoD4pq\nfgghlCOSaQgh9Jaimh9CCGVo4QugIpmGENqDiGQaQgiliGp+CCGUIEqmIYRQghbOpZFMQwhtQtR6\nW5KmaGoyVTozsl3zvalDCHOmVu+Aavg9oCQNlfS4pAuBR4A9JD0s6RFJpxa2e1fSaZImSfqzpPUk\njZP0TOE+LUMl3S7p/vzYIC8fkbe9WtJjki7Jibtyb5c7JT0o6R5J80vqm491r6SHJH270eclhFCd\nang0S7NKpisB/w38CxgPrAO8AdwiaVvb1wIDgFttHy7pD8CJwCbAqsDvSHcTfAXYxPaHklYCLgMq\n98teC1gNeAm4A/iSpHuAK4Cdbd8raQHgA2Bv4C3b60qaB7hD0i22ny0GLWk0MBpg4JDF63JiQghd\ni5n2P+2ftsdL2gYYZ3sKgKRLgI2Aa4FpwM15+4eBqbanS3oYGJqXzwX8QtKawAxg5cIx7rH9Qt7v\nxPyet4DJ+WZZ2H47r98UWEPSDvm9g0gJ/xPJ1PYYYAzAIius5hLOQwhhVrRuLm1aMn2vhm2m264k\nrJnAVADbMyVV4j4EeBkYRmqy+LDw/qmF5zPo/rMK+J7tsTXEFUJokhbOpY1vM+3gHuArkoZI6gvs\nCtw2C+8fRCppzgT2APpW2f5xYHFJ6wLk9tJ+pPtn7y9prrx8ZUkDZvGzhBDqSKrt0SxN7c23PVnS\nEcBfSX90brT9f7Owi3OA30v6FqlJoNsSr+1pknYGzpLUn9ReOhL4NakZ4P7cUTUF2HZWP08Iob5i\naFSB7eeA1QuvLyN1HHXcbmDh+bGdrbP9JLBGYdUP8/JxwLjC9gcUnt8LfLGT0I7MjxBCi2rdVBqD\n9kMIbaSFC6aRTEMI7UGopYdGNbsDKoQQZgtRMg0htI0WLphGMg0htI+YzzSEEHpJcXfSEEIoSSTT\nEELovVau5kdvfgihbZR1OamkzfJUoE/lqzA7rp9H0hV5/d2ShlbbZyTTEELbKCOZ5nlAzgY2J03p\nuaukVTtstjfwhu0VgZ8Bp1JFJNMQQttQDf9qsB7wlO1nbE8DLge26bDNNqR5kwGuBr6mKhMD6ONZ\n7sKskDQF+GcvdzMEeLWEcHqrFeJohRigNeJohRig93Esa3uRsoKRdDMppmrm5ZPTcY7JcxFX9rMD\nsJntffLrPYAvFOfwkPRI3qYyJ/LTeZsuz0d0QPVQGb8kkibYHl59y/pqhThaIYZWiaMVYmilOCps\nb9bsGLoT1fwQwpzmRWDpwuul8rJOt8lzHg8CXutup5FMQwhzmnuBlSQtJ2luYBfSPeWKriPdpw5g\nB9L96LptE41qfnONqb5JQ7RCHK0QA7RGHK0QA7ROHKWy/ZGkA0h32OgL/Nb2JEnHAxNsXwf8BrhI\n0lPA66SE263ogAohhBJENT+EEEoQyTSEEEoQyTSEEEoQybRFSFpH0g8Kr1t3Roc5VHwnoTuRTFuA\npHWAycCFkj4PYNvN+s8rqdPfi1ZIJpKWbNax83fyZUn7NCuGjlrhOylqtXgaKZJpk0n6CvBXYDrw\nFnCOpLOE2rTuAAATUklEQVSgeQnV9swc256Svidp92bGk2PpI2kQcJukZl4JMzfwJUn9m504JKky\n9lHS5vmxUJNiGSxp7mpjMWdnkUyb7zPAT4GvkQYJfwtYRdLp0LwEJukbwBGk35FD8hi8hseTrz7B\n9kzbbwGnAOtLmrurEnSdvUD6zoY2848LpO8CQNJewGnAkcAxkjZsZBySvg+cBVwuaStJ/Rt5/FYR\nybT5Hge+AZwM3GP7WWBfYJik0+Dj/zSNImkbYGPgm7bPBHYGvi7p2EbFI2lIPtZHkoblkmk/4A5g\nTWAu2zMbkcwkfV7SpZIWtf0EcA1wgqT5ml0Sk7QlsB3weWAk8D6wtaQvN+j43wC2sL0b6bLMLWx/\n0Ihjt5pIps33b+AJ4EFgeUlL2H4O2AvYWNJJ9Q6gkpAKiWkVUjJdI1fdniYl/FGSjmpAPPMAx0k6\nJy86AfgtqQT/Eql9+T8l5TrFUDknS5KugHkPOFHSecC7wMuk67W7bGOuZ1z5+XzA2sBXgHVsTwV+\nTmoy2lXS+g2IYwipaeoA0nk6OK8vbbaotmE7Hk18AH3yz02BS0lJdNG8bBnSNGb1PL4Kz1cB5svP\ndwduBtYH+uVlywHLN+Cc9APWAC4EDsnLlicl0JtIl/rdBszd8TOUeU6ArYGrKt8BsAKwG3A78CZw\nVoN/V4rfVX9SYWge4FjgD8Dn87rFgR9Xfo/qFMuw/HP7fD5uAvrmZT8Azqm8nlMeTQ9gTnoAw4Et\nC6/nKv7CASNIE9J+B1ikwbF9BxgPXAJcQSp1jQJuBDaqJNQGxFH547IxqZT1EHBUYf36pIknXgQO\nrWMcGwCPFJLGXMC8+fmSwIb5PNX9j0s+ZjGRHpz/8I7L52kYcAhpEuM18zZ1S2TAd4H78+/IcsC1\nOYFuDOwBTARWa9Tvbqs8oprfILkqOBC4T9LSALan254haQNJZ9geR/oLPwyYVud4Fiw83wDYD9iJ\n1IkxkVTSuYxU6jiABk2K49QOuiGpVPpn4GJSh9xP8/q7bF8N/BewqtItKOphHeBWYEquwl4JXC9p\nftsv2r6dlGAXrdPxP8GVjCptTUpY3yf9ruwIrEw6T48Bh+WZkGbWIw5Jm5NqT9vZfsupjf//AQNI\nHaibk9raJ9Xj+C2t2dl8TnjwcbWxD6kKdhuwZ162BqkTaovC9gPrHM/KpARZqSavA5yXn8+VH78B\nNs3LBjf4fO0I/Dg/HwB8EfgLcEJhm91JU6nNW/J3VGnSWAG4hfSH5SDgS6Sq6/C8fkVSO/eKdT4X\nI4G9C68PBs4ovP4mqZS4EDC43t8V8HXg6Py8Px9X7Svnr38jf1da6REl0zorjgUElrI9GTiT1OO6\nC/Ac6T/LTZVSlu136xzWdFI1cYVcKn2OdI+bfZ1Ky9PzNsvm7V+vZzCd9Mi/Aewp6bO237M9HngF\nWFfSKnmbt4DdbX9IL1W+I0mbAkfkQfnPAlsBX3Ma0fA+8NX8E9tPASPzz3p6DLhVH9/wbSKwcOHi\njouBp0m/W6/Z7nYC4xK8R/pu1rD9gVPNajSpLRnPoT35EPOZ1l0lkUr6HrBTribdQqqG7QvMsH1V\n3rwuVbOK3Pv7ge1nJS0B7AOINEZwO+BKScuSExfwP8XPUKeYKolsY1Ip7AngLtLdIE+XdDRgYBHg\nANuP5ZhuKCuGfPzNSZ/3ANIN1obl15MlbUQaTXCw7X9I6mt7hu0pZcXQUeW82H5B0lzA25JOsn2i\npO2BHSUNJyX3NYB6xjKK1N5/Dald/WTgLEk/BxYG9ieVkOdokUwbQOkGXqOAHSulTkmVhPp9SX1s\nX1HnpDWA1JH0gaQ1SKW/C0lDnr4NnE9q7xpNGpT+rQaUuiqJbAvgRNLA81Gk5HAyMB/wS1L78c8r\nibRDab/X8iDzbwO7ktpAXyJ1rPw4x/UqqRR8d455RlnH7iKewcC2wG8k7Uy6cePqwO2SXgcOA/Yk\ndfjMBXzD9r/rFMuOpNse/43U3HEjqS35HVIb+/vMqW2kHTW7nWF2fPDJntcBpPa9ffLr+Qrr5gG2\nBJZuQEz9SL/8E0jVwhXy8tVICeNkYI0mnKs+wHGkYWAjc3yLF9b3B+bveF57ecwBpOo7pNEBq5I6\nB1cC7iTNvj6QNJ70BGBAE87Lb0njae8AFs7LViTVGg4sbDdfHWPYELgPWCm/3pY0+/5+fDyErtRh\nae38iDbTkhVLTZK+S6rKLwYcKGkh2+/ndXsCa9u+0fbz9YwH0pVEpNLF26Qe+lXyFTyTSCXUeUhX\nOTX0UkCneQDmBi4ijZfczvZkSVvmq2um2X4nb1tWaXQacKikW0nt13M51Rj65HULkYY/3Qpcb/u9\nko5bVaH9+Jek2xXPbft1+E877ZeA/5H0o7zs/TrFURklMYBUEsb2tcANpLbjXST1K/E7aXuRTEtW\nSKTfJg0Vucr26aQbdP1W0mqS9gYOJVW166aTxL4dqTd2HGlo0bZ50/dIYwV/5Tp3IBSuLFojd35B\nGttq4Ebbz+dLIX8GvO6Sq9T5nEwHTgLWAl62/WBe/SpwN6lz7mbgbNv3lHn87uKC/zR79CONr10B\neEHSnYVNXyJdtnl1HWP5DLCA0/CvPYAF9fGlxNcBFwB/zH+gQxb3gKqDXLq7jFS6mEAa6rMUqT1w\nHKkKeZQb1M4k6UBSe+B+th/M8e1FqtYuDnwO2Nj2qw2KZxPSgPz3SEOerid1MH0X+IA0iuBIl9jJ\nlI+rQrJaiNS0cCFwl+198jafA2YACzYykRb+6H0H+CzpD8lxedm1pI6e35A6DTd3nUZ8SDoE2Ix0\nDsbaPlPSF0hDsl6wfXg9jjtbaHY7w+z6IHXkPEBKFKeS2plOJXWqzNXAOOYmJfbl8ut58s/+pA6p\nI2nA1SoUxnGSBnl/PsdwNqmjZ1g+N8sXYi29PQ7YJB9zVH49EJgE/CLHdD11HufbTWzfITXFrEAq\nJf+W3H5MKqlfBKxex+PvSxoD3Z/UNjoTOC6v+zKpk3JIM85NOzyaHsDs+gDmJQ0vqnQe7Eaat7Ru\nHQb5OJ+4fjv/vKOSPArr1m3COdmWVIW+m3yRArA0aWjWmeRLN+t4/A1IHSoHk8ZrHk3qDZ+PVEIe\nB2zbhPPSh1TyvJA0kuJ7wJ9Il6teTxpD+p8/hHWMY2tS+/5BpDkJViH11h+T15dygcTs+mh6ALP7\nI/9H2Rt4uJ6linysYiI9IJfADiK13f6KPC8AaXTBfdRxIoxOYlstl7p2Io0ceIg001EloZ4LfLaO\nx1+e1G49qhDP73NCnTd/T0M6nscGfVeV+QgGkK5GG1d4/Sap5F632kz+Y1K58muRfJ4qk6acT5oH\nYeFG/a606yPGmdbfvKTq0k62H63ngVz5X5ra3XYkJc0JwD2kToPT84Dv9YEdbL9Sz3gqJK1Mqto/\nYvtK0sUBk4FzJR1g+25JBzlNIVcvS5C+h50l3WJ7Ur4g4GekRHqyc5tx5TzWU+G72gVYXdJVTu3Z\n7wN9JS1PasseC5zv1GlWOkmHA+vl5wc6jaR4mTQB90jSOVvHeURB6Fp0QDVA2YPMqxxrAeB0UvLa\niTQQfwqprfJq4ElgitNlrfWMo9ipMhA4itQm+RNgvNOEJoeTSs0bAO+5xJ77ylVKkpYCXrP9QU7q\n+5GGP52ZE8eqpOaQ+8o6dpW4iudlJ9J5uRHYBvgR6Qqj0aThR0sA29frj3A+NxeSmj2+Sfp9WZuU\nXDclfS+jbT9Uj+PPbiKZzoaUJldehTQhxlfzsJvXgDNIJbC6lHIKx6/0mq9PuqLoHdu3SjqO1It+\nKXB33mao02TYZR17WdLv9XP5yqqTSO2hC9geLWk10qiKuYFT6v1HpUNsxUS6LKlK/3Quke6R4zqN\nNIHLAsB02y/UKZZvkCZGGWL7J3nZ/wJbABvZniJpgBs4xrbdxTjT2VCuLr8P9FOaEGNLUofGBfVO\npPn4Vrrp3RjSIPMTJf3a9o9JSX0v0kxQlJxIh5KqxYvnMaw/AXYhXUm0jaTrnIajXUwa1zqorGPX\nEFsxkR5IunDiGNLVX9i+iNQ+eRKwvu1n65hItyedmy1Id3PYMsfwfVIn3F/zoP26XBAwu4qS6Wwq\nl04PJl2iuQRpXoB/NOjYfUjDsa6wfU1edhfpiqLjSDfFO6/M6msufX+DlKTPII2euJnUO34iqQp7\nPvC27W0kDXT9Z+eqxNbf+WIIpUlTRpHOw2DScKh3bB+S1+9EKrX/s06xjCYNiTuE1Ll1KKnT6a+2\nb8zbLNqo9vTZSZRMZ1O5dHo6qRS4ab0TaeHKphGkpPYKnyzZ7AUsYXsacHjZ7YC51PdnUqfbI8Cf\nclvf9sCpuQT8N2BZSWs3MJF+jnTp5dySFiMNA1uUNIXgQ6TxrfMr3VsK21fWK5FmA0h/aIbnWsol\npDb1rSX9V96mbjNQzc4imc7GnOYmfd72iw04lpVmgf8Z8C/SPJznKt2QDtK17kMlDSJVsevhbdKF\nEm+TZp4CmEqat3UHUpPDjrbvr9PxOzOANNRoKOnqrn3ysq0BbE8kDQubmpNtXUhaVtI8tn9G6uC6\nXNKw3JRwGWmC8gdyTFFd7YEYGhVKkXvs9wK+6zRV3d1Kt2seK2ksqX3uMNtv1SsGp0lTtpC0DPCX\nXFq+iJTANgHOtf1kvY5fVGkjtT1B6U6dhwMvkOZIPZrUZilJl+VtHq7X0DBJB5PazWdIuo1UOp4B\n3CJpc9v3S/p5maMp5kTRZhpKoTRf6ljgeNu3FHr0/5t0gcA8tu9r1DAxSWuRRg38r+1fV9otG3H8\nzo6RO8R2JFWhzyRdPnsOqQnisjrGsg7pJo1f4ePpBhe3fYjSzFP75GXT8x+j0EORTENplO4mMJjU\n8fRoHhp1DOmqo5ebEM86pLG1Xy1z1ECVYxZ77fcnDVGbQLpB4Yqk0vtLpLbS1YB/17GzaX/SRDYr\n2N49L1uH9J0cZfsRSYNd/1udzBGizTSU6RrS79QYSSeThiD9ohmJFCAPxF+jUYk0H7OSSEeQpq+b\nTBoEfyLwDGnmp5VJ7Zb31DGRbksa+H8jsKjSjP2Vc/IuqTQKdZ4Gck4SJdNQqlzdX5c0YcZzuf20\nmfE07OqzwjH3IA092jMPyB8O7ECagPtY0tR/r9Trj0zu9BsP3Gb7m7mpZSQpsT8MHEGaaKaeowbm\nOJFMQ+iljglb0tKkYU8X2T4wL1uHNL70PeBHDWi33Z40SmAf29dJWps0G9WrwIW2H67n8edE0Zsf\nQi90aCM9gHTju4dJcyLcJOlF26fmzrcZwEuNKCnbvkbSNOAneUjUVaRbNPeJjqb6iGQaQi8UEmlx\npq6HSKXCvYFf5Gvcj8ljShsZ2w05gY+RNNP27yOR1k90QIXQS0ozda1Nmgdge9JEJUOBrUiX9O4u\naXDlKrFGsv1H0giCBxp97DlNtJmGUAJ1PlPXm6TOnoud77AaZl9RzQ+hBLanKk3sXJmpa1nSRCs3\nRSKdM0TJNISSNHOmrtB8kUxDKJGkuUjT/s1sxAQzoXVEMg0hhBJEb34IIZQgkmkIIZQgkmkIIZQg\nkmkIIZQgkmkIIZQgkmnoEUkzJE2U9IikqyTN14t9jZB0Q37+dUlHdLPtgvk6+Fk9xrGSDqt1eYdt\nLsj3kKr1WEMlPTKrMYb2Fsk09NQHtte0vTowDdivuFLJLP9+2b7O9indbLIg6fbIIbSUSKahDLcD\nK+YS2eOSLiTdbnlpSZtKukvS/bkEOxBA0maSHpN0P2lyEPLyUZJ+kZ8vJukPkh7Mjw2AU0h3G50o\n6bS83eGS7pX0kKTjCvs6StITkv4OfLbah5C0b97Pg5J+36G0PVLShLy/rfL2fSWdVjj2t3t7IkP7\nimQaekVSP9LcnZXJhlcCzrG9Gmki5KOBkbbXJt0L6VBJ8wLnkW53vA7piqHO/Jw0W/ww0qxMk0gT\nhzydS8WHS9o0H3M9YE1gHUkb5cmYd8nLtiDN/l/NNbbXzcd7lDSFXsXQfIwtSbewnjevf8v2unn/\n+0parobjhNlQTHQSeqq/pMr8nLeT7m20BPBP2+Pz8i+S7jV0R559bm7gLtLsSs9Wbrss6WLSPZE6\n2hj4FkC+DfFbkhbqsM2m+VGZYm4gKbnOD/zB9vv5GNfV8JlWl3QiqSlhIOluqxVX5rlAn5T0TP4M\nmwJrFNpTB+VjP1HDscJsJpJp6KkPbK9ZXJAT5nvFRcCfbO/aYbtPvK+XBJxs+1cdjnFwD/Z1AbBt\nvm/TKGBEYV3H666dj/0928Wki6ShPTh2aHNRzQ/1NB74kqQVId1sT9LKwGPAUEkr5O127eL9fwH2\nz+/tK2kQ8A6p1FkxFtir0Ba7pKRFgb8B20rqL2l+UpNCNfMDk/NkJbt3WLejpD455uWBx/Ox98/b\nI2llpRsKhjlQlExD3diekkt4l+Xp6QCOtv2EpNHAjXkO0Nv5ZIKsOIh0y429gRnA/rbvknRHHnr0\nx9xu+jngrlwyfhf4pu37JV0BPAi8Qpr9vpr/B9wNTMk/izH9C7gHWADYz/aHkn5Naku9P08GPQXY\ntrazE2Y3MWtUCCGUIKr5IYRQgkimIYRQgkimIYRQgkimIYRQgkimIYRQgkimIYRQgkimIYRQgv8P\nCjqkIMxm7lEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f75e8c12ed0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predict(n_gram_vectorizer, logreg, test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### TF-IDF\n",
    "\n",
    "[Term Frequency - Inverse Document Frequency](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) is a little more advanced way to count words in a document.\n",
    "It adjusts for document length, word frequency and most importantly for frequency of a particular word in a particular document.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.54 s, sys: 10.3 ms, total: 1.55 s\n",
      "Wall time: 1.56 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tf_vect = TfidfVectorizer(\n",
    "    min_df=2, tokenizer=nltk.word_tokenize,\n",
    "    preprocessor=None, stop_words='english')\n",
    "train_data_features = tf_vect.fit_transform(train_data['plot'])\n",
    "\n",
    "logreg = linear_model.LogisticRegression(n_jobs=1, C=1e5)\n",
    "logreg = logreg.fit(train_data_features, train_data['tag'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'caesar',\n",
       " u'cage',\n",
       " u'caine',\n",
       " u'cal',\n",
       " u'calhoun',\n",
       " u'california',\n",
       " u'californians',\n",
       " u'called',\n",
       " u'calling',\n",
       " u'callous']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_vect.get_feature_names()[1000:1010]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy 0.465020576132\n",
      "confusion matrix\n",
      " [[23  2 12  2  2  1]\n",
      " [ 3 10 10  1  3  4]\n",
      " [ 9  6 49  0 21  1]\n",
      " [ 3  4  1  4  2  2]\n",
      " [ 1  2 20  0 11  1]\n",
      " [ 9  2  5  1  0 16]]\n",
      "(row=expected, col=predicted)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAAEmCAYAAADfpHMGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe8XFW5//HPN4UQAoQSVHqocgEJQkBBwYiBS1MQQZpo\nKEaQjqAI/JAqIF4UAUVAQIpU0UuToGIQqQkQSqQ3CUQIRTohJN/fH2sN2Zl7zplJzp6WPO+85pWZ\nvffs/cyec56z1tprryXbhBBC6J0+rQ4ghBDmBpFMQwihBJFMQwihBJFMQwihBJFMQwihBJFMQwih\nBJFM5yKSBkq6TtIbkq7qxX52lXRzmbG1iqSNJD3W6jgaQdIxki7Jz5eT9LakviUf41lJI8vc59wq\nkmkLSNpF0vj8wz9Z0p8kfb6EXW8PfBxY3PYOc7oT25fa3qyEeBpKkiWt3NM2tm+z/ck53P+zkl6W\nNKiwbC9JY+dkf41k+1+2F7Q9vdWxzKsimTaZpEOAnwM/JiW+5YBfAtuUsPvlgcdtf1jCvjqepH4l\n7KYvcGAJsUhS/L7NxeLLbSJJg4HjgH1tX2P7HdvTbF9n+7C8zQBJP5f0Yn78XNKAvG6EpEmSvpdL\nTJMl7Z7XHQscDeyYS7x7FquBeZuhuTTXL78eJelpSW9JekbSroXl/yi8b0NJ43LzwThJGxbWjZV0\nvKTb835uljSkm89fif/7hfi3lbSlpMclvSbpiML260u6U9J/8rZnSpovr/t73uyB/Hl3LOz/B5L+\nDVxQWZbfs1I+xjr59VKSpkga0cPXdipwqKRFuvlMtc7NiZJuB94FVszLTpB0R477OkmLS7pU0pt5\nH0ML+zhd0vN53b2SNuomjo++W0kb5H1XHu9LejZv10fS4ZKekvSqpCslLVbYz26SnsvrjuzhvIRq\ntuPRpAewOfAh0K+HbY4D7gI+BiwB3AEcn9eNyO8/DugPbEn6JV00rz8GuKSwr+rXQwED/YBBwJvA\nJ/O6JYE18vNRwD/y88WA14Hd8vt2zq8Xz+vHAk8BqwID8+uTu/lslfiPzvF/G5gC/A5YCFgDeA9Y\nIW+/LvDZfNyhwCPAQYX9GVi5i/2fAgzI8YwAJhW2+TbwT2ABYAzw0x6+i2eBkcA1wAl52V7A2Nk4\nN//Kn6tf/sxjgSeBlYDBOZbH83H6ARcBFxRi+AaweF73PeDfwPzV32/xu636DP2BW4GT8usDST9f\ny+Rz9GvgsrxudeBtYOO87rR8Pke2+nenEx5RMm2uxYFX3HM1fFfgONsv254CHEv6Za2YltdPs30j\n6Yd/jtoEgRnAmpIG2p5se2IX22wFPGH7Ytsf2r4MeBT4cmGbC2w/bvs94Epg7R6OOQ040fY04HJg\nCHC67bfy8f8JDAOwfa/tu/JxnyX94n+hjs/0I9tTczyzsH0uKZndTfoDUk/p62hgf0lLVC2v59xc\naHtiXj8tL7vA9lO23wD+BDxl+y/55+Iq4NOFeC+x/Wp+//+QktzsfN+/AN4qfM69gSNtT7I9lZSQ\nt8+1le2B623/Pa/7f6TzGeoQybS5XgWG1GjLWwp4rvD6ubzso31UJeN3gQVnNxDb7wA7kn65Jku6\nQdJqdcRTiWnpwut/z0Y8r3rmRZJKsnupsP69yvslrSrpekn/lvQmqZ25yyaEgim236+xzbnAmsAZ\nOWn0yPbDwPXA4VWr6jk3z3exy+rP2+XnB5B0qKRHcjPCf0il2VrnoPLe75BK5rvYriTF5YE/5KaT\n/5BK+9NJ7fdLFePNPyOv1nOsEMm02e4EpgLb9rDNi6Qf+Irl8rI58Q6pOlvxieJK22Nsb0oqoT1K\nSjK14qnE9MIcxjQ7fkWKaxXbCwNHAKrxnh6HQZO0IOkC4G+AY4rthTX8iNREUEyU9ZybOR6WLbeP\nfh/4OqkpZxHgDWqfg8p7jwe2sf1mYdXzwBa2Fyk85rf9AjAZWLawjwVItalQh0imTZSrdUcDZ+UL\nLwtI6i9pC0k/yZtdBhwlaYl8Iedo4JLu9lnDBGBjpT6Ig4EfVlZI+rikbZS6/UwlNRd0VaW7EVhV\nqTtXP0k7ktrWrp/DmGbHQqR23bdzqXmfqvUvASvO5j5PB8bb3gu4ATi7njfZfhK4AjigsLjR52Yh\nUpvlFKCfpKOBhWu9SdKypOaWb9p+vGr12cCJkpbP2y4hqdKT5Gpga0mfzxf6jiNyRN3iRDVZbvc6\nBDiK9EvyPLAf8Me8yQnAeOBB4CHgvrxsTo71Z1ICeBC4l1l/yfvkOF4EXiO1RVYnK2y/CmxNuvjx\nKqmktLXtV+Ykptl0KLALqc3vXNJnKToG+G2usn691s5y0ticmZ/zEGAd5V4MdTiOdOEOaMq5GQPc\nRLpA9RzwPl03G1T7EqnafnXhin6lPfx04FrgZklvkS5GfSZ/nonAvqQLgpNJF9MmlfRZ5nqyY3Do\nEELorSiZhhBCCSKZhhBCCSKZhhBCCSKZhhBCCcoYCGKepP4LWAMGtzSGT62yVO2NmmDyWzX7vTfc\nkgsNaHUIbaOPanZDbYoJ99/7iu3qu8bmWN+Fl7c//D83tf0ffm/KGNubl3XcekUynUMaMJgBw/Zo\naQw33fijlh6/4oS/PtnqEDjqSz2OxDdPGdi/1CFN59iig/pV3x3WK/7wPQZ8smYPON6fcFZdd4iV\nLZJpCKEzSNCnPf5QdCWSaQihc7TxkLCRTEMInaNN2oO7Esk0hNAhFCXTEELoNRFtpiGE0HuKan4I\nIZQiqvkhhFCCKJmGEEIvRT/TEEIoSVTzQwiht6JrVAgh9J6Avu1bzW/fNF8yScMl/aKbdRtJmihp\ngqSlJV3d7PhCCHWQaj9aZJ4pmdoeT5qoriu7AifZrswCun1zogoh1K+9q/ntG1mdJA2SdIOkByQ9\nLGlHSetJuiMvu0fSQpJGSPo/U/BK2os0L/nxki6VNFTSw83/JCGEmqJk2lCbAy/a3gogzw9/P7Cj\n7XGSFga6HVHW9nmSPg9cb/tqSUO721bSaGA0APPVnL48hFCmNu8a1fElU9Lc8ptKOkXSRsBywGTb\n4wBsv2n7wzIOZPsc28NtD1f/BcrYZQhhdqhP7UeLdHwytf04sA4pqZ4AbFfrPZLG5ItN5zU6vhBC\niaKa3ziSlgJes32JpP8A3wWWlLReruYvRFU13/Z/tyLWEEJvtPcFqI5PpsCngFMlzQCmAfuQeqSd\nIWkgKZGObGF8IYQyxBB8jWV7DDCmi1WfrXo9Nj+62seowvNngTVLCS6EUKLySqaSNgdOB/oC59k+\nuWr9KOBU4IW86EzbPTYLdnwyDSHMQ0poE5XUFzgL2BSYBIyTdK3tf1ZteoXt/erdbyTTEELnKKea\nvz7wpO2nASRdDmwDVCfT2dK+rbkhhFAk1ds1aoik8YXH6Ko9LQ08X3g9KS+r9jVJD0q6WtKytcKL\nkmkIoXPUV81/xfbwXh7pOuAy21MlfQf4LbBJT2+IkmkIoWNIqvmowwtAsaS5DDMvNAFg+1XbU/PL\n84B1a+00kmkIoSOkWr5qPuowDlhF0gqS5gN2Aq6d9VhasvDyK8AjtXYa1fwQQoeou+TZI9sfStqP\n1KWyL3C+7YmSjgPG274WOEDSV4APgdeAUbX2G8k0hNAxykimALZvBG6sWnZ04fkPgR/Ozj4jmYYQ\nOkafPu3bMhnJNITQGZQfbSqS6RxafaUlufKqI1oaw4F/iDGsK+br2x4llpfenFp7owZ7YWq3w/d2\nNJXUZtookUxDCB0jkmkIIZQg2kxDCKG3os00hBDKEdX8EELopbgAFUIIJanzdtGWiGQaQugMimp+\nCCGUIpJpCCH0klB0jQohhFK0b8E0kmkIoUNEm2kIIZQjkmkIIZQgukY1kaQRwAe278iv9wbetX1R\nSwMLIfRalEybawTwNnAHgO2zWxpNCKEUszFhXku0bz+DKpL+KOleSRMr82BL2lzSfZIekPRXSUOB\nvYGDJU2QtJGkYyQdmrdfW9JdeS7sP0haNC8fK+kUSfdIelzSRq36nCGE7pU0O2lDdFLJdA/br0ka\nCIyT9L/AucDGtp+RtFhefzbwtu2fAkj6UmEfFwH72741T571I+CgvK6f7fUlbZmXj2zaJwsh1CXa\nTMtxgKSv5ufLAqOBv9t+BsD2az29WdJgYBHbt+ZFvwWuKmxyTf7/XmBoN/sYnY/Lkksv29UmIYQG\nimp+L+WLSiOBDWwPA+4HJpR8mMp8E9Pp5o+M7XNsD7c9fLHFh5R8+BBCj9Te1fyOSKbAYOB12+9K\nWg34LDA/sLGkFQAkLZa3fQtYqHoHtt8AXi+0h+4G3Fq9XQihPaXbSWs/WqVTqvk3AXtLegR4DLgL\nmEKqcl8jqQ/wMrApcB1wtaRtgP2r9vMt4GxJCwBPA7s3Kf4QQgnauJbfGcnU9lRgi25W/6lq28eB\ntQqLbiusm0Aq1Vbvf0Th+St002YaQmitdm4z7YhkGkIIqc201UF0r1PaTEMI8zgBffuq5qOufaU+\n6o9JelLS4T1s9zVJljS81j4jmYYQOkYZV/Ml9QXOIjUdrg7sLGn1LrZbCDgQuLue2CKZhhA6Q67m\n13rUYX3gSdtP2/4AuBzYpovtjgdOAd6vZ6eRTEMIHUGU1s90aeD5wutJednMY0nrAMvavqHe+OIC\nVAihQ9Tdj3SIpPGF1+fYPqfuo6SulqcBo2YnukimIYSOUWfJ8xXbPV0weoF0S3rFMnlZxULAmsDY\nfLxPANdK+ortYpKeRSTTEEJnKK9r1DhglXz35AvATsAulZX5bsmP7heXNBY4tKdECpFMQwgdQlDK\n7aK2P5S0HzAG6Aucb3tiHkluvO1r52S/kUxDCB2jrDugbN8I3Fi17Ohuth1Rzz4jmYYQOkY73wEV\nyXQO2TBjhlsawylb/59+xi2x+qaHtjoELvrGma0OAYDrxj3X6hDYd8MVWh1CY8RUzyGE0Huqv2tU\nS0QyDSF0jDYumEYyDSF0jqjmhxBCL0nldI1qlEimIYSOESXTEEIoQRvn0kimIYTOESXTEELoJSm6\nRoUQQinauGAayTSE0Dn6tHE27TaZSlq4pzfafrP8cEIIoXttnEt7LJlOBEwa+aqi8trAcg2MK4QQ\nZiFB305sM7W9bHfrQgihFdr5an5dE+pJ2knSEfn5MpLWbWxYjSdphKTrWx1HCKF+Jc1O2hA1k6mk\nM4EvArvlRe8CZzcyqBBCqCagr1Tz0Sr1lEw3tP0d8tzRtl8D5iszCEnflPSgpAckXSxpqKRb8rK/\nSloub3ehpF9JukvS07l0eb6kRyRdWNjfZpLulHSfpKskLZiXby7pUUn3AdvlZX0kPSFpicLrJyuv\nQwhtoo5pnlvZDFBPMp2Wpz41gKTFgRllBSBpDeAoYBPbw4ADgTOA39peC7gU+EXhLYsCGwAHA9cC\nPwPWAD4laW1JQ/L+RtpeBxgPHCJpfuBc4MvAuqQZB7E9A7gE2DXvfyTwgO0pXcQ6WtJ4SeNff+2V\nsk5BCKFOHV3NB84Cfg8sIelY4B/AKSXGsAlwle1X4KOS7wbA7/L6i4HPF7a/zraBh4CXbD+UE+JE\nYCjwWWB14HZJE4BvAcsDqwHP2H4iv/+Swj7PB76Zn+8BXNBVoLbPsT3c9vBFFxvS1SYhhAYRqZ9p\nrUer1Oy0b/siSfeSSmwAO9h+uLFh9Whq/n9G4XnldT9gOvBn2zsX3yRp7e52aPt5SS9J2gRYn5ml\n1BBCG2nn20nruppPmg51GvDBbLynXrcAO+TmAyQtBtxBmssaUmK7bTb2dxfwOUkr5/0NkrQq8Cgw\nVNJKebudq953Hqm0epXt6XP0SUIIDVNPFb+tq/mSjgQuA5YClgF+J+mHZQVgeyJwInCrpAeA04D9\ngd0lPUjqRXDgbOxvCjAKuCy//05gNdvvA6OBG/IFqJer3notsCDdVPFDCK3X0dV8Ulvip22/CyDp\nROB+4KSygrD9W+C3VYs36WK7UYXnzwJrdrPuFmC9Lt5/E6nttCvDSBeeHq0/8hBCM7VvJb++ZDq5\nart+edlcQ9LhwD5EW2kIbUt06O2kkn5G6g71GjBR0pj8ejNgXHPCaw7bJwMntzqOEEIPWtyPtJae\nSqaVK/YTgRsKy+9qXDghhNC9Ns6lPQ508ptmBhJCCD0ps5ovaXPgdFJPpfNy7bS4fm9gX1JXy7eB\n0bb/2dM+a7aZ5q5EJ5I6ws9fWW571dn9ACGE0BtlVPMl9SXdjLQpMAkYJ+naqmT5O9tn5+2/Qupl\ntHlP+62nz+iFpO5CArYArgSumN0PEEIIvaU6HnVYH3jS9tO2PwAuB7YpblA1+P0g8u30PaknmS5g\ne0w+wFO2jyIl1RBCaBqp7n6mQypjaOTH6KpdLQ08X3g9KS+rOp72lfQU8BPggFrx1dM1amoe6OSp\n3I7wArBQHe8LIYRS1Xk76Su2h/f2WLbPAs6StAtp8KRv9bR9Pcn0YFIx9wBS2+lg0mAgIYTQVCVd\nzX8BKM4kskxe1p3LgV/V2mk9A53cnZ++xcwBokMIoalEabeLjgNWkbQCKYnuBOwyy7GkVWw/kV9u\nBTxBDT112v8DPTS62t6ujqDnWu9Pn85jr77V0hgemfJOS49f8dhfftrqEPjp2CdbHQIAm6/U+jHF\nH32xtT+XDVPSQCa2P5S0HzCG1DXqfNsTJR0HjLd9LbCfpJGkAZ5ep0YVH3oumZ7Z+7BDCKE8ZU1L\nYvtG4MaqZUcXntc9uFJFT532/zq7OwshhEYR7T07aT0XoEIIoS208TgnkUxDCJ1B6tBRo6pJGmB7\nau0tQwihMdo4l9Y10v76kh4idw2QNEzSGQ2PLIQQqnT0tCWkaZa3Bl4FsP0A8MVGBhVCCNU6fnZS\noI/t56quosWEcyGEpuvbxtX8epLp85LWB5yHrtofeLyxYYUQwqzU4pJnLfUk031IVf3lgJeAv+Rl\nIYTQVG2cS+u6N/9lZs5hH0IILSGgXxtfzq9npP1z6eIefdvVYwSGEEJDdXTJlFStr5gf+CqzDqwa\nQgiNp/buZ1pPNX+WKUokXQz8oxHBSDqA1B57n+3ZmsNe0hG2f9yIuEII7UH1TkzSAvX0M622AvDx\nsgPJvgtsOruJNDui7GBCCO0jtZnWfrRKPXdAvS7ptfz4D/Bn4IdlByLpbGBF4E+SfiDpTkn3S7pD\n0ifzNqMkXSPpJklPSPpJXn4yMFDSBEmX5mV/lHSvpImVOWAk9ZV0oaSHJT0k6WBJK0m6rxDHKsXX\nIYT2Ianmo1V6rOYrRTaMmUP6z7Bdc5a+OWF77zyX9ReBD4D/yYO4jgR+DHwtb7o28GlgKvCYpDNs\nHy5pP9trF3a5h+3XJA0kTeX6e2AosLTtNfPnW8T2fyS9IWlt2xOA3UmzsYYQ2ki6A6rVUXSvx5Jp\nTpw32p6eHw1JpF0YDFwl6WHgZ8AahXV/tf2G7feBfwLLd7OPAyQ9ANxFmu9lFeBpYEVJZ+TEXZnO\n9Txg93xTwo7A77raoaTRlRkP33z91V5+xBDCbKnjvvx2vzd/gqRPNzySWR0P/C2XIL9M6kVQURy5\najpdlK4ljQBGAhvYHgbcD8xv+3VSSXsssDcpiQL8njR99dbAvba7zJS2z7E93PbwhRddfM4/XQhh\ntlX6mdZ6tEpPc0D1s/0hqUo9Ls8f/Q7pM9n2Og2MazAzmxZG1fmeaZL6256W3/+67XclrQZ8FkDS\nEOAD27+X9BhwCYDt9yWNIc1AuGeJnyOEUKJO7Wd6D7AO8JUmxVL0E+C3ko4CbqjzPecAD+aLR3sA\ne0t6BHiMVNUHWBq4QFKlRF68kHYpqQ/tzb0NPoTQCKJPG3eN6imZCsD2U02KBdtD89NXgFULq47K\n6y8ELixsv3Xh+Q+AHxTes0U3h+muRP154ALbMSJWCG0ojbTf6ii611MyXULSId2ttH1aA+JpiTyt\n9UrAJq2OJYTQvU4dNaovsCC0cbm6JLa/2uoYQgg9S7OTtjqK7vWUTCfbPq5pkYQQQg2dWjJt36hD\nCPMc0bkj7X+paVGEEEItoqW3i9bS7bUx2681M5AQQqhFdTzq2o+0uaTHJD0p6fAu1h8i6Z+SHpT0\nV0nd3Wn5kTbuaBBCCDOVNTtpvm38LFL3ydWBnSWtXrXZ/cBw22sBV5P6vvcokmkIoWP0Ue1HHdYH\nnrT9tO0PgMuBbYob2P6b7Xfzy7uAZWrttJ6R9kMIoQ3UPcTeEEnjC6/PsX1O4fXSzDpbyCTgMz3s\nb0/gT7UOGsk0hNARRN1V6VdsDy/lmNI3gOHAF2ptG8k0hNAxSupn+gJpWM6KZZg5sNJH8ljKRwJf\nsD21en21SKZzaKEB/fjiyku0NIbPLLdYS49fscom32t1CLw+7sxWhwDAnpdPaHUInPHVNVsdQmOU\n1zVqHLCKpBVISXQnYJdZDpWGHf01sHme7r6mSKYhhI4wG9X8HuUZPPYDxpBumz/f9kRJxwHjbV8L\nnEq6nf6qnMD/ZbvHEfQimYYQOkZZnfZt3wjcWLXs6MLzkbO7z0imIYSO0c5zQEUyDSF0hFTNb99s\nGsk0hNAx2vjW/EimIYROIRQl0xBC6J00BF8k0xBC6B1FNT+EEEoRyTSEEHopqvkhhFCSuAAVQggl\naOOCaSTTEELnaOeSaduMtC9pb0nfLGlfR1S9vqOM/YYQWkeIvqr9aJW2Saa2z7Z9UUm7myWZ2t6w\npP2GEFold42q9WiVhiZTSX+UdK+kiZJG52VvSzpR0gOS7pL08bz8GEmH5udjJf1M0nhJj0haT9I1\nkp6QdEKN/Z8MDJQ0QdKllWPm/yXpVEkPS3pI0o55+Yh8zKslPSrpUrXznLIhzKPKmp20ERpdMt3D\n9rqkYf8PkLQ4MAi4y/Yw4O/At7t57wd56oGzgf8F9gXWBEbl/XS5f9uHA+/ZXtv2rlX73A5YGxgG\njAROlbRkXvdp4CDSbIUrAp+rDkjS6Jzgx7/6yiuzfzZCCHOs0jVqXq3mHyDpAdLsfssCqwAfANfn\n9fcCQ7t577X5/4eAibYn56kDnmbmlANd7b8nnwcusz3d9kvArcB6ed09tifZngFM6Cou2+fYHm57\n+OJDhtQ4VAihdG1cNG3Y1XxJI0ilvw1svytpLDA/MM2282bTe4ihMufKjMLzyut+Pex/ThWP0VNc\nIYQWmVev5g8GXs+JbjXgs03c/zRJ/bt4z23AjpL6SloC2Bi4p+S4QggNMq9egLqJVIJ8BDiZVBVv\n1v7PAR6sXIAq+APwIPAAcAvwfdv/LjmuEEKDtHMybVhVNrdvbtHFqgUL21wNXJ2fH1NYPqLwfCww\ntqt13ewf2z8AflB4vWD+38Bh+VHcvvoY+3X9qUIIrZKaRNu3mh/tgiGEzhBD8IUQQjnaOJdGMg0h\ndAqVNtVzI0QyDSF0jDbOpZFMQwidodW3i9YSyTSE0DHauZrfNqNGhRBCLWX1M5W0uaTHJD0p6fAu\n1m8s6T5JH0ravp59RjINIXSMMm7Nl9QXOIvUT311YGdJq1dt9i9gFPC7emOLan4IoTOU12i6PvCk\n7acBJF0ObAP8s7KB7Wfzuhn17jSSaQihIwjoU189foik8YXX59g+p/B6aeD5wutJwGd6G18k0zn0\n77em8uNbnmxpDAd8boWWHr9izBXHtToE3pn6YatDAOA3O63d6hBY4hu/bXUIDVNnwfSVPBZyU0Uy\nDSF0jnKq+S8wc0xkgGXysl6JC1AhhI6hOv7VYRywiqQVJM0H7MTMwejnWCTTEELH6KPaj1psfwjs\nB4wBHgGutD1R0nGSvgKQ552bBOwA/FrSxFr7jWp+CKFzlNRn3/aNwI1Vy44uPB9Hqv7XLZJpCKEj\nxHimIYRQhjqr8a0SyTSE0DkimYYQQm/VfbW+JSKZhhA6RhsPGhXJNITQGUQk0xBCKEVU80MIoQRR\nMg0hhBK0cS6NZBpC6BBq72lLWppMlc6MbNc9AGsIYd7U7hegmj7QiaShee6Vi4CHgd0kPSTpYUmn\nFLZ7W9KpkiZK+ouk9SWNlfR0YTCCoZJuy3O13Cdpw7x8RN72akmPSro0J+7KAAZ3SHpA0j2SFpLU\nNx9rnKQHJX2n2eclhFBbGdOWNEqrSqarAN8izbNyF7Au8Dpws6Rtbf8RGATcYvswSX8ATgA2Jc3Z\n8lvSkFkvA5vafl/SKsBlQGVQ2E8DawAvArcDn5N0D3AFsKPtcZIWBt4D9gTesL2epAHA7ZJutv1M\nMWhJo4HRAAstsVRDTkwIoXt1jrTfEq1Kps/ZvkvSNsBY21MAJF0KbAz8EfgAuClv/xAw1fY0SQ8B\nQ/Py/sCZktYGpgOrFo5xj+1Jeb8T8nveACbnEWGw/WZevxmwVmEWwsGkhD9LMs1TH5wD8IlV1nQJ\n5yGEMDvaN5e2LJm+U8c202xXEtYMYCqA7RmSKnEfDLwEDCM1WbxfeP/UwvPp9PxZBexve0wdcYUQ\nWqSNc2nLB4e+B/iCpCF5+tWdgVtn4/2DSSXNGcBuQN8a2z8GLClpPYDcXtqPNEjsPpL65+WrSho0\nm58lhNBAUn2PVmnp1XzbkyUdDvyN9EfnBtv/Oxu7+CXwe0nfJDUJ9Fjitf2BpB2BMyQNJLWXjgTO\nIzUD3JcvVE0Btp3dzxNCaKzoGlWQ56Nes/D6MtKFo+rtFiw8P6ardbafANYqrPpBXj4WGFvYfr/C\n83HAZ7sI7Yj8CCG0qfZNpdFpP4TQQdq4YBrJNITQGYTaumtUqy9AhRDCXCFKpiGEjtHGBdNIpiGE\nzhHjmYYQQi8pZicNIYSSRDINIYTea+dqflzNDyF0jLJuJ5W0eR4K9Ml8F2b1+gGSrsjr75Y0tNY+\nI5mGEDpGGck0jwNyFrAFaUjPnSWtXrXZnsDrtlcGfgacQg2RTEMIHUN1/KvD+sCTtp+2/QFwObBN\n1TbbkMZNBrga+JJqDAygmaPchdkhaQrwXC93MwR4pYRweqsd4miHGKA94miHGKD3cSxve4mygpF0\nEymmWuZn1uE4z8ljEVf2sz2wue298uvdgM8Ux/CQ9HDepjIm8lN5m27PR1yAmkNl/JBIGm97eO0t\nG6sd4mjqWEVDAAAUS0lEQVSHGNoljnaIoZ3iqLC9eatj6ElU80MI85oXgGULr5fJy7rcJo95PBh4\ntaedRjINIcxrxgGrSFpB0nzATqQ55YquJc1TB7A9aT66HttEo5rfWufU3qQp2iGOdogB2iOOdogB\n2ieOUtn+UNJ+pBk2+gLn254o6ThgvO1rgd8AF0t6EniNlHB7FBegQgihBFHNDyGEEkQyDSGEEkQy\nDSGEEkQybROS1pX0/cLr9h3RYR4V30noSSTTNiBpXWAycJGkTwHYdqt+eSV1+XPRDslE0tKtOnb+\nTj4vaa9WxVCtHb6TonaLp5kimbaYpC8AfwOmAW8Av5R0BrQuodqekWPbXdL+knZtZTw5lj6SBgO3\nSmrlnTDzAZ+TNLDViUOSKn0fJW2RH4u2KJbFJc1Xqy/m3CySaet9AvgJ8CVSJ+FvAqtJOg1al8Ak\nfQ04nPQzcnDug9f0ePLdJ9ieYfsN4GRgA0nzdVeCbrBJpO9saCv/uED6LgAk7QGcChwBHC1po2bG\nIel7wBnA5ZK2ljSwmcdvF5FMW+8x4GvAScA9tp8Bvg0Mk3QqzPylaRZJ2wCbAN+wfTqwI/AVScc0\nKx5JQ/KxPpQ0LJdM+wG3A2sD/W3PaEYyk/QpSb+T9DHbjwPXAMdLWqDVJTFJWwFfBT4FjATeBb4s\n6fNNOv7XgC1t70K6LXNL2+8149jtJpJp6/0beBx4AFhR0lK2nwX2ADaRdGKjA6gkpEJiWo2UTNfK\nVbenSAl/lKQjmxDPAOBYSb/Mi44HzieV4F8ktS9/VFJuUAyVc7I06Q6Yd4ATJJ0LvA28RLpfu9s2\n5kbGlZ8vAKwDfAFY1/ZU4BekJqOdJW3QhDiGkJqm9iOdp4Py+tJGi+oYtuPRwgfQJ/+/GfA7UhL9\nWF62HGkYs0YeX4XnqwEL5Oe7AjcBGwD98rIVgBWbcE76AWsBFwEH52UrkhLojaRb/W4F5qv+DGWe\nE+DLwFWV7wBYCdgFuA34D3BGk39Wit/VQFJhaABwDPAH4FN53ZLAjyo/Rw2KZVj+f7t8Pm4E+uZl\n3wd+WXk9rzxaHsC89ACGA1sVXvcv/sABI0gD0n4XWKLJsX0XuAu4FLiCVOoaBdwAbFxJqE2Io/LH\nZRNSKetB4MjC+g1IA0+8ABzSwDg2BB4uJI3+wPz5+dLARvk8NfyPSz5mMZEelP/wjs3naRhwMGkQ\n47XzNg1LZMC+wH35Z2QF4I85gW4C7AZMANZo1s9uuzyimt8kuSq4IHCvpGUBbE+zPV3ShpJ+bnss\n6S/8MOCDBsezSOH5hsDewNdJFzEmkEo6l5FKHfvRpEFxnNpBNyKVSv8CXEK6IPeTvP5O21cD/w2s\nrjQFRSOsC9wCTMlV2CuB6yQtZPsF27eREuzHGnT8WbiSUaUvkxLW90g/KzsAq5LO06PAoXkkpBmN\niEPSFqTa01dtv+HUxv//gEGkC6hbkNraJzbi+G2t1dl8Xngws9rYh1QFuxXYPS9bi3QRasvC9gs2\nOJ5VSQmyUk1eFzg3P++fH78BNsvLFm/y+doB+FF+Pgj4LPBX4PjCNruShlKbv+TvqNKksRJwM+kP\ny4HA50hV1+F5/cqkdu6VG3wuRgJ7Fl4fBPy88PobpFLiosDijf6ugK8AR+XnA5lZta+cv4HN/Flp\np0eUTBus2BcQWMb2ZOB00hXXnYBnSb8sN1ZKWbbfbnBY00jVxJVyqfRZ0hw333YqLU/L2yyft3+t\nkcF0cUX+dWB3SZ+0/Y7tu4CXgfUkrZa3eQPY1fb79FLlO5K0GXB47pT/DLA18CWnHg3vAl/M/2P7\nSWBk/r+RHgVu0cwJ3yYAixVu7rgEeIr0s/Wq7R4HMC7BO6TvZi3b7znVrEaT2pLxPHolH2I804ar\nJFJJ+wNfz9Wkm0nVsG8D021flTdvSNWsIl/9fc/2M5KWAvYCROoj+FXgSknLkxMX8NPiZ2hQTJVE\ntgmpFPY4cCdpNsjTJB0FGFgC2M/2ozmm68uKIR9/C9Ln3Y80wdqw/HqypI1JvQkOsv1PSX1tT7c9\npawYqlXOi+1JkvoDb0o60fYJkrYDdpA0nJTc1wIaGcsoUnv/NaR29ZOAMyT9AlgM2IdUQp6nRTJt\nAqUJvEYBO1RKnZIqCfV7kvrYvqLBSWsQ6ULSe5LWIpX+LiJ1efoOcAGpvWs0qVP6N5tQ6qoksi2B\nE0gdz0eRksNJwALAr0jtx7+oJNKq0n6v5U7m3wF2JrWBvki6sPKjHNcrpFLw3Tnm6WUdu5t4Fge2\nBX4jaUfSxI1rArdJeg04FNiddMGnP/A12/9uUCw7kKY9/jupueMGUlvyW6Q29neZV9tIq7W6nWFu\nfDDrlddBpPa9vfLrBQrrBgBbAcs2IaZ+pB/+8aRq4Up5+RqkhHESsFYLzlUf4FhSN7CROb4lC+sH\nAgtVn9deHnMQqfoOqXfA6qSLg6sAd5BGX1+Q1J/0eGBQC87L+aT+tLcDi+VlK5NqDQcUtluggTFs\nBNwLrJJfb0safX9vZnahK7VbWic/os20ZMVSk6R9SVX5jwMHSFrU9rt53e7AOrZvsP18I+OBdCcR\nqXTxJukK/Wr5Dp6JpBLqANJdTk29FdBpHID5gItJ/SW/anuypK3y3TUf2H4rb1tWafQD4BBJt5Da\nr/s71Rj65HWLkro/3QJcZ/udko5bU6H9+Fek6Yrns/0afNRO+zngp5J+mJe926A4Kr0kBpFKwtj+\nI3A9qe14J0n9SvxOOl4k05IVEul3SF1FrrJ9GmmCrvMlrSFpT+AQUlW7YbpI7F8lXY0dS+patG3e\n9B1SX8Ffu8EXEAp3Fq2VL35B6ttq4Abbz+dbIX8GvOaSq9T5nEwDTgQ+Dbxk+4G8+hXgbtLFuZuA\ns2zfU+bxe4oLPmr26EfqX7sSMEnSHYVNXyTdtnl1A2P5BLCwU/ev3YBFNPNW4muBC4E/5T/QIYs5\noBogl+4uI5UuxpO6+ixDag8cS6pCHukmtTNJOoDUHri37QdyfHuQqrVLAv8FbGL7lSbFsympQ/47\npC5P15EuMO0LvEfqRXCES7zIlI+rQrJalNS0cBFwp+298jb/BUwHFmlmIi380fsu8EnSH5Jj87I/\nki70/IZ00XALN6jHh6SDgc1J52CM7dMlfYbUJWuS7cMacdy5QqvbGebWB+lCzv2kRHEKqZ3pFNJF\nlf5NjGM+UmJfIb8ekP8fSLogdQRNuFuFQj9OUifvT+UYziJd6BmWz82KhVhLb48DNs3HHJVfLwhM\nBM7MMV1Hg/v59hDbd0lNMSuRSsnnk9uPSSX1i4E1G3j8b5P6QA8ktY3OAI7N6z5Pukg5pBXnphMe\nLQ9gbn0A85O6F1UuHuxCGre0YRcM8nFmuX87/397JXkU1q3XgnOyLakKfTf5JgVgWVLXrNPJt242\n8Pgbki6oHETqr3kU6Wr4AqQS8lhg2xaclz6kkudFpJ4U+wN/Jt2ueh2pD+lHfwgbGMeXSe37B5LG\nJFiNdLX+6Ly+lBsk5tZHywOY2x/5F2VP4KFGlirysYqJdL9cAjuQ1Hb7a/K4AKTeBffSwIEwuoht\njVzq+jqp58CDpJGOKgn1bOCTDTz+iqR261GFeH6fE+r8+XsaUn0em/RdVcYjGES6G21s4fV/SCX3\nhtVm8h+Typ1fS+TzVBk05QLSOAiLNetnpVMf0c+08eYnVZe+bvuRRh7Ild/S1O62AylpjgfuIV00\nOC13+N4A2N72y42Mp0LSqqSq/cO2ryTdHDAZOFvSfrbvlnSg0xByjbIU6XvYUdLNtifmGwJ+Rkqk\nJzm3GVfOYyMVvqudgDUlXeXUnv0u0FfSiqS27DHABU4XzUon6TBg/fz8AKeeFC+RBuAeSTpn6zr3\nKAjdiwtQTVB2J/Max1oYOI2UvL5O6og/hdRWeTXwBDDF6bbWRsZRvKiyIHAkqU3yx8BdTgOaHEYq\nNW8IvOMSr9xX7lKStAzwqu33clLfm9T96fScOFYnNYfcW9axa8RVPC9fJ52XG4BtgB+S7jAaTep+\ntBSwXaP+COdzcxGp2eMbpJ+XdUjJdTPS9zLa9oONOP7cJpLpXEhpcOXVSANifDF3u3kV+DmpBNaQ\nUk7h+JWr5huQ7ih6y/Ytko4lXUX/HXB33mao02DYZR17edLP9bP5zqoTSe2hC9seLWkNUq+K+YCT\nG/1HpSq2YiJdnlSlfyqXSHfLcZ1KGsBlYWCa7UkNiuVrpIFRhtj+cV72P8CWwMa2p0ga5Cb2se10\n0c90LpSry+8C/ZQGxNiKdEHjwkYn0nx8K016dw6pk/kJks6z/SNSUt+DNBIUJSfSoaRq8ZK5D+uP\ngZ1IdxJtI+lap+5ol5D6tQ4u69h1xFZMpAeQbpw4mnT3F7YvJrVPnghsYPuZBibS7UjnZkvSbA5b\n5Ri+R7oI97fcab8hNwTMraJkOpfKpdODSLdoLkUaF+CfTTp2H1J3rCtsX5OX3Um6o+hY0qR455ZZ\nfc2l76+RkvTPSb0nbiJdHT+BVIW9AHjT9jaSFnTjR+eqxDbQ+WYIpUFTRpHOw+Kk7lBv2T44r/86\nqdT+XINiGU3qEncw6eLWIaSLTn+zfUPe5mPNak+fm0TJdC6VS6enkUqBmzU6kRbubBpBSmovM2vJ\nZg9gKdsfAIeV3Q6YS31/IV10exj4c27r2w44JZeA/w4sL2mdJibS/yLdejmfpI+TuoF9jDSE4IOk\n/q0LKc0the0rG5VIs0GkPzTDcy3lUlKb+pcl/XfepmEjUM3NIpnOxZzGJn3e9gtNOJaVRoH/GfAv\n0jicZytNSAfpXvehkgaTqtiN8CbpRok3SSNPAUwljdu6PanJYQfb9zXo+F0ZROpqNJR0d9deedmX\nAWxPIHULm5qTbUNIWl7SANs/I13gulzSsNyUcBlpgPL7c0xRXZ0D0TUqlCJfsd8D2NdpqLq7laZr\nHiNpDKl97lDbbzQqBqdBU7aUtBzw11xavpiUwDYFzrb9RKOOX1RpI7U9XmmmzsOASaQxUo8itVlK\n0mV5m4ca1TVM0kGkdvPpkm4llY6nAzdL2sL2fZJ+UWZvinlRtJmGUiiNlzoGOM72zYUr+t8i3SAw\nwPa9zeomJunTpF4D/2P7vEq7ZTOO39Ux8gWxHUhV6NNJt8/+ktQEcVkDY1mXNEnjF5g53OCStg9W\nGnlqr7xsWv5jFOZQJNNQGqXZBBYnXXh6JHeNOpp019FLLYhnXVLf2i+W2WugxjGLV+33IXVRG0+a\noHBlUun9RVJb6RrAvxt4sWkf0kA2K9neNS9bl/SdHGn7YUmLu/FTncwTos00lOka0s/UOZJOInVB\nOrMViRQgd8Rfq1mJNB+zkkhHkIavm0zqBH8C8DRp5KdVSe2W9zQwkW5L6vh/A/AxpRH7K+fkbVJp\nFBo8DOS8JEqmoVS5ur8eacCMZ3P7aSvjadrdZ4Vj7kbqerR77pA/HNieNAD3MaSh/15u1B+ZfNHv\nLuBW29/ITS0jSYn9IeBw0kAzjew1MM+JZBpCL1UnbEnLkro9XWz7gLxsXVL/0neAHzah3XY7Ui+B\nvWxfK2kd0mhUrwAX2X6okcefF8XV/BB6oaqNdD/SxHcPkcZEuFHSC7ZPyRffpgMvNqOkbPsaSR8A\nP85doq4iTdHcJy40NUYk0xB6oZBIiyN1PUgqFe4JnJnvcT869yltZmzX5wR+jqQZtn8fibRx4gJU\nCL2kNFLXOqRxALYjDVQyFNiadEvvrpIWr9wl1ky2/0TqQXB/s489r4k20xBKoK5H6voP6WLPJc4z\nrIa5V1TzQyiB7alKAztXRupanjTQyo2RSOcNUTINoSStHKkrtF4k0xBKJKk/adi/Gc0YYCa0j0im\nIYRQgriaH0IIJYhkGkIIJYhkGkIIJYhkGkIIJYhkGkIIJYhkGuaIpOmSJkh6WNJVkhboxb5GSLo+\nP/+KpMN72HaRfB/87B7jGEmH1ru8apsL8xxS9R5rqKSHZzfG0NkimYY59Z7ttW2vCXwA7F1cqWS2\nf75sX2v75B42WYQ0PXIIbSWSaSjDbcDKuUT2mKSLSNMtLytpM0l3Srovl2AXBJC0uaRHJd1HGhyE\nvHyUpDPz849L+oOkB/JjQ+Bk0myjEySdmrc7TNI4SQ9KOrawryMlPS7pH8Ana30ISd/O+3lA0u+r\nStsjJY3P+9s6b99X0qmFY3+ntycydK5IpqFXJPUjjd1ZGWx4FeCXttcgDYR8FDDS9jqkuZAOkTQ/\ncC5puuN1SXcMdeUXpNHih5FGZZpIGjjkqVwqPkzSZvmY6wNrA+tK2jgPxrxTXrYlafT/Wq6xvV4+\n3iOkIfQqhuZjbEWawnr+vP4N2+vl/X9b0gp1HCfMhWKgkzCnBkqqjM95G2luo6WA52zflZd/ljTX\n0O159Ln5gDtJoys9U5l2WdIlpDmRqm0CfBMgT0P8hqRFq7bZLD8qQ8wtSEquCwF/sP1uPsa1dXym\nNSWdQGpKWJA022rFlXks0CckPZ0/w2bAWoX21MH52I/Xcawwl4lkGubUe7bXLi7ICfOd4iLgz7Z3\nrtpulvf1koCTbP+66hgHzcG+LgS2zfM2jQJGFNZV33ftfOz9bReTLpKGzsGxQ4eLan5opLuAz0la\nGdJke5JWBR4FhkpaKW+3czfv/yuwT35vX0mDgbdIpc6KMcAehbbYpSV9DPg7sK2kgZIWIjUp1LIQ\nMDkPVrJr1bodJPXJMa8IPJaPvU/eHkmrKk0oGOZBUTINDWN7Si7hXZaHpwM4yvbjkkYDN+QxQG9j\n1gRZcSBpyo09genAPrbvlHR77nr0p9xu+l/Anblk/DbwDdv3SboCeAB4mTT6fS3/D7gbmJL/L8b0\nL+AeYGFgb9vvSzqP1JZ6Xx4MegqwbX1nJ8xtYtSoEEIoQVTzQwihBJFMQwihBJFMQwihBJFMQwih\nBJFMQwihBJFMQwihBJFMQwihBP8fhaALNtntVMcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f75e6691fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predict(tf_vect, logreg, test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "White box vectorizer and classifier are great! We can see what are the most important words for sci-fi. This makes it very easy to tune and debug."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'halloween',\n",
       " u'adventurous',\n",
       " u'animated',\n",
       " u'animals',\n",
       " u'tale',\n",
       " u'taken',\n",
       " u'forest',\n",
       " u'anna',\n",
       " u'adventure',\n",
       " u'prince']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_influential_words(tf_vect, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Things to try with bag of words\n",
    "\n",
    "10 mins for exercises.\n",
    "\n",
    "For more insight into the model print out the most influential words for a particular plot.\n",
    "\n",
    "Try n-grams with TF-IDF.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Averaging word vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's use more complex features rather than just counting words.\n",
    "\n",
    "A great recent achievement of NLP is the [word2vec embedding](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf). See Chris Moody's [video](https://www.youtube.com/watch?v=vkfXBGnDplQ) for a great introduction to word2vec. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First we load a word2vec model. It has been pre-trained by Google on a 100 billon word Google News corpus. You can play with this model using a fun [web-app](http://rare-technologies.com/word2vec-tutorial/#app).\n",
    "\n",
    "Link to the web-app: http://rare-technologies.com/word2vec-tutorial/#app\n",
    "\n",
    "Vocabulary size: 3 mln words. \n",
    "\n",
    "__Warning__: 3 mins to load, takes 4 GB of RAM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-01-30 21:15:53,002 : INFO : loading projection weights from ~/GoogleNews-vectors-negative300.bin.gz\n",
      "2017-01-30 21:18:35,760 : INFO : loaded (3000000, 300) matrix from ~/GoogleNews-vectors-negative300.bin.gz\n",
      "2017-01-30 21:18:35,775 : INFO : precomputing L2-norms of word weight vectors\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 52s, sys: 2.63 s, total: 2min 55s\n",
      "Wall time: 2min 58s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "wv = Word2Vec.load_word2vec_format(\n",
    "    \"~/GoogleNews-vectors-negative300.bin.gz\",\n",
    "    binary=True)\n",
    "wv.init_sims(replace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'Word2Vec' object has no attribute 'layer_size'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-6053b9616fb5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mwv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayer_size\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m: 'Word2Vec' object has no attribute 'layer_size'"
     ]
    }
   ],
   "source": [
    "wv.layer1_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Example vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-01-30 21:19:10,258 : WARNING : direct access to vocab will not be supported in future gensim releases, please use model.wv.vocab\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[u'slang',\n",
       " u'Murtuza',\n",
       " u'padlock',\n",
       " u'Daniel_Gotlin',\n",
       " u'Moreland_Hills',\n",
       " u'HR_Generalist',\n",
       " u'violate_Simone_Celestin',\n",
       " u'SS_FINISHED',\n",
       " u'symbol_USBI',\n",
       " u'@_Sulekha_Movies',\n",
       " u'unhonored',\n",
       " u'Rufai_Alkali',\n",
       " u'SinoCast_via',\n",
       " u'Shengshui_Temple_Kindergarten',\n",
       " u'AmericanTowns',\n",
       " u'Omtool_undertakes',\n",
       " u'Pembleton',\n",
       " u'ArenaBowl_XXII',\n",
       " u'Waltons',\n",
       " u'sleeper_timbers']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from itertools import islice\n",
    "list(islice(wv.vocab, 13000, 13020))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we have a vector for each word. How do we get a vector for a sequence of words (aka a document)?\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The most naive way is just to take an average. [Mike Tamir](https://www.youtube.com/watch?v=7gTjYwiaJiU) has suggested that the resulting vector points to a single word summarising the whole document. For example all words in a book\n",
    " ‘A tale of two cities’ should add up to 'class-struggle’"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/naivedoc2vec.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "def word_averaging(wv, words):\n",
    "    all_words, mean = set(), []\n",
    "    \n",
    "    for word in words:\n",
    "        if isinstance(word, np.ndarray):\n",
    "            mean.append(word)\n",
    "        elif word in wv.vocab:\n",
    "            mean.append(wv.syn0norm[wv.vocab[word].index])\n",
    "            all_words.add(wv.vocab[word].index)\n",
    "\n",
    "    if not mean:\n",
    "        logging.warning(\"cannot compute similarity with no input %s\", words)\n",
    "        # FIXME: remove these examples in pre-processing\n",
    "        return np.zeros(wv.layer1_size,)\n",
    "\n",
    "    mean = gensim.matutils.unitvec(np.array(mean).mean(axis=0)).astype(np.float32)\n",
    "    return mean\n",
    "\n",
    "def  word_averaging_list(wv, text_list):\n",
    "    return np.vstack([word_averaging(wv, review) for review in text_list ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For word2vec we apply a different tokenization. We want to preserve case as the vocabulary distingushes lower and upper case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def w2v_tokenize_text(text):\n",
    "    tokens = []\n",
    "    for sent in nltk.sent_tokenize(text, language='english'):\n",
    "        for word in nltk.word_tokenize(sent, language='english'):\n",
    "            if len(word) < 2:\n",
    "                continue\n",
    "            tokens.append(word)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "test_tokenized = test_data.apply(lambda r: w2v_tokenize_text(r['plot']), axis=1).values\n",
    "train_tokenized = train_data.apply(lambda r: w2v_tokenize_text(r['plot']), axis=1).values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'wv' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-33-ebb3a4132632>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mu'time'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34mu''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34mu'X_train_word_average = word_averaging_list(wv,train_tokenized)\\nX_test_word_average = word_averaging_list(wv,test_tokenized)'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m   2113\u001b[0m             \u001b[0mmagic_arg_s\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvar_expand\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstack_depth\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2114\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2115\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2116\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2117\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-59>\u001b[0m in \u001b[0;36mtime\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python2.7/site-packages/IPython/core/magic.pyc\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m    186\u001b[0m     \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 188\u001b[0;31m         \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    189\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    190\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python2.7/site-packages/IPython/core/magics/execution.pyc\u001b[0m in \u001b[0;36mtime\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n\u001b[1;32m   1178\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1179\u001b[0m             \u001b[0mst\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1180\u001b[0;31m             \u001b[0;32mexec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mglob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlocal_ns\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1181\u001b[0m             \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1182\u001b[0m             \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<timed exec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'wv' is not defined"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "X_train_word_average = word_averaging_list(wv,train_tokenized)\n",
    "X_test_word_average = word_averaging_list(wv,test_tokenized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's see how KNN and logistic regression classifiers perform on these word-averaging document features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "knn_naive_dv = KNeighborsClassifier(n_neighbors=3, n_jobs=1, algorithm='brute', metric='cosine' )\n",
    "knn_naive_dv.fit(X_train_word_average, train_data.tag)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "predicted = knn_naive_dv.predict(X_test_word_average)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "evaluate_prediction(predicted, test_data.tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "KNN is even worse than the naive 'everything is comedy' baseline! Let's see if logistic regression is better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "logreg = linear_model.LogisticRegression(n_jobs=1, C=1e5)\n",
    "\n",
    "logreg = logreg.fit(X_train_word_average, train_data['tag'])\n",
    "predicted = logreg.predict(X_test_word_average)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Great! It gives __54%__ accuracy. Best that we have seen so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "evaluate_prediction(predicted, test_data.tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now just for fun let's see if text summarisation works on our data. Let's pick a plot and see which words it averages to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "test_data.iloc()[56]['plot']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Hmm... The summarisation doesn't work here. Any ideas why? Hint: look at the area where the average ends up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "wv.most_similar(positive=[X_test_word_average[56]], restrict_vocab=100000, topn=30)[0:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Word2vec things to try"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "10 mins exercise\n",
    "\n",
    "Remove stop-words. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def w2v_tokenize_text(text):\n",
    "    tokens = []\n",
    "    for sent in nltk.sent_tokenize(text, language='english'):\n",
    "        for word in nltk.word_tokenize(sent, language='english'):\n",
    "            if len(word) < 2:\n",
    "                continue\n",
    "            if word in stopwords.words('english'):\n",
    "                continue\n",
    "            tokens.append(word)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What accuracy do you get?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "### More word2vec things to try\n",
    "\n",
    "Experiment with other pre-trained models - see nice [list](https://github.com/3Top/word2vec-api/) from 3Top.\n",
    "\n",
    "\n",
    "Use Gensim's GloVe converter.\n",
    "\n",
    "\n",
    "Do IDF weighting in the averaging function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Doc2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A [paper](https://cs.stanford.edu/~quocle/paragraph_vector.pdf) by Google suggests a model for document classification called Paragraph Vectors Doc2Vec or Doc2vec in short. It is very similar to word2vec. \n",
    "\n",
    "It introduces 'a tag' - a word that is in every context in the document.\n",
    "\n",
    "For our first try we tag every plot with its genre. This makes it 'semi-supervised' learning - the genre labels is just one objective among many."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from gensim.models import Doc2Vec\n",
    "from gensim.models.doc2vec import TaggedDocument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_tagged = train_data.apply(\n",
    "    lambda r: TaggedDocument(words=tokenize_text(r['plot']), tags=[r.tag]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_tagged = test_data.apply(\n",
    "    lambda r: TaggedDocument(words=tokenize_text(r['plot']), tags=[r.tag]), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is what a training entry looks like - an example plot tagged by 'sci-fi'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_tagged.values[50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "trainsent = train_tagged.values\n",
    "testsent = test_tagged.values\n",
    "\n",
    "# simple gensim doc2vec api\n",
    "doc2vec_model = Doc2Vec(trainsent, workers=1, size=5, iter=20, dm=1)\n",
    "\n",
    "train_targets, train_regressors = zip(\n",
    "    *[(doc.tags[0], doc2vec_model.infer_vector(doc.words, steps=20)) for doc in trainsent])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Interesting thing about doc2vec is that we need to run gradient descent during prediction to infer the vector for an unseen document. An unseen document is initially assigned a random vector and then this vector fit by gradient descent. Because of this randomness we get different vectors on re-runs of the next cell.\n",
    "\n",
    "Consequently, the accuracy of logistic regression changes when the test set vectors change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "test_targets, test_regressors = zip(\n",
    "    *[(doc.tags[0], doc2vec_model.infer_vector(doc.words, steps=20)) for doc in testsent])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "logreg = linear_model.LogisticRegression(n_jobs=1, C=1e5)\n",
    "logreg = logreg.fit(train_regressors, train_targets)\n",
    "evaluate_prediction(logreg.predict(test_regressors), test_targets, title=str(doc2vec_model))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "KNN gives a lower accuracy than logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time \n",
    "knn_test_predictions = [\n",
    "    doc2vec_model.docvecs.most_similar([pred_vec], topn=1)[0][0]\n",
    "    for pred_vec in test_regressors\n",
    "]\n",
    "evaluate_prediction(knn_test_predictions, test_targets, str(doc2vec_model))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Doc2vec gives us a vector for each genre so we can see which genres are close together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "doc2vec_model.docvecs.most_similar('action')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Words surrounding the 'sci-fi' tag describe it pretty accurately!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "doc2vec_model.most_similar([doc2vec_model.docvecs['sci-fi']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Doc2vec exercise\n",
    "\n",
    "10 mins\n",
    "\n",
    "Find the random seed that gives the best prediction. :)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "seed = 20\n",
    "\n",
    "doc2vec_model.seed = seed\n",
    "doc2vec_model.random = random.RandomState(seed)\n",
    "\n",
    "\n",
    "test_targets, test_regressors = zip(\n",
    "    *[(doc.tags[0], doc2vec_model.infer_vector(doc.words, steps=20)) for doc in testsent])\n",
    "\n",
    "\n",
    "logreg = linear_model.LogisticRegression(n_jobs=1, C=1e5, random_state=42)\n",
    "logreg = logreg.fit(train_regressors, train_targets)\n",
    "evaluate_prediction(logreg.predict(test_regressors), test_targets, title=str(doc2vec_model))\n",
    "print doc2vec_model.seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Doc2vec things to try\n",
    "Try tagging every sentence with a unique tag 'SENT_123' and then apply KNN. \n",
    "\n",
    "Try multiple tags per plot as in this repo published __today__ : https://github.com/sindbach/doc2vec_pymongo\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Deep IR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'Deep IR' is a technique developed by  [“Document Classification by Inversion of Distributed Language Representations”, Matt Taddy](http://arxiv.org/pdf/1504.07295v3.pdf). Matt has contributed a gensim [tutorial](https://github.com/piskvorky/gensim/blob/develop/docs/notebooks/deepir.ipynb) - great source of more in depth information.\n",
    "\n",
    "In short the algorithm is:\n",
    "\n",
    "1. Train a word2vec model only on comedy plots.\n",
    "\n",
    "2. Trian another model only on sci-fi, another on romance etc. Get 6 models - one for each genre.\n",
    "\n",
    "3. Take a plot and see which model fits it best using Bayes' Theorem\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The tokenization is different from other methods. The reason for this is that we are following an original approach in the paper. The purpose of this tutorial is to see how the models behave out of the box.\n",
    "\n",
    "We just clean non-alphanumeric characters and split by sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "contractions = re.compile(r\"'|-|\\\"\")\n",
    "# all non alphanumeric\n",
    "symbols = re.compile(r'(\\W+)', re.U)\n",
    "# single character removal\n",
    "singles = re.compile(r'(\\s\\S\\s)', re.I|re.U)\n",
    "# separators (any whitespace)\n",
    "seps = re.compile(r'\\s+')\n",
    "\n",
    "# cleaner (order matters)\n",
    "def clean(text): \n",
    "    text = text.lower()\n",
    "    text = contractions.sub('', text)\n",
    "    text = symbols.sub(r' \\1 ', text)\n",
    "    text = singles.sub(' ', text)\n",
    "    text = seps.sub(' ', text)\n",
    "    return text\n",
    "\n",
    "# sentence splitter\n",
    "alteos = re.compile(r'([!\\?])')\n",
    "def sentences(l):\n",
    "    l = alteos.sub(r' \\1 .', l).rstrip(\"(\\.)*\\n\")\n",
    "    return l.split(\".\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def plots(label):\n",
    "    my_df = None\n",
    "    if label=='training':\n",
    "        my_df = train_data\n",
    "    else:\n",
    "        my_df = test_data\n",
    "    for i, row in my_df.iterrows():\n",
    "        yield {'y':row['tag'],\\\n",
    "        'x':[clean(s).split() for s in sentences(row['plot'])]}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "# The corpus is small so can be read into memory\n",
    "revtrain = list(plots(\"training\"))\n",
    "revtest = list(plots(\"test\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# shuffle training set for unbiased word2vec training\n",
    "np.random.shuffle(revtrain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def tag_sentences(reviews, stars=my_tags):  \n",
    "    for r in reviews:\n",
    "        if r['y'] in stars:\n",
    "            for s in r['x']:\n",
    "                yield s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example `sci-fi` sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "next(tag_sentences(revtrain, my_tags[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We train our own 6 word2vec models from scratch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time \n",
    "## training\n",
    "from gensim.models import Word2Vec\n",
    "import multiprocessing\n",
    "\n",
    "## create a w2v learner \n",
    "basemodel = Word2Vec(\n",
    "    workers=multiprocessing.cpu_count(), # use your cores\n",
    "    iter=100, # iter = sweeps of SGD through the data; more is better\n",
    "    hs=1, negative=0, # we only have scoring for the hierarchical softmax setup\n",
    "    \n",
    "    )\n",
    "print(basemodel)\n",
    "basemodel.build_vocab(tag_sentences(revtrain)) \n",
    "from copy import deepcopy\n",
    "genremodels = [deepcopy(basemodel) for i in range(len(my_tags))]\n",
    "for i in range(len(my_tags)):\n",
    "    slist = list(tag_sentences(revtrain, my_tags[i]))\n",
    "    print(my_tags[i], \"genre (\", len(slist), \")\")\n",
    "    genremodels[i].train(  slist, total_examples=len(slist) )\n",
    "# get the probs (note we give docprob a list of lists of words, plus the models)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we will compute most likely class for a plot using Bayes' Theorem formula."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='images/deep_ir_bayes.png' width=600>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For any new sentence we can obtain its _likelihood_ (lhd; actually, the composite likelihood approximation; see the paper) using the [score](https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Word2Vec.score) function in the `word2vec` class.  We get the likelihood for each sentence in the first test review, then convert to a probability over star ratings. Every sentence in the review is evaluated separately and the final star rating of the review is an average vote of all the sentences. This is all in the following handy wrapper. (from the original [tutorial](https://github.com/piskvorky/gensim/blob/develop/docs/notebooks/deepir.ipynb) by Matt Taddy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "docprob takes two lists\n",
    "* docs: a list of documents, each of which is a list of sentences\n",
    "* models: the candidate word2vec models (each potential class)\n",
    "\n",
    "it returns the array of class probabilities.  Everything is done in-memory.\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def docprob(docs, mods):\n",
    "    # score() takes a list [s] of sentences here; could also be a sentence generator\n",
    "    sentlist = [s for d in docs for s in d]\n",
    "    # the log likelihood of each sentence in this review under each w2v representation\n",
    "    llhd = np.array( [ m.score(sentlist, len(sentlist)) for m in mods ] )\n",
    "    # now exponentiate to get likelihoods, \n",
    "    lhd = np.exp(llhd - llhd.max(axis=0)) # subtract row max to avoid numeric overload\n",
    "    # normalize across models (stars) to get sentence-star probabilities\n",
    "    prob = pd.DataFrame( (lhd/lhd.sum(axis=0)).transpose() )\n",
    "    # and finally average the sentence probabilities to get the review probability\n",
    "    prob[\"doc\"] = [i for i,d in enumerate(docs) for s in d]\n",
    "    prob = prob.groupby(\"doc\").mean()\n",
    "    return prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "## predict\n",
    "probs = docprob( [r['x'] for r in revtest], genremodels )  \n",
    "predictions = probs.idxmax(axis=1).apply(lambda x: my_tags[x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "tag_index = 0\n",
    "col_name = \"out-of-sample prob positive for \" + my_tags[tag_index]\n",
    "probpos = pd.DataFrame({col_name:probs[[tag_index]].sum(axis=1), \n",
    "                        \"true genres\": [r['y'] for r in revtest]})\n",
    "probpos.boxplot(col_name,by=\"true genres\", figsize=(12,5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "target = [r['y'] for r in revtest]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "evaluate_prediction(predictions, target, \"Deep IR with word2vec\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Performance is worse than for a naive predictor that says that everything is `comedy`.\n",
    "\n",
    "### Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "\n",
    "It is because we train each word2vec model from scratch on a very small sample of about 30k words.\n",
    "\n",
    "This model needs more data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Word Mover's Distance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='images/wmd_gelato.png'>\n",
    "\n",
    "Image from \n",
    "http://tech.opentable.com/2015/08/11/navigating-themes-in-restaurant-reviews-with-word-movers-distance/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Word Mover's Distance is a new algorithm developed in by [Matt Kusner](http://jmlr.org/proceedings/papers/v37/kusnerb15.pdf). There is Matt's code on [github](https://github.com/mkusner/wmd) and also Gensim can compute WMD similarity in this [PR](https://github.com/piskvorky/gensim/pull/659)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For KNN the best code is from [Vlad Niculae's blog](http://vene.ro/blog/word-movers-distance-in-python.html). He is a contributor to sklearn and did great integration of WMD with sklearn KNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Warning__: Write 7 GB file on disk to use memory mapping."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## __This part requires Python 3__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "data_folder = '/data/'\n",
    "\n",
    "fp = np.memmap(data_folder + \"embed.dat\", dtype=np.double, mode='w+', shape=wv.syn0norm.shape)\n",
    "fp[:] = wv.syn0norm[:]\n",
    "\n",
    "with smart_open(data_folder + \"embed.vocab\", \"w\") as f: \n",
    "    for _, w in sorted((voc.index, word) for word, voc in wv.vocab.items()):\n",
    "        print(w.encode('utf8'), file=f)\n",
    "del fp, wv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W = np.memmap(data_folder + \"embed.dat\", dtype=np.double, mode=\"r\", shape=(3000000, 300))\n",
    "with smart_open(data_folder + \"embed.vocab\", mode=\"rb\") as f:\n",
    "    vocab_list = [line.strip() for line in f]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vocab_dict = {w: k for k, w in enumerate(vocab_list)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sklearn KNN integration with WMD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"%%file word_movers_knn.py\"\"\"\n",
    "\n",
    "# Authors: Vlad Niculae, Matt Kusner\n",
    "# License: Simplified BSD\n",
    "\n",
    "import numpy as np\n",
    "from sklearn.metrics import euclidean_distances\n",
    "from sklearn.externals.joblib import Parallel, delayed\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.utils import check_array\n",
    "from sklearn.cross_validation import check_cv\n",
    "from sklearn.metrics.scorer import check_scoring\n",
    "from sklearn.preprocessing import normalize\n",
    "\n",
    "from pyemd import emd\n",
    "\n",
    "\n",
    "class WordMoversKNN(KNeighborsClassifier):\n",
    "    \"\"\"K nearest neighbors classifier using the Word Mover's Distance.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    \n",
    "    W_embed : array, shape: (vocab_size, embed_size)\n",
    "        Precomputed word embeddings between vocabulary items.\n",
    "        Row indices should correspond to the columns in the bag-of-words input.\n",
    "\n",
    "    n_neighbors : int, optional (default = 5)\n",
    "        Number of neighbors to use by default for :meth:`k_neighbors` queries.\n",
    "\n",
    "    n_jobs : int, optional (default = 1)\n",
    "        The number of parallel jobs to run for Word Mover's Distance computation.\n",
    "        If ``-1``, then the number of jobs is set to the number of CPU cores.\n",
    "    \n",
    "    verbose : int, optional\n",
    "        Controls the verbosity; the higher, the more messages. Defaults to 0.\n",
    "        \n",
    "    References\n",
    "    ----------\n",
    "    \n",
    "    Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, Kilian Q. Weinberger\n",
    "    From Word Embeddings To Document Distances\n",
    "    The International Conference on Machine Learning (ICML), 2015\n",
    "    http://mkusner.github.io/publications/WMD.pdf\n",
    "    \n",
    "    \"\"\"\n",
    "    _pairwise = False\n",
    "\n",
    "    def __init__(self, W_embed, n_neighbors=1, n_jobs=1, verbose=False):\n",
    "        self.W_embed = W_embed\n",
    "        self.verbose = verbose\n",
    "        super(WordMoversKNN, self).__init__(n_neighbors=n_neighbors, n_jobs=n_jobs,\n",
    "                                            metric='precomputed', algorithm='brute')\n",
    "\n",
    "    def _wmd(self, i, row, X_train):\n",
    "        \"\"\"Compute the WMD between training sample i and given test row.\n",
    "        \n",
    "        Assumes that `row` and train samples are sparse BOW vectors summing to 1.\n",
    "        \"\"\"\n",
    "        union_idx = np.union1d(X_train[i].indices, row.indices)\n",
    "        W_minimal = self.W_embed[union_idx]\n",
    "        W_dist = euclidean_distances(W_minimal)\n",
    "        bow_i = X_train[i, union_idx].A.ravel()\n",
    "        bow_j = row[:, union_idx].A.ravel()\n",
    "        return emd(bow_i, bow_j, W_dist)\n",
    "    \n",
    "    def _wmd_row(self, row, X_train):\n",
    "        \"\"\"Wrapper to compute the WMD of a row with all training samples.\n",
    "        \n",
    "        Assumes that `row` and train samples are sparse BOW vectors summing to 1.\n",
    "        Useful for parallelization.\n",
    "        \"\"\"\n",
    "        n_samples_train = X_train.shape[0]\n",
    "        return [self._wmd(i, row, X_train) for i in range(n_samples_train)]\n",
    "\n",
    "    def _pairwise_wmd(self, X_test, X_train=None):\n",
    "        \"\"\"Computes the word mover's distance between all train and test points.\n",
    "        \n",
    "        Parallelized over rows of X_test.\n",
    "        \n",
    "        Assumes that train and test samples are sparse BOW vectors summing to 1.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X_test: scipy.sparse matrix, shape: (n_test_samples, vocab_size)\n",
    "            Test samples.\n",
    "        \n",
    "        X_train: scipy.sparse matrix, shape: (n_train_samples, vocab_size)\n",
    "            Training samples. If `None`, uses the samples the estimator was fit with.\n",
    "        \n",
    "        Returns\n",
    "        -------\n",
    "        dist : array, shape: (n_test_samples, n_train_samples)\n",
    "            Distances between all test samples and all train samples.\n",
    "        \n",
    "        \"\"\"\n",
    "        n_samples_test = X_test.shape[0]\n",
    "        \n",
    "        if X_train is None:\n",
    "            X_train = self._fit_X\n",
    "\n",
    "        dist = Parallel(n_jobs=self.n_jobs, verbose=self.verbose)(\n",
    "            delayed(self._wmd_row)(test_sample, X_train)\n",
    "            for test_sample in X_test)\n",
    "\n",
    "        return np.array(dist)\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        \"\"\"Fit the model using X as training data and y as target values\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : scipy sparse matrix, shape: (n_samples, n_features)\n",
    "            Training data. \n",
    "\n",
    "        y : {array-like, sparse matrix}\n",
    "            Target values of shape = [n_samples] or [n_samples, n_outputs]\n",
    "\n",
    "        \"\"\"\n",
    "        X = check_array(X, accept_sparse='csr', copy=True)\n",
    "        X = normalize(X, norm='l1', copy=False)\n",
    "        return super(WordMoversKNN, self).fit(X, y)\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\"Predict the class labels for the provided data\n",
    "        Parameters\n",
    "        ----------\n",
    "        X : scipy.sparse matrix, shape (n_test_samples, vocab_size)\n",
    "            Test samples.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        y : array of shape [n_samples]\n",
    "            Class labels for each data sample.\n",
    "        \"\"\"\n",
    "        X = check_array(X, accept_sparse='csr', copy=True)\n",
    "        X = normalize(X, norm='l1', copy=False)\n",
    "        dist = self._pairwise_wmd(X)\n",
    "        return super(WordMoversKNN, self).predict(dist)\n",
    "    \n",
    "    \n",
    "class WordMoversKNNCV(WordMoversKNN):\n",
    "    \"\"\"Cross-validated KNN classifier using the Word Mover's Distance.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    W_embed : array, shape: (vocab_size, embed_size)\n",
    "        Precomputed word embeddings between vocabulary items.\n",
    "        Row indices should correspond to the columns in the bag-of-words input.\n",
    "\n",
    "    n_neighbors_try : sequence, optional\n",
    "        List of ``n_neighbors`` values to try.\n",
    "        If None, tries 1-5 neighbors.\n",
    "\n",
    "    scoring : string, callable or None, optional, default: None\n",
    "        A string (see model evaluation documentation) or\n",
    "        a scorer callable object / function with signature\n",
    "        ``scorer(estimator, X, y)``.\n",
    "\n",
    "    cv : int, cross-validation generator or an iterable, optional\n",
    "        Determines the cross-validation splitting strategy.\n",
    "        Possible inputs for cv are:\n",
    "          - None, to use the default 3-fold cross-validation,\n",
    "          - integer, to specify the number of folds.\n",
    "          - An object to be used as a cross-validation generator.\n",
    "          - An iterable yielding train/test splits.\n",
    "        For integer/None inputs, StratifiedKFold is used.\n",
    "\n",
    "    n_jobs : int, optional (default = 1)\n",
    "        The number of parallel jobs to run for Word Mover's Distance computation.\n",
    "        If ``-1``, then the number of jobs is set to the number of CPU cores.\n",
    "\n",
    "    verbose : int, optional\n",
    "        Controls the verbosity; the higher, the more messages. Defaults to 0.\n",
    "\n",
    "    Attributes\n",
    "    ----------\n",
    "    cv_scores_ : array, shape (n_folds, len(n_neighbors_try))\n",
    "        Test set scores for each fold.\n",
    "\n",
    "    n_neighbors_ : int,\n",
    "        The best `n_neighbors` value found.\n",
    "\n",
    "    References\n",
    "    ----------\n",
    "\n",
    "    Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, Kilian Q. Weinberger\n",
    "    From Word Embeddings To Document Distances\n",
    "    The International Conference on Machine Learning (ICML), 2015\n",
    "    http://mkusner.github.io/publications/WMD.pdf\n",
    "    \n",
    "    \"\"\"\n",
    "    def __init__(self, W_embed, n_neighbors_try=None, scoring=None, cv=3,\n",
    "                 n_jobs=1, verbose=False):\n",
    "        self.cv = cv\n",
    "        self.n_neighbors_try = n_neighbors_try\n",
    "        self.scoring = scoring\n",
    "        super(WordMoversKNNCV, self).__init__(W_embed,\n",
    "                                              n_neighbors=None,\n",
    "                                              n_jobs=n_jobs,\n",
    "                                              verbose=verbose)\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        \"\"\"Fit KNN model by choosing the best `n_neighbors`.\n",
    "        \n",
    "        Parameters\n",
    "        -----------\n",
    "        X : scipy.sparse matrix, (n_samples, vocab_size)\n",
    "            Data\n",
    "        y : ndarray, shape (n_samples,) or (n_samples, n_targets)\n",
    "            Target\n",
    "        \"\"\"\n",
    "        if self.n_neighbors_try is None:\n",
    "            n_neighbors_try = range(1, 6)\n",
    "        else:\n",
    "            n_neighbors_try = self.n_neighbors_try\n",
    "\n",
    "        X = check_array(X, accept_sparse='csr', copy=True)\n",
    "        X = normalize(X, norm='l1', copy=False)\n",
    "\n",
    "        cv = check_cv(self.cv, X, y)\n",
    "        knn = KNeighborsClassifier(metric='precomputed', algorithm='brute')\n",
    "        scorer = check_scoring(knn, scoring=self.scoring)\n",
    "\n",
    "        scores = []\n",
    "        for train_ix, test_ix in cv:\n",
    "            dist = self._pairwise_wmd(X[test_ix], X[train_ix])\n",
    "            knn.fit(X[train_ix], y[train_ix])\n",
    "            scores.append([\n",
    "                scorer(knn.set_params(n_neighbors=k), dist, y[test_ix])\n",
    "                for k in n_neighbors_try\n",
    "            ])\n",
    "        scores = np.array(scores)\n",
    "        self.cv_scores_ = scores\n",
    "\n",
    "        best_k_ix = np.argmax(np.mean(scores, axis=0))\n",
    "        best_k = n_neighbors_try[best_k_ix]\n",
    "        self.n_neighbors = self.n_neighbors_ = best_k\n",
    "\n",
    "        return super(WordMoversKNNCV, self).fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how well it performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_tokenized = test_data.apply(lambda r: w2v_tokenize_text(r['plot']), axis=1).values\n",
    "train_tokenized = train_data.apply(lambda r: w2v_tokenize_text(r['plot']), axis=1).values\n",
    "\n",
    "flat_train_tokenized = [item for sublist in train_tokenized for item in sublist]\n",
    "flat_test_tokenized = [item for sublist in test_tokenized for item in sublist]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To speed up performance we focus only on the words that are both in Google News model and in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the word2vec model was loaded with strings as byte-arrays so need to convert\n",
    "def convert_to_vocab_bytes(s):\n",
    "     return bytes(\"b'\" + s + \"'\", encoding='utf-8')    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vect = CountVectorizer(stop_words=\"english\").fit(flat_train_tokenized)\n",
    "common = [word for word in vect.get_feature_names() if convert_to_vocab_bytes(word) in vocab_dict]\n",
    "W_common = W[[vocab_dict[ convert_to_vocab_bytes(w)] for w in common]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mass of the pile of mud in the Word Mover's Distance is determined by how many times a word appears in a document. So we need a CountVectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_tokenized[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vect = CountVectorizer(vocabulary=common, dtype=np.double)\n",
    "X_train = vect.fit_transform(train_tokenized)\n",
    "X_test = vect.transform(test_tokenized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "knn = WordMoversKNN(n_neighbors=1,W_embed=W_common, verbose=5, n_jobs=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "knn.fit(X_train, train_data['tag'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Warning__: 10 minutes runtime on 7 cores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time \n",
    "predicted = knn.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only 2% above the naive baseline unfortunately. WMD achieves good results on sentiment analysis in the published paper. Maybe it works better for sentiment than for topic classification that we use it here. Or maybe preprocessing can be tuned here. It is hard to debug a black box method!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "evaluate_prediction(predicted, test_data['tag'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above we shown how to run 'hello-world' in 7 different document classification techniques. It is just a beginning of exploration of their features... There are a lot of parameters that can be tuned to get the best possible results out of them. The 'hello-world' run is in no way an indication of their best peformance. The goal of this tutorial is to show the API so you can start tuning them yourself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of the box \"no tuning\" accuracy of bag of words is not far behind more advanced techniques. \n",
    "Tune them and the pre-processing for them well first and only then reach for more advanced methods if more accuracy is absolutely needed."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}