Global Attention and word embedding demonstration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Global Attention and word embedding demonstration\n",
    "\n",
    "This will briefly cover embedding and attention layer. Lemmatization and Stemming are not present here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow.keras as keras\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.layers import Input, Embedding, Dense, LSTM\n",
    "from keras.models import Model\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset import\n",
    "\n",
    "We import the amazon_cells_labelled csv dataset. Available <a href=\"https://archive.ics.uci.edu/ml/datasets/Sentiment+Labelled+Sentences\">here</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\leximus\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\ipykernel_launcher.py:1: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1000"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"amazon_cells_labelled.txt\", header=None, delimiter=\"\\\\t\")\n",
    "len(df) #1000 samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "So there is no way for me to plug it in here in the US unless I go by a converter.\n",
      "converted to:\n",
      "[33, 117, 5, 53, 214, 11, 47, 8, 155, 4, 19, 337, 19, 1, 546, 416, 2, 241, 190, 6, 812]\n"
     ]
    }
   ],
   "source": [
    "#We use a one hot encoder Each sentence is converted to a token encoding.\n",
    "#In other term, we replace all \"King\" by the same numerical value.\n",
    "t = Tokenizer()\n",
    "t.fit_on_texts(df[0])\n",
    "text_matrix = t.texts_to_sequences(df[0])\n",
    "print(df[0][0])\n",
    "print(\"converted to:\")\n",
    "print(text_matrix[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, 33 is equals to \"So\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'so'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.index_word[33]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need all our input to have the same size. In this case, we look for the longest sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The longest sentence is composed of 30 tokens.\n"
     ]
    }
   ],
   "source": [
    "s_length = 0\n",
    "for i in range(len(text_matrix)):\n",
    "    s_length = max(len(text_matrix[i]), s_length)\n",
    "print(\"The longest sentence is composed of \"+str(s_length)+\" tokens.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Padding\n",
    "\n",
    "If the longest sentence is made of 30 tokens, there are some which are smaller. We will thus add a \"0\" (not used in the dictionnary) to signify \"empty token\" there.\n",
    "\n",
    "This will help us to complete the vector of the other sizes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33, 117, 5, 53, 214, 11, 47, 8, 155, 4, 19, 337, 19, 1, 546, 416, 2, 241, 190, 6, 812]\n",
      "[ 33 117   5  53 214  11  47   8 155   4  19 337  19   1 546 416   2 241\n",
      " 190   6 812   0   0   0   0   0   0   0   0   0]\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "print(text_matrix[0])\n",
    "\n",
    "text_pad = pad_sequences(text_matrix, maxlen=s_length, padding='post')\n",
    "\n",
    "print(text_pad[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, a bunch of \"0\" have been added to the vector to get an array of 32 elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We know look for the size of our dictionnary, since we need it to configure the embedding layer.\n",
    "nb_tokens = len(t.word_index) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "latent_dim = 32 #What is the size of the abstract representation we want to represent each word?\n",
    "#Every word can be represented by has much number you want.\n",
    "#Some of this neuron might generate the gender (King = 0 vs Queen = 1),\n",
    "# another the social status (King = 1 vs fool = 0), etc...\n",
    "#This is a hidden layer where the network will try to capture the differences.\n",
    "\n",
    "inp = Input(shape=(s_length,)) #We create a vector of 32 dimensions as input\n",
    "\n",
    "\n",
    "#input_dim : Size of the vocabulary dict\n",
    "#input_length : Size of the sentences.\n",
    "#output_dim : the latent dim variable, the size of the representation.\n",
    "x = Embedding(input_dim = nb_tokens, \\\n",
    "              input_length = s_length, \\\n",
    "              output_dim = latent_dim,\\\n",
    "              embeddings_regularizer = keras.regularizers.l2(.001))(inp) #Regularization to help learning\n",
    "\n",
    "x = LSTM(64, dropout = 0.2, recurrent_dropout = 0.2)(x) #A simplement LSTM network made of 64 cells\n",
    "\n",
    "out = Dense(1, activation='sigmoid')(x) #Our output neuron for the sentiment analysis.\n",
    "\n",
    "model = Model(inp, out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_14 (InputLayer)        (None, 30)                0         \n",
      "_________________________________________________________________\n",
      "embedding_13 (Embedding)     (None, 30, 32)            60128     \n",
      "_________________________________________________________________\n",
      "lstm_13 (LSTM)               (None, 64)                24832     \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 85,025\n",
      "Trainable params: 85,025\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 800 samples, validate on 200 samples\n",
      "Epoch 1/15\n",
      "800/800 [==============================] - 3s 4ms/step - loss: 0.7305 - acc: 0.4713 - val_loss: 0.7170 - val_acc: 0.4550\n",
      "Epoch 2/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.7076 - acc: 0.5112 - val_loss: 0.7032 - val_acc: 0.4550\n",
      "Epoch 3/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.6982 - acc: 0.5112 - val_loss: 0.6998 - val_acc: 0.4550\n",
      "Epoch 4/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.6941 - acc: 0.5112 - val_loss: 0.6961 - val_acc: 0.4550\n",
      "Epoch 5/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.6933 - acc: 0.5112 - val_loss: 0.6944 - val_acc: 0.4550\n",
      "Epoch 6/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.6929 - acc: 0.5275 - val_loss: 0.6943 - val_acc: 0.4550\n",
      "Epoch 7/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.6927 - acc: 0.5312 - val_loss: 0.6957 - val_acc: 0.4550\n",
      "Epoch 8/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.6863 - acc: 0.5800 - val_loss: 0.6815 - val_acc: 0.5500\n",
      "Epoch 9/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.5452 - acc: 0.7450 - val_loss: 0.5783 - val_acc: 0.7500\n",
      "Epoch 10/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.3412 - acc: 0.8850 - val_loss: 0.5539 - val_acc: 0.7650\n",
      "Epoch 11/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.2080 - acc: 0.9438 - val_loss: 0.5598 - val_acc: 0.8050\n",
      "Epoch 12/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.1305 - acc: 0.9725 - val_loss: 0.6236 - val_acc: 0.7950\n",
      "Epoch 13/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0934 - acc: 0.9812 - val_loss: 0.8347 - val_acc: 0.7700\n",
      "Epoch 14/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0796 - acc: 0.9862 - val_loss: 0.9760 - val_acc: 0.7650\n",
      "Epoch 15/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0861 - acc: 0.9812 - val_loss: 0.8390 - val_acc: 0.7900\n"
     ]
    }
   ],
   "source": [
    "#We build and fit the model\n",
    "\n",
    "model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])\n",
    "\n",
    "train_no_attention_history = model.fit(x = text_pad, y = df[1], \\\n",
    "                          batch_size = 64, epochs = 15, \\\n",
    "                          verbose = 1, shuffle = True, \\\n",
    "                          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2e2b8372198>]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4nOV57/HvLcmSLO+y5FWSN2zjBYxBmMVZoMTESQPOAsRAWyglbtqQ5NC0OaYL4TjnSmh60qQLaUOIAyc1GOokRKQmLAGaBLxIJoA9MsayAUsa77IlW160zN0/ZpyMhWSNrZHeWX6f65rL824z9/iSfvPqed73eczdERGR7JATdAEiIjJwFPoiIllEoS8ikkUU+iIiWUShLyKSRRT6IiJZRKEvIpJFFPoiIllEoS8ikkXygi6gq5KSEp88eXLQZYiIpJVNmzYdcPfS3vZLudCfPHkyNTU1QZchIpJWzOzdRPZT846ISBZR6IuIZBGFvohIFlHoi4hkEYW+iEgWUeiLiGQRhb6ISBZJKPTNbLGZbTOzOjNb3s32CjN70cx+Y2ZvmNlH47bdEztum5l9OJnFi8jA2tzQzKoN71K37wiaajU99XpzlpnlAg8Ai4AGoNrMqty9Nm63vwWecPd/M7PZwFpgcuz5UmAOMAF43sxmuHtnsj+IiPSvV+oO8McPV3OyIwLAmGEFXDltNFdOK+HK80ZTNqoo4AolEYnckbsAqHP3nQBmthpYAsSHvgPDY89HAOHY8yXAanc/CbxtZnWx11uXhNpFZIDUvNPEnzxSw+TRQ/jmTfMIhZt5ue4gv647yJOvRX/dK4qLWHjeaK6YVsKV00ZTMrQg4KqlO4mE/kSgPm65Abisyz73Ac+a2eeBIcCH4o5d3+XYiedUqYgE4vX6w9z+g2rGjyjkh3cuYMywQuZOHMGnL63A3dm+7yiv1B3g5R0H+dkbu3lsYzQuZo4dxpXnRf8SuGxqMcMLBwX8SQQSC33rZl3XxrybgYfd/ZtmdgXwQzObm+CxmNkyYBlARUVFAiWJyEDYuruFP1q5kVFDBrHqM5cxZljhadvNjBljhzFj7DBuXziFzoizpbGZV3Yc5JUdB3hs4y5+8PI75BhcUDaSK6eNZuG0Ei6ZNIrB+bkBfarslkjoNwDlcctl/K755pQ/ARYDuPs6MysEShI8Fnd/EHgQoLKyUr1DIimgbt8R/uChDRTl5/LonZczfsTgXo/JzTHmlY9kXvlI/uyqaZzs6OQ3uw5HvwTqDvC9X+7k317aQX5uDhdPGsmV00pYeN5oLiwbyaBcXUw4EKy3HngzywPeAq4BGoFq4BZ3D8Xt8zTwuLs/bGazgF8QbcaZDTxKtB1/Qmz99DN15FZWVrpG2RQJ1jsHWrnpu+uIODzxp5cztXRoUl736MkOqt9p4pW6A7yy4yC1u1twhyH5uSxdUMGXrp1BUX7KDf6bFsxsk7tX9rZfr/+77t5hZncBzwC5wEp3D5nZCqDG3auALwHfM7O7iTbf3O7Rb5OQmT1BtNO3A/icrtwRSW2Nh49z60MbaO+MsHrZFUkLfIChBXlcPXMMV88cA8Ch1jbW7zzIc7V7+f6v3+bZ2j3c/8kLWXheSdLeU07X65n+QNOZvkhw9rac4KbvrqOptY3HPnM5cyeOGLD33rDzIMt/vJm3D7Ty6cpy/vr3ZzFicPZ0/rZ3RjjU2saY4YW979yNRM/01YgmIgAcOHqSW763ngNHTvLIHQsGNPABLps6mqe/+H4++8FprHm1gUX/+N88G9ozoDUEZXNDM9f9y6/50//YRCTSvyfiCn0R4fCxNv7w+xtpPHyc799+KRdXjAqkjsJBuSz/yPk8+ecLGT20gGU/3MTnHn2V/UdOBlJPfzvR3snXn97Kkgd+TVNrG3/2wWnk5HR30WPyqMdEJMu1nGjntpUb2bHvKA/dVsnlU0cHXRIXlI2g6q6FfPe/d/DPv6jj5boDfOW62Xz8oomY9W8oDpT1Ow9yT6w56+YF5Sz/yMA0Z+lMXySLHWvr4I4fVBMKt/CdWy/mAzN6nVd7wAzKzeGu35vO2i++j6klQ7j78df544eraTx8POjS+uTIiXb+5iebWfrgejojzqN3XsbXP3nhgPVfqCNXJEudaO/kjoerWb/zIP9y88X8/oXjgy6pR50R54fr3uEbz2zDgOUfOZ9bL5vU700hyfbim/v4659sZm/LCe5YOIW/SOIlqol25Cr0RbJQW0eEP/1hDS+9tZ9v3jiPT15cFnRJCalvOsZf/2Qzv9p+gAWTi7n/Uxck9ZLS/tLU2saKp0I8+VqY6WOG8o0bLmR+kvtNFPoi0q2Ozgife/RVngnt5WufuIBbLkuvoU/cnTWbGvjqz2o50RHh7g/N4DPvn0JeCt7R6+489cZu7qsK0XK8nc9dfR5/fvU0CvKSPwRF0m7OEpHM0RlxvvSfr/NMaC/3fmx22gU+RMf7ubGynA/OKOXvfrqFv//5m/zX5jDf+NQ8Zk8Y3vsLDJA9zSf42ye38PzWvcwrG8Hff+Yyzh8XfH060xfJEpGIs/zHb/BETQNfXjyTP7/qvKBLSoqnN+/m734a4vCxNj77wWnc9XvnUTgouMHc3J3V1fV87b+20h6J8KVFM7njfVPI7ef+B53pi8hvuTv3PRXiiZoGvnDN9IwJfICPXDCeK6aN5qs/28q/vljH01t2840bLuSSScUDXsu7B1tZ/qPNrNt5kMunFnP/Jy9kcsmQAa/jTFKvEUxEksrd+frTb/L/173Lsg9M5e4PTQ+6pKQbWZTPN2+axyN3LOBEe4Qb/n0d91WFaD3ZMSDv3xlxHvrVTj787V+ypbGZr33iAh698/KUC3zQmb5IxvvW89t58Jc7+aMrJnHPR87PmJubuvPBGaU8c/cH+Iefv8kj697h2dAerphWQnnxYMpHFVExuojyUUWMGVaQtMs9t+05wpd/9Aav1x/mmvPH8H8/MTehYaiDotAXyWDfeamOf/7Fdm6qLOO+6+ZkdOCfMrQgj/+zZC4fmzeBbz//Fi/XHWDvkRPEd1/m5+VQNnIw5cVFv/1CKC8uoqI4+qUwoqj3G6XaOiI88GId33mpjmGFg/jnm+dz3YXjU/7/WKEvkqF+8PLbfOPn21hy0QS+/skL0+5Gpr66dHIxq+68HICTHZ00HjpO/aHj7Go6RkPTMeoPHWNX0zFeqz9M8/H2044dVpgX/cvg1JdC7MugvLiIslGDeXPPEb685nXe2nuUJRdN4CvXzaF4SH4QH/OsKfRFMkgk4qzfeZDHqut56vUwi+eM45s3zuv3K0dSXUFeLlNLh/Z4I1fLiXbqm47FHsepPxR9vn3fEV7cto+THZH3HDNueCHfv62Sa2aN7e/yk0qhL5IB9jSfYM2meh6vqae+6TjDC/O4831T+PLi81PypqVUM7xwEHMmjGDOhPcOJx2JOAeOnvztXwb1Tccx4LaFk9NysneFvkiaau+M8MKb+3iiup4Xt+0j4nDF1NH85bUz+fCccYFeq55JcnKMMcMLGTO8MJDLQJNNoS+SZt4+0Mrj1fWs2dTAgaMnGTOsgD+7aho3VZYzaXTqXSIoqUWhL5IGjrd18vSW3ayurmfj203k5hhXzxzD0kvLuWpmqZpwJGEKfZEUtqWxmcer63nytUaOnOhg0ugivrx4JjdcXHbOc6lKdkso9M1sMfBPQC7wkLvf32X7t4CrY4tFwBh3Hxnb1glsjm3b5e7XJ6NwkUzVfLydqtcaWV1dTyjcQkFeDh+9YDw3VZZz+dTilL8OXFJbr6FvZrnAA8AioAGoNrMqd689tY+73x23/+eB+XEvcdzdL0peySKZx93Z8HYTT1TX81+bd3OyI8Ls8cNZsWQOS+ZNTOhmIZFEJHKmvwCoc/edAGa2GlgC1Paw/83AV5JTnkhmc3ceeeUdHln3Lm8faGVYQR43Vpax9NIK5k587+WDIn2VSOhPBOrjlhuAy7rb0cwmAVOAF+JWF5pZDdAB3O/uT3Zz3DJgGUBFRfqN7y1yrjY3NnPfU7VcVD6Sb944j49eMJ7B+brUUvpPIqHfXQNiT4PwLwXWuHtn3LoKdw+b2VTgBTPb7O47Tnsx9weBByE6nn4CNYlkhDcamgH4l5vnU15cFHA1kg0Suc6rASiPWy4Dwj3suxR4LH6Fu4dj/+4EXuL09n6RrBYKtzBi8CDKRqXuqIySWRIJ/WpguplNMbN8osFe1XUnM5sJjALWxa0bZWYFseclwEJ67gsQyTq14WZmjx+uK3JkwPQa+u7eAdwFPANsBZ5w95CZrTCz+MsvbwZW++nzL84CaszsdeBFom36Cn0RosMobN1zhLkTg583VbJHQtfpu/taYG2Xdfd2Wb6vm+NeAS7oQ30iGWvH/qO0dUS6HeRLpL/o3m2RgIQaWwCYM0Fn+jJwFPoiAQmFWygclNPjGO8i/UGhLxKQLeFmZo0fnvUTnMjAUuiLBCAScbaGW9S0IwNOoS8SgPpDxzhyskOduDLgFPoiAQiFo524cxX6MsAU+iIB2NLYTF6OMWOcOnFlYCn0RQIQCrdw3pihFORpcDUZWAp9kQHm7oTCzRo6WQKh0BcZYPuOnOTA0TZduSOBUOiLDLBQODqcsq7ckSAo9EUG2KnhF2aNHxZwJZKNFPoiAywUbmFKyRCGFWreWxl4Cn2RAbYl3MxstedLQBT6IgOo+Vg7DYeOqxNXAqPQFxlAod3qxJVgKfRFBlBtWGPoS7AU+iIDaEtjM+OGF1IytCDoUiRLKfRFBlBIwylLwBIKfTNbbGbbzKzOzJZ3s/1bZvZa7PGWmR2O23abmW2PPW5LZvEi6eR4Wyc79h9ljoZfkAD1OjG6meUCDwCLgAag2syq3L321D7ufnfc/p8H5seeFwNfASoBBzbFjj2U1E8hkgbe3NNCxNWeL8FK5Ex/AVDn7jvdvQ1YDSw5w/43A4/Fnn8YeM7dm2JB/xywuC8Fi6SrLerElRSQSOhPBOrjlhti697DzCYBU4AXzvZYkUxXG25mxOBBTBw5OOhSJIslEvrdzdrsPey7FFjj7p1nc6yZLTOzGjOr2b9/fwIliaSfULiFuROHY6aJ0CU4iYR+A1Aet1wGhHvYdym/a9pJ+Fh3f9DdK929srS0NIGSRNJLe2eEN3cf0U1ZErhEQr8amG5mU8wsn2iwV3XdycxmAqOAdXGrnwGuNbNRZjYKuDa2TiSr1O07SltnRO35Erher95x9w4zu4toWOcCK909ZGYrgBp3P/UFcDOw2t097tgmM/sq0S8OgBXu3pTcjyCS+kLqxJUU0WvoA7j7WmBtl3X3dlm+r4djVwIrz7E+kYwQCjczeFAuU0o0EboES3fkigyAUGMLs8YPIzdHnbgSLIW+SD+LRJza3S3qxJWUoNAX6We7mo5x9GQHcyeqPV+Cp9AX6We/68TVmb4ET6Ev0s+2hJvJyzGmj1UnrgRPoS/Sz0LhFqaPHUZBXm7QpYgo9EX6k7sTamxmrq7PlxSh0BfpR3tbTnKwtU03ZUnKUOiL9KNQODYRuiZOkRSh0BfpR6FwC2Ywa7zO9CU1KPRF+lEo3MyU0UMYWpDQiCci/U6hL9KPtjS2MFvt+ZJCFPoi/eTwsTYaDx/XTVmSUhT6Iv2kNnYnroZfkFSi0BfpJxp+QVKRQl+kn2wJNzN+RCHFQ/KDLkXktxT6Iv0kFG7RTVmSchT6Iv3gWFsHO/cfVdOOpByFvkg/2Lr7CBHXnLiSehIKfTNbbGbbzKzOzJb3sM9NZlZrZiEzezRufaeZvRZ7VHV3rEimqdXwC5Kier1N0MxygQeARUADUG1mVe5eG7fPdOAeYKG7HzKzMXEvcdzdL0py3SIpLRRuYVTRICaMKAy6FJHTJHKmvwCoc/ed7t4GrAaWdNnnM8AD7n4IwN33JbdMkfQS7cQdgZkmQpfUkkjoTwTq45YbYuvizQBmmNnLZrbezBbHbSs0s5rY+o/3sV6RlNfeGWHbniNqz5eUlMgoUN2dqng3rzMduAooA35lZnPd/TBQ4e5hM5sKvGBmm919x2lvYLYMWAZQUVFxlh9BJLVs33uUts6IxtyRlJTImX4DUB63XAaEu9nnp+7e7u5vA9uIfgng7uHYvzuBl4D5Xd/A3R9090p3rywtLT3rDyGSSk6NoT9XnbiSghIJ/WpguplNMbN8YCnQ9SqcJ4GrAcyshGhzz04zG2VmBXHrFwK1iGSwULiFovxcpoweEnQpIu/Ra/OOu3eY2V3AM0AusNLdQ2a2Aqhx96rYtmvNrBboBP7K3Q+a2ZXAd80sQvQL5v74q35EMlEo3Mys8cPJyVEnrqSehGZ2cPe1wNou6+6Ne+7AX8Qe8fu8AlzQ9zJF0kMk4tSGW7jhkrKgSxHplu7IFUmid5uO0drWqeEXJGUp9EWSaEtjtBNXV+5IqlLoiyRRKNzCoFxjxthhQZci0i2FvkgShcLNzBg7jPw8/WpJatJPpkiSuEc7cXUnrqQyhb5IkuxpOcHB1jZ14kpKU+iLJEmo8dScuDrTl9Sl0BdJklC4BTOYNV6hL6lLoS+SJKFwM1NKhjCkIKF7HkUCodAXSZJTY+iLpDKFvkgSHGpto/HwceaqPV9SnEJfJAlqd5/qxNWZvqQ2hb5IEpwafkFX7kiqU+iLJEEo3MKEEYWMGpIfdCkiZ6TQF0mCULiZOZopS9KAQl+kj461dbDzQKuadiQtKPRF+mjr7hbc1Ykr6UGhL9JHoXD0yp25E3WmL6lPoS/SR6HGFoqH5DNueGHQpYj0KqHQN7PFZrbNzOrMbHkP+9xkZrVmFjKzR+PW32Zm22OP25JVuEiqCO1uZs6E4ZhpInRJfb0OEmJmucADwCKgAag2syp3r43bZzpwD7DQ3Q+Z2ZjY+mLgK0Al4MCm2LGHkv9RRAZeW0eEbXuOcMf7pgRdikhCEjnTXwDUuftOd28DVgNLuuzzGeCBU2Hu7vti6z8MPOfuTbFtzwGLk1O6SPC27ztCe6czV524kiYSCf2JQH3cckNsXbwZwAwze9nM1pvZ4rM4ViRtnerE1eWaki4SGQO2u4ZK7+Z1pgNXAWXAr8xsboLHYmbLgGUAFRUVCZQkkhpCjc0Myc9l8ughQZcikpBEzvQbgPK45TIg3M0+P3X3dnd/G9hG9EsgkWNx9wfdvdLdK0tLS8+mfpFAhcItzBo/nJwcdeJKekgk9KuB6WY2xczygaVAVZd9ngSuBjCzEqLNPTuBZ4BrzWyUmY0Cro2tE0l7kYizdXcLczX8gqSRXpt33L3DzO4iGta5wEp3D5nZCqDG3av4XbjXAp3AX7n7QQAz+yrRLw6AFe7e1B8fRGSgvXOwlda2TmarPV/SSELzurn7WmBtl3X3xj134C9ij67HrgRW9q1MkdSzRZ24koZ0R67IOQqFm8nPzWH6mGFBlyKSMIW+yDmqDbcwY9xQ8vP0ayTpQz+tIufA3aMToY9XJ66kF4W+yDnY3XyCptY25mhkTUkzCn2Rc/C7O3F1pi/pRaEvcg5C4WbMYNZ4deJKelHoi5yDLY0tTC0ZQlF+Qlc9i6QMhb7IOagNN+tOXElLCn2Rs3SotY1w8wndlCVpSaEvcpbUiSvpTKEvcpa2hJsBDb8g6UmhL3KWQuEWJo4czMii/KBLETlrCn2RsxQKN+ssX9KWQl/kLLSe7ODtA61qz5e0pdAXOQtbd7fgrvZ8SV8KfZGzcOrKHV2jL+lKoS9yFkLhZkYPyWfs8IKgSxE5Jwp9kbOwpbGF2ROGY6aJ0CU9KfRFEtTWEWH7viNq2pG0ptAXSdBbe4/Q3unqxJW0llDom9liM9tmZnVmtryb7beb2X4zey32uDNuW2fc+qpkFi8ykGo1/IJkgF7HhTWzXOABYBHQAFSbWZW713bZ9XF3v6ublzju7hf1vVSRYG0JNzO0II9JxUVBlyJyzhI5018A1Ln7TndvA1YDS/q3LJHUEwq3MHv8cHJy1Ikr6SuR0J8I1MctN8TWdfUpM3vDzNaYWXnc+kIzqzGz9Wb28e7ewMyWxfap2b9/f+LViwyQA0dP8kbDYeZXjAy6FJE+SST0uzut8S7LTwGT3f1C4HngkbhtFe5eCdwCfNvMpr3nxdwfdPdKd68sLS1NsHSRgbNmUwPtnc6NlWVBlyLSJ4mEfgMQf+ZeBoTjd3D3g+5+Mrb4PeCSuG3h2L87gZeA+X2oV2TARSLOoxt2sWBKMeeN0Zy4kt4SCf1qYLqZTTGzfGApcNpVOGY2Pm7xemBrbP0oMyuIPS8BFgJdO4BFUtqv6w6wq+kYt15WEXQpIn3W69U77t5hZncBzwC5wEp3D5nZCqDG3auAL5jZ9UAH0ATcHjt8FvBdM4sQ/YK5v5urfkRS2qoN71I8JJ/Fc8cFXYpIn/Ua+gDuvhZY22XdvXHP7wHu6ea4V4AL+lijSGD2tpzg+a37uPP9UyjIyw26HJE+0x25ImfweHU9nRHn5kvVtCOZQaEv0oPOiLN64y7eP72EySVDgi5HJCkU+iI9eGnbPsLNJ9SBKxlFoS/Sg1UbdjFmWAHXzBobdCkiSaPQF+lGw6FjvLhtH5++tJxBufo1kcyhn2aRbqzeWI8BSxeoaUcyi0JfpIv2zgiP19Rz9cwxTBw5OOhyRJJKoS/SxfO1e9l/5CS3qANXMpBCX6SLVRt2MWFEIVfNHBN0KSJJp9AXifPOgVZ+XXeAmxdUkKtx8yUDKfRF4jy2cRe5OcanLy3vfWeRNKTQF4k52dHJEzX1LJo1ljHDC4MuR6RfKPRFYn6+ZQ+HjrVz6+XqwJXMpdAXiVm1YReTRhexcFpJ0KWI9BuFvgiwfe8RNr7dxC0LKjTxuWQ0hb4I0bP8/NwcbrhEc+BKZlPoS9Y73tbJj15tYPHccYweWhB0OSL9SqEvWe9nb4Q5cqJDQyhLVlDoS9ZbtWEX540ZyoIpxUGXItLvEgp9M1tsZtvMrM7Mlnez/XYz229mr8Ued8Ztu83MtscetyWzeJG+CoWbea3+MLcsqMBMHbiS+XqdGN3McoEHgEVAA1BtZlXuXttl18fd/a4uxxYDXwEqAQc2xY49lJTqRfro0Q27KMjL4VMXqwNXskMiZ/oLgDp33+nubcBqYEmCr/9h4Dl3b4oF/XPA4nMrVSS5jp7s4MnfNHLdvAmMKBoUdDkiAyKR0J8I1MctN8TWdfUpM3vDzNaY2amBSxI9VmTA/fS1RlrbOjWEsmSVREK/u4ZO77L8FDDZ3S8EngceOYtjMbNlZlZjZjX79+9PoCSRvnF3Vq3fxazxw5lfPjLockQGTCKh3wDEDzlYBoTjd3D3g+5+Mrb4PeCSRI+NHf+gu1e6e2VpaWmitYucs9cbmqnd3cKtl6kDV7JLIqFfDUw3sylmlg8sBaridzCz8XGL1wNbY8+fAa41s1FmNgq4NrZOJFCr1r/LkPxcPj5frY2SXXq9esfdO8zsLqJhnQusdPeQma0Aaty9CviCmV0PdABNwO2xY5vM7KtEvzgAVrh7Uz98DpGENR9r56k3wnxifhlDC3r9FRDJKAn9xLv7WmBtl3X3xj2/B7inh2NXAiv7UKNIUv34Nw2caI/oDlzJSrojV7KKu/Pohl3MKx/J3Ikjgi5HZMAp9CWrVL9ziO37juosX7KWQl+yyqoN7zKsMI/rLpwQdCkigVDoS9Zoam3j6c17+NTFZQzOzw26HJFAKPQla6zZVE9bZ0R34EpWU+hLVohEoh24CyYXM2PssKDLEQmMQl+ywis7DvLOwWM6y5esp9CXrPDoxncZVTSIxXPHBV2KSKAU+pLx9rWc4NnQXm6sLKdwkDpwJbsp9CXjPVFTT0fEuXmBmnZEFPqS0TojzmMb61l43mimlAwJuhyRwCn0JaP98q39NB4+zq2XTQq6FJGUoNCXjLZqw7uUDitg0eyxQZcikhIU+pKxwoeP88Kb+7ipsoxBufpRFwGFvmSw1dX1OLD0UnXgipyi0JeM1N4ZYfXGXVw1o5Ty4qKgyxFJGQp9yUi/2LqPfUdOqgNXpAuFvmSkVRveZfyIQq6aWRp0KSIpRaEvGWfXwWP8avsBll5aQZ46cEVOk9AcuWa2GPgnohOjP+Tu9/ew3w3AfwKXunuNmU0GtgLbYrusd/fP9rXo7hw+1saN/76uP15a0kzLiXZyc4xPX1oedCkiKafX0DezXOABYBHQAFSbWZW713bZbxjwBWBDl5fY4e4XJaneHuXkGNPHDu3vt5E0ccmkYsaNKAy6DJGUk8iZ/gKgzt13ApjZamAJUNtlv68C3wD+MqkVJmh44SC+c+slQby1iEjaSKTBcyJQH7fcEFv3W2Y2Hyh39591c/wUM/uNmf23mb3/3EsVEZG+SuRM37pZ57/daJYDfAu4vZv9dgMV7n7QzC4BnjSzOe7ectobmC0DlgFUVOhGGhGR/pLImX4DEN8jVgaE45aHAXOBl8zsHeByoMrMKt39pLsfBHD3TcAOYEbXN3D3B9290t0rS0t1iZ2ISH9JJPSrgelmNsXM8oGlQNWpje7e7O4l7j7Z3ScD64HrY1fvlMY6gjGzqcB0YGfSP4WIiCSk1+Ydd+8ws7uAZ4hesrnS3UNmtgKocfeqMxz+AWCFmXUAncBn3b0pGYWLiMjZM3fvfa8BVFlZ6TU1NUGXISKSVsxsk7tX9rafblcUEckiCn0RkSyScs07ZrYfeLcPL1ECHEhSOf0tnWqF9Ko3nWqF9Ko3nWqF9Kq3L7VOcvdeL39MudDvKzOrSaRdKxWkU62QXvWmU62QXvWmU62QXvUORK1q3hERySIKfRGRLJKJof9g0AWchXSqFdKr3nSqFdKr3nSqFdKr3n6vNePa9EVEpGeZeKYvIiI9yJjQN7PFZrbNzOrMbHnQ9ZyJmZWb2YtmttXMQmb2xaBr6o2Z5caGyO5u+OyUYmYjzWyNmb0Z+z++IuiaemJmd8d+BraY2WNmllIzv5jZSjPbZ2Zb4tYVm9lzZrY99u+oIGs8pYda/yH2c/CGmf3EzEYGWWO87uqN2/aXZuZmVpLs982I0I+b3esjwGzgZjObHWxVZ9QBfMndZxEdlfRzKV4vwBcSDYr4AAAC/0lEQVSJTn2ZDv4J+Lm7nw/MI0XrNrOJRGebq3T3uUTHtloabFXv8TCwuMu65cAv3H068IvYcip4mPfW+hww190vBN4C7hnoos7gYd5bL2ZWTnSmwl398aYZEfrEze7l7m3Aqdm9UpK773b3V2PPjxANpYlnPio4ZlYG/D7wUNC19MbMhhMd6O/7AO7e5u6Hg63qjPKAwWaWBxRx+rDlgXP3XwJdB0lcAjwSe/4I8PEBLaoH3dXq7s+6e0dscT3RoeFTQg//txCdn+TLxM1bkkyZEvq9zu6VqmKTx8/nvXMLp5JvE/0hjARdSAKmAvuBH8Saox4ysyFBF9Udd28E/h/RM7rdQLO7PxtsVQkZ6+67IXoCA4wJuJ5E3QE8HXQRZ2Jm1wON7v56f71HpoT+GWf3SlVmNhT4EfC/us4mlirM7GPAvtgkOOkgD7gY+Dd3nw+0kjrND6eJtYUvAaYAE4AhZvYHwVaVmczsb4g2q64KupaemFkR8DfAvf35PpkS+r3N7pVyzGwQ0cBf5e4/DrqeM1gIXB+bFW018Htm9h/BlnRGDUCDu5/6y2kN0S+BVPQh4G133+/u7cCPgSsDrikRe81sPEDs330B13NGZnYb8DHgVk/ta9SnET0BeD32+1YGvGpm45L5JpkS+mec3SvVmJkRbXPe6u7/GHQ9Z+Lu97h7WWxWtKXAC+6esmej7r4HqDezmbFV1wC1AZZ0JruAy82sKPYzcQ0p2uncRRVwW+z5bcBPA6zljMxsMfC/ic7mdyzoes7E3Te7+5i4WQgbgItjP9NJkxGhH+uoOTW711bgCXcPBVvVGS0E/pDoWfNrscdHgy4qg3weWGVmbwAXAV8LuJ5uxf4aWQO8Cmwm+vuYUnePmtljwDpgppk1mNmfAPcDi8xsO9GrTO4PssZTeqj1X4nO4/1c7Pfs3wMtMk4P9fb/+6b2XzsiIpJMGXGmLyIiiVHoi4hkEYW+iEgWUeiLiGQRhb6ISBZR6IuIZBGFvohIFlHoi4hkkf8B2t/DExUrc+EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(train_no_attention_history.history[\"val_acc\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Layer\n",
    "import keras.backend as K\n",
    "\n",
    "class Attention(Layer):\n",
    "    def __init__(self,**kwargs):\n",
    "        super(Attention,self).__init__(**kwargs)\n",
    "\n",
    "    def build(self,input_shape):\n",
    "        self.weight = self.add_weight(name=\"watt\", shape=(input_shape[-1], 1), initializer=\"normal\")\n",
    "        self.bias = self.add_weight(name=\"batt\", shape=(input_shape[1], 1), initializer=\"zeros\")        \n",
    "        super(Attention, self).build(input_shape)\n",
    "\n",
    "    def call(self,x):\n",
    "        att = K.expand_dims(K.softmax(K.squeeze(K.tanh(K.dot(x,self.weight) + self.bias), axis=-1)), axis=-1)\n",
    "        return K.sum(x*att, axis=1)\n",
    "\n",
    "    def compute_output_shape(self,input_shape):\n",
    "        return (input_shape[0], input_shape[-1])\n",
    "\n",
    "    def get_config(self):\n",
    "        return super(Attention,self).get_config()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_15 (InputLayer)        (None, 30)                0         \n",
      "_________________________________________________________________\n",
      "embedding_14 (Embedding)     (None, 30, 32)            60128     \n",
      "_________________________________________________________________\n",
      "lstm_14 (LSTM)               (None, 30, 64)            24832     \n",
      "_________________________________________________________________\n",
      "attention_9 (Attention)      (None, 64)                94        \n",
      "_________________________________________________________________\n",
      "dense_12 (Dense)             (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 85,119\n",
      "Trainable params: 85,119\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "inp = Input(shape=(s_length,))\n",
    "x = Embedding(input_dim = nb_tokens, \\\n",
    "              input_length = s_length, \\\n",
    "              output_dim = latent_dim,\\\n",
    "              embeddings_regularizer = keras.regularizers.l2(.001))(inp)\n",
    "\n",
    "x = LSTM(64, return_sequences = True, dropout = 0.2, recurrent_dropout = 0.2)(x)\n",
    "x = Attention()(x)\n",
    "\n",
    "out = Dense(1, activation='sigmoid')(x) #Our output neuron for the sentiment analysis.\n",
    "\n",
    "model = Model(inp, out)\n",
    "model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 800 samples, validate on 200 samples\n",
      "Epoch 1/15\n",
      "800/800 [==============================] - 4s 5ms/step - loss: 0.7305 - acc: 0.5075 - val_loss: 0.7185 - val_acc: 0.4550\n",
      "Epoch 2/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.7077 - acc: 0.5150 - val_loss: 0.7041 - val_acc: 0.4550\n",
      "Epoch 3/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.6960 - acc: 0.5162 - val_loss: 0.7005 - val_acc: 0.4600\n",
      "Epoch 4/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.6755 - acc: 0.5950 - val_loss: 0.6486 - val_acc: 0.6600\n",
      "Epoch 5/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.4699 - acc: 0.8013 - val_loss: 0.5786 - val_acc: 0.7450\n",
      "Epoch 6/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.3029 - acc: 0.8962 - val_loss: 0.7135 - val_acc: 0.7300\n",
      "Epoch 7/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.2250 - acc: 0.9300 - val_loss: 0.6067 - val_acc: 0.7550\n",
      "Epoch 8/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.1604 - acc: 0.9575 - val_loss: 0.5412 - val_acc: 0.7900\n",
      "Epoch 9/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0882 - acc: 0.9888 - val_loss: 0.7759 - val_acc: 0.7700\n",
      "Epoch 10/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0831 - acc: 0.9825 - val_loss: 0.8523 - val_acc: 0.7800\n",
      "Epoch 11/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0711 - acc: 0.9825 - val_loss: 0.5363 - val_acc: 0.8250\n",
      "Epoch 12/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0776 - acc: 0.9875 - val_loss: 0.6414 - val_acc: 0.8350\n",
      "Epoch 13/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0806 - acc: 0.9812 - val_loss: 0.7106 - val_acc: 0.8000\n",
      "Epoch 14/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0615 - acc: 0.9862 - val_loss: 0.6803 - val_acc: 0.8150\n",
      "Epoch 15/15\n",
      "800/800 [==============================] - 1s 1ms/step - loss: 0.0490 - acc: 0.9937 - val_loss: 0.6976 - val_acc: 0.8250\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x2e2ba916e10>]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8lPW1+PHPSUIIYU3YSQgBBQTZiaCitb0WpVbFpbW41aWW295L3Vq32moL1trebt6fVotK3aXWWopclOIuKmYCEjBBICZAJgESCAlbyHp+f3wnMoSEDJDkmcmc9+s1r8w88zwzJ5Cc+ea7HVFVjDHGRIcYrwMwxhjTfizpG2NMFLGkb4wxUcSSvjHGRBFL+sYYE0Us6RtjTBSxpG+MMVHEkr4xxkSRkJK+iMwQkQ0ikicidzfxfJqIvCMin4rIWhG5IHA8XUQqRWRN4PZ4a38DxhhjQictrcgVkVhgIzAd8AM+4EpVzQ06Zz7wqao+JiKjgaWqmi4i6cASVR0TakB9+vTR9PT0Y/0+jDEmqq1atWqnqvZt6by4EF5rCpCnqvkAIrIQmAnkBp2jQI/A/Z5A8bGFe0h6ejpZWVnHe7kxxkQlEdkSynmhdO+kAIVBj/2BY8F+AVwjIn5gKfCjoOeGBrp93hORs5sJdraIZIlIVmlpaShxG2OMOQ6hJH1p4ljjPqErgadVNRW4AHhORGKAbUCaqk4EbgdeFJEeja5FVeeraoaqZvTt2+JfJ8YYY45TKEnfDwwOepzKkd033wNeBlDVj4EEoI+qVqnqrsDxVcAXwIgTDdoYY8zxCSXp+4DhIjJUROKBWcDiRudsBc4FEJFRuKRfKiJ9AwPBiMgwYDiQ31rBG2OMOTYtDuSqaq2IzAGWAbHAAlXNEZG5QJaqLgZ+DDwhIrfhun6uV1UVka8Ac0WkFqgDfqCqZW323RhjjDmqFqdstreMjAy12TvGGHNsRGSVqma0dJ6tyDXGmChiSd8Y452aGvjkE/h//w/WrfM6mqgQyuIsY4xpHfv3w8qV8MEH7rZyJRw44J7r0gWefRa+9S1vY+zgLOkbY9pOWRmsWHEoya9aBbW1IALjx8NNN8HZZ8Mpp8Ds2fDtb8O8eXDvve4c0+os6RtjWo/ffyjBf/ABfPaZOx4fD1OmwB13uCR/5pnQs+fh1779tvsQ+PnPYf16eOopSEho/++hg7Okb4w5PqqwcePhSb6gwD3XrRtMmwazZrkkP2VKywk8IQGeew5Gj3Yt/S++gEWLYMCAtv9eooglfWNM6IqL4e9/P5TkS0rc8b59XXK/+Wb3dfx4iDuO9CICP/0pjBwJ117rPixee829XkdWWwtr1sDevfC1r7XpW1nSN8aEZssW13ovKoL0dDj/fJfgzz7bJenW7IO//HIYOhQuvti954svuvsdRWUlZGYe+vD86CPYtw/GjYPs7DZ9a0v6xpiW7dgB06e72Tc+H2S0uAboxE2a5BLjzJlwySXw0ENuTCASB3grKuDDDw8leZ8Pqqvdc2PHwne/e+gDtI1Z0jfGHF15OcyY4QZp33yzfRJ+g0GD4L334IYb4K673ADv449D587tF8Px2L798LGO7Gw3BhIX5/79brnFJfhp0yA5uV1Ds6RvjGnegQNw0UWQk+P61s88s/1jSEyEhQth1Cj45S8hLw9efdWNI4QDVTeA/f77h5L8pk3uucREOOMMuP9+l+SnToWuXT0N15K+MaZpNTVu3vyHH7qke/753sUiAr/4hZvPf8MNLnm+9hqcemr7x6LqpqIGJ/niwG7zSUkuuc+e7b5OmgSdOrV/jEdhSd8Yc6T6erjuOli6FP7yF7jiCq8jcmbNcgO8l1ziWtB/+xt84xvt894bNsALL7hbfmCH+JQUOOecQ/3xo0dDTHjvbmNJ3xhzOFX40Y/gpZfg1792rdZwMnWqG+C9+GK48EL4/e9dH3lbDPDu2OH+ynn+ecjKcgn93HPdtNL/+A83iynCBpYt6RtjDnffffDnP7uZMnfd5XU0TRs82G3vcO21cNttboD3kUdapytl/363KOz552H5cqirg4kT3YfLrFlucDmCWdI3xhzyxz/CAw/A974Hv/lNeLdiu3aFV16Bn/3M/UWyaZN7fDyzYWpr3cyk5593CX//fhgyxH3oXX2167bpICzpG2Ocp5+G2293u1z+5S/hnfAbxMTAgw+6mT033eS6fpYscYvFWqLqNoB7/nnXlVVSAr16uSR/zTVuOmWY988fj5CSvojMAB7GlUt8UlUfavR8GvAM0Ctwzt2qujTw3D24wul1wM2quqz1wjfGtIpFi1zrfvp0lwRjY484pbaunrlLclm1ZTeT0pI4bWgyU9KTGdAzDDZFu/ZaGDYMLr0UTj/dbRXx9a83fW5+vlvh+/zzbnA2Pt6NDVxzDVxwQfivAThBLZZLDBQ23whMB/y4QulXqmpu0DnzgU9V9TERGQ0sVdX0wP2XgCnAIOBNYISq1jX3flYu0Zh29vbbbgbMxImui6NbtyNOqa6t5+aXPuWNnO1MTOvFhu17OVDtfo3TkhM5LT2ZKUOTOC09maF9uiJe/ZWwebNbV7B+vSvM8sMfuuO7dsHLL7tE/9FH7tg557hEf/nlbqplhAu1XGIoLf0pQJ6q5gdeeCEwE8gNOkeBHoH7PYHApFVmAgtVtQooEJG8wOt9HNJ3YYxpWz6f2+ZgxAg3PbOJhH+wpo4fPL+KdzeU8vMLR/O9s4ZSW1dP7rY9ZBaUkVlQxjsbSvjHaj8Afbp1/vID4LT0ZEYN7EFsTDt9CKSnu6R+1VXwX//lirSUl8Prr7t1B6NHu/7/K690ffZRKJSknwIUBj32A1MbnfML4N8i8iOgK9Dwd1UKsLLRtSmN30BEZgOzAdLS0kKJ2xhzonJzXQu/b19YtqzJAdD9VbXc9EwWKwt28evLxnLlFPf7GRcbw7jUXoxL7cVNZw9DVfmidB+ZBbvJLNiFb/Nulq7bDkD3znFMTk8K/DWQzLjUnnSOO7L7qNV07+66q+66y824GTTI7f55zTVut85IGKtoQ6Ek/ab+hRr3CV0JPK2qvxeRM4DnRGRMiNeiqvOB+eC6d0KIyRhzIjZvhvPOc1Mcly9vchpiRWUNN/w1k2x/BX+4YjyXTkxt9uVEhJP7defkft25aqr7YCgqr8RXUMYnBWX4Npfx7oYNAMTHxTBhcC+mpCdz2tBkJg9JolvnVp5TEhsLv/udm3bap0+TYxTRKpR/aT8wOOhxKoe6bxp8D5gBoKofi0gC0CfEa40x7Sl4x8z33oOTTjrilLL91Xx3wSds2L6XR6+ayIwxA4/5bVJ6dSFlYgqXTEz58jV9m8vwFZSRubmMx977gkfeySNGYPSgHnzntDSumpLWul1B/fu33mt1EKEM5MbhBnLPBYpwA7lXqWpO0DmvA39T1adFZBTwFq4bZzTwIocGct8ChttArjEeKS93RTo2bnQt/CY2UCvZc5BrnvqELbsO8Pg1k/naKf3aJJT9VbWs3robX0EZ720sJdtfwamDejB35hgmD4n8gdX2FupAbotJP/BiFwB/wk3HXKCqvxKRuUCWqi4OzNJ5AuiG6765U1X/Hbj2XuBGoBa4VVVfP9p7WdI3YevgQTe9L1Lnbh844DZN++QTt1lZExuoFZVXcvUTKynZW8WT12Vw5kl92iU0VWXJ2m386v/Ws33PQb49OZW7vnEKfbp17OmTwQ7W1FG6t4rByYnHdX2rJv32ZEnfhKUPP4TLLoMePdwA4Xe/6z4AIkV1tduk7I033F4yTWygtmXXfq564hP2HKzh6RtOY/KQ9t3nHVzr/3/f3sRTHxSQGB/Lj88bydVT04iLjdAP2hCoKstzdzB3SS69Ejvx2pyzjmvKa6hJv+P+SxrTWp57zm2u1aOHW7H5/e+7fvCHH3at53BXXw/XX++mLT7+eJMJf9OOvXz78Y85UF3LS98/3ZOED9C1cxz3fGMUb9z6Fcal9uL+xTlc9MiHZG0u8ySetrZ5535ueNrH7OdW0TU+jp99c3Sbr3GwpG9Mc+rr3W6K3/0unHWW6xbJzHTTG086CW691c31fvBBVw4vHAXvmPnQQ03umJlTXMF35q9EgYWzz2BMSs/2j7ORk/t147nvTeHPV0+i/EA133r8Y25/eQ2le6u8Dq1VVFbX8btlGzjvj++TtXk3P79wNEtuPovTh/Vu+zdX1bC6TZ48WY3x3L59qpdeqgqqs2erVlcfec6KFaoXXODO6dFD9ac/VS0paf9Yj+ZnP3Px3Xlnk0+v3lKmY+9/Q8948E3NL93XzsGFZn9VjT70+no9+af/p2Pue0MXrMjXmto6r8M6LvX19fr6umI989dv6ZC7luitCz/VHRWVrfLauDHWFnOs50m+8c2SvvHc1q2qEyaoxsSoPvywan390c9fvVr1W99SFVHt0kX1lltUCwvbJ9aj+f3v3a/4TTc1+T18/MVOHf3z1/Urv31bC8v2exDgsckr2avXPLlSh9y1RM//43v6Sf4ur0M6Jl+0cfyhJn0byDUmWGam25Zg//5jr8r0+eeuC+X5590Mn+uuc4O+J5/cdvEGq6x08X/wgSvlt3y52zFz4cIjFie9t7GU/3wui9SkRF64aSr9e4TBpmkhUFWW5Wxn3pL1FJVXcunEFO75xin0C+P4D1TX8sjbeTzxQT4JcbHcft4Irj19SKsPTtvsHWOO1cKFrv7qwIEnVn9182b4n/+Bp55y+7185ztwzz0wdmyrhktFhZtV1FCn1edzs3TAvdeMGTBv3hG7Rv47ZztzXvz0y37z3hE4LbKyuo5H38lj/vv5xMfFcOvXh3Pdmel0CqNZPqrK659t54EluRRXHOTySanc/Y1T6Nu9bf69LekbEypVV3R77lxX5/TVV93S/RO1fTv84Q/w2GOwb58r73fvvTBlyvG93o4dh1rxH3wA2dku9rg4yMg4VKd12rRmC4n8a00Rt7+czdiUnjxzwxR6JoZX0e5jVbBzP798LYd3N5Qysn93fjnz1PYZDG1BXsk+frE4hxV5Oxk1sAfzZp5KRnrbzoiypG9MKCor3XTGl192Xx9/vPX3Uy8rc9v8Pvww7N7taqzeey989avNb/6lCgUFhyf5TZvcc4mJrih4Q5KfOtVVkWrBy75C7np1LaelJ7Pg+tNaf78bj2hgnvsvX8ulqLySmRMG8dMLRnnSZdWwzmDBigISOsXyk3ZcZ2BJ35iWFBe7BUtZWa404E9+0rY7MO7d6ypS/f737q+A0093yf+b33RJPifnUFfN+++7+MDt9X7WWfCVr7gkP2nSMdeCfeajzdy/OIezh/dh/rUZdInveBuQVVbX8di7eTz+fj6dYoRbvz6C66e1T5ePqvJ/67bxwBLvVhRb0jfmaFavdt0t5eWuitLFF7ffex88CH/9q/ug2bLFDfTu2uX+CgBISXHJvSHJjx59Qls/PPbuF/zmjc+ZPro/j1w1sW23NQ4DW3bt55ev5fL25yWc1LcrZ5zUm0G9upDSqwupSV0Y1KsL/bontNrGbnkle7nvXzl89MUuT/cOsqRvTHNefdWV1+vd2w3Yjh/vTRw1NW7R1DPPwNChh5J8enqr/MWhqvxx+Ub+9+08Lho/iD9cMT6sBjrb2pu5O3j03TwKdu6n/EDNYc/FxQgDeia4nUCT3AdCw/2GD4iETkf/cNxXVcv/vuW6chLjY7nj/JFcNXVI+xWMacSSvjGNqbqqSffe67pW/vlPGDDA66jahKryq/9bz5MrCrgiI5VfXzbOs2QUDvZX1VJcXom/vJLi8kqKdldSFHR/+56D1DdKhb27xn/5gdDwQTAo8NfCF6X7eHDpenbsqeI7GYO5c8ZIz2dBtWa5RGMi38GDbs+c55+Hq6+GJ5+EhPCd2328tu46wKI1RSxaU0R+6X6uPzOd+y4cTUwUJ3xwe/oM79+d4f27N/l8bV092/ccpGh3JcUVhz4UisoPsnHHXt7ZUMLBmvrDrhmb0pPHr5nMxLTI2gbakr7p+HbsgEsvhY8/hgcecPvpdKCSebv3V7Nk3TYWfVrEqi1uXGDq0GT++6snc9mkFO+KlEeQuNgYUpMSSU1qeltjVWX3gZovPwxE4Ouj+kfkX0+W9E3HtnYtXHQRlJbC3//uVqh2AAdr6nhrfQn//LSIdzeUUFuvjOjfjbtmnMLFEwaR0quL1yF2KCJCctd4krvGMzbV+w3pToQlfdNxvfYaXHWV2xL5gw9g8mSvIzohdfXKJ/m7WLSmiNfXbWdvVS39e3TmxrOGcsmEFEYN7G6tetMiS/qm41F1c+HvvNPNaV+8uMnC35Fi/bY9LPq0iH+tKWb7noN06xzHjDEDuHRiCqcP6x2RXQzGOyElfRGZATyMK5f4pKo+1Oj5PwJfCzxMBPqpaq/Ac3XAusBzW1W1HSdEm6hTVOT2uX/lFdeV88wzbgVrhCkur2RxdjGLPi3i8+17iYsRvjqyLz+7cBRfH9W/xemExjSnxaQvIrHAo8B0wA/4RGSxquY2nKOqtwWd/yNgYtBLVKrqhNYL2ZgmVFfDn/7k9s+prYVf/Qruvjui6tlWVNbwxmfbWPRpMSsLdqEKk9J6MW/mqXxz3CCSu0ZQeUYTtkJp6U8B8lQ1H0BEFgIzgdxmzr8SuL91wjMmBG+9BXPmuK2NL7rIJf9hw7yOKmTvbyxloW8rb64vobq2nqF9unLruSO4ZOIghvRueU8dY45FKEk/BSgMeuwHpjZ1oogMAYYCbwcdThCRLKAWeEhVFzVx3WxgNkBaWlpokRtTWAg//rGblTNsGCxZ4vaxiSAbd+zlur9mkpwYz1VT0rhkYgrjU3vagKxpM6Ek/aZ++ppbxjsLeEVV64KOpalqsYgMA94WkXWq+sVhL6Y6H5gPbkVuCDGZaFZd7bYsnjfP1bH95S/doG0ELrbK2rwbVfjHD88kvY+16k3bC6XD0w8MDnqcChQ3c+4s4KXgA6paHPiaD7zL4f39JhIcPAj//d9ue+CSEm9j+fe/XYGQe+6B6dMhNxfuuy8iEz7AWn85vRI7MaR35A02m8gUStL3AcNFZKiIxOMS++LGJ4nISCAJ+DjoWJKIdA7c7wNMo/mxABOunnwS/vxnuPlmN/Xxm990G4UdONB+MWzd6mbjnH++a90vXQqLFrmNyiLYmsJyxqX2su4c025aTPqqWgvMAZYB64GXVTVHROaKSPD0yyuBhXr4Dm6jgCwRyQbewfXpW9KPJFVVru7r2WfDunVwxx3u61VXQf/+rg7s8uVQV9fyax3v+z/4IIwa5RL9Aw+49z+W2rVh6kB1LZtK9jE+wld4mggTSvX09rxNnjz5GOq/mzb32GOqoLp8+aFjdXWq776retNNqj17uucHDFC97TbVVatU6+tb571ff111+HD3+pddprp5c+u8bpjILNilQ+5aostztnsdiukAgCwNIcdGziRm0/6qq91WxGec4Ur8NYiJgXPOgSeecBWg/vEPd84jj7itDk491bXON28+vvfdsgUuu+xQa/6NN9x7DBlywt9SOMkuLAdg3GBr6Zv2Y0nfNO/ZZ11f+n33Nb8rZUKCS9Cvvuo+AP7yF1dU/N57DxUGmT//UFWoozl40HXfjBoFy5a5D45161w/fgeU7a9gUM8E+nWPzEFoE5ks6Zum1dS4pHvaaaEn3eRkmD3b1XctKHCrYnfuhP/8T1espOHD4eDBI69duhTGjIGf/9wNFK9f72botHaR8jCy1u8GcY1pT5b0TdNeeMEl7qO18o8mPd3tW5+T4+rRzpnj9rO//HL3AfD978N770F+Psyc6RJ9XJybkvn3v0MHX6S3e381W3YdYPxgS/qmfdkum+ZIDXvXTJx44itcRdzrTJwIv/0tvP22q1710ktuKihA166uSPitt0J8dOwvs7aoAsBm7ph2Z0nfHGnhQsjLc10xrTl/PDbWLaiaPt3N+1+82P0l8IMfQGpq671PBMguLEcExljSN+3Mkr45XF2da+WPHeu6XdpK165w5ZVt9/phbq2/nGF9utIjoZPXoZgoY0nfHO6VV9xulS+/HFHbEkcSVWVNYQVfGdHH61BMFLLfanNIfb3bxGz0aDfgatrEtoqD7NxXxXibuWM8YC19c8g//+n62F980Vr5bahhUZbN3DFesN9s49TXu6pTI0bAFVd4HU2Hlu2voFOsMGpgd69DMVHIWvrGee01WLvWrcKNtfqrbSm7sJxRA3vQOc7+nU37s5a+AVXXyj/ppKieUdMe6uuVz4oqGGdTNY1HrKVv3BYIq1fDU0+5VbGmzeTv3M/eqlobxDWesZZ+tFN1M3aGDIFrr/U6mg7PBnGN16xZF+2WL4dPPnG7Y3ayhUJtba2/nK7xsZzUt5vXoZgoZS39aKbqiooPHuwqYJk2t8ZfwZiUnsTGWHlE442Qkr6IzBCRDSKSJyJ3N/H8H0VkTeC2UUTKg567TkQ2BW6WWcLJO+/ARx/B3Xd36C2Mw0V1bT3ri/dY147xVIvdOyISCzwKTAf8gE9EFmtQrVtVvS3o/B8BEwP3k4H7gQxAgVWBa0OoqGHa3Ny5rtD5jTd6HUlU+Hz7Hqrr6m0Q13gqlJb+FCBPVfNVtRpYCBxtJ64rgZcC988HlqtqWSDRLwdmnEjAppW895673Xmnq35l2ly2322nbNM1jZdCSfopQGHQY3/g2BFEZAgwFHj7WK4VkdkikiUiWaWlpaHEbU7UvHnQv78rZmLaRXZhOb27xpOa1MXrUEwUCyXpNzXipM2cOwt4RVXrjuVaVZ2vqhmqmtG3b98QQjIn5MMP4a234I47IDHR62iihiuP2BNpzRoFxhyjUJK+Hxgc9DgVKG7m3Fkc6to51mtNe5k3zxUv/8EPvI4kauyrqmVTyT4bxDWeCyXp+4DhIjJUROJxiX1x45NEZCSQBHwcdHgZcJ6IJIlIEnBe4JjxSmYmLFsGP/mJK2Ri2sVnRRWoYoO4xnMtzt5R1VoRmYNL1rHAAlXNEZG5QJaqNnwAXAksVFUNurZMRObhPjgA5qpqWet+C+aYzJsHycnwX//ldSRRpWElrg3iGq+FtCJXVZcCSxsdu6/R4180c+0CYMFxxmda0+rVsGQJPPAAdLdtfdvTWn8FqUld6N3N1kMYb9mK3Ggybx706gVz5ngdSdRZU1hu/fkmLFjSjxbZ2bBoEdxyC/S0Lob2tHNfFUXllYy3rh0TBizpR4uGLp1bbvE6kqiz1h/YWdMGcU0YsKQfDT77DF55BW6+GZKSvI4m6mQXVhAjMCbFWvrGe5b0o8GvfuWmZ952W8vnmlaX7S9neL/udO1sO5kb71nS7+g+/xz+9jc3eNu7t9fRRB1VZa3fyiOa8GFJv6N78EHo0gVuv93rSKKSf3clZfurbeaOCRuW9DuyvDx44QX44Q+hXz+vo4lK2TaIa8KMJf2O7MEHIT7ebblgPJFdWE58XAwjB9hiOBMeLOl3VAUF8OyzMHs2DBjgdTRRK9tfweiBPYiPs181Ex7sJ7Gj+vWvITbWFUkxnqirVz4rqmCC9eebMGJJvyPasgWefhpuuglSmqx3Y9pBXsk+DlTX2cwdE1Ys6XdEv/mN+3rXXd7GEeUadta0mTsmnFjS72iKiuCpp+CGGyAtzetoolq2v5zuneMY2tvqFpjwYUm/o/ntb6G+Hu65x+tIol62v5xxg3sSE2PlEU34sKTfkai6efnf/jakp3sdTVQ7WFPH59v2Ms7m55swY0m/IykogF274JxzvI4k6uVu20NtvdqiLBN2Qkr6IjJDRDaISJ6I3N3MOVeISK6I5IjIi0HH60RkTeB2RG1d04oyM93XKVO8jcOw9stBXJu5Y8JLi9v+iUgs8CgwHfADPhFZrKq5QecMB+4BpqnqbhEJXvNfqaoTWjlu0xSfDxISYMwYryOJetn+Cvp178yAHgleh2LMYUJp6U8B8lQ1X1WrgYXAzEbnfB94VFV3A6hqSeuGaUKSmQkTJ0KnTl5HEvWy/eWMS+2FiA3imvASStJPAQqDHvsDx4KNAEaIyIcislJEZgQ9lyAiWYHjlzT1BiIyO3BOVmlp6TF9AyagttYVPj/tNK8jiXoVlTXkl+5ngnXtmDAUSlWHppoq2sTrDAe+CqQCH4jIGFUtB9JUtVhEhgFvi8g6Vf3isBdTnQ/MB8jIyGj82iYUublw4ID154eBz4oqAGzmjglLobT0/cDgoMepQHET5/xLVWtUtQDYgPsQQFWLA1/zgXeBiScYs2mKz+e+Wkvfc2sCg7i2/YIJR6EkfR8wXESGikg8MAtoPAtnEfA1ABHpg+vuyReRJBHpHHR8GpCLaX2ZmdCrF5x8steRRL21/nLSeyfSKzHe61CMOUKL3TuqWisic4BlQCywQFVzRGQukKWqiwPPnSciuUAdcIeq7hKRM4G/iEg97gPmoeBZP6YV+XyQkQExtvTCa9mFFUwdlux1GMY0KaRKzaq6FFja6Nh9QfcVuD1wCz7nI2DsiYdpjqqyEtautQ3WwsCOPQfZvueg9eebsGXNwo5gzRqoq7P+/DDQsLOmzdwx4cqSfkdgK3HDxlp/BbExwuiBlvRNeLKk3xH4fDBokLsZT2X7yxnZvztd4mO9DsWYJlnS7wgyM62VHwZUlezCcttvx4Q1S/qRbvdu2LTJ+vPDwOZdB9hzsNZ21jRhzZJ+pMvKcl+tpe+5tf6GRVmW9E34sqQf6RpW4mZkeBuHYU1hOQmdYhjRv5vXoRjTLEv6kS4zE0aMcKtxjafW+isYM6gncbH2a2XCl/10Rjqfz/rzw0BNXT05xRWMH2wfvia8WdKPZEVFUFxs/flhYOOOvRysqbdN1kzYs6QfyWxnzbCx1u+2U7aZOybcWdKPZJmZEBcHE6wapdeyC8vp2aUTQ3oneh2KMUdlST+S+Xwwdix06eJ1JFEv21/BuNSeVh7RhD1L+pGqvt4lfevP91xldR0bd+xlgg3imghgST9S5eVBRYX154eBnOIK6urVFmWZiGBJP1LZzppho6E84nibuWMigCX9SOXzQWIijBrldSRRb62/goE9E+jXI8HrUIxpUUhJX0RmiMgGEcnCcTc1AAASMUlEQVQTkbubOecKEckVkRwReTHo+HUisilwu661Ao96mZkwebKbvWM8le0vt6maJmK0mPRFJBZ4FPgGMBq4UkRGNzpnOHAPME1VTwVuDRxPBu4HpgJTgPtFJKlVv4NoVFMDn35q/flhoPxANVt2HWCcbadsIkQoLf0pQJ6q5qtqNbAQmNnonO8Dj6rqbgBVLQkcPx9YrqplgeeWAzNaJ/Qotm4dVFVZf34YyA4syppgLX0TIUJJ+ilAYdBjf+BYsBHACBH5UERWisiMY7gWEZktIlkiklVaWhp69NHKVuKGjbWBQdwxNohrIkQoSb+p1Sba6HEcMBz4KnAl8KSI9ArxWlR1vqpmqGpG3759QwgpymVmQu/eMHSo15FEvWx/OSf17UqPhE5eh2JMSEJJ+n5gcNDjVKC4iXP+pao1qloAbMB9CIRyrTlWDTtr2upPT6kqaworbBDXRJRQkr4PGC4iQ0UkHpgFLG50ziLgawAi0gfX3ZMPLAPOE5GkwADueYFj5njt3w85OdafHwa2VRxk574q207ZRJQW5/upaq2IzMEl61hggarmiMhcIEtVF3MouecCdcAdqroLQETm4T44AOaqallbfCNRY/VqtwWD9ed77lB5ROvPN5EjpEneqroUWNro2H1B9xW4PXBrfO0CYMGJhWm+1LAS15K+59YUVtApVhg1sIfXoRgTMluRG2l8PkhLg/79vY4k6q31l3PKgB4kdIr1OhRjQmZJP9JkZlp/fhior1fW+SsYb4uyTISxpB9Jdu6EggLr2gkD+Tv3s7eq1nbWNBHHkn4kaViUZS19z2UHFmXZHvom0ljSjyQ+n5ubP3my15FEvbX+chLjYzmpbzevQzHmmFjSjySZmW4r5e7dvY4k6q3xVzA2pSexMbZAzkQWS/qRQvXQSlzjqeraetYX77FFWSYiWdKPFFu3QkmJ9eeHgc+376G6rt62XzARyZJ+pLCdNcNGw3bKthLXRCJL+pEiMxPi42HcOK8jiXrZheX07hpPalIXr0Mx5phZ0o8UPh+MHw+dO3sdSdRb6y9nXGpPxHY5NRHIkn4kqKuDrCzrzw8D+6pq2VSyzwZxTcSypB8JNmyAffusPz8MfFZUgSo2iGsiliX9SNCws6a19D3XsBLXBnFNpLKkHwl8Prcga+RIryOJemv9FaQmdaF3NxtbMZHJkn4kyMyEjAyIsf8ur60pLLf+fBPRLIuEu6oqyM62/vwwsHNfFUXllYy3rh0TwUJK+iIyQ0Q2iEieiNzdxPPXi0ipiKwJ3G4Keq4u6Hjj2rqmJdnZUFNj/flhoKE8og3imkjWYrlEEYkFHgWmA37AJyKLVTW30al/U9U5TbxEpapOOPFQo5StxA0b2YUVxAiMSbGWvolcobT0pwB5qpqvqtXAQmBm24ZlvpSZ6UojDh7sdSRRL9tfzvB+3enaOaTS0saEpVCSfgpQGPTYHzjW2OUislZEXhGR4AyVICJZIrJSRC5p6g1EZHbgnKzS0tLQo48GDTtr2upPTx2ormXVlt1WNMVEvFCSflPZRhs9fg1IV9VxwJvAM0HPpalqBnAV8CcROemIF1Odr6oZqprRt2/fEEOPAhUV8Pnn1p8fBv6xuoi9B2u54rRUr0Mx5oSEkvT9QHDLPRUoDj5BVXepalXg4RPA5KDnigNf84F3gYknEG90WbXK7aNv/fmeqq9X/rqigPGDezEpLcnrcIw5IaEkfR8wXESGikg8MAs4bBaOiAwMengxsD5wPElEOgfu9wGmAY0HgE1zbBA3LLy7sYT8nfu5cVq6bbJmIl6LI1KqWisic4BlQCywQFVzRGQukKWqi4GbReRioBYoA64PXD4K+IuI1OM+YB5qYtaPaU5mJgwbBr17ex1JVFuwYjMDeiRwwdiBLZ9sTJgLaRqCqi4FljY6dl/Q/XuAe5q47iNg7AnGGL18Ppg2zesootrn2/ewIm8nd84YSadYW8toIp/9FIer7duhsNC6djz21xWb6dIplqumpHkdijGtwpJ+uGroz7eZO57Zua+Kf64p4vLJKfRKjPc6HGNahSX9cJWZ6TZYm2iTnbzywsqtVNfWc8O0oV6HYkyrsaQfrnw+GDMGunb1OpKoVFVbx3Mrt/C1kX05qW83r8MxptVY0g9HqodW4hpPvJa9jZ37qrjxLGvlm47Fkn44ys+HsjLrz/eIqvLUigJG9u/OWSf38TocY1qVJf1w1FAe0Vr6nliZX8b6bXu48SxbjGU6Hkv64cjng4QE16dv2t1TKwpI7hrPzAlN7StoTGSzpB+OMjPdrJ1OnbyOJOps3rmftz7fwTVT00joFOt1OMa0Okv64aa2Flavtv58jzz90WbiYoRrTh/idSjGtAlL+uEmJwcqK60/3wMVlTW8nFXIReMH0a9HgtfhGNMmLOmHG1uJ65mXfYUcqK7jRluMZTowS/rhJjMTevWCk0/2OpKoUltXz9MfbWbq0GSrgWs6NEv64cbKI3piWc4Oisor+Z4txjIdnCX9cHLgAKxbZ/35HljwYQFDeidy7qj+XodiTJuypB9O1qyBujrrz29nawrLWbVlN9efmU5sjP2FZTo2S/rhxFbiemLBigK6d47j2xmDWz7ZmAgXUtIXkRkiskFE8kTk7iaev15ESkVkTeB2U9Bz14nIpsDtutYMvsPx+SAlBQYN8jqSqLGtopKl67bxndMG061zSIXkjIloLf6Ui0gs8CgwHfADPhFZ3ESt27+p6pxG1yYD9wMZgAKrAtfubpXoO5rMTGvlt7NnPtpCvSrXnZnudSjGtItQWvpTgDxVzVfVamAhMDPE1z8fWK6qZYFEvxyYcXyhdnBlZZCXZ/357ehAdS0vZW5lxpgBDE5O9DocY9pFKEk/BSgMeuwPHGvschFZKyKviEhD52io15qsLPfVWvrt5h+ri6iorLHFWCaqhJL0m5rOoI0evwakq+o44E3gmWO4FhGZLSJZIpJVWloaQkgdUMNK3IwMb+OIEvX1yl9XFDA+tSeThyR5HY4x7SaUpO8Hgqc1pALFwSeo6i5VrQo8fAKYHOq1gevnq2qGqmb07ds31Ng7lsxMGDHCrcY1be69jaXk79zPjWcNtT3zTVQJJen7gOEiMlRE4oFZwOLgE0RkYNDDi4H1gfvLgPNEJElEkoDzAsdMMFWX9K0/v908taKAAT0SuGDswJZPNqYDaXH2jqrWisgcXLKOBRaoao6IzAWyVHUxcLOIXAzUAmXA9YFry0RkHu6DA2Cuqpa1wfcR2YqKYPt2689vJxu272VF3k7unDGSTrG2VMVEl5AmJqvqUmBpo2P3Bd2/B7inmWsXAAtOIMaOz3bWbFcLVhSQ0CmGq6akeR2KMe3OmjnhIDMT4uJgwgSvI+nwdu6r4p9rirh8Uiq9EuO9DseYdmdJPxz4fDBunKuLa9rUi59spbq2nhtsmqaJUpb0vVZff2g7ZdOmqmrrePbjLXx1ZF9O7tfN63CM8YQlfa9t2gR79lh/fjt4LXsbO/dV2Z75JqpZ0vea7azZLlSVBSsKGNG/G2ed3MfrcIzxjCV9r/l80LUrjB7tdSQd2sr8MnK37eHGabYYy0Q3S/pey8yESZMgNtbrSDq0BR8WkNw1nksm2tZPJrpZ0vdSdbWrlmX9+W1q8879vLl+B1dPTSOhk324muhmSd9L69ZBVZX157expz/aTFyMcO3pQ7wOxRjPWdL3kq3EbXMVlTW8nFXIReMG0a+HrYMwxpK+lzIzoXdvSE/3OpIO62VfIQeq67jRpmkaA1jS95bP51r5NpukTdTW1fP0R5uZOjSZMSk9vQ7HmLBgSd8r+/ZBbq7157ehf+fuoKi80lr5xgSxpO+V1avdFgzWn99mnlpRQFpyIl8f1d/rUIwJGyFtrWxaSU0N+P2wdSs8+6w7Zi39NrGmsJxVW3Zz34WjiY2x7jNjGljSb01797qEvmWLuzW+X1zsWvcNxo+Hfv28i7cDW7CigO6d47jitMEtn2xMFLGkHypVKClpPqFv2QK7dx9+TadOMHgwpKXBuefCkCHu/pAhh26m1W2rqGTpum1cf2Y63Trbj7gxwUL6jRCRGcDDuHKJT6rqQ82c9y3g78BpqpolIum4erkbAqesVNUfnGjQTanwb2d3xhlt8dJ0qq2mb0Up8bU1hx3fn9CVHUn9KenVj5JR57j7Sf3YkTSAkqT+7O6eRH1MoxWgFcBaYG0xTdSIN61g78Fa6lW57sx0r0MxJuy0mPRFJBZ4FJgO+AGfiCxW1dxG53UHbgY+afQSX6hqm5eEkk5xlA05qU1euy42jtzkfuzqM5DdvQdQ1mcgu/oM5GBi83uyJwduxhuThyQzODnR6zCMCTuhtPSnAHmqmg8gIguBmUBuo/PmAb8FftKqEYaoR/8+TPrkTS/e2hhjIkYoUzZTgMKgx/7AsS+JyERgsKouaeL6oSLyqYi8JyJnH3+oxhhjTlQoLf2m5rvpl0+KxAB/BK5v4rxtQJqq7hKRycAiETlVVfcc9gYis4HZAGlpaSGGbowx5liF0tL3A8Hz3lI5fASyOzAGeFdENgOnA4tFJENVq1R1F4CqrgK+AEY0fgNVna+qGaqa0bdv3+P7TowxxrQolKTvA4aLyFARiQdmAYsbnlTVClXto6rpqpoOrAQuDsze6RsYCEZEhgHDgfxW/y6MMcaEpMXuHVWtFZE5wDLclM0FqpojInOBLFVdfJTLvwLMFZFaoA74gaqWtUbgxhhjjp2oastntaOMjAzNysryOgxjjIkoIrJKVTNaOs82XDPGmChiSd8YY6JI2HXviEgpsOUEXqIPsLOVwmlrkRQrRFa8kRQrRFa8kRQrRFa8JxLrEFVtcfpj2CX9EyUiWaH0a4WDSIoVIiveSIoVIiveSIoVIive9ojVuneMMSaKWNI3xpgo0hGT/nyvAzgGkRQrRFa8kRQrRFa8kRQrRFa8bR5rh+vTN8YY07yO2NI3xhjTjA6T9EVkhohsEJE8Ebnb63iORkQGi8g7IrJeRHJE5BavY2qJiMQGtshuavvssCIivUTkFRH5PPBv3DYl1VqBiNwW+Bn4TEReEpEEr2MKJiILRKRERD4LOpYsIstFZFPga5KXMTZoJtb/CfwcrBWRf4pILy9jDNZUvEHP/UREVET6tPb7doikH1Td6xvAaOBKERntbVRHVQv8WFVH4XYl/e8wjxfgFlzpy0jwMPCGqp4CjCdM4xaRFFy1uQxVHYPb22qWt1Ed4WlgRqNjdwNvqepw4K3A43DwNEfGuhwYo6rjgI3APe0d1FE8zZHxIiKDcZUKt7bFm3aIpE9QdS9VrQYaqnuFJVXdpqqrA/f34pJSytGv8o6IpALfBJ70OpaWiEgP3EZ/TwGoarWqlnsb1VHFAV1EJA5IJMwKJ6vq+0DjTRJnAs8E7j8DXNKuQTWjqVhV9d+qWht4uBK3NXxYaObfFlx9kjsJqlvSmjpK0m+xule4ChSPn8iRtYXDyZ9wP4T1XgcSgmFAKfDXQHfUkyLS1eugmqKqRcDvcC26bUCFqv7b26hC0l9Vt4FrwAD9PI4nVDcCr3sdxNGIyMVAkapmt9V7dJSkf9TqXuFKRLoB/wBubVxNLFyIyIVASaAITiSIAyYBj6nqRGA/4dP9cJhAX/hMYCgwCOgqItd4G1XHJCL34rpVX/A6luaISCJwL3BfW75PR0n6LVX3Cjsi0gmX8F9Q1Ve9jucopgEXB6qiLQT+Q0Se9zako/IDflVt+MvpFdyHQDj6OlCgqqWqWgO8CpzpcUyh2CEiAwECX0s8jueoROQ64ELgag3vOeon4RoA2YHft1RgtYgMaM036ShJ/6jVvcKNiAiuz3m9qv7B63iORlXvUdXUQFW0WcDbqhq2rVFV3Q4UisjIwKFzgVwPQzqarcDpIpIY+Jk4lzAddG5kMXBd4P51wL88jOWoRGQGcBeumt8Br+M5GlVdp6r9gqoQ+oFJgZ/pVtMhkn5goKahutd64GVVzfE2qqOaBlyLazWvCdwu8DqoDuRHwAsishaYADzocTxNCvw18gqwGliH+30Mq9WjIvIS8DEwUkT8IvI94CFguohsws0yecjLGBs0E+sjuDreywO/Z497GmSQZuJt+/cN7792jDHGtKYO0dI3xhgTGkv6xhgTRSzpG2NMFLGkb4wxUcSSvjHGRBFL+sYYE0Us6RtjTBSxpG+MMVHk/wO1dMvIqvZTEwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_attention_history = model.fit(x = text_pad, y = df[1], \\\n",
    "                          batch_size = 64, epochs = 15, \\\n",
    "                          verbose = 1, shuffle = True, \\\n",
    "                          validation_split=0.2)\n",
    "plt.plot(train_no_attention_history.history[\"val_acc\"])\n",
    "plt.plot(train_attention_history.history[\"val_acc\"], color=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see how the model with Attention Layer (in red) captured faster the distribution and achieved even better result."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}