pratical-rl/week4_approx/dqn_atari.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-Network implementation\n",
    "\n",
    "This notebook shamelessly demands you to implement a DQN - an approximate q-learning algorithm with experience replay and target networks - and see if it works any better this way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#XVFB will be launched if you run on a server\n",
    "import os\n",
    "if type(os.environ.get(\"DISPLAY\")) is not str or len(os.environ.get(\"DISPLAY\"))==0:\n",
    "    !bash ../xvfb start\n",
    "    %env DISPLAY=:1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Frameworks__ - we'll accept this homework in any deep learning framework. This particular notebook was designed for tensorflow, but you will find it easy to adapt it to almost any python-based deep learning framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's play some old videogames\n",
    "![img](https://s17.postimg.org/y9xcab74f/nerd.png)\n",
    "\n",
    "This time we're gonna apply approximate q-learning to an atari game called Breakout. It's not the hardest thing out there, but it's definitely way more complex than anything we tried before.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing game image \n",
    "\n",
    "Raw atari images are large, 210x160x3 by default. However, we don't need that level of detail in order to learn them.\n",
    "\n",
    "We can thus save a lot of time by preprocessing game image, including\n",
    "* Resizing to a smaller shape, 64 x 64\n",
    "* Converting to grayscale\n",
    "* Cropping irrelevant image parts (top & bottom)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gym.core import ObservationWrapper\n",
    "from gym.spaces import Box\n",
    "\n",
    "from scipy.misc import imresize\n",
    "from skimage.color import rgb2gray\n",
    "import cv2\n",
    "\n",
    "class PreprocessAtari(ObservationWrapper):\n",
    "    def __init__(self, env):\n",
    "        \"\"\"A gym wrapper that crops, scales image into the desired shapes and optionally grayscales it.\"\"\"\n",
    "        ObservationWrapper.__init__(self,env)\n",
    "        \n",
    "        self.img_size = (64, 64)\n",
    "#         self.observation_space = Box(0.0, 1.0, self.img_size)\n",
    "        self.observation_space = Box(0.0, 1.0, (self.img_size[0], self.img_size[1], 1))\n",
    "\n",
    "\n",
    "    def _observation(self, img):\n",
    "        \"\"\"what happens to each observation\"\"\"\n",
    "        \n",
    "        # Here's what you need to do:\n",
    "        #  * crop image, remove irrelevant parts\n",
    "        #  * resize image to self.img_size \n",
    "        #     (use imresize imported above or any library you want,\n",
    "        #      e.g. opencv, skimage, PIL, keras)\n",
    "        #  * cast image to grayscale\n",
    "        #  * convert image pixels to (0,1) range, float32 type\n",
    "        \n",
    "        # crop image (top and bottom, top from 34, bottom remove last 16)\n",
    "        img = img[34:-16, :, :]\n",
    "        \n",
    "         # resize image\n",
    "        img = cv2.resize(img, self.img_size)\n",
    "        # grayscale\n",
    "        img = img.mean(-1, keepdims=True)\n",
    "        # convert pixels to range (0,1)\n",
    "#         img = img/255\n",
    "        img = img.astype('float32') / 255.\n",
    "        \n",
    "#         print (img.shape)\n",
    "#         # resize image\n",
    "# #         img = cv2.resize(img, self.img_size)\n",
    "# #         img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "# #         \n",
    "#         img = imresize(img, self.img_size)\n",
    "#         print (img.shape)\n",
    "# #         img = rgb2gray(img)\n",
    "# #         img = img[:, :, np.newaxis]\n",
    "#         img = img.mean(-1, keepdims=True)\n",
    "#         print (img.shape)\n",
    "#         img = img.astype('float32') / 255.\n",
    "#         print (img.shape)\n",
    "        \n",
    "        return img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 (210, 160, 3) uint8\n",
      "Discrete(4) Box(210, 160, 3)\n",
      "3 (64, 64, 1) float32\n",
      "Formal tests seem fine. Here's an example of what you'll get.\n",
      "(64, 64)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAEICAYAAAB/KknhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFG9JREFUeJzt3X20VXWdx/H3Rx4CE0XkhijmtQEzWinKXY6mlY8tKlPWqhztCYsV2tQsXNmYVtPoyjVla3pwZpoazJJRUwklkOlBQ8Esl3oRK5AIMgwQ5GIQklqh3/lj/y5ubvdyD9x9zuHy+7zWOuvs5/09e5/P2Q9nn30UEZhZXvZrdgFm1ngOvlmGHHyzDDn4Zhly8M0y5OCbZSiL4EsKSWObXUeuJF0l6eZm12EvyyL4fSFptaSzml1H1SRdJOmBZtdhzeHg9xOSBja7hj3RX+ve1/Xb4Ev6kKS7Su0rJX2v1L5G0oTSKGelYbZI+rokpeH+TtK9kp6RtEnSLZKGp343Aa8G7pK0TdLl3dSxVNI7S+2D0nSOT+3nSlqW5rtQ0utKw+50CCLpRknXpObTJK2V9ClJG4DvdDPviyQ9IOnfJW2W9DtJbyv1P0jSDZLWS1on6RpJA1IN3wROTq9ri6Sj0vN+adzrJW0sTesmSZem5sMkzZP0B0mrJH2kNNxVkmZLulnSVuCiLjUPknSrpDskDe7mNR0i6S5JWyU9kmp+oNT/jan7H9PzG0v9Fkr6vKSfSXpW0t2SRqZ+rWl5T5H0+7SOPlMa90RJD6ZlsF7Sf3VXXxp2SHp9z6ThH5E0alfLvDTuhyUtT+vrx5KO7G4edRcR/fIBvAbYQvHhdRjwJLC21G8zsF9qD2A+MJwiyB3ApNRvLHA28AqgBbgf+FppPquBs3ZRx+XA7aX284BfpeajgT+l6Q9Kw64CBpfqGlsa90bgmtR8GrAduDbVNrSbeV8E/BX4CDAA+CjwFKDUfw7wP8ArgVcBDwMXl8Z9oMv0fg9MTM0rgCeA15X6HZ+a7wf+GxgCTEjL84zU76pU0+S0boambjen5v9Lr3NAD8vztvTYHxgPrOmsExiR1usHgIHAhan9kNR/IfDbtNyHpvYvpn6taXlfn/odB/y59PomAiel6bYCy4FLe6jxYuCuVOOANO6BNSzz89L6f12az2eBnzclP80OcB/DvwY4AbgAmJEW8jHAh4B5peECOLXUPgu4oodpTgaWlNpXs+vgHwY8W1rxs4HLU/O/ALNKw+4HrANOK9W1q+D/BRiyi3lfBKwqte+fpnkoMCq9sYeW+l8I3Fcat2vwbwI+kcZfAXwJuAQ4ipc/ZI8AXgSGlcb7AnBjar4KuL/LdK8C5gGLgP8gfTB183oGUHxovLbU7RpeDv4HgIe7jPMgcFFqXgh8ttTvH4EfpebWtGzGlPo/DFzQQy2XAnN66Pdh4OfAsV2697bMfwhM7fJ+eA44stHZ6e/HX4soAjI2NW8B3gKcnNrLNpSanwMOAEi7aNcBbwKGUayMzbUWEBFPSfoZ8C5Jc4C3AdNT7849kc5hX5K0Bji8xsl3RMQLvQyz43VFxHPpCOYAiq3jIGB96gbFa1uzi2ktAs4F1lJs1RdShO0F4Kep/sOAP0TEs6XxngTaSu3dzeOkVM+Fkd713Wih2BKWxy8377Q8S/MuL89u13Nv/SUdDXyF4nXsn+pY3EOdN1F8AN6WDgtvBj4DHMmul/mRwHWSvlyallL9XV9XXfXbY/ykM/hvSs2LKIL/Fv42+D35N4otwRsi4kDg/RQro1MtP1+cmcZ7D/BgRKxL3Z+iWNkApPMKR1Bs9aF44+1fms6hXabbl59OrqHY+oyMiOHpcWBEvH4X015EsSxPS80PAKew8/J8ChghaVhpvFfz8mvqadp3U+wZLOg8Hu5GB8XhzZhStyNKzTstzx7mvae+AfwaGJfeB59m5/fBDhHx14i4OiLGA28EzgE+SO/LfA3Fbv/w0mNoRPy8gvp3y74Q/NMpdq3WAj8FJgGHAEtqnMYwYBvwR0mHA//cpf/TFOcMduX7FIcc04H/LXWfBbxD0pmSBgGXUbwxOlf0Y8B70wm3SRQBq0RErKcI25clHShpPxUnMjvn8TQwpnwCKyJWAs9TfIgtioitabh3kYIfEWtS/V9IJ7mOBaZSbPV6q+lLwHcpwj+ym/4vAncCV0naX9IxFIHq9APgaEnvlTRQ0j9QnAeYvxuLpifDgK3AtjTfj/Y0oKTTJb0hnbTbSnF48lINy/ybwJWSXp+mc5Ck91RQ+27r18GPiN9QhPanqX0rxQmpn6U3US2upgjtHylOPN3Zpf8XgM+ms7ef7KGO54E7KI6F7yx1X0ERov8ENgHvBN4ZEX9Jg0xP3bYA76P4AKnSB4HBwOMUhy+zgdGp373AMmCDpE2lcRYBz6SAd7YLeLQ0zIUUx8xPUZzM+teI+EktBUXE5yle508kjehmkI8DB1Hskt8E3ErxYUlEPEOxdb0MeIbiZOk5EbGpm+nsrk8C76U4X3M9cPsuhj2UYllupTgJuCjVCrtY5hExh+Jk7W3pG4+lFIeGDdd59tf6SNLngKMj4v3NrmVfIula4NCImNLsWvYl/XqLv7dIW66pFN8sWB9IOkbSsSqcSLFc5zS7rn2Ng99H6eKVNcAPI+L+ZtezDxhGcbj0J4rd7S8Dc5ta0T7Iu/pmGerTFl/SJEkr0mWbV1RVlJnV1x5v8dNXGb+huBx1LfAIxcUZj/c0zsiRI6O1tbXXaS9e3NN1E2b5mjhxYq/DrF69mk2bNnV7/UFZX67cO5HictEnACTdRnEtco/Bb21tpb29vdcJl656MrOkluy0tbX1Ogz0bVf/cHa+nHIt3VyKKmmapHZJ7R0dHX2YnZlVpe5n9SNiRkS0RURbS0tLvWdnZjXoS/DXsfN11GOo5pppM6uzvgT/EWCcihs4DKb4aey8asoys3ra45N7EbFd0seBH1P8jvrbEbGsssrMrG769Hv8iPgBxS+mzKwf8SW7Zhly8M0y5OCbZcjBN8uQg2+WIQffLEMOvlmGHHyzDDn4Zhly8M0y5OCbZcjBN8uQg2+WIQffLEMOvlmGHHyzDDn4Zhly8M0y5OCbZcjBN8uQg2+WIQffLEMOvlmG+nRf/Xp59NFHm12C2T7NW3yzDPUafEnflrRR0tJStxGS7pG0Mj0fXN8yzaxKtWzxbwQmdel2BbAgIsYBC1K7mfUTvQY/Iu4H/tCl83nAzNQ8E5hccV1mVkd7eow/KiLWp+YNwKieBpQ0TVK7pPaOjo49nJ2ZVanPZ/UjIiTFLvrPAGYAtLW19TjcTkUN3Cu/bDDbZ+zpFv9pSaMB0vPG6koys3rb0+DPA6ak5inA3GrKMbNGqOXrvFuBB4HXSloraSrwReBsSSuBs1K7mfUTvR5MR8SFPfQ6s+JadhgyZEi9Jm1m+Mo9syw5+GYZcvDNMuTgm2XIwTfLkINvlqG98trYF154odklmO3TvMU3y5CDb5YhB98sQw6+WYYcfLMM7ZVn9f0jHbP68hbfLEMOvlmGHHyzDO2Vx/iSml2C2T7NW3yzDDn4Zhly8M0y5OCbZcjBN8uQg2+Wob3y67yVK1c2uwSzvc7YsWMrm5a3+GYZquUvtI6QdJ+kxyUtkzQ9dR8h6R5JK9PzwfUv18yqUMsWfztwWUSMB04CPiZpPHAFsCAixgELUruZ9QO9Bj8i1kfEo6n5WWA5cDhwHjAzDTYTmFyvIs2sWrt1jC+pFTgeeAgYFRHrU68NwKgexpkmqV1Se0dHRx9KNbOq1Bx8SQcAdwCXRsTWcr+ICCC6Gy8iZkREW0S0tbS09KlYM6tGTV/nSRpEEfpbIuLO1PlpSaMjYr2k0cDGqoryB4RZfdVyVl/ADcDyiPhKqdc8YEpqngLMrb48M6uHWrb4pwAfAH4l6bHU7dPAF4FZkqYCTwLn16dEM6tar8GPiAeAnu6McWa15ZhZI/jKPbMMOfhmGdorf6SzePHiZpdgttdpa2urbFre4ptlyME3y5CDb5ahvfIY/5JLLml2CWZ7nYsvvriyaXmLb5YhB98sQw6+WYYcfLMMOfhmGXLwzTLk4JtlyME3y5CDb5YhB98sQw6+WYYcfLMMOfhmGXLwzTLk4JtlyME3y5CDb5YhB98sQ7X8d94QSQ9L+oWkZZKuTt2PkvSQpFWSbpc0uP7lmlkVatni/xk4IyKOAyYAkySdBFwLfDUixgKbgan1K9PMqtRr8KOwLbUOSo8AzgBmp+4zgcl1qdDMKlfTMb6kAemfcjcC9wC/BbZExPY0yFrg8B7GnSapXVJ7R0dHFTWbWR/VFPyIeDEiJgBjgBOBY2qdQUTMiIi2iGhraWnZwzLNrEq7dVY/IrYA9wEnA8Mldd6XfwywruLazKxOajmr3yJpeGoeCpwNLKf4AHh3GmwKMLdeRZpZtWr5J53RwExJAyg+KGZFxHxJjwO3SboGWALcUMc6zaxCvQY/In4JHN9N9ycojvfNrJ/xlXtmGXLwzTLk4JtlyME3y5CDb5YhB98sQw6+WYYcfLMMOfhmGXLwzTLk4JtlyME3y5CDb5YhB98sQw6+WYYcfLMMOfhmGXLwzTLk4JtlyME3y5CDb5YhB98sQw6+WYYcfLMMOfhmGao5+OmvspdImp/aj5L0kKRVkm6XNLh+ZZpZlXZniz+d4s8yO10LfDUixgKbgalVFmZm9VNT8CWNAd4BfCu1CzgDmJ0GmQlMrkeBZla9Wrf4XwMuB15K7YcAWyJie2pfCxze3YiSpklql9Te0dHRp2LNrBq9Bl/SOcDGiFi8JzOIiBkR0RYRbS0tLXsyCTOrWK9/kw2cApwr6e3AEOBA4DpguKSBaas/BlhXvzLNrEq9bvEj4sqIGBMRrcAFwL0R8T7gPuDdabApwNy6VWlmlerL9/ifAj4haRXFMf8N1ZRkZvVWy67+DhGxEFiYmp8ATqy+JDOrN1+5Z5YhB98sQw6+WYYcfLMMOfhmGXLwzTLk4JtlyME3y5CDb5YhB98sQw6+WYYcfLMMOfhmGXLwzTLk4JtlyME3y5CDb5YhB98sQw6+WYYcfLMMOfhmGXLwzTLk4JtlyME3y5CDb5ahmv5JR9Jq4FngRWB7RLRJGgHcDrQCq4HzI2Jzfco0syrtzhb/9IiYEBFtqf0KYEFEjAMWpHYz6wf6sqt/HjAzNc8EJve9HDNrhFqDH8DdkhZLmpa6jYqI9al5AzCquxElTZPULqm9o6Ojj+WaWRVq/bfcUyNinaRXAfdI+nW5Z0SEpOhuxIiYAcwAaGtr63YYM2usmrb4EbEuPW8E5lD8PfbTkkYDpOeN9SrSzKrVa/AlvVLSsM5m4K3AUmAeMCUNNgWYW68izaxatezqjwLmSOoc/rsR8SNJjwCzJE0FngTOr1+ZZlalXoMfEU8Ax3XT/RngzHoUZWb15Sv3zDLk4JtlyME3y5CDb5YhB98sQw6+WYYcfLMMOfhmGXLwzTLk4JtlyME3y5CDb5YhB98sQw6+WYYcfLMMOfhmGXLwzTLk4JtlyME3y5CDb5YhB98sQw6+WYYcfLMMOfhmGXLwzTJUU/AlDZc0W9KvJS2XdLKkEZLukbQyPR9c72LNrBq1bvGvA34UEcdQ/J3WcuAKYEFEjAMWpHYz6wdq+bfcg4A3AzcARMRfImILcB4wMw02E5hcryLNrFq1bPGPAjqA70haIulb6e+yR0XE+jTMBop/1f0bkqZJapfU3tHRUU3VZtYntQR/IHAC8I2IOB74E1126yMigOhu5IiYERFtEdHW0tLS13rNrAK1BH8tsDYiHkrtsyk+CJ6WNBogPW+sT4lmVrVegx8RG4A1kl6bOp0JPA7MA6akblOAuXWp0MwqN7DG4f4JuEXSYOAJ4EMUHxqzJE0FngTOr0+JZla1moIfEY8Bbd30OnN3Zvb888+zdOnS3RnF9nETJ07c0Xz99ddXPv0TTjih8mk2y5IlS3od5rnnnqtpWr5yzyxDDr5Zhhx8swzVenLPrC62bdu2o3nx4sVNrCQv3uKbZcjBN8uQiqttGzQzqYPiO/+RwKaGzbh7e0MN4Dq6ch072906joyIXq+Nb2jwd8xUao+I7q4LyKoG1+E6mlWHd/XNMuTgm2WoWcGf0aT5lu0NNYDr6Mp17KwudTTlGN/Mmsu7+mYZcvDNMtTQ4EuaJGmFpFWSGnZXXknflrRR0tJSt4bfHlzSEZLuk/S4pGWSpjejFklDJD0s6RepjqtT96MkPZTWz+3p/gt1J2lAup/j/GbVIWm1pF9JekxSe+rWjPdIQ25l37DgSxoAfB14GzAeuFDS+AbN/kZgUpduzbg9+HbgsogYD5wEfCwtg0bX8mfgjIg4DpgATJJ0EnAt8NWIGAtsBqbWuY5O0ylu2d6pWXWcHhETSt+bN+M90phb2UdEQx7AycCPS+1XAlc2cP6twNJS+wpgdGoeDaxoVC2lGuYCZzezFmB/4FHg7ymuEBvY3fqq4/zHpDfzGcB8QE2qYzUwsku3hq4X4CDgd6ST7vWso5G7+ocDa0rta1O3Zqnp9uD1IqkVOB54qBm1pN3rxyhuknoP8FtgS0RsT4M0av18DbgceCm1H9KkOgK4W9JiSdNSt0avlz7dyn53+OQeu749eD1IOgC4A7g0IrY2o5aIeDEiJlBscU8Ejqn3PLuSdA6wMSL2ht/jnhoRJ1Acin5M0pvLPRu0Xvp0K/vd0cjgrwOOKLWPSd2apSm3B5c0iCL0t0TEnc2sBSCKf0W6j2KXerikzns0NGL9nAKcK2k1cBvF7v51TaiDiFiXnjcCcyg+DBu9Xhp2K/tGBv8RYFw6YzsYuIDiFt3N0vDbg0sSxV+RLY+IrzSrFkktkoan5qEU5xmWU3wAvLtRdUTElRExJiJaKd4P90bE+xpdh6RXShrW2Qy8FVhKg9dLNPJW9vU+adLlJMXbgd9QHE9+poHzvRVYD/yV4lN1KsWx5AJgJfATYEQD6jiVYjftl8Bj6fH2RtcCHAssSXUsBT6Xur8GeBhYBXwPeEUD19FpwPxm1JHm94v0WNb53mzSe2QC0J7WzfeBg+tRhy/ZNcuQT+6ZZcjBN8uQg2+WIQffLEMOvlmGHHyzDDn4Zhn6f5Mtynay14XmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "#spawn game instance for tests\n",
    "env = gym.make(\"BreakoutDeterministic-v0\") #create raw env\n",
    "\n",
    "tmp = env.reset()\n",
    "print (tmp.ndim, tmp.shape, tmp.dtype)\n",
    "print (env.action_space, env.observation_space)\n",
    "\n",
    "env = PreprocessAtari(env)\n",
    "\n",
    "observation_shape = env.observation_space.shape\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "obs = env.reset()\n",
    "print (obs.ndim, obs.shape, obs.dtype)\n",
    "\n",
    "#test observation\n",
    "assert obs.ndim == 3, \"observation must be [batch, time, channels] even if there's just one channel\"\n",
    "assert obs.shape == observation_shape\n",
    "assert obs.dtype == 'float32'\n",
    "assert len(np.unique(obs))>2, \"your image must not be binary\"\n",
    "assert 0 <= np.min(obs) and np.max(obs) <=1, \"convert image pixels to (0,1) range\"\n",
    "\n",
    "print (\"Formal tests seem fine. Here's an example of what you'll get.\")\n",
    "\n",
    "plt.title(\"what your network gonna see\")\n",
    "\n",
    "print (obs[:, :, 0].shape)\n",
    "plt.imshow(obs[:, :, 0], interpolation='none', cmap='gray');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frame buffer\n",
    "\n",
    "Our agent can only process one observation at a time, so we gotta make sure it contains enough information to fing optimal actions. For instance, agent has to react to moving objects so he must be able to measure object's velocity.\n",
    "\n",
    "To do so, we introduce a buffer that stores 4 last images. This time everything is pre-implemented for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from framebuffer import FrameBuffer\n",
    "def make_env():\n",
    "    env = gym.make(\"BreakoutDeterministic-v4\")\n",
    "    env = PreprocessAtari(env)\n",
    "    env = FrameBuffer(env, n_frames=4, dim_order='tensorflow')\n",
    "    return env\n",
    "\n",
    "env = make_env()\n",
    "env.reset()\n",
    "n_actions = env.action_space.n\n",
    "state_dim = env.observation_space.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAEICAYAAADBfBG8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEbhJREFUeJzt3X2wVPV9x/H3RwjGQSL4EKpwFbDEVtOEGGKc+JC0JojWBu2DwbbRJLZoI9NkNJOIdqx1mhljAtZMqhYjDXZUtDE+tKNGQlOdjvEBlOADPiBihEFQUFA0GuDbP87vxsP1Lnfv/nbds+vnNbNzz/7Ow/7O3P3c3znnnv2uIgIza9wu7e6AWadziMwyOURmmRwis0wOkVkmh8gsk0PUhSTtL+k1SUPa3Zf3Aocog6Tpku6XtEXS+jT9VUlqZ78i4lcRsXtEbGtnP94rHKIGSToHuAz4LvA7wGjgTOAIYFgbu2bvtojwY5APYA9gC/BnAyz3x8DDwGbgeeDC0rxxQABfTvNepgjhJ4BlwCvAD/ps7yvA8rTsT4EDarxu77aHpuf/C/wzcC/wGvBfwF7AtalvDwLjSutflvq0GVgCHFWatxswP/VhOfBNYHVp/n7ATcCLwLPA37f799Xy90O7O9CJD2AqsLX3TbqT5T4D/AHFiP8RYB1wYprX+0a/Eng/MAX4NXAL8EFgDLAe+HRafhqwAvh9YCjwD8C9NV63vxCtAA5MfwAeB54CPpu2dQ3w76X1/zqFbChwDvAC8P4072LgbmAUMDYFfnWat0sK3QUUo/EEYCVwbLt/Zy19P7S7A534SG+yF/q03ZtGjzeAo2us9y/ApWm6940+pjR/A/CF0vObgK+n6TuA00vzdgFep5/RqEaIzi/Nnw3cUXr+J8DSnezvy8BH0/QOoQD+phSiTwK/6rPurHJAu/Hhc6LGbAD2ljS0tyEiPhURI9O8XQAkfVLSzyW9KGkTxeHa3n22ta40/UY/z3dP0wcAl0l6RdIrwEZAFCNWPep9HSR9Q9JySZvSa+1R6vd+FId6vcrTBwD79fYxrXsexfli13KIGvML4E2KQ6yduQ64DeiJiD0oDt0avXL3PHBGRIwsPXaLiHsb3F6/JB1FcZ5zMjAq/WHYxNv9XktxGNerp08fn+3TxxERcXwz+1g1DlEDIuIV4J+AyyX9uaQRknaRNAkYXlp0BLAxIn4t6TDgLzNe9kpglqRDACTtIekvMrZXywiK870XgaGSLgA+UJp/Y+rHKEljgJmleQ8Ar0r6lqTdJA2R9GFJn2hBPyvDIWpQRFwCnE3xV3tdevwb8C2K8yOArwIXSXqV4mT7xozXuxn4DrBA0mbgUeC4hnegtp8Cd1JceHiO4mJH+ZDtImA1xZW3nwE/phiVieL/UicAk9L8l4AfUhwOdi2lkz+zhkj6O2B6RHy63X1pF49ENiiS9pV0RDp8PYjiEvjN7e5XOw0deBGzHQyjOGwdT3FJfwFweVt71GYtO5yTNJXiP99DgB9GxMUteSGzNmtJiNLdw08Bn6M4CX0QOCUiHm/6i5m1WasO5w4DVkTESgBJCyj+p9JviCT56oZV0UsRsc9AC7XqwsIYdrwsupo+/1mXNEPSYkmLW9QHs1zP1bNQ2y4sRMRcYC54JLLO1qqRaA073g4yNrWZdZ1WhehBYKKk8ZKGAdMp7iEz6zotOZyLiK2SZlLcQjIEmBcRj7XitVplzpw5g17n7LPPztpG3/WbtY1cffsw0H62og+D7dO7qWXnRBFxO3B7q7ZvVhW+Y6FOrRgl2jHaWfP53jmzTB6JbNA8+u3II5FZJo9ENqCBrny910cmj0RmmTwS1akZf22rsg1rLo9EZpkcIrNMlShU4ru4raKWRMTkgRbySGSWqRIXFsaOHdvWGwjN+lPve9IjkVkmh8gsk0NklskhMsvkEJllajhEknrSF1g9LukxSV9L7RdKWiNpaXp09XfTmOVc4t4KnBMRD0kaASyRtDDNuzQivpffPbPqazhEEbGW4lvTiIhXJS2n/q8+NOsaTTknkjQO+Bhwf2qaKWmZpHmSRtVY57cVULds2dKMbpi1RXaIJO3O299yvRm4guKr3idRjFSz+1svIuZGxOSImDx8+PD+FjHrCFkhkvQ+igBdGxE/AYiIdRGxLSK2A1dRFLc361o5V+cEXA0sj4g5pfZ9S4udRPHdomZdK+fq3BHAF4FHJC1NbecBp6Rv0Q5gFXBGVg/NKi7n6tz/Aepnlque2ntKJT4KMRB/TMJaoVn1Knzbj1kmh8gsk0NklskhMsvkEJllcojMMjlEZpkcIrNMDpFZJofILJNDZJbJITLL5BCZZXKIzDI5RGaZsj9PJGkV8CqwDdgaEZMl7QncAIyj+HTryRHxcu5rmVVRs0aiP4yISaVvFTsXWBQRE4FF6blZV2rV4dw0YH6ang+c2KLXMWu7ZoQogLskLZE0I7WNThVSAV4ARjfhdcwqqRk1Fo6MiDWSPggslPREeWZERH9fbJwCNwNg1Kh+i6SadYTskSgi1qSf64GbKYo1ruutP5d+ru9nPVdAta6QWwF1ePpGCCQNB6ZQFGu8DTgtLXYacGvO65hVWe7h3Gjg5qIYKkOB6yLiTkkPAjdKOh14Djg583XMKisrRBGxEvhoP+0bgGNytm3WKXzHglmmjqiAet/Uqe3ugnWhe5u0HY9EZpkcIrNMDpFZJofILJNDZJapI67Obf/dze3ugllNHonMMjlEZpkcIrNMDpFZJofILJNDZJapIy5xb/zA6+3ugllNHonMMjlEZpkaPpyTdBBFldNeE4ALgJHA3wIvpvbzIuL2hntoVnENhygingQmAUgaAqyhqPbzZeDSiPheU3poVnHNOpw7BngmIp5r0vbMOkazrs5NB64vPZ8p6VRgMXBObjH7jb/3Vs7qZv17qTmbyR6JJA0DPg/8Z2q6AjiQ4lBvLTC7xnozJC2WtHjLli253TBrm2Yczh0HPBQR6wAiYl1EbIuI7cBVFBVR38EVUK1bNCNEp1A6lOstH5ycRFER1axrZZ0TpdLBnwPOKDVfImkSxbdFrOozz6zr5FZA3QLs1afti1k9MuswHXHv3HXb9293F6wLTWnSdnzbj1kmh8gsk0NklskhMsvkEJll6oirc28tuLDdXbBuNKU5X67ikcgsk0NklskhMsvkEJllcojMMjlEZpk64hL3/9x5eLu7YF3ohClzmrIdj0RmmRwis0wOkVmmukIkaZ6k9ZIeLbXtKWmhpKfTz1GpXZK+L2mFpGWSDm1V582qoN6R6EfA1D5t5wKLImIisCg9h6L6z8T0mEFRQsusa9UVooi4B9jYp3kaMD9NzwdOLLVfE4X7gJF9KgCZdZWcc6LREbE2Tb8AjE7TY4DnS8utTm07cPFG6xZNubAQEUFRImsw67h4o3WFnBCt6z1MSz/Xp/Y1QE9pubGpzawr5YToNuC0NH0acGup/dR0le5wYFPpsM+s69R124+k64HPAHtLWg38I3AxcKOk04HngJPT4rcDxwMrgNcpvq/IrGvVFaKIOKXGrGP6WTaAs3I6ZdZJfMeCWSaHyCyTQ2SWySEyy+QQmWVyiMwyOURmmRwis0wOkVkmh8gsk0NklskhMsvkEJllcojMMjlEZpkcIrNMDpFZpgFDVKP66XclPZEqnN4saWRqHyfpDUlL0+PKVnberArqGYl+xDurny4EPhwRHwGeAmaV5j0TEZPS48zmdNOsugYMUX/VTyPirojYmp7eR1EWy+w9qRnnRF8B7ig9Hy/pYUl3Szqq1kqugGrdIuub8iSdD2wFrk1Na4H9I2KDpI8Dt0g6JCI29103IuYCcwF6enoGVT3VrEoaHokkfQk4AfirVCaLiHgzIjak6SXAM8CHmtBPs8pqKESSpgLfBD4fEa+X2veRNCRNT6D4epWVzeioWVUNeDhXo/rpLGBXYKEkgPvSlbijgYsk/QbYDpwZEX2/ksWsqwwYohrVT6+usexNwE25nTLrJL5jwSyTQ2SWySEyy+QQmWVyiMwyOURmmRwis0wOkVkmh8gsk0NklskhMsvkEJllcojMMjlEZpkcIrNMDpFZJofILFOjFVAvlLSmVOn0+NK8WZJWSHpS0rGt6rhZVTRaARXg0lKl09sBJB0MTAcOSetc3lu4xKxbNVQBdSemAQtS6axngRXAYRn9M6u8nHOimamg/TxJo1LbGOD50jKrU9s7uAKqdYtGQ3QFcCAwiaLq6ezBbiAi5kbE5IiYPHz48Aa7YdZ+DYUoItZFxLaI2A5cxduHbGuAntKiY1ObWddqtALqvqWnJwG9V+5uA6ZL2lXSeIoKqA/kddGs2hqtgPoZSZOAAFYBZwBExGOSbgQepyh0f1ZEbGtN182qoakVUNPy3wa+ndMps07iOxbMMjlEZpkcIrNMDpFZJofILJNDZJbJITLL5BCZZXKIzDI5RGaZHCKzTA6RWSaHyCyTQ2SWySEyy+QQmWVyiMwyNVoB9YZS9dNVkpam9nGS3ijNu7KVnTerggE/Hk5RAfUHwDW9DRHxhd5pSbOBTaXln4mISc3qoFnV1VNj4R5J4/qbJ0nAycAfNbdbZp0j95zoKGBdRDxdahsv6WFJd0s6qtaKroBq3aKew7mdOQW4vvR8LbB/RGyQ9HHgFkmHRMTmvitGxFxgLkBPT09k9sOsbRoeiSQNBf4UuKG3LRWy35CmlwDPAB/K7aRZleUczn0WeCIiVvc2SNqn96tUJE2gqIC6Mq+LZtVWzyXu64FfAAdJWi3p9DRrOjseygEcDSxLl7x/DJwZEfV+LYtZR2q0AioR8aV+2m4Cbsrvllnn8B0LZpkcIrNMDpFZJofILJNDZJbJITLL5BCZZXKIzDLl3oDaFJuGbOe/R77W0Lr3TZ2a9dqH33ln1vrWuT51111N2Y5HIrNMDpFZJofILFMlzoly+JzG2s0jkVmmjh+JzBrVrKMYRbS/vIGk9nfC7J2WRMTkgRby4ZxZpno+Ht4j6eeSHpf0mKSvpfY9JS2U9HT6OSq1S9L3Ja2QtEzSoa3eCbN2qmck2gqcExEHA4cDZ0k6GDgXWBQRE4FF6TnAcRQFSiYCM4Armt5rswoZMEQRsTYiHkrTrwLLgTHANGB+Wmw+cGKangZcE4X7gJGS9m16z80qYlDnRKmc8MeA+4HREbE2zXoBGJ2mxwDPl1Zbndr6buu3FVAH2WezSqk7RJJ2p6jk8/W+FU2juMQ3qCtsETE3IibXc/XDrMrqCpGk91EE6NqI+ElqXtd7mJZ+rk/ta4Ce0upjU5tZV6rn6pyAq4HlETGnNOs24LQ0fRpwa6n91HSV7nBgU+mwz6z7RMROH8CRFIdqy4Cl6XE8sBfFVbmngZ8Be6blBfwrRR3uR4DJdbxG+OFHBR+LB3rvRoTvWDDbCd+xYPZucIjMMjlEZpkcIrNMVfk80UvAlvSzW+xN9+xPN+0L1L8/B9SzsUpcnQOQtLib7l7opv3ppn2B5u+PD+fMMjlEZpmqFKK57e5Ak3XT/nTTvkCT96cy50RmnapKI5FZR3KIzDK1PUSSpkp6MhU2OXfgNapH0ipJj0ha2vtJ3VqFXKpI0jxJ6yU9Wmrr2EI0NfbnQklr0u9oqaTjS/Nmpf15UtKxg37Bem71btUDGELxkYkJwDDgl8DB7exTg/uxCti7T9slwLlp+lzgO+3u5076fzRwKPDoQP2n+BjMHRQfeTkcuL/d/a9zfy4EvtHPsgen992uwPj0fhwymNdr90h0GLAiIlZGxFvAAopCJ92gViGXyomIe4CNfZo7thBNjf2pZRqwICLejIhngRUU78u6tTtEdRU16QAB3CVpiaQZqa1WIZdOkVWIpqJmpkPQeaXD6+z9aXeIusWREXEoRc29syQdXZ4ZxXFDx/4vodP7n1wBHAhMAtYCs5u14XaHqCuKmkTEmvRzPXAzxeFArUIunaKrCtFExLqI2BYR24GrePuQLXt/2h2iB4GJksZLGgZMpyh00jEkDZc0oncamAI8Su1CLp2iqwrR9DlvO4nidwTF/kyXtKuk8RSVex8Y1MYrcCXleOApiqsi57e7Pw30fwLF1Z1fAo/17gM1CrlU8QFcT3GI8xuKc4LTa/WfBgrRVGR//iP1d1kKzr6l5c9P+/MkcNxgX8+3/ZhlavfhnFnHc4jMMjlEZpkcIrNMDpFZJofILJNDZJbp/wELY+HY9h8UEwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAACDCAYAAACdg+BGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAE4tJREFUeJzt3Xu4VXWdx/H3R45oKGIKQwgapJbRM95CIzPzyfJCljrPZJoplUVlNebUlN2prLQms55pSiYNb2lmloyTqZlkVxVTUUQULQMERETFTA34zh+/34HFdl/P2WdvWOfzep79nLXWb12+67vX+u7fXnvtfRQRmJnZ5m+LbgdgZmbt4YJuZlYSLuhmZiXhgm5mVhIu6GZmJeGCbmZWEi7oJSRpvKSQ1NPtWFoh6QRJ1w3QureSdI+kMX1c/gOSlkt6StKO7Y5vILR6HEg6Q9KjkpYNdGx1YviepM82Oe9MSWe0sO49Jf2+79Ft+lzQmyBptqRVkrbq4DZD0m6d2l6nVSs2EXFJRBw6QJucBtwUEUsr4hgqab6kxXVi3RI4Gzg0IraNiJUDFGPXSNoF+CgwMSJe1MyLgaTpki5uZxwR8f6I+FI71lV5DkXEXOBxSW9ux/o3RS7oDUgaD7wWCOAtXQ1mE6Jkczp+3g9cVGX6fwArGiw7GtgamFetcXN7J1TDLsDKiHikWwFIGtKBzVwCvK8D2+mOiPCjzgP4HPA7Ug/t6oq2HYH/BZ4EbgXOAH5baN8DuB54DFgAHFtomwl8B/g/YDVwM7BrbruJ9ALyN+Ap4G1V4toC+AzwEPAIcCEwIreNz8tPAx4GlgIfKyy7PzAnx70cOLvQNhn4PfA4cCdwcKFtNvDlnI+/A58A5lTEdRowKw+/Cbg9b2cRML0w319zjE/lx6uBd1bk74Cc1yfy3wMqYvlSjmU1cB0wssZzuEuOt6di+gRgPnAEsLjGsi/Nz0NvrL/K0wP4IHA/8Oc87Vt5P58EbgNeW1jPdODHwMU53rvyuj+Zn79FpHcAvfOPAM7Lz90S0rE1JLftBvw65+VR4Ec1Yu89DnrqrRN4Q87PuryPM6s9PxXrPhx4DvhHbr8zT98JmEU65hcC761zbs0Evgv8POf4DXnaGYV5Pp7jfRh4T45pt76eQ8DYvK9bdbu2DEi96nYAm/ojH5SnAK/MB+/oQttl+TEMmJhPyt/mtm3y+LuAHmCffPJNzO0zgZWk4tpD6jlcVlj3+gO3RlzvzrG9BNgWuBK4KLf1nsiX5jj+mdQLfUNu/wNwYh7eFpich8fmmKaQXjDemMdH5fbZ+UR/RY55RD6Rdi/EdStwXB4+OG97C2BP0ovH0RUx9hSWfWchfzsAq4AT87aOz+M7FmJ5gFQUX5DHz6yRqzcB86pMvxo4JsdZtaDXiTVIL9Y7AC/I095BepHvIV2+WAZsndumA88Ah+X2C4E/A58GtgTeS35hyPP/FDg3P3//BNwCvC+3XZqX24L0zuHAZuJusM6NclBtn6usfzpwccW0m4D/znHtTTruXl9j+ZmkF6XXFPZlJrmgk140lpGOt2GkF8PKgt7yOUR6wd2z27VlIB5dD2BTfgAHkor4yDx+L3BaHh6S215WmH99Dx14G/CbivWdC3w+D88Evl9omwLcWxhvVNBvAE4pjL8sx9NTOBn3KLR/DTgvD98EfIGKHi2px31RxbRrgal5eDbwxYr2i4HP5eHdSQV+WI2YzwG+mYefVzDYuKCfCNxSsfwfgHcWYvlMoe0U4Bc1tnsC8MeKaccA1+Thg+lbQa9aqArzrAL2ysPTgesLbW8m9Rx7e93D8zq3J13ieZb8QpHbjwduzMMXAjOAcQ22vz7uJta5UQ6q7XOV9U+nUNCBnYG1wPDCtK8CM2ssPxO4sMq03oJ+PvDVQttuPL+gt3wOkd6dHFQvd5vrY3O6BtoNU4HrIuLRPP7DPA1gFOlEWVSYvzj8YuBVkh7vfZAKy4sK8xTvJnia1Ftu1k6kyy29HmLDiVstnofyMgAnk3q290q6VdKRhZjfWhHzgUDxzpDiOiHl5Pg8/HbgZxHxNICkV0m6UdIKSU+QrmOP7OP+9e7D2MJ4s/lbRSqY5Li2Ib3A/VuTsdSyUS4kfSx/wPpEzt0INt7f5YXhvwOPRsTawjikfXgxqde+tPA8nEvqVUO6DCHgFknzJL27iVgbrbMddgIei4jVhWmVz1mlyuOpcn21zq9efTmHhpMuKZZOGT7MGRCSXgAcCwwp3Ma1FbC9pL2Au4E1wDjgvty+c2EVi4BfR8QbByjEh0knaa9dcjzLc0y98dxbaH8YICLuB47PH2r+C3BFvhVvEamH/t46242K8euBUZL2JhX20wptPwT+CzgiIp6RdA4bClzlehrtX+8+/KLBctXMBSZI6omINaR3EuOB30gCGAqMyM/z5Ij4S5PrXb8Pkl5LKrSHkC7vrJO0ilR4W7WI1JsemePdeKMRy0iXaJB0IPBLSTdFxMK+rrOKRs9PtXkeBnaQNLxQ1Hch9Yj7sp2lbDiWYePzq08kjSU93wv6u65NkXvotR1Nevs4kXQtcG/g5cBvgJNyz+pKYLqkYZL2AE4qLH818FJJJ0raMj/2k/TyJre/nHR9vJZLgdMkTZC0LfAV0odjxZP1szm2V5Cu5f8IQNI7JI2KiHVs6KmsI10+ebOkwyQNkbS1pIMlFU+qjUTEP0gf9n2ddD35+kLzcFKP7RlJ+5N68L1W5G3W2sefk/L3dkk9kt5Gei6urpOTWjEuJn3esH+edDepOPQ+r+8h5Xtv6vcY6xlOekFdAfRI+hywXV9WFOnWyuuAb0jaTtIWknaV9DoASW8tPCerSEVxXX/WWUWj5wdSzsb33u0UEYtIH6h/NR87e5LeDfb11sbLgXdJermkYUBT96dXxFcZ/+tIH2w/28eYNmku6LVNBX4QEX+NiGW9D1KP84R8q9qHSG+rl5FuibuU1Asi91AOBY4j9VyWAWeRevnNmA5ckN8eH1ul/fy8zZtIH649A3y4Yp5fkwrZDcB/RkTvl3YOB+ZJeop0Z8ZxEfH3fEIeBXyKdEIvIt3W1+g4+SHpDoUfV7ygnAJ8UdJq0t1Cl/c25MsyXwZ+l/dxcnGFke71PpL04eJKUu/3yMLlr1adS7ouT0SsqXhOHwPW5fG1dddS27Wkdw/3kS4zPEPfXxwgdQ6GAveQivYVbLj0tR9wc37+ZgGnRsSD/VznRho9P9mP89+Vkv6Uh48nvft5mPQh7Ocj4pdNxFYthmuAbwM3ko7jP+amZovxdJ5/Dp0AfK8v8WwOlD8ksDaQdBbwooiY2nBm66j8pbDbgUOi4stFtnnI727vJt1y2Mxlo8rl9wTOjYhXtz24TYQLej/kyyxDSfcU70e6TPCeiPhZVwMzKwlJx5DOq2HABaR3Ukd3N6pNly+59M9w0nX0v5GuT38DuKqrEZmVy/tIX7x6gPSZ1ge6G86mrV89dEmHk67BDiHdD3pmuwIzM7PW9Lmg599duI/0bcLFpG8IHh8R97QvPDMza1Z/LrnsDyyMiAcj4jnSV+CPak9YZmbWqv58sWgsG9+WtRh4Vb0FRu4wJMbvvOX68fvmDuvH5gefl+75dN1257M19fLpXLbGx2Z7VebztrnPPhoRoxotN+DfFJU0jfSrf+wytodbrt3wZa/Ddtp7oDdfKtdee0fdduezNfXy6Vy2xsdme1Xmc8iYhZU/g1FVfy65LGHjr+KOo8pXfCNiRkRMiohJo3bsxM8dm5kNTv0p6LcCu+evng8lfSNyVnvCMjOzVvX5kktErJH0IdJXnocA50dE1f/oYmZmA69f19Aj4uekb3GZmVmX+ZuiZmYl4YJuZlYSLuhmZiXhgm5mVhIu6GZmJeGCbmZWEl39J9EH3PlcNzdfOs5n+ziX7eV8doZ76GZmJdHVHvqwIaX8x9td43y2j3PZXs5nZ7iHbmZWEl3toY/qWd3NzZeO89k+zmV7OZ+d4R66mVlJdLWHvmLN8G5ufjP0SN1W57NVtfPpXLbKx2Z71c9nLe6hm5mVhAu6mVlJ+EPREnE+28e5bC/nszO6WtC3YF03N186zmf7OJft5Xx2RsNLLpJ2lnSjpHskzZN0ap6+g6TrJd2f/75w4MM1M7NamrmGvgb4aERMBCYDH5Q0ETgduCEidgduyONmZtYlDS+5RMRSYGkeXi1pPjAWOAo4OM92ATAb+EQrG7982X6tzD7onbTdNXXbnc/W1Munc9kaH5vt1SiftbR0l4uk8cA+wM3A6FzsAZYBo/sUgZmZtUXTBV3StsBPgI9ExJPFtogIIGosN03SHElzVqxc269gzcystqbucpG0JamYXxIRV+bJyyWNiYilksZQ46tNETEDmAEwaa+tNyr6e45Y0ufA7fmcz/ZxLtvL+eyMZu5yEXAeMD8izi40zQKm5uGpwFXtD8/MzJrVTA/9NcCJwF2S7sjTPgWcCVwu6WTgIeDYVjd++fx9W11kUPvK6Ll1253P1tTLp3PZGh+b7dUon7U0c5fLbwHVaD6kT1vNdn37HY1nsg0ert/sfLaoTj6dyxb52GyvBvmsxb/lYmZWEi7oZmYl4YJuZlYSLuhmZiXhgm5mVhIu6GZmJeGCbmZWEi7oZmYl4YJuZlYSLuhmZiXhgm5mVhIu6GZmJeGCbmZWEi7oZmYl4YJuZlYSLuhmZiXhgm5mVhJNF3RJQyTdLunqPD5B0s2SFkr6kaShAxemmZk10koP/VRgfmH8LOCbEbEbsAo4uZ2BmZlZa5oq6JLGAW8Cvp/HBbweuCLPcgFw9EAEaGZmzWm2h34O8HFgXR7fEXg8Itbk8cXA2DbHZmZmLWhY0CUdCTwSEbf1ZQOSpkmaI2nOipVr+7IKMzNrQk8T87wGeIukKcDWwHbAt4DtJfXkXvo4YEm1hSNiBjADYNJeW0dbojYzs+dp2EOPiE9GxLiIGA8cB/wqIk4AbgT+Nc82FbhqwKI0M7OG+nMf+ieAf5e0kHRN/bz2hGRmZn3RzCWX9SJiNjA7Dz8I7N/+kMzMrC/8TVEzs5JwQTczKwkXdDOzknBBNzMrCRd0M7OScEE3MysJF3Qzs5JwQTczKwkXdDOzknBBNzMrCRd0M7OScEE3MysJF3Qzs5JwQTczKwkXdDOzknBBNzMrCRd0M7OSaKqgS9pe0hWS7pU0X9KrJe0g6XpJ9+e/LxzoYM3MrLZme+jfAn4REXsAewHzgdOBGyJid+CGPG5mZl3SsKBLGgEcRP4n0BHxXEQ8DhwFXJBnuwA4eqCCNDOzxprpoU8AVgA/kHS7pO9L2gYYHRFL8zzLgNEDFaSZmTXWTEHvAfYFvhsR+wB/o+LySkQEENUWljRN0hxJc1asXNvfeM3MrIZmCvpiYHFE3JzHryAV+OWSxgDkv49UWzgiZkTEpIiYNGrHIe2I2czMquhpNENELJO0SNLLImIBcAhwT35MBc7Mf69qtK7la4dyzqrxVdvWvP6VHPTNP9Rd/vd7DW20iVL7woqJG42P6Hm64TIH3Plc3fbBnNNiPitz2Shvl8yfBMCE4+a2P7DNUKNjs5l8OpcbVOYTFja1XMOCnn0YuETSUOBB4F2k3v3lkk4GHgKObXJdZmY2AJoq6BFxBzCpStMh7QpkyyefXd/rqWUCfgVvlXPaN43ypge26VAk5eB8dobS55kd2pi0GljQsQ1uukYCj3Y7iC5zDhLnwTmAxjl4cUSMarSSZi+5tMuCiKj/Uj0ISJoz2PPgHCTOg3MA7cuBf8vFzKwkXNDNzEqi0wV9Roe3t6lyHpyDXs6DcwBtykFHPxQ1M7OB40suZmYl0bGCLulwSQskLZQ0aH5qV9JfJN0l6Q5Jc/K00v+WvKTzJT0i6e7CtKr7reTb+diYK2nf7kXePjVyMF3Sknw83CFpSqHtkzkHCyQd1p2o20vSzpJulHSPpHmSTs3TB9uxUCsP7T0eImLAH8AQ4AHgJcBQ4E5gYie23e0H8BdgZMW0rwGn5+HTgbO6HecA7PdBpN/8ubvRfgNTgGsAAZOBm7sd/wDmYDrwsSrzTsznxVakXzh9ABjS7X1oQw7GAPvm4eHAfXlfB9uxUCsPbT0eOtVD3x9YGBEPRsRzwGWk31MfrEr/W/IRcRPwWMXkWvt9FHBhJH8Etu/94bfNWY0c1HIUcFlEPBsRfyb9eMf+AxZch0TE0oj4Ux5eTfrnOGMZfMdCrTzU0qfjoVMFfSywqDC+mPo7UyYBXCfpNknT8rTB+lvytfZ7sB0fH8qXE84vXG4rfQ4kjQf2AW5mEB8LFXmANh4P/lB04B0YEfsCRwAflHRQsTHS+6tBd6vRYN1v4LvArsDewFLgG90NpzMkbQv8BPhIRDxZbBtMx0KVPLT1eOhUQV8C7FwYH5enlV5ELMl/HwF+Snrb1NRvyZdQrf0eNMdHRCyPiLURsQ74Hza8jS5tDiRtSSpil0TElXnyoDsWquWh3cdDpwr6rcDukibkn+A9DpjVoW13jaRtJA3vHQYOBe4m7fvUPFtTvyVfErX2exZwUr7DYTLwROHteKlUXA8+hnQ8QMrBcZK2kjQB2B24pdPxtZskkf4f8fyIOLvQNKiOhVp5aPvx0MFPeaeQPtl9APh0tz917tA+v4T0SfWdwLze/QZ2BG4A7gd+CezQ7VgHYN8vJb2F/Afp+t/JtfabdEfDd/KxcRcwqdvxD2AOLsr7ODeftGMK838652ABcES3429TDg4kXU6ZC9yRH1MG4bFQKw9tPR78TVEzs5Lwh6JmZiXhgm5mVhIu6GZmJeGCbmZWEi7oZmYl4YJuZlYSLuhmZiXhgm5mVhL/D0hkAdpWwBB9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for _ in range(50):\n",
    "    obs, _, _, _ = env.step(env.action_space.sample())\n",
    "\n",
    "\n",
    "plt.title(\"Game image\")\n",
    "plt.imshow(env.render(\"rgb_array\"))\n",
    "plt.show()\n",
    "plt.title(\"Agent observation (4 frames left to right)\")\n",
    "plt.imshow(obs.transpose([0,2,1]).reshape([state_dim[0],-1]));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a network\n",
    "\n",
    "We now need to build a neural network that can map images to state q-values. This network will be called on every agent's step so it better not be resnet-152 unless you have an array of GPUs. Instead, you can use strided convolutions with a small number of features to save time and memory.\n",
    "\n",
    "You can build any architecture you want, but for reference, here's something that will more or less work:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](https://s17.postimg.org/ogg4xo51r/dqn_arch.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "from keras.layers import Conv2D, Dense, Flatten, Input\n",
    "\n",
    "class DQNAgent:\n",
    "    def __init__(self, name, state_shape, n_actions, epsilon=0, reuse=False):\n",
    "        \"\"\"A simple DQN agent\"\"\"\n",
    "        with tf.variable_scope(name, reuse=reuse):\n",
    "            \n",
    "#             < Define your network body here. Please make sure you don't use any layers created elsewhere >\n",
    "            self.network = keras.models.Sequential()\n",
    "            self.network.add(Conv2D(16, (3, 3), strides=2, activation='relu', input_shape=state_shape))\n",
    "            self.network.add(Conv2D(32, (3, 3), strides=2, activation='relu'))\n",
    "            self.network.add(Conv2D(64, (3, 3), strides=2, activation='relu'))\n",
    "            self.network.add(Flatten())\n",
    "            self.network.add(Dense(256, activation='relu'))\n",
    "            self.network.add(Dense(n_actions, activation='linear'))\n",
    "            \n",
    "            # prepare a graph for agent step\n",
    "            self.state_t = tf.placeholder('float32', [None,] + list(state_shape))\n",
    "            self.qvalues_t = self.get_symbolic_qvalues(self.state_t)\n",
    "            \n",
    "        self.weights = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope=name)\n",
    "        self.epsilon = epsilon\n",
    "\n",
    "    def get_symbolic_qvalues(self, state_t):\n",
    "        \"\"\"takes agent's observation, returns qvalues. Both are tf Tensors\"\"\"\n",
    "#         < apply your network layers here >\n",
    "#         qvalues = < symbolic tensor for q-values >\n",
    "        qvalues = self.network(state_t)\n",
    "\n",
    "        assert tf.is_numeric_tensor(qvalues) and qvalues.shape.ndims == 2, \\\n",
    "            \"please return 2d tf tensor of qvalues [you got %s]\" % repr(qvalues)\n",
    "        assert int(qvalues.shape[1]) == n_actions\n",
    "        \n",
    "        return qvalues\n",
    "    \n",
    "    def get_qvalues(self, state_t):\n",
    "        \"\"\"Same as symbolic step except it operates on numpy arrays\"\"\"\n",
    "        sess = tf.get_default_session()\n",
    "        return sess.run(self.qvalues_t, {self.state_t: state_t})\n",
    "    \n",
    "    def sample_actions(self, qvalues):\n",
    "        \"\"\"pick actions given qvalues. Uses epsilon-greedy exploration strategy. \"\"\"\n",
    "        epsilon = self.epsilon\n",
    "        batch_size, n_actions = qvalues.shape\n",
    "        random_actions = np.random.choice(n_actions, size=batch_size)\n",
    "        best_actions = qvalues.argmax(axis=-1)\n",
    "        should_explore = np.random.choice([0, 1], batch_size, p = [1-epsilon, epsilon])\n",
    "        return np.where(should_explore, random_actions, best_actions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = DQNAgent(\"dqn_agent\", state_dim, n_actions, epsilon=0.5)\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try out our agent to see if it raises any errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(env, agent, n_games=1, greedy=False, t_max=10000):\n",
    "    \"\"\" Plays n_games full games. If greedy, picks actions as argmax(qvalues). Returns mean reward. \"\"\"\n",
    "    rewards = []\n",
    "    for _ in range(n_games):\n",
    "        s = env.reset()\n",
    "        reward = 0\n",
    "        for _ in range(t_max):\n",
    "            qvalues = agent.get_qvalues([s])\n",
    "            action = qvalues.argmax(axis=-1)[0] if greedy else agent.sample_actions(qvalues)[0]\n",
    "            s, r, done, _ = env.step(action)\n",
    "            reward += r\n",
    "            if done: break\n",
    "                \n",
    "        rewards.append(reward)\n",
    "    return np.mean(rewards)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate(env, agent, n_games=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experience replay\n",
    "For this assignment, we provide you with experience replay buffer. If you implemented experience replay buffer in last week's assignment, you can copy-paste it here __to get 2 bonus points__.\n",
    "\n",
    "![img](https://s17.postimg.org/ms4zvqj4v/exp_replay.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The interface is fairly simple:\n",
    "* `exp_replay.add(obs, act, rw, next_obs, done)` - saves (s,a,r,s',done) tuple into the buffer\n",
    "* `exp_replay.sample(batch_size)` - returns observations, actions, rewards, next_observations and is_done for `batch_size` random samples.\n",
    "* `len(exp_replay)` - returns number of elements stored in replay buffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from replay_buffer import ReplayBuffer\n",
    "exp_replay = ReplayBuffer(10)\n",
    "\n",
    "for _ in range(30):\n",
    "    exp_replay.add(env.reset(), env.action_space.sample(), 1.0, env.reset(), done=False)\n",
    "\n",
    "obs_batch, act_batch, reward_batch, next_obs_batch, is_done_batch = exp_replay.sample(5)\n",
    "\n",
    "assert len(exp_replay) == 10, \"experience replay size should be 10 because that's what maximum capacity is\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_and_record(agent, env, exp_replay, n_steps=1):\n",
    "    \"\"\"\n",
    "    Play the game for exactly n steps, record every (s,a,r,s', done) to replay buffer. \n",
    "    Whenever game ends, add record with done=True and reset the game.\n",
    "    :returns: return sum of rewards over time\n",
    "    \n",
    "    Note: please do not env.reset() unless env is done.\n",
    "    It is guaranteed that env has done=False when passed to this function.\n",
    "    \"\"\"\n",
    "    # State at the beginning of rollout\n",
    "    s = env.framebuffer\n",
    "    \n",
    "    # Play the game for n_steps as per instructions above\n",
    "    total_reward = 0.0\n",
    "    greedy = False\n",
    "    \n",
    "    for t in range(n_steps):\n",
    "        # get agent to pick action given state s\n",
    "        qvalues = agent.get_qvalues([s])\n",
    "        action = qvalues.argmax(axis=-1)[0] if greedy else agent.sample_actions(qvalues)[0]\n",
    "        next_s, r, done, _ = env.step(action)\n",
    "\n",
    "        # update agent on current transition. Use agent.update\n",
    "#         agent.update(s, a, r, next_s)\n",
    "        \n",
    "        if exp_replay is not None:\n",
    "            # store current <s,a,r,s'> transition in buffer\n",
    "            exp_replay.add(s, action, r, next_s, done)\n",
    "            \n",
    "            # sample replay_batch_size random transitions from replay, \n",
    "            # then update agent on each of them in a loop\n",
    "#             states, actions, rewards, next_states, is_done = exp_replay.sample(replay_batch_size)\n",
    "            \n",
    "#             for idx in range(replay_batch_size):\n",
    "#                 agent.update(states[idx], actions[idx], rewards[idx], next_states[idx])\n",
    "                    \n",
    "        total_reward += r\n",
    "        if done:\n",
    "            s = env.reset()\n",
    "        else:\n",
    "            s = next_s\n",
    "    \n",
    "    return total_reward    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "# testing your code. This may take a minute...\n",
    "exp_replay = ReplayBuffer(20000)\n",
    "\n",
    "play_and_record(agent, env, exp_replay, n_steps=10000)\n",
    "\n",
    "# if you're using your own experience replay buffer, some of those tests may need correction. \n",
    "# just make sure you know what your code does\n",
    "assert len(exp_replay) == 10000, \"play_and_record should have added exactly 10000 steps, \"\\\n",
    "                                 \"but instead added %i\"%len(exp_replay)\n",
    "is_dones = list(zip(*exp_replay._storage))[-1]\n",
    "\n",
    "assert 0 < np.mean(is_dones) < 0.1, \"Please make sure you restart the game whenever it is 'done' and record the is_done correctly into the buffer.\"\\\n",
    "                                    \"Got %f is_done rate over %i steps. [If you think it's your tough luck, just re-run the test]\"%(np.mean(is_dones), len(exp_replay))\n",
    "    \n",
    "for _ in range(100):\n",
    "    obs_batch, act_batch, reward_batch, next_obs_batch, is_done_batch = exp_replay.sample(10)\n",
    "    assert obs_batch.shape == next_obs_batch.shape == (10,) + state_dim\n",
    "    assert act_batch.shape == (10,), \"actions batch should have shape (10,) but is instead %s\"%str(act_batch.shape)\n",
    "    assert reward_batch.shape == (10,), \"rewards batch should have shape (10,) but is instead %s\"%str(reward_batch.shape)\n",
    "    assert is_done_batch.shape == (10,), \"is_done batch should have shape (10,) but is instead %s\"%str(is_done_batch.shape)\n",
    "    assert [int(i) in (0,1) for i in is_dones], \"is_done should be strictly True or False\"\n",
    "    assert [0 <= a <= n_actions for a in act_batch], \"actions should be within [0, n_actions]\"\n",
    "    \n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Target networks\n",
    "\n",
    "We also employ the so called \"target network\" - a copy of neural network weights to be used for reference Q-values:\n",
    "\n",
    "The network itself is an exact copy of agent network, but it's parameters are not trained. Instead, they are moved here from agent's actual network every so often.\n",
    "\n",
    "$$ Q_{reference}(s,a) = r + \\gamma \\cdot \\max _{a'} Q_{target}(s',a') $$\n",
    "\n",
    "![img](https://s17.postimg.org/x3hcoi5q7/taget_net.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_network = DQNAgent(\"target_network\", state_dim, n_actions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_weigths_into_target_network(agent, target_network):\n",
    "    \"\"\" assign target_network.weights variables to their respective agent.weights values. \"\"\"\n",
    "    assigns = []\n",
    "    for w_agent, w_target in zip(agent.weights, target_network.weights):\n",
    "        assigns.append(tf.assign(w_target, w_agent, validate_shape=True))\n",
    "    tf.get_default_session().run(assigns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It works!\n"
     ]
    }
   ],
   "source": [
    "load_weigths_into_target_network(agent, target_network) \n",
    "\n",
    "# check that it works\n",
    "sess.run([tf.assert_equal(w, w_target) for w, w_target in zip(agent.weights, target_network.weights)]);\n",
    "print(\"It works!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning with... Q-learning\n",
    "Here we write a function similar to `agent.update` from tabular q-learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# placeholders that will be fed with exp_replay.sample(batch_size)\n",
    "obs_ph = tf.placeholder(tf.float32, shape=(None,) + state_dim)\n",
    "actions_ph = tf.placeholder(tf.int32, shape=[None])\n",
    "rewards_ph = tf.placeholder(tf.float32, shape=[None])\n",
    "next_obs_ph = tf.placeholder(tf.float32, shape=(None,) + state_dim)\n",
    "is_done_ph = tf.placeholder(tf.float32, shape=[None])\n",
    "\n",
    "is_not_done = 1 - is_done_ph\n",
    "gamma = 0.99"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take q-values for actions agent just took"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "current_qvalues = agent.get_symbolic_qvalues(obs_ph)\n",
    "current_action_qvalues = tf.reduce_sum(tf.one_hot(actions_ph, n_actions) * current_qvalues, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute Q-learning TD error:\n",
    "\n",
    "$$ L = { 1 \\over N} \\sum_i [ Q_{\\theta}(s,a) - Q_{reference}(s,a) ] ^2 $$\n",
    "\n",
    "With Q-reference defined as\n",
    "\n",
    "$$ Q_{reference}(s,a) = r(s,a) + \\gamma \\cdot max_{a'} Q_{target}(s', a') $$\n",
    "\n",
    "Where\n",
    "* $Q_{target}(s',a')$ denotes q-value of next state and next action predicted by __target_network__\n",
    "* $s, a, r, s'$ are current state, action, reward and next state respectively\n",
    "* $\\gamma$ is a discount factor defined two cells above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute q-values for NEXT states with target network\n",
    "# next_qvalues_target = agent.get_qvalues(next_obs_ph)\n",
    "next_qvalues_target = target_network.get_symbolic_qvalues(next_obs_ph)\n",
    "\n",
    "# compute state values by taking max over next_qvalues_target for all actions\n",
    "next_state_values_target = tf.reduce_max(next_qvalues_target, axis=-1)\n",
    "\n",
    "# compute Q_reference(s,a) as per formula above.\n",
    "reference_qvalues = rewards_ph + gamma * next_state_values_target * is_not_done\n",
    "\n",
    "# Define loss function for sgd.\n",
    "td_loss = (current_action_qvalues - reference_qvalues) ** 2\n",
    "td_loss = tf.reduce_mean(td_loss)\n",
    "\n",
    "train_step = tf.train.AdamOptimizer(1e-3).minimize(td_loss, var_list=agent.weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Splendid!\n"
     ]
    }
   ],
   "source": [
    "for chk_grad in tf.gradients(reference_qvalues, agent.weights):\n",
    "    error_msg = \"Reference q-values should have no gradient w.r.t. agent weights. Make sure you used target_network qvalues! \"\n",
    "    error_msg += \"If you know what you're doing, ignore this assert.\"\n",
    "    assert chk_grad is None or np.allclose(sess.run(chk_grad), sess.run(chk_grad * 0)), error_msg\n",
    "\n",
    "assert tf.gradients(reference_qvalues, is_not_done)[0] is not None, \"make sure you used is_not_done\"\n",
    "assert tf.gradients(reference_qvalues, rewards_ph)[0] is not None, \"make sure you used rewards\"\n",
    "assert tf.gradients(reference_qvalues, next_obs_ph)[0] is not None, \"make sure you used next states\"\n",
    "assert tf.gradients(reference_qvalues, obs_ph)[0] is None, \"reference qvalues shouldn't depend on current observation!\" # ignore if you're certain it's ok\n",
    "print(\"Splendid!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Main loop\n",
    "\n",
    "It's time to put everything together and see if it learns anything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import trange\n",
    "from IPython.display import clear_output\n",
    "import matplotlib.pyplot as plt\n",
    "from pandas import DataFrame\n",
    "moving_average = lambda x, span, **kw: DataFrame({'x':np.asarray(x)}).x.ewm(span=span, **kw).mean().values\n",
    "%matplotlib inline\n",
    "\n",
    "mean_rw_history = []\n",
    "td_loss_history = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_replay = ReplayBuffer(10**5)\n",
    "play_and_record(agent, env, exp_replay, n_steps=10000)\n",
    "\n",
    "def sample_batch(exp_replay, batch_size):\n",
    "    obs_batch, act_batch, reward_batch, next_obs_batch, is_done_batch = exp_replay.sample(batch_size)\n",
    "    return {\n",
    "        obs_ph:obs_batch, actions_ph:act_batch, rewards_ph:reward_batch, \n",
    "        next_obs_ph:next_obs_batch, is_done_ph:is_done_batch\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "buffer size = 39010, epsilon = 0.47074\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALcAAAEICAYAAAAdhWWLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAH4RJREFUeJztnXt4XFW99z+/mVxnmrTNJE3Tppnh2lI4lLYIrR45BV5EBEQRj3Jpfc+j4usjKoqKl/OIr0fPxSM+h/PyHoXjDYoiykV9q4IKBETLrdBW6KQU2qRJO2mSJu1M7pOZ9f6x96RDm0wmyezL7KzP8+TJ7Nlr7/Xbe76zZq3f2uv3E6UUGo0X8TltgEZjFVrcGs+ixa3xLFrcGs+ixa3xLFrcGs+ixe1iRKRZRD7itB3Fiha3xrNocWchBrbfExEpsbtON9VvFbZ9kCLSKiKfF5GdIjIgIj8QkXoR+Z2IJETkjyKyMKv8OhH5i4gcEZEdIrIha98/iEjUPG6viHwsa98GEekQkVtEpEtEYiLyDznsahaRb4rIn4FB4GQRmW/aFxORAyLyDRHxm+XbRGSt+fp6EVEicqa5/WER+aX5+jwR2WraHxORO0WkLKteJSKfEJE9wB7zvUtEpEVEjorInYDksPtrIvKgiDxg3oeXRGRV1v4lIvKQiHSLyD4R+dQEx94nInHgf05w/pCI/D8RiYvIC+Y9eCZr/x0i0m7u3yYibz/u/L8wz58Qkb+KyOki8iXzM2kXkXdklZ/0fs8KpZQtf0Ar8CxQDywFuoCXgNVABfAEcJtZdilwGHgXxhfwEnO7ztx/OXAKxof/dxiiXGPu2wCMAV8HSs1zDAILJ7GrGdgPnAmUmMc8AtwFBIFFwPPAx8zy9wK3mK/vBt4APp617zPm67XAOvOcESAK3JxVrwL+ANQAlUAtkACuMW34jHkdH5nE7q8ByazynwP2ma99wDbgq0AZcDKwF7j0uGPfY5atnOD8PzP/AsBKoB14Jmv/DUDIvL5bgE6gIuv8w8Cl5v57Tdu+Ytr3UWBf1rkmvd+z0pzN4r4+a/sh4LtZ258Efmm+vhXYfNzxjwEfmuTcvwQ+nSXuIaAka38XsC6HuL+etV0PjGR/4MC1wJPm6w8DvzZfR4GPAD8zt9swv2QT1HMz8Mhx4r4oa3sT8GzWtgAdU4g7u7wPiAFvB84H9h9X/kvAj7KOfTrHZ+U3xb88671vZIt7gmP6gFVZ5/9D1r4rgX7Ab25Xmde/YKr7PZs/u/tah7JeD02wPc98HQbeLyJXZu0vBZ4EEJHLgNuA0zE+1ADw16yyh5VSY1nbg1nnnoj2rNdhs66YyHivwJdV5ing2yLSgCGCnwO3iUgEmA9sN208HfgOcK5pXwlGazpZvUuyt5VSSkTayU12+bSIdJjnUcASETmSVdYP/GmSuo+nzrQ3u8ybyovI5zC+6Jn6qjF+fTIc/9n2KKVSWdtgfCZLyH2/Z4xbBxLtGC33R4/fISLlGK3+JuBXSqmk2c+dtH+aB9mPRrZjtCS1x31BjIJKvS4igxi/NE8rpeIi0gnciNGypc2i3wVeBq5VSiVE5GaMLsRk9caAZVnXKdnbk5Bd3gc0AgcxujP7lFKn5Tg21+Og3eY5GoHXJqjr7cAXgIuBV80vVh8z+wxy3u/Z4FZvyX3AlSJyqYj4RaTCHCg2YvQhyzE/ALMVf0euk00HpVQM+D1wu4hUi4hPRE4Rkb/LKvYUcJP5H4yuTfY2GD+9caBfRFYAH5+i6t8AZ4rI1WJ4Lz4FLJ7imLVZ5W/GEMmzGH3WhIjcKiKV5j08S0TeMsX5ADBb2IeBr4lIwLR/03HXNobxGZSIyFcxWu5pk+f9nhGuFLdSqh24Cvgyxg1sBz4P+JRSCYwP/ucY/bzrgF8X2IRNGF+iXWYdDwINWfufwviAn55kG4wB3nUYg8T/Bh7IVaFSqgd4P/CvGIPn04A/T2Hnr4APmDZuBK5WSiVNcV4BnIMxkOsBvo/RbcqXm8zyncBm4H6MLw8Y459HMVr1NozB42y6EVPd7xkhZgdeU2SIyNeAU5VSN9hU378Bi5VSH7KjvkLgypZb4zwiskJEzhaD8zAGj484bdd0cOuAUuM8VRhdkSUYno/bMbpBRYPulmg8i+6WaDyLJd2S2tpaFYlETnh/YGCAYDBoRZWuwMvX56Zr27ZtW49Sqm6qcpaIOxKJ8OKLL57wfnNzMxs2bLCiSlfg5etz07WJSFs+5XS3RONZtLg1nkWLW+NZtLg1nkWLW+NZtLg1nkWLW+NZtLg1U9LaM8DO7oKuI7AFLW7NlNzx+B7u3D5CsT2HpMWtmZJdB+OMpqArMTJ1YRehxa3JychYije6+wGje1JMaHFrcrLnUD9jaaM70nZ40GFrpocWtyYn0Vh8/HXr4eJqufVKHE1OorEEFaU+qkpU0bXcWtyanLR0xlm+uJr0UIK23uJquXW3RDMpSimisTgrG6pYFBDaegaLyh2oxa2ZlEPxEfoGk6xYXE19wEdiZIzegVGnzcqbvMQtIgvMkLctYoQOXm+1YRrnyQwmz2ioZlHAiJTWWkT97nxb7juAR5VSK4BVGNFNNR5nlynuFQ1VLAoYUmkrIo/JlANKEZkPXIAZoFwpNQoUz2+TZsZEY3EaF1ZSXVFKXUAQKa6WOx9vyUkY8fp+ZEbu34YRC/tNX2ERuREj0in19fU0NzefcKL+/v4J3/cKXru+bW8Msjjoo7m5mZHBAWrKfbyway/NpQedNi0/pgrgjRFfegw439y+A/inXMesXbtWTcSTTz454ftewUvXNzQ6pk764hZ1++93K6WMa7vuv7eqq+58xmHLlAJeVHkEn8+nz90BdCilnjO3HwTWFP5rpnETrx1KkFawsqFq/L1wKFhUfe4pxa2U6gTaRWS5+dbFGKFmNR4m4ylZsfhY2O1wTYC+wSRHB5NOmTUt8p2h/CTwEzMb115g0uxgGm8QjSUIlvlpqgmMvxcOGRGn2noHODuwwCnT8iYvcSultmP0vTVzhF2xOMsXV+HzHcsEEqk1hN56eJCzG90vbj1DqTkBZU67n9Hw5kwgmVZ8f5H0u7W4NSdw4MgQieGxE8QdKCuhvrq8aHzdWtyaE2iJJQBOEDcUl8dEi1tzAsc8JVUn7AvXBHTLrSleop1xwqEAwfIT/Q2R2iDdiREGRtwf6kGLW3MC0ViCMxZPnFYyHDIGlcWwKkeLW/MmBkfHaD08MGF/GyBi+rr3F8GqHC1uzZto6UygFJzRcGJ/G6ApdMzX7Xa0uDVvInuBwkRUV5QSCpYVhcdEi1vzJlpiCaoqSmhcWDlpmaZQgNYe3XJrioxoLM4Zi6sRkUnLRIrE163FrRknnVa0dCZYMUl/O0M4FCAWH2Y4mbLJspmhxa0Zp6NviP6RE6fdjycSCqIUdPS5u2uixa0ZZ9cUg8kMGV+32/vdWtyacaKxOD6B5fW5uyUZX7fbYwdqcWvGicbiRGqDVJb5c5ZbECilqqLE9bOUWtyacaKdJz7DPREiQiQU1C23pjhIDCdp7x1iZR7iBqPfrVtuTVGwu9N4hnuix1wnIhIKcuDIEMlU2kqzZoUWtwaYetr9eMKhAKm04kDfkJVmzQotbg0Au2IJ5leW0jC/Iq/ykVr3e0y0uDWAOe3eUJVz2j2bcI37n+vW4taQSit2dyby7pIA1FWVU1nq1y23xt20HR5gKJmalrhFxPUeEy1uDS2mpyRfN2AGtz8dqMWtIRqL4/cJpy6aN63jwrUB2nuHSKXdmScnr3BqItIKJIAUMKaU0qHVPEQ0Fufk2iAVpbmn3Y8nEgoymkoTOzpE48LA1AfYzHRS9V2olOqxzBKNY0RjCdaGF077uGyPiRvFrbslc5yjg0kOHBma1mAyQ9jlvu58W24F/F5EFHCXUuru4wvotCHFeX0tvcZqmmT3Ppqb2yctN9G1pZWixAd/enk3S4f2WWnmzMgn/QKw1Py/CNgBXJCrvE4bUjz88Jm9KnzrFnXo6FDOcpNd28W3N6uP3vOCBZZNDgVMG4JS6oD5vwt4BDjPkm+axnZaYglCwTLqqspndHwkFGB/rzt93VOKW0SCIlKVeQ28A3jFasM09pB5hjvfaffjCZvPdSsXps3Op+WuB54RkR3A88BvlFKPWmuWxg7GUml2dybyfsx1IsKhAMPJNF2JkQJaVhimHFAqpfZiZA3WeIzWwwOMjKVn5CnJkMmT09ozQH11fk8U2oV2Bc5hduUIMp8vERdHfdXinsNEY3FK/dOfds9m6YJKSnziSl+3FvccJhqLc0rdPMpKZi6DEr+PxoWVtLnQY6LFPYdpiSWm/STgRLg1T44W9xylb2CUzvjwrPrbGSKhAG09g65zB2pxz1HGkzpNEfQyH5pCQRIjY/QOjM76XIVEi3uOkm9cwHyIuDTbghb3HCUaS1BXVU7tvJlNu2cznhPeZf1uLe45ykTpr2fKsppKRNzn69binoMkU2le7+qfNKnTdCkv8bNkfqVuuTXO80Z3P6OpdEHcgBkite7LLKzFPQfJldt9pjTVuM/XrcU9B4nG4pT5fZxkLhMrBJFQgL7BJEcHkwU752zR4p6D7IrFOa1+HqX+wn384x4TF2UW1uKeg0Rj0wudlg+RWvc9HajFPcfoTozQ0z9ScHE3jYd50C23xiGOxeEujBswQ6CshPrqcld5TLS45xgZcRfSDZgh7DKPiRb3HKOlM0HD/AoWBMoKfu5wyF2+bi3uOUY0Fp/VguBcRGqDdCdGGBgZs+T800WLew4xMpYyp90L3yWBY5mF3RLHRIt7DvF6Vz9jaWWZuCMuezpQi3sOEbVg2j2bJpc9163FPYeIxuJUlBZ22j2b6opSaoJluuXW2E80Fmd5fRV+38xCp+VDOBSgtafIWm4R8YvIyyKyxUqDNNaglCroAoXJcFOenOm03J8GolYZorGWrsQIfYNJy9yAGcKhALH4MMPJlKX15ENe4haRRuBy4PvWmqOxikIuCM5FJBREKejoc75rkm/L/R/AFwD3ZrHX5ORYKAdrxZ3xdbuh3z1llFcRuQLoUkptE5ENOcrptCEuvr6ntg8TqhBefu7PMzo+32tLjBqBeR5/ficlXaUzqqtgTJV6AfgXoANoBTqBQeC+XMfotCHu4+Lbm9WHfzzz9B75Xls6nVZn3fao+sdH/jrjuqaCQqUNUUp9SSnVqJSKAB8EnlBK3WDVl01TeIaTKfZ297OywI+5ToSIEDGzLTiN9nPPAfYc6ietrB9MZgi7JE/OtMStlGpWSl1hlTEaa7BrMJkhEgrS0TdEMuWs/0G33HOAXbE4gTL/eMZfqwmHAqTSigN9Q7bUNxla3HOAaCzO8sVV+Cycds9mPE+Ow/1uLW6Po2yads/GLXlytLg9zsGjw8SHx2wVd11VOZWlft1ya6wlejCzINh6N2AGETE8Jrrl1lhJxlOyfLF9LTfgCl+3FrfHaelM0FQTYF75lE9aFJRwbYD23iFSaefy5GhxexxjMGlflyRDuCbIaCpN7Khz7kAtbg8zODrGvsMDtg4mM7jBY6LF7WF2dyZQNk67ZxOudd7XrcXtYTKr3a0InTYVDdUVlJX4HPWYaHF7mGgsTlV5CY0LK22v2+cTmmoCuuXWWEM0FmdFQxUi9ky7H08kFNB9bk3hUUrR0plghc3+7Wyaagxft3IobbYWt0fp6Buif8TeaffjidQGGE6m6UqMOFK/FrdH2WVRkPnpMP50YI8z/W4tbo8SjcURgeUWxynJxbiv26FVOVrcHiUai3NSKEigzN5p92yWLqikxCeORaDS4vYoVmQsmy4lfh+NCysdi/qqxe1BEsNJ9vcOOtrfztDkYOxALW4P8tohY2bSSTdghkgoQFvPoCPuQC1uD7IrE2R+ifPiDoeCJEbG6B0Ytb1uLW4PEo3Fqa4oYcn8CqdNcdRjosXtQTILgp2ads8m7GCeHC1uj5FOK3Z3Ou8pybCsphIRZ6K+anF7jLbeQQZHU4485joR5SV+lsyvdGfLLSIVIvK8iOwQkVdF5H/bYZhmZkRtCjI/HZzKLJxPyz0CXKSUWgWcA7xTRNZZa5ZmprTE4vgETquf57Qp44Qd8nXnE8JYKaX6zc1S88+5Jc2anOyKJTi5bh4VpX6nTRknEgrQN5jk6FDS1nrzzYnjF5HtQBfwB6XUc9aapZkpdodOy4eMx8TuJWd5PVWjlEoB54jIAuARETlLKfVKdhmdNsT56xtIKg4cGWL9orGC2zGba+tOGKGMf/unFzjcYOODXPmkX8j+A74KfC5XGZ02xBmefaNHhW/dop5oOVTwc8/m2gZGkip86xb1fx5/rSC2UKi0ISJSZ7bYiEglcAnQYuUXTjMzMp4St7gBMwTKSlhUVW67xySf34gG4B4R8WP00X+ulNJZhF1INJagJljGoqpyp005AScyC08pbqXUTmC1DbZoZklLZ5wVi51b7Z6LcCjAU69121qnnqH0CKm0Yvch90y7H0+kNkhXYoTB0THb6tTi9gj7egYYTqZdK+6wA7EDtbg9QtQFq91zEa6x/+lALW6PEI3FKfEJpy5yz7R7Nk2ZnPC65dZMl2gszqmL5lFe4p5p92zmV5ZSEyzTLbdm+rhhtftUhG2OHajF7QH6BkbpjA+zwsEAPPlg+Lq1uDXTINrpvme4JyIcCnDw6BDDyZQt9Wlxe4BMkPliELdS0NFnT+utxe0BorE4tfPKqXPhtHs2xwJjanFr8sSpjGXTJWJzTngt7iInmUqz51C/654EnIiFgVKqKkrYb1MMEy3uImdv9wCjKfdOu2cjImZmYS1uTR64cbV7Lgxft+6WaPIg2hmnzO/j5Lqg06bkRTgUoKNviGQqbXldWtxFTjSW4NRF8yj1F8dHGQ4FSaUVB/qsT5tdHHdEMyluXO2eCzs9JlrcRUxP/wjdiZGicANmyER9tcNjosVdxLh1QXAu6qrKqSz12zKRo8VdxBSbpwQMd6BdHhMt7iImGkuwuLqChcEyp02ZFkZgTC1uTQ4yud2LjUgoSHvvEKm0tSEntbiLlNGxNG909xdVlyRDOBRkNJUmdtRad6AWd5Hyelc/yZQqSnGPe0wsnobX4i5SjnlKiq9bEq7N+Lq1uDUTEI3FKS/xjU+KFBMN1RWUlfgs95jkEwhzmYg8KSK7zLQhn7bUIk1eRDvjLF9cRUmRTLtn4/MJyxZWWu4xyefOjAG3KKVWAuuAT4jISkut0uREKWWsdndBhuCZYsdi4XzShsSUUi+ZrxNAFFhqqVWanHQnRugdGC1KN2CGsCluZWHa7GmFuReRCEbE1xPShujMCvZd3893G6mm0917aW5us7w+KPy1jfYmGUqm+NVjT7KgwqKuVT4R6s1v1zxgG3D1VGV1ZgXr2LLjoArfukV9+eGdlteVTaGvrXl3lwrfukU9t/fwtI+lUJkVAESkFHgI+IlS6mFrvmaaqdjdmeDzD+5gTdMCbrvyTKfNmRWR8diB1g0q8/GWCPADIKqU+o5llmhycnQoycc2v0iwvITv3rCWspLi85Jks2RBJX6fWOoOzOcOvQ3YCFwkItvNv3dZZpHmBNJpxWcf2E5H3xD/df0a6qsrnDZp1pT6fTQurLR0IieftCHPAO7LQzGHuOPxPTze0sXXrzqTt0RqnDanYFidWbi4f9vmAH/cdYg7Ht/D+9Y0snFd2GlzCkrEjPqqLHIHanG7mL3d/Xzmge2ctbSab773LFcmcpoN4VCQxPAYfYPWpM3W4nYp/SNj3Lh5G6UlPr53w1pX5XIvFFZ7TLS4XYhSis//Ygd7u/u589rVNC4MOG2SJRxLAqXFPWf47lNv8LtXOvniZSt466m1TptjGY0LA4hYF/VVi9tlPP1aN99+bDdXnN3AR99+stPmWEpFqZ8l8yt1yz0XaO8d5JP3v8zp9VV865qzPTeAnIhwKECbRTFMtLhdwtBoihs3b0MpxV0b1xIom9YzbUVL2MJHX7W4XYBSii89vJOWzjh3XLt6PAPBXCASCtA7MMrRocK7A7W4XcCP/tzKL7cf5LP/43QuXL7IaXNsJWzhYmEtbod5du9hvvnbKJesrOcTF57qtDm2E7YwMKYWt4PEjg5x009fIhwK8J2/X4XP5/0B5PFY6evW4naIkbEU/+u+lxgaTXH3xrVUVZQ6bZIjBMpKWFRVbsmgcm4MyV3Ibb96lR3tR/jeDWs5dVHxroUsBFYtFtYttwP89Ln9/OyFdj5x4Sm886zFTpvjOFYFxtTitpmX9vdx269f4YLT6/jsJcudNscVhEMBuhIjDI6OFfS8Wtw20pUY5uP3baNhfiX/+cFz8M/BAeREZDwmhe6aaHHbRDKV5qafvEx8aIy7Nq5lQaC4YmpbSWRc3IXtmugBpU188zdRnm/t5T+vXV2UkVmtpGncHahb7qLjoW0d/PgvrXzkb0/i3auWOG2O65hfWUpNsKzgi4W1uC3mlQNH+fIjf2X9ySG+eNkKp81xLVbkydHitpDegVE+tnkboWAZd163uigjstpFuCaguyXFwlgqzSfvf4nu/hG+t3EtoXnlTpvkasKhIAePDjGcTBXsnFrcFvHvj+3mz68f5hvvOYuzGxc4bY7ridQGUAo6+grXemtxW8CWnQe56+m93LCuib8/d5nT5hQFVvi6tbgLzO7OBF94cCdrwwv56hXFHazSTo7lhLdR3CLyQxHpEpFXClarRxlIqvFglf91/ZqiD1ZpJwsDpVRVlBTUY5LP3f8x8M6C1ehR0mnFXTtH6Ogb4rseCVZpJ5m02ba23Eqpp4He2VbUPzLGQ3tGOXDE2sSaTjA0muKffxtlZ3eK265cybkeClZpJ4UOjFmw6fep0obs6B5jyxujbPnXJ1i9yM/FTaWsDPmKOnxB12Cax/cneebAGANJeGu9onF4H83NrU6bVnDsSIki/aO09yb54xNPUlKAh8oKJm6l1N3A3QDnnnuu2rBhw5v2bwAa5z3BG7KEn73Qzr+/OMwpdUE2rY9w9ZqlRbMSJZ1WPPVaN/dubaX5tW78Ilx6VgMfWh9hoHUHF154odMmWkJzczPHf6aFpmteO1v27uS0VecVJAKArQ9OhSp9vG/DCj518Wn89q8x7tnaxm2/fpVvPdrC1Wsa2bQ+zGn17lyVcmRwlF+82MF9z7XRdniQuqpyPnXRaVx3ftN4/7q5rXh/hdxAtsek6MSdoaLUz9VrGrl6TSM72o9w79Y2Hnixnc3PtrH+5BCb1oe5ZGW9K6arXzlwlM1b2/jVjgMMJ9O8JbKQz71jOZeeuVh7QwpM5E2Lhetmfb4pxS0i92P0KmpFpAO4TSn1g1nXbLJq2QJuX7aAr1x+Bg+80M59z7bx8Z+8RMP8Cq4/v4kPvKWJuip7p65Hx9L87pUY925tY1tbHxWlPt67eikb10VYuUQ/rmoVdVXlVJb6CxYYM5+0IdcWpKYpqAmW8fENp3DjBSfzREsX925t5du/f407Ht/D5X/TwMb1EdY0LbB0ANp5dJifPtfGT59vp6d/hHAowD9efgbvX7uM+YHiGBMUMxl3YKE8Jq5brOD3CZesrOeSlfW80d3P5q1tPLStg19uP8hZS6vZtD7Cu1ctKVgwdqUUz+3r5d6trTz26iHSSnHh8kVsWh/mgtPq5mQsEScJhwK83tVfkHO5TtzZnFI3j6+9+0w+f+lyHnn5APdubeULD+7kn38b5QPnLuOGdWGW1cwsMPvAyBiPvHyAzVvb2H0owfzKUj78tydxw/nh8ZUhGvuJhII82dJNKq1mvcbU1eLOECwv4YZ1Ya4/v4ln9/ay+dlWvv/MPu7+014uWr6ITW+N8PZTa/NqZbN/DRIjY6xsqOZb7zubK1ctobLMe6k5io1wKMhoKk1nfJilCypnda6iEHcGEWH9KSHWnxIidnSInz63n/uf38+Hfvg8kVCAjesjXLO2kfmVb+4fp9JqvB//pz09lPqFd/1NA5vWh1nTtLCoJ5K8xrjHpGdgbok7m4b5ldzyjuXcdNGpPPpKJ/f8pZV/2rKLbz+2m/esXsqm9WHqqyvGPTAHjgyxuLqCz15yOh88bxmLqvSzH26kaTwJ1CBvnWVc0KIVd4byEj9XnbOUq85ZyisHjnLv1lYefqmD+5/fT4lPGEsr1p1cw1cuP4NLVtZT6gLfuWZyGuZXUub3FcRjUvTizuaspfP51jWr+NJlZ/CLbe10J0a4Zu0yli9256yn5kT8PuG9q5cSqS3SGUqrWRgs48YLTnHaDM0M+bdrzi7IefRvtMazaHFrPIsWt8azaHFrPIsWt8azaHFrPIsWt8azaHFrPIsopQp/UpFuoG2CXbVAT8ErdA9evj43XVtYKTXlOjRLxD1pZSIvKqXOta1Cm/Hy9RXjteluicazaHFrPIvd4r7b5vrsxsvXV3TXZmufW6OxE90t0XgWLW6NZ7FN3CLyThHZLSKvi8gX7arXakRkmYg8KSK7RORVEfm00zYVGhHxi8jLIrLFaVumgy3iFhE/8H+By4CVwLUistKOum1gDLhFKbUSWAd8wkPXluHTQNRpI6aLXS33ecDrSqm9SqlR4GfAVTbVbSlKqZhS6iXzdQJDBEudtapwiEgjcDnwfadtmS52iXsp0J613YGHBJBBRCLAauA5Zy0pKP8BfAFIO23IdNEDygIhIvOAh4CblVJxp+0pBCJyBdCllNrmtC0zwS5xHwCyEzI2mu95AhEpxRD2T5RSDzttTwF5G/BuEWnF6EpeJCL3OWtS/tgyiSMiJcBrwMUYon4BuE4p9arllVuMGLHY7gF6lVI3O22PVYjIBuBzSqkrnLYlX2xpuZVSY8BNwGMYA66fe0HYJm8DNmK0atvNv3c5bZRGT79rPIweUGo8ixa3xrNocWs8ixa3xrNocWs8ixa3xrNocWs8y/8HbPH+qdK4tBIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAAEICAYAAABoLY4BAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXmYFNXV/z9n9n2AGfZtQEAEVFDEDRV3hRiM0cQlicYkxiTmF+P2Eo1GfRODZveNJjHRaEziEqMRxS0K4y4gyiICMsCw78MMs299f39UVXd1T28z0zO9zPk8zzxTy62qe7u6v3Xq3HPPFWMMiqIoSvxJi3cFFEVRFAsVZEVRlARBBVlRFCVBUEFWFEVJEFSQFUVREgQVZEVRlARBBTkJEZGrROSdXrpWmYgYEckIsf9WEflLb9Sls4jIt0Xkt3G4bp2IjO3t6yYDIpItIutEZGC865KIqCD3MPaP0/nziEija/0KEblTRFpFpNb++0xEfi8iQ+Nd92gwxtxjjPlmpHIiUi4iEcvFChHJAn4M/KK3rulgjCkwxmzq7esmA8aYZuARYF6865KIqCD3MPaPs8AYUwBsBS5wbfuHXewpY0whMAD4AjAEWJ4sotzTiEVnv6tzgXXGmB09UadUJtTbUAz5J3CliGT38HWSDhXkBMIY02qMWQN8GdgH3BjNcSJykogsE5Ea+/9Jrn1Xicgm2/reLCJX2NvHicib9jH7ReSpCJe5QkS22mVvc53/ThH5u72cIyJ/F5EDIlJt12WwiPwMOAX4vf1m8Pso6l0uIj8TkXeBBuBGEVke0O4bROT5EPU9H3jTVdZxvXxdRLaJyEERuVZEjhORVXZ9f+8qnyYiPxaRLSKyV0T+JiLF9r6XReS6gLqsFJGL7GUjIuPs5UdF5AERWWjfgyUicpjruHNEZL39GTxo35OgbxIiMkNE3rfrust+k8qy9/1BRH4ZUP55EbnBXh4mIv8WkX329+D/BdzDZ+x7dwi4Kty1oqm3iFwtImvtz/lVERnt7DPGbAcOAieEuHd9F2OM/vXSH1AJnBWw7U7g70HK3g0sCXGeq4B37OUBWF/urwIZwGX2egmQDxwCDrfLDgUm28tPALdhPZRzgJkhrlUGGODPQC5wNNAMHBFYf+DbwAtAHpAOHAsU2fvKgW+6zhuy3q7yW4HJ9v5soMq5rl3mY+CLIeq9DLgkSDv+aLf3HKAJ+A8wCBgO7AVOs8tfDVQAY4EC4FngcXvf14B3XeeeBFQD2fa6AcbZy48CB4AZdjv+ATxp7yu1789F9r4fAK3uzymgTcdiiViG3Z61wPX2vlOBbYDY6/2BRmCYfY+XA3cAWXabNgHnuu5hK3ChXTY3wrXC1hvr7aQCOMLe/2PgvYC2LAD+X7x/k4n2pxZy4rITS7QiMQfYYIx53BjTZox5AlgHXGDv9wBTRCTXGLPLWBY4WD+g0cAwY0yTMSZSJ+FdxphGY8xKYCWWMAfSivUgGGeMaTfGLDfGHOpivQEeNcassfc3A08BXwEQkclYQvFiiPP3A2qDbP9fu72vAfXAE8aYvcZybbwNTLPLXQH82hizyRhTB/wIuNR+nX8OmOqy+q4AnrXrGIznjDFLjTFtWII81d4+G1hjjHnW3nc/sDvEObA/zw/sz6MS+BNwmr37bawHwSn2+sXA+8aYncBxwEBjzN3GmBZj+bf/DFzqOv37xpj/GGM89n0Od61I9b4W+LkxZq29/56Azwuse9MvVFv7KirIictwLIswEsOALQHbtgDDjTH1WO6Pa4Fd9mvzRLvMLYAAS0VkjYhcHeE67h9cA5bVGMjjwKvAkyKyU0TuE5HMztbbtb4tYP9jwOUiIliW9dNhRPAgUBhk+x7XcmOQdaddgfXbgmXtDTbG1AIL8QnaZVhCG4pQn90wXG00lum4PdRJRGSCiLwoIrtt18I9WNaqc+yTdl0ALnfVaTQwzHY/VItINXArMNh1er/POty1oqj3aOB3rmtVYX3X3Pe2EOutQnGhgpyAiNWBdQGW1ROJnVg/ADejgB0AxphXjTFnY7kr1mFZRhhjdhtjvmWMGYblanjQ8Xt2FWP5wO8yxkwCTgI+h/V6D5b1FnW9gx1jjPkAaMGyAi/HegCEYhUwoVMNCF+/UUAbPgF/ArhMRE7EcoEs7sI1dgEjnBX7QTMidHH+gHUPxxtjirBEVVz7nwAuti3R44F/29u3AZuNMf1cf4XGmNmuYwPvT7hrRar3NuDbAdfLNca85ypzBNabluJCBTmBEJEMETkC64c1BPh1FIe9BEwQkcvt47+M5dN8UawOtbkiko/l963DcmEgIpeIiPMjOoj1g/R0s/6ni8iRIpKO5WNsdZ1zD5bvMmK9I1zmb8DvgdYIbpaX8L1id4UngB+KyBgRKcCyEJ+yX8Gd84/G8vU/ZYzpyme3EDhSRC60XSHfw7rvoSjE+lzr7Ded77h3GmM+BvYDfwFeNcY4FuhSoFZE/kdEckUkXUSmiMhxXbxWpHr/EfiR7VZCRIpF5BJnp4gMx3LHfRDm+n0SFeTE4MsiUgfUYHV2HACOtf1/YTHGHMCyRG+0j7sF+JwxZj/W/b0By9qrwhIo54d1HLDEvu4C4Aem+7GzQ4BnsH7Ia7GiHBwr9ndY1ttBEbk/Qr3D8TgwBfh7hHIvABNFZFiXWmLFyj4OvAVsxuoA/L6z03aVPAuchRXG1Wnstl4C3If1GUwCPsR6eAbjJqw3g1qsN51gkTH/DKyTMaYd67OearfFEe3iMNULea1I9TbGPAfci+W6OgR8ghX14nA58FgYd1OfxemRVZSkQERysaIhjjHGbIhQ9hpgkjHm+l6pXDexXVXbgSuMMV1xgcSFztRbrNjjlcCpxpi9vVG/ZKKnA8AVJdZ8B1gWSYwBjDEP9UJ9uoWInAsswepQvBnLT5vwr/JdrbdtFU+MVK6vooKsJA0iUon1w78wzlWJJSdiuReygE+BC40xjfGtUlQka70TGnVZKIqiJAjaqacoipIgJJzLorS01JSVlXnX6+vryc/Pj1+FYkQqtCMV2gCp0Q5tQ+IQ2I7ly5fvN8Z0Lb1oLMdhx+Lv2GOPNW4WL15sUoFUaEcqtMGY1GiHtiFxCGwH8KHRXBaKoijJjQqyoihKgqCCrCiKkiCoICuKoiQIKsiKoigJggqyoihKgqCCrCiKkiD0CUF+r2I/m/bVxbsaiqIoYUm4kXo9weV/WQJA5fw5ca6JoihKaPqEhawoipIMqCAriqIkCCrIiqIoCYIKsqIoSoLQpwT55n/prOOKoiQufUqQ/7V8O63t3ZrpXlEUpcfoU4IM0NDSHu8qKIqiBCXlBdkEzBnYqIKsKEqCkvKCXL5+n996Q0tbnGqiKIoSnpQX5DaPv4WsLgtFURKVlBfk/Kx0v/WmVkuQPR7DTf9ayertNfGolqIoSgdSXpB31jT5rTfagry3tplnlm/nm39bFo9qKYqidCDlBfmmgNhjp1MvK8NqekubhsEpipIYpLwgB7J8y0Ha2j2Ivd7absKWVxRF6S36RPpNN396axMt7R6+O2scoBayoiiJQ58TZID3Kg6wdtchAFp05J6iKAlCnxTk2qZW1u+pjXc1FEVR/OhzPmSA2mYdHKIoSuKRshayx2N4YdXOoPvUb6woSiKSsoL8x7c2ct8r64PuU7+xoiiJSFQuCxE5T0TWi0iFiMwLsj9bRJ6y9y8RkbKA/aNEpE5EbopNtSOzYmt1yH1GI90URUlAIgqyiKQDDwDnA5OAy0RkUkCxbwAHjTHjgN8A9wbs/zXwcverGz1vrNvbm5dTFEXpNtFYyDOACmPMJmNMC/AkMDegzFzgMXv5GeBMEREAEbkQ2AysiU2VI7NiWzXtnujNYCe/haIoSjyRwHzBHQqIXAycZ4z5pr3+VeB4Y8x1rjKf2GW22+sbgeOBJuC/wNnATUCdMeaXQa5xDXANwODBg4998sknvfvq6uooKCjoVKMqDrbz0yVNkQvaTCpJ45bjcjt1jc7SlXYkGqnQBkiNdmgbEofAdpx++unLjTHTu3Kunu7UuxP4jTGmzjaYg2KMeQh4CGD69Olm1qxZ3n3l5eW416Nh8K5DsOTtqMt/esDT6Wt0lq60I9FIhTZAarRD25A4xLId0QjyDmCka32EvS1Yme0ikgEUAwewrOSLReQ+oB/gEZEmY8zvu13zIDS0tJGZnsbqHR1Tag7vl8uO6saeuKyiKEpMiEaQlwHjRWQMlvBeClweUGYBcCXwPnAxsMhYvpBTnAIicieWy6JHxBhg0h2vcvyYASzZXOW3/YyJg/jdpVM5af4iapusQSHpadIpP7OiKEpPE7FTzxjTBlwHvAqsBZ42xqwRkbtF5PN2sYeBEhGpAG4AOoTG9RaBYgyQlZ5GYU4m4wb5/DyZ6aFdKIqiKPEgKh+yMeYl4KWAbXe4lpuASyKc484u1C8mOANBcjJ8s4f0y81id2v0HX+Koig9TZ/IZeEMlc51Ted00riSeFVHURQlKH1KkP0I4T5uafNw3T8/YuO+up6tlKIoSgB9QpCbbZfFwYYW77bA2agdVm6v5sVVu7jlmVW9UjdFURSHPiHIjoV87xeP8m4LHJ332prdABTlZAJQ09jaS7VTFEWx6COCbInvhMGF3PE5Kw1HU4Ab45rHlwNWOBzAIRVkRVF6mT4hyHdcMNm7nJ1pNTlU/gpnKLkTr6woitJb9AlBPm3CQO9yth361hxCkNttQW7UhEOKovQyKS/IP7nAP1NoVobV5FCC69Hc9YqixImUF+QvTBvut56XaVnI9c0hBFmz1yuKEidSXpCdqAmHvGxLkEMlGnIL8oG65p6rmKIoSgApLch3z51MWpp/zoqC7NCjxVvbPbjDk5t1MlRFUXqRlBbk5taOgpqX5S/Ij39jhne5vrnNLwNcQ4t27CmK0nuktCAHm1060EIeWpzjXa5rbsM9g0qjCrKiKL1Iygjy+t21HbYNLMjusM3xITvkuizmuuY2P5dFQ4vGIiuK0nukjCDf/MzKDtsumT6iw7b8AJdFvisDXAeXhcYiK4rSi6SMIOdlpXfYFmwev/SATj53Ss7aJnVZKIoSP1JIkLs2X2tWuu8j6OiyUEFWFKX3SCFB7mghR4Pbiq5ravMOnYbQ+S4URVF6gpQR5EDf8PlThnT6HPUt7X4DQ1SQFUXpTVJGkHMDLORjR/eP+tjXfngqAA3NbXg8KsiKosSHlBHkQJdFsA69UEwYXEhWRhp1Lf4+ZM34pihKb5K6gtzJ4wuyM6hvbvNzWTS26NBpRVF6j5QR5JzMQAu5c8cXZGdQ1+TvslALWVGU3iRlBDkwa2Y4PXaHujkU5mRQ2+TvslAfsqIovUnqCDL+ihzOh5xtJ6n/01eP9W5zBNkJe0tPEx0YoihKr5I6gmzrseNLTgtjIjvz6h0zyheJUZiTyaGmVu9IvfysdHVZKIrSq6SOINv/vb7kMBby1TPHAJDvSjRUlJNpuyysM+VlZajLQlGUXqVr440TEMdCzrUFOZwP+buzxvHdWeP8thXmZHCoqRUnY2dedroKsqIovUrKWMiOZesMEOlslEVRTgZ1zW2027OcFmRnqMtCUZReJWUE2cFnIXdOkQtzMjHGyvgGli9aBVlRlN4kZQTZ6YzLsTvswnXqBaMo1/LeVDe0AlZuDB0YoihKb5JCgmz9dzr1OuuyKLRnp65ptAU5O0NnDFEUpVdJHUG2/3sCR4hESWGOZSGv3F4NQElBFg0t7dqxpyhKr5E6gmzr8LsVBwB4Y+3eTh3vWMirttcAMLDQmo/vQH1LjGqoKIoSntQR5ICRevvqmjt1vGMhOwzIywKgxvYpK4qi9DSpI8i2Hmemi996tBTZFrJDvzxrvbZJBVlRlN4hdQTZ/v/hbWdz2MB8HrjimE4dH2ghF+daFrITBqcoitLTpMxIPcckLs7L5I0bZ3X68MD0nV4LuVktZEVReoeUspA7G+oWjuJcS5B/v6gididVFEUJQ1SCLCLnich6EakQkXlB9meLyFP2/iUiUmZvnyEiK+y/lSLyhdhW30e7x3R6lpBwlBZYURaDCnNieFZFUZTQRHRZiEg68ABwNrAdWCYiC4wxn7qKfQM4aIwZJyKXAvcCXwY+AaYbY9pEZCiwUkReMMbE1DH7o2dX88TSraR3dnheGNLThOH9chnaTwVZUZTeIRoLeQZQYYzZZIxpAZ4E5gaUmQs8Zi8/A5wpImKMaXCJbw7QtVEbEXASzsdQj0kTKz1nQ7MODFEUpXeIRpCHA9tc69vtbUHL2AJcA5QAiMjxIrIGWA1cG2vrGHyC3NmEQoE4eTDAmnEkLyuDeh0+rShKL9HjURbGmCXAZBE5AnhMRF42xjS5y4jINcA1AIMHD6a8vNy7r66uzm89GDu2W6PpWts9EcuGI0s8OBUrLy+HpiYqq023zukQTTsSnVRoA6RGO7QNiUMs2xGNIO8ARrrWR9jbgpXZLiIZQDFwwF3AGLNWROqAKcCHAfseAh4CmD59upk1a5Z3X3l5Oe71YCxtWgebNmIgYtlw9F+2mEMHGsA+z0v7V/LWZ/u7dU6HaNqR6KRCGyA12qFtSBxi2Y5oXBbLgPEiMkZEsoBLgQUBZRYAV9rLFwOLjDHGPiYDQERGAxOBypjU3IV7brzukJ9lPZ+e/97JgBX65mR/UxRF6WkiWsh2hMR1wKtAOvCIMWaNiNwNfGiMWQA8DDwuIhVAFZZoA8wE5olIK+ABvmuM2R/rRhTlZkYuFAUF2dbH4SSmL8rJpLG1nZY2D1kZKROyrShKghKVD9kY8xLwUsC2O1zLTcAlQY57HHi8m3WMiCOk3cWZ9LS+2erIK87z5Uh2sr8piqL0FClh9uVlpUcuFAXOYJBWe6ZTJ7/FIU0wpChKL5ASuSxiJci3XzCJIcU5nHXEYAAKsi0LWWORFUXpDVJDkGPksijKyeTGcw73rjsujLpmjUVWFKXnSQmXRW5mbCzkQBzfdL0KsqIovUBKCHIsc1i4yXcEWUfrKYrSC6SEIPcUjoWsLgtFUXoDFeQw5PeSy+JXr63n+RWBgx8VRelrpESnXk+Rl+l06vVslMX/2Unw504NzNmkKEpfQi3kMKSlCQXZGdTFYF69pjbDEbe/wqJ1e7zbahpbaWzRkDpFUSzUQo5AfnZ6TFwWexo8NLa2c/WjH3JcWX/+de1JHH3Xa4zonxuDWiqKkgqohRyB/OyMmHTqVTX5cvMvqzzoXd5+sLHb51YUJTVQQY5AYYwE+XcfNcegNoqipDIpI8j982KT8S2Q/OyMHomyqNhbG/NzKoqS3KSMD3nRjbOo7oHcxfnZGVTVN3TqmA82HeDY0f3JTA/9vNtW1dFV4fEY0npokIuiKIlP6ljI+VmMKc2P+XnbPYZ1u6O3Ziv313PpQx9w67OraWnz8PA7m2mzs8e5aW7rGF3R0KoRF4rSl0kZC7mnWLLJmonqzgVreGn1LpbedlZUx724aheHDynkpwvXYkzHybYPNnS05uub22KW21lRlOQjZSzknuK7p48D4NH3KtlbG33HXGNrO20eS4gXrNzZYf/BhpYO2x55Z3MXa6koSiqgghyBocU5fuvB3A9u3LawY+1WB7GGg23L7qGsdYqiJAcqyBHID3Ah1IcZRt3Q0sY7Fb4pA50ZRzzBXBb1HS3kAT0UKaIoSnKgDssIBPp0a5tbvXPtBXLbc5/w3Me+JEF59izWtUGGXgeLCNGscorSt1ELOQKBFnIwcXWoPFDvt+5EUtTY4vutU8Z491U3tJAREOJWq4KsKH0aFeQIBFrINWFinYtz/S3nqgC3xIwxJd7lQ41ttLtcGaUFWWHFXlGU1EcFOQKBghzM9+swtNg/UdD+gKiM/Ox07vjcJI4r609NYyvGwPFjBvDi92fGLKucoijJiwpyBJyJTh2CxQ87BM5+vT9AvNNFuHrmGI4a0Y/qRmvfyeNKmTK8mIKc2OTMUBQleVFBjkB+VoCFHCR+2CFQkJ9Zvt1v3Q5Lpignk6ZWK3zOmQ+wMDtTLWRF6eOoIEcgLU0odLktAv3CbnIDBLmlzT9m2TnWCYcDSBNLkItyM7xWc1Nre8R4Z0VRUg8V5ChwRz+Es5Az08J/nIOLsgEocnX+OfmHSgqyOVBnnXvi7a/wrb992NXqKoqSpKggd5KaMD5kQ8cBIA4FmTC9bIC17PJLOxZyaUE2VQ0tXst48fp9saiuoihJhApyFIwd6MsiF85CDjIgz0v/nLSg5Rwf8sDCbIwJ7xJRFCW1UUGOgtlThnqXw+VcDqPHNLf79p4+cZB32SvIBVkAnUpgpChKaqGCHAX9XEOlw7ksguWscHCnwMjJDO6yANhfl1yCvHbXIZ7+cFu8q6EoKYHmsoiC/nlZ3uVDTa0YYxDpOLNHOJdFc5v/zqz0NFraPX4uC4D9dT6XRV0S5Ec+/3dvA/Cl6SPjXBNFSX7UQo4Ct4Xc2m5oDDKzR1V9C794db13PXAmpsAkcU6CovQAC3mfy2WxL4ncF+GGlCuKEh0qyFHQz2UhQ3DxueeltX7r00b191sPNJ772aFvzhx6+dkZ5Gam8/j7ld4yydDB5zxItlV1bt5BRVE6ooIcBYEzWh9q7Diizi3SL//gFL524ugI57RE3j0PalNbOztrmrzrySDIg2xXy74k830rSiKighwF0VjIF04d7l2eMLiQgbblGArHZZHm8kU74uZQVZ/4Itc/32rHgbrEf3goSqKjghwFgWk1I/lLBSgtDC/IRTnWOZtbPR22OVTVJ75f1nlYVYeJz1YUJToSuws/QUgP6KE7FESQ3bmNRfyt3ctmjKRu/26/8kW51kd/qMl3roKc6HMvJwr5dv6OZHCvKEqioxZyF/hkZ02HbcZPkMXPqv75RUfxxQn+bo9Txw8EYOKQIu+2ziTDTwTerdjPzmrL562CrCjdRy3kKDlh7AD21jazaV99UKEMHBQSLE7ZzekTB7HstrO88cfQUZCDWeKJxBV/WeJdDjekXFGU6IjKQhaR80RkvYhUiMi8IPuzReQpe/8SESmzt58tIstFZLX9/4zYVr/3ePKaE1l04ywmDysKOmvIm0GSAf38oiP5+zeOD3nOgQF+5uwM/9vhdmckOgeTwN+tKIlORAtZRNKBB4Czge3AMhFZYIz51FXsG8BBY8w4EbkUuBf4MrAfuMAYs1NEpgCvAsNJYgYWZgcN8frPip0AvHL9Kd5tl80Y1alzHz6kCNjpXX97w/6uVTIO7D7UFLmQoihhicZCngFUGGM2GWNagCeBuQFl5gKP2cvPAGeKiBhjPjbGOAqzBsgVkfDhBwlOqZ23uL65jY+3HuywPzBSojN8+9SxnDt5sN+2xpaOowITkV01jX5+dEVROk80PuThgDt7zHYg8D3cW8YY0yYiNUAJloXs8EXgI2NMB/NSRK4BrgEYPHgw5eXl3n11dXV+6/GmoaqFfYdaueGvb/BqZRv3npLL4Hzfc23JB+/7pdp0iLYds/p7eNW1/uLrbzIwLzH6XsO1obXd8PLr5eRlhvedJwKJ9p3qCtqGxCGW7eiVTj0RmYzlxjgn2H5jzEPAQwDTp083s2bN8u4rLy/HvR5vNqRt4qXNa9nZmg/UMGzCkfagiBUAnDrzZEqCDAqJth17DzXBO29410dNPJrjx5bEqPbdo0MbXlnot78ycyQXTh3OsH7+s28nGon2neoK2obEIZbtiMb02gG4U3mNsLcFLSMiGUAxcMBeHwE8B3zNGLOxuxWONyV23mInquLjrdVc/9QK7/78bmZncwZazLBnF7n278u7db6eZmhxDg9ecQwA972ynjn3vx3nGilK8hKNIC8DxovIGBHJAi4FFgSUWQBcaS9fDCwyxhgR6QcsBOYZY96NVaXjiWP9tttTSP/6v5/57Q+MlOgsWRlpVM6fw5+/Nh2AUSX5EY6ID46/+MvHjWREf59FfDBMvmhFUcITUT2MMW3AdVgREmuBp40xa0TkbhH5vF3sYaBERCqAGwAnNO46YBxwh4issP8GkcSU5DtJgTr6Sq8+eUzE+ONoKc7LZPygAoYW5cTkfLHGeSCliTAgPytCaUVRoiGq92tjzEvASwHb7nAtNwGXBDnup8BPu1nHhMKJHXZikceU5rN5fz1gDZmOJSP651J5oD62J40RdfZM3OlpQkl+UgfOKErCkBjd90mEkzbTmdnDnUQ+iNHcLYb1y03YOfZueWYVYLU/Nys9QmlFUaJBBbmTZGWkUZybSUu7laXNsRQBmts8oQ7rEqUF2RxsaKGtPbbnjQUVe+sA//YritI9VJC7gBNpEcjeQ7G1ZksLsjAGqhIwT8TZ9gCW2UcO6bBPB4goStdQQe4CpSF8pntrYzt82JkeKRGTvw+wXTcn2DHSN5w9wbuvIUlGFypKoqGC3AV2VDcG3R7raYycELv9CTg9kh1kgWA5zscO9IXnaSpORekaKshdIKQgx7gDrtR2jSSihWzsaVudyJL+rmmuErUjUlESHRXkLvCLi48Kuv03X5oa0+s4FvL1T61gTZCk+PHEcRM7gjym1Gch3/fKOppa1W2hKJ1FBbkLfO6oYR22/fbLUzn/yKExvU6Ra0qnbzz6YUzP3V2cjjtnktZh/XJ5/nsnA7BkcxW/e2ND3OqmKMmKCnIXCBZ3G+tBIdY5hX727NSJNiOHz4fsY8rwYu/y9oPB3TqKooRGBbmbOFZsrIZMBzJ6QB7gH+P80daDPL8iML9T7+K4LNJc7XYPJ69NotlOFCVR0Dn1ukn//CwONbXFfJSew8DCHMDff3zRg+8BMHdq/CZfcbLdhXoOJfp8gIqSiKiF3E2cdJlCzyhyaYhBKOBL8BMPnCuHejOobdIRfIrSWVSQu8m4gQUAVNX3TKiXO5Na4Ai4ePqVjTFh/eYqyIrSeVSQu8jT3z6RH541gePHWInkS4PMEhILnE498M+wBvEbMPJexX62VTX4+Y8dCm2fuvqQFaXzqA+5i8wYM4AZYwZgjGHMwHymj+7fI9dxT5p6z0truevzUyjJz2JvbTP7apuZ2DGVRI9z+V+WAJARxHFelJNJbVMb9Tp8WlE6jVrI3UREOK604EP+AAAgAElEQVRsQI9FWZw/xRfb/MTSbfzqtfVeazzWIwM7S7Am3zr7CO+yJ44+bkVJRlSQE5zivEzW3HWud33l9mpvtrl4CHK1y2/d2t5RcOccNZSfXDAJSLzYaUVJdFSQkwD3xKm7aprIswemxMOHHM2cec6s0x9squrp6ihKSqGCnGTsqm7yDsqIh4Xc3BbZNzx+kBV5su1gQ09XR1FSChXkJKOl3eMdtvzxtmqWbu5dK7S5NfLsJWMHFlCcm8m2KhVkRekMKshJyOtr9wCw5UADX/rT+716bfcQ7sKc0EE6ZSV5bDmggqwonUHD3lKAFduqmTqyX69cq67Z8iHfd/FRYUP9ykrzeW/jgV6pk6KkCmohJwlzwqT2/M7fl/daPa6204COH1TAWHuUYjCq6lvYV9usbgtF6QQqyEnCz74whRPt+esCOXdy748OGW5HUoTivClWnRz3iqIokVFBThL65WXxyFXHBd0XLt63uqGFY/73vyyrjG3n36CinLD7Jw4pBOCuFz6N6XUVJZVRQU4igiXGB9h7KHT424a9dVTVt/DThWtjUoeykjw+f3THGVMCOWaU5V8+toeGlCtKKqKCnGRcfOyIDtv21jaFLF9gDypZua06JtdvbTdkpEceJi4inDNpsOZFVpROoFEWScYvLzmaq04qozAng9N+UQ6En+XZxDidRGu7h6z06J7j6WnChr11NLS0kZelXzVFiYRayEnIlOHFjC7J569fP44ZYwZQ29QWcpZng0+RV2/v3szV7R7DwYYWb1L+SByos3zb//1UO/YUJRpUkJOY0w8f5HVhhBpG7baQN+6r69b1alsNre2GYf3Cd+g53HPRkYAvj7OiKOFRQU5yBhVaqThvfW51UF+yW5BruymMtXYwR7TJ+MeU5pOeJuyqDu3jVhTFhwpykjPYDj97e8N+ZvzsDX7x6jq//W6XxZod3XNZ1DRb5yrJj85lkZ4mDCnKYUd1Y7euqyh9BRXkJGfC4EK/9QcWb/Rbd1vI63bXBj3Hk0u3ctovFvvN2betqoEHyyv8tr2zw4qYKC2MfrqqUQPy+KSbDwJF6SuoICc56UGmUXLjDrI4FGKeu3nPrmbLgQb21/kGmPzwqRXc98p6Kvb6/M719uEj++dFXb+jRhSz5UADbe2Rs8QpSl9HBTkFmG93ngXDsXDHDsy3cyl3jIMbWmy5Pba78hfnZFqDULa73A25GTC2NJ+sjOi/NpOGFdHS7uGzPd3rUFT6FlsPNFA2byErYhQ/nyyoIKcAZ08a7Lfujmpw5Hd4v1waW9upDjLjh5OXYvtBn/iOHGBtcycHWrq7vdMRE45L5bM9wd0lihKM1z7dDcCFD7wb55r0LirIKUBJQNRD5f5677JjEY+w3QxbgmRfc6Zccne+OSLtCLJznnCDUIIxdmA+RTkZLF6/t1PHKX2bfy7d6l0O9laXqqggpwjXnzXeu+xODO98l53cxbf/55MOx+ZnW+6Jhat2ebdl2KPxnJzGziwlY0vzO1Wv7Ix0Tp0wkOdX7NRZqJWo2bTPZ1SUr9/n505LZVSQU4Trz5rAp3efiwhs2OtzDzgSOKjIsqI/2dkx4sER7dU7arzWtbPNWffYGy46Znin67bB9h8/+l5lp4/tKW55ZiVX/OWDeFdDiYKvP7qMmfcu5rv/WM4fyjdGPqCTNLZEnieyt4hKkEXkPBFZLyIVIjIvyP5sEXnK3r9ERMrs7SUislhE6kTk97GtuhJIXlYGY0rzWbvrkHebI6yC8NUTRlOQndHhFdC9+ue3N1nbbCmvb2mnqbWddtu6FYmcWCiQB66YBlhzAMaSDXtqKZu3kOVbDnb62Kc/3M67FQeob27jPx/voFWjQBKK/nmZfOWEUX7bXlq9m3tfWRfTkZ8vrd7FEXe8wrrdhyIX7gUiCrKIpAMPAOcDk4DLRGRSQLFvAAeNMeOA3wD32tubgNuBm2JWYyUsY0sLqNzvdlk4QmpNq1Tb1OYX3gaW+A62LehPbTF3i3TlgXqvhRwpzC4Y4wYVMmpAHi+s3ElVfejczeXr99LQEv2PzRH4372xoVP18bgaN/knr3L9Uyt4cHHsLS+l6xggPcTD/40YTnrgTBL85vp9MTtnd4jGQp4BVBhjNhljWoAngbkBZeYCj9nLzwBniogYY+qNMe9gCbPSCxTlZrB+Ty0t9mSkjvQIMGloEYCfBQ2W+Dpf/o+3VtvbfKK1aV+914cc6kcSiVw7jO63r38WdP/+umau+usyvv149NNROaMU3/qscz+meW93HDn4mxD1SmaMMTz+fiU1SZgC1eMxiAj98jI77OvsAzgcYwdafSKbXR3h8SSanIjDgW2u9e3A8aHKGGPaRKQGKAH2R1MJEbkGuAZg8ODBlJeXe/fV1dX5rScrvdWOqn1WFMSvnl5ETgasPWD5x1auXMmwAuv5++p7K/Ds9H3Rd+5qprm5nZOGZfDezjYW/ncxm7b6fsRvLF2N2W2V37RpI+UeXw94tNx8tOGbr8G/P9xCdt0uTh7u/0M72GQ9QN7esJ83Fi2OyhJfs9/n+4v2s61u8rC3IXjnYrTXTQSCfZ/2NngYlOezsZ6vaOG5ilZuf34ND52dR1YUeax7k3C/ida2Nnbs2M61UzJYttvwwa4278CkLfvrY/JbOtDoodL+fXxUsYPy8q7NqhPL33ZCJKk1xjwEPAQwffp0M2vWLO++8vJy3OvJSm+1Y+CEGsrvf4eHVjf7uR2mTZvK8WMGcPv7ryH9hjBrlm8wyYK9K9jSWMWXTz2c955cQU3RYZSVNcOGzxhYmI2noJSTTpoEb/yXw8ePY9bJY7pUt6lr32XFtmr+vLqF264422/fzupGKF8EQNmRx3FYmAlUHdI37IMPlwJwyqmnRSWmq7fXQPk7AGy8Zzbn/+4tBhZm827FASZMPZ6RA6IfhRhPAr9Pb6zdwy2PWRPQbv75bESEq15Z6N0/f4Ww6MZZJBLhfhPpi19l1MiRXPu5SVxrb2tp8/Db1z/jobc2cdLMU8MOUDpQ10ybx3jfogJZs7OGq+5/x7u+vyWjy7/PWP62o3FZ7ABGutZH2NuClhGRDKAY0Dng44DjlggM3RSsDrnDBhX4DYe2Cls+5mkjrdC4BSt3eI+fOKSQj7ce9Ppd07phQRbm+J7/gT3bbr9uh/qFwB1F99aG6NwW1Y2WD/vpb59Ieprw2g9P41unjAXgwfKKqM6RaGyrauD/Fvnq7sSTuz9vdxhZMuAxhsBvWlZGGhMGF9LmMRFdDMf+9HWOv+eNkO6amoABUlX1LQnRsReNIC8DxovIGBHJAi4FFgSUWQBcaS9fDCwyfSmaO4EQEb40veM0T050xLiBBWwM+HEarCiMUSWWdfjBpipvlMX00QPYUtXg/WJ355XeHcO8ab+/6HpcQQ6f7ozuh+H+ii2vjC7S4t/LtwNWL77DCfZs3gfqWrjx6ZV8tLXzURvx4h9LtnDKfYv9hhh/suMQ++ua6Z+XxYljSzhj4iCgowglMsYEf/g7Iz///dH2qM4TaoRoZhDr+oklW+M+ojSiIBtj2oDrgFeBtcDTxpg1InK3iHzeLvYwUCIiFcANgDc0TkQqgV8DV4nI9iARGkqMcQTGjTMz9diBBeyrbabWlWjIYwxOX90YWzQd6/PwIQUYA2f86k3AEuuucu7kId7lj7b6h8C1u8R15fbowuMCk+83tbZz9q/f5O0w1vIaW+xHuBIkOXk7Xvt0D//+aDsXPfheVNePN8YYbnuu40Cfh9/ZxPSfvs7WqgYy0oVTxpcC8POXYzPRbW/g/k66cTrhHnprU9jjnXLPrwh8mbdoa+9oLz72/hbO+c1bzH95XZAjeoeo4pCNMS8ZYyYYYw4zxvzM3naHMWaBvdxkjLnEGDPOGDPDGLPJdWyZMWaAMabAGDPCGKPzwvcwpx8+qMO2aSP7AT7B3ew3vBrS7G//eVMs0Vy4aiciHdN7Xnva2C7X66Rxpbx+w2mANWJwp2uotttl8fHW6qiGyzpW/PB+uazZeYj3Nx5gw946bnh6ZchjJgwpZEiehJzBO5kIDCF88poTAFjmelvYcbCRr51YBhBU4BIVY6y3tkByMtM5rsxyrYUb+ekkzPr7B1t5b2PH2AJPmO/XH9+MXwikjtRLQfq7Esi/O+8MKufPYZDdueFYDrc8s8pbxnJZWIwfZHWmbdxXjwCjS3zZ3QSYPKy4W3UbN6jAm7jopPmLvOF5zo/r6BHF1DS2+iU6CoXj5pgyvIitVQ18/dFlQOjprACaW9vJDBJtcO8X/TPmdTUMqrmtnedX7OiVYeLOgJg/fuVYKufP4YSxJd5Zxh027a8nPU04eVwJn+yIv480WgyGUN6xS461urSC5WVxaHd9/pf/eYn3Ad/S5uGTHTXe/efYibluOe9wv+O3hTl3T6KCnOLkBPjKRtlRBOt217JpXx0/ePJjXli506vIF071HxqdniaMsyMeYhU19cuLj/YuT/jxy4DPZeFY8O9WRI6YdH5yU4I8JG7/zydB8z83tXrICvKtnzt1OFedVMZfrzoOIKzbIxzPr9jJD55cwbf+9mHYcu9vPEDZvIVsOdD1zrY99oPH6cgFePPmWX5lnAfwlOHFrN9dG3Iy3ETD43prC2TSsODx9IHHD3RNpLDKnuD3T29u5HP/9w5PLbMieb8z6zA2/3y2nzsNrBF88UAFOUW56/OTGVqcQ2GOf7yv4y8Fyy/8/IqdADTZUQ9pacJlMywLxDEyDrOt5li98k50CYiDY+2ecYRlsUSTB9d57ZwyoqMgP/7BFo668zXe3+gf7NPU2k4wb0VOZjp3fn4yp00YSF5WOu9VdC1IyLFQ31gXPrvdva9YfsrTflHOm/bAlsXr9vLUsuhjvPcdaiJNYHj/XO+2koJsfnfpVEoLsllz17m8YbuIThhbQku7p0dyQcSKA3XNnDx/ET9/eW1IHzJYb1npaRK289cYw/hBBSy+aRbgux/OvJILbcFNTxNExK/DOSsjzc+d1puoIKcoV55Uxvs/OjNorOZvvzy1w7adNb7BlHMDrOTP7KmfWmOU7qE4N5N7vmC5CEQsd4Ujro5F/+SybTS3hbfmHDfg4MLQs2Bf9mf/BEJNbe1hB0ikpQmzjxzKuxv3dynto9uqCzcM/AjXQ+nKR5ayans1X390Gf/z79VeN04k9hxqprQgu0Pky9ypw/nwx2eRn53hja450e7ojeUot1jyP8+s4tifvs6O6kb+9OYm24ccnJzMdMYNLPAO8wfrQeuelabdY0gT8b5x3f/GBjwew+gS/zjzfrmWe8+do2XayH4sj1OkjQpyH+TCacOZfeSQkPvHDfIflOEkB4ollx8/ivu+eBTGWLkyHJ9eepowzO6QWRzBynTn6fjWKWOYMryIs44YzPfPGBfymKZWD5kRvvWThhZR29TWpaRFbhH/YwhrtLmtnSeW+lvCK11vBJ+GeRV38BjDUx9ui3r2lpxMKw1qsORSicBTH27rsO2lT3aHLD9pWBGL1u31hvJd+MC7nHrfYu9+jyts7qJploFx479Wen37A+x+FvfbRZkt1lOGF1Oxty4u6WJVkPsoD15hdQQ9992TOuwLnFV63KDCDmViwdRRVuRH+fp9fgNPFt88i8KcDF5bEz6JTL3jZhHhtjmTePH7p/CXK6fzw7Mm8IcrjvGW2+2y/i2XRXjfi/MjvfiP73e6Tf5zGPos5JXbqnn90z14PIbDf/yKd3txruVS+tAl/tE8CFbu87U9Wk4+rIS65jbueH5N1Mf0Bo++uzno9m/MDD0idNbhAwG4Y4EV9rdudy07a5r43j8+4sIH3qWqvsX74LnjAivSdunmKu+D//UbTqNy/hy/t4sXvj+T8ptmMWFwAU2tHrbFIQezCnIfZ9qo/lxz6li/BPfBUmyuvvMcfnFqboft3cHpLLz7xU9pteNC00TIzkhn5rhSlmwOH/N8079W2vX1356WJpx/5FCuO92ylJ/92DeIIBoL+ZxJg8m3Hc1uMY8G58EyrDiHJ5dt9XYszn3gXb75tw857ZeL/cqv/Mk5fP7oYX4Jkj6KQpB311vXecTuhIwGJ0Lm8Q+2RH1MT1Pb1MqdL/giYd338tLjRgY5wuKCo4YBePtAHBau3sWKbdVsrWrg7Q1Wx3C/vCy+dcoYdtY0UtNoPSSDJckqzMmkrDSf8Xao5/oQs7T3JCrICrfOPoLrz5rgt+0L04YzxJUHoDAnk4F5sf26pKWJ95X7+098BPh+KMePGcCO6sagM0V4PIapd7/mO08II/Gbp1gW1n2vrHeFPQXv1HMjIvzzW1ZMb2fndHO8AVedXEZTq4eH3/a3/rZV+TqL7vviUQAcP3YAB12j6KIZKVjTbMjJTOvgXgrHzPGlDLIjD6obQqdB7U2OvNN3Hyvnz2Hzz+fw0FePZf5FR4bNve3ufK5uaPH7rjpkuL4Ypx8+CGN8Wf0kzFd54pBCstLTuuSy6i4qyEpQfvPlqXxw65k9fp1lt50FWB1UAGn2N3LGGKsTamkQK3nT/nq/yVrT04J/jfvl+VwvTu7k5jYPmVEM/3ZCq3YfaurUQAHHQj5mlDV44cVVlgUXOPXV+z86gy/ZFuCJrpGVhdkZ7Kpp4puPhQ+ba/EYb0rTzvCA7cpZFME/39tcc6pvwNE5k4dw6YxRYUpbOFbyx9uq/UZ6Oiy4bqZ3ecaYAX77wqWRzcvK4AvThjMiDommVJCVuFKcm8lZR/hGFjo+0YlDCinKyeCGp1dSNm+hXwdLRoCgOhOyBuPF71s/yosefA9jjC3IkeuVmZ7G/144BYD5L6/zmwA2GgbkZ3H1yWPYUd1Ia7uHfNeAjdKCLIYW++o8pjTfa+E5ccOvr90Tdmqh1na6JMhHj7D89jc8vTLuMcm1LdY9PXfyYG459/AIpTty9Mh+pAl8vOUgHo/x+pXBsradhypYc0SeO9k3O3uknCz3XnwUXz1hdKfr1F1UkJW4U1bisx6dH0pamnBcmc+qOfou36ut2xr69mljw0YaTHb9KN+xB5tEGZjg94M8ef6iqGJTPd7ID2HqqH40tXpYv7uWdo/xxigHztgiInxw65n8+WvTeezqGd4HTrhY7OZ24xdTHi1ZGWne0ZjRxHr3JLe/a32eFx0zwjupbmfIz87AY+D+RRUcbGhh1IA8hhTlhHRhPXC5r6O3M52hvYkKshJ3Dh/ii+Jwz203yhUzWtvc5vUDe3vPPzeJm84Jb1mJCE/Y/uBr/mbNRhKNy8LBPfLtpPmLIs695zwr0sSXP+TjbdV4jOHEwyzXRODrs8PZkwbTLy+LpbYbxxk8EoyWdrokyAD/uvZEAC596AP2HIrfZD7VzdaHNXNcabfP5TFW7HH5zbP45K5zg5bJSE9jya1ncv9l06IOF+xtErNWSp/iC9N8A1EGFviGuzoWqpPXd8FKyx/raOLgohwyo7CsTjyshPGDCmi0X9E7o2OjBuR5LUqA8be9zNOumNk1O2u8I+3AN7pREEb0z6W0IIvydXu902Qtu+0sHv16+MgIJ0Z2xbZqv8EObprbDXldTJDk9q0ff88bHHH7K2FKd6SlzeOXK+KrDy+hbN5C1u+ujWpQy6trdnPeb98iQyArPc3PndNZ1v/0PO/yP5ZsJScznbys0OcbXJTD548e1uXr9TQqyErcyUhPo3L+HNbefZ435AisVKGV8+fwzi1nAHD3C5/y0FsbqbdHwHXmLffbpx3mXY7Gh+wgIvz3htP8ks/c8swqNu6rwxjDnPvf4cpHlnpH5bkHq4gIowbk8ca6vazfU0tampVfIZxgOPzo/ImANTt2MBraoCi343xz0eJ0pgI0traHTcgUyIQfv+wn4k542bm/fYvDb3854vHffnw563bX0magpZuzfWdnpPPK9acA+MWeJysqyErCEColZnFeJhcfO4ID9S3c89I6b77izvgBjx3d37sc2CkYDd+dNY53553hXT/zV28y5kcvedeX2HmiHZeFUzUnnam1LfrrXjLdisD4z8fB8/k2tBq/GUE6y8DCbObZog9WovtocDpXW9o9PPLO5g4TzBrjn23PGMNTy7ZS1+wbJDMmIOKku0wcUkTl/Dmcf+TQmJ43HqggK0mBkwPXTWdmL3GLQFVj16yy4f1yWXnHOUH3OUOhnRzNzsPimlN9lnlnHiAD8q3ZPpZWVvm5Bxya2ronyADfnDmG604fx4D8LBau8mU3W7vrEGXzFgb1YT/hSn5094uf8rVHrDkNLz1uJF88xpqp5rifvc7n/u9tNu+vZ8PeOv7n36uZ8pNX+eu7m3n90z1+qU2dHM6KhQqykhTMCpJ0v7M95R/dfjZnThzEicO6LmTFQaalB2u2kZ3VjbR5fC4Lh7vnTgagspM5lp2HULAE641tUJDddZcFWK6im849nK8cP4oNe+som7eQsnkLOf93bwPwh/KN/HOJT4DL1+8NOkMJwJ2fn8yvvuRLq/rJjkM89l6l38wcd73wKQ+/4xso8+h5+UFnt+nLqCArScHgohw23TObyvlzvPGmne3UGpCfxcNXHUdJbve+9nd8zsqN8Ond51I5fw432zG0J81f5BUs98PitAlWfVfvqOnUdb5lD5b46sNL/azK5rZ22gwUZMdm1pNvzAw9C8ytz61mV40VnnbfK+u9291vChccPcwb8fGtU3z5Jx59r5LZ97/td773N1lpTb9+clm3652KqCArSYOTveueLxzJrbMn+sUp9yZXzxzD5p/P9nbOfe/0jtnlBrmSo48uyefkcSVB056Gw53L+vRfllM2byGt7R4O2HHMgbODdJXivEyW3uY/KrN/XqbX175w1S7+/sEWPMZQmJ1B5fw5FOdlsvnns3nwimP4X/sNAOC2OZOonD+H012DNAD++BX/Djfnoab4o4KsJB3D+uVyzamHBZ2VuLcI7KA76wjfKLBbzju8w/5/fPMELpzmn2c6GgKF7H+eWeXtgBsUJH9DVxlUmMONZ1v5TGYdPpAlt57F6juteN6fLlzLj//zCet213oTvIP1Gcw+cqhfGJ3DH75yrN/6eVOGcvYk32fUmQ7OvkRsHrGK0sf5/eXT2FHdyGEDo0/2Ew3nTRnKK9efwnm/tV79n3VFXZw3OXRO667w/TPHM3ZgATPHl3oHTnzv9MN4YLEvl8fNUQ5xzslMp3L+HJ79aLt3FN6fvzadp5dto6CbnZGpjFrIihIDcjLTYy7GDhOHFPHi92f65WoAeuQNYc5RQ705mgFuPneidx7GH5w5Pqh7JhwXHTPCbyDGl44byewUCE/rKfRRpShJwJThxdx49uGUr+/a5Kvd4ZXrT2FfbTOjS2IbP6x0RAVZUZKEI0cUs+5/z+PmZ1YxNr1rk7B2hbysDEaXqFT0BuqyUJQkIicznf+7bBrTBqlApiIqyIqiKAmCCrKiKEqCoIKsKIqSIKggK4qiJAgqyIqiKAmCCrKiKEqCoIKsKIqSIKggK4qiJAhiTMfZCOKJiOwD3PPJlAIdM3QnH6nQjlRoA6RGO7QNiUNgO0YbYwaGKhyOhBPkQETkQ2PM9HjXo7ukQjtSoQ2QGu3QNiQOsWyHuiwURVESBBVkRVGUBCEZBPmheFcgRqRCO1KhDZAa7dA2JA4xa0fC+5AVRVH6CslgISuKovQJVJAVRVEShIQWZBE5T0TWi0iFiMyLd33CISKVIrJaRFaIyIf2tgEi8l8R2WD/729vFxG5327XKhE5JvzZe7Tej4jIXhH5xLWt0/UWkSvt8htE5MoEaMOdIrLDvh8rRGS2a9+P7DasF5FzXdvj9n0TkZEislhEPhWRNSLyA3t70tyLMG1ItnuRIyJLRWSl3Y677O1jRGSJXaenRCTL3p5tr1fY+8sitS8kxpiE/APSgY3AWCALWAlMine9wtS3EigN2HYfMM9engfcay/PBl4GBDgBWBLHep8KHAN80tV6AwOATfb//vZy/zi34U7gpiBlJ9nfpWxgjP0dS4/39w0YChxjLxcCn9l1TZp7EaYNyXYvBCiwlzOBJfZn/DRwqb39j8B37OXvAn+0ly8FngrXvnDXTmQLeQZQYYzZZIxpAZ4E5sa5Tp1lLvCYvfwYcKFr+9+MxQdAPxGJy1S8xpi3gKqAzZ2t97nAf40xVcaYg8B/gfN6vvYWIdoQirnAk8aYZmPMZqAC67sW1++bMWaXMeYje7kWWAsMJ4nuRZg2hCJR74UxxtTZq5n2nwHOAJ6xtwfeC+cePQOcKSJC6PaFJJEFeTiwzbW+nfA3N94Y4DURWS4i19jbBhtjdtnLu4HB9nKit62z9U7U9lxnv84/4rzqkwRtsF95p2FZZkl5LwLaAEl2L0QkXURWAHuxHmobgWpjTFuQOnnra++vAUroQjsSWZCTjZnGmGOA84Hvicip7p3GeodJuhjDZK038AfgMGAqsAv4VXyrEx0iUgD8G7jeGHPIvS9Z7kWQNiTdvTDGtBtjpgIjsKzaib1x3UQW5B3ASNf6CHtbQmKM2WH/3ws8h3UT9ziuCPv/Xrt4orets/VOuPYYY/bYPyoP8Gd8r4oJ2wYRycQSsn8YY561NyfVvQjWhmS8Fw7GmGpgMXAillvIme7bXSdvfe39xcAButCORBbkZcB4u2czC8tZviDOdQqKiOSLSKGzDJwDfIJVX6eX+0rgeXt5AfA1u6f8BKDG9VqaCHS23q8C54hIf/t19Bx7W9wI8Ml/Aet+gNWGS+2e8THAeGApcf6+2T7Hh4G1xphfu3Ylzb0I1YYkvBcDRaSfvZwLnI3lD18MXGwXC7wXzj26GFhkv82Eal9oeqvnsit/WD3Jn2H5b26Ld33C1HMsVm/qSmCNU1csP9IbwAbgdWCA8fXiPmC3azUwPY51fwLrNbIVy8f1ja7UG7gaq9OiAvh6ArThcbuOq+wfxlBX+dvsNqwHzk+E73Zvjj8AAABzSURBVBswE8sdsQpYYf/NTqZ7EaYNyXYvjgI+tuv7CXCHvX0slqBWAP8Csu3tOfZ6hb1/bKT2hfrTodOKoigJQiK7LBRFUfoUKsiKoigJggqyoihKgqCCrCiKkiCoICuKoiQIKsiKoigJggqyoihKgvD/AdTJ0riRQdfLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  3%|▎         | 2999/100000 [04:34<2:12:28, 12.20it/s]"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-82-c38557ed92b4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0;31m# train\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m     \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss_t\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtrain_step\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtd_loss\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msample_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexp_replay\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m64\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m     \u001b[0mtd_loss_history\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss_t\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    927\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    928\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 929\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    930\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    931\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1150\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1151\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1152\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1153\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1154\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1326\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1327\u001b[0m       return self._do_call(_run_fn, feeds, fetches, targets, options,\n\u001b[0;32m-> 1328\u001b[0;31m                            run_metadata)\n\u001b[0m\u001b[1;32m   1329\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1330\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1332\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1333\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1334\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1335\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1336\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1317\u001b[0m       \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_extend_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1318\u001b[0m       return self._call_tf_sessionrun(\n\u001b[0;32m-> 1319\u001b[0;31m           options, feed_dict, fetch_list, target_list, run_metadata)\n\u001b[0m\u001b[1;32m   1320\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1321\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_call_tf_sessionrun\u001b[0;34m(self, options, feed_dict, fetch_list, target_list, run_metadata)\u001b[0m\n\u001b[1;32m   1405\u001b[0m     return tf_session.TF_SessionRun_wrapper(\n\u001b[1;32m   1406\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1407\u001b[0;31m         run_metadata)\n\u001b[0m\u001b[1;32m   1408\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1409\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_call_tf_sessionprun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "for i in trange(10**5):\n",
    "    \n",
    "    # play\n",
    "    play_and_record(agent, env, exp_replay, 10)\n",
    "    \n",
    "    # train\n",
    "    _, loss_t = sess.run([train_step, td_loss], sample_batch(exp_replay, batch_size=64))\n",
    "    td_loss_history.append(loss_t)\n",
    "    \n",
    "    # adjust agent parameters\n",
    "    if i % 500 == 0:\n",
    "        load_weigths_into_target_network(agent, target_network)\n",
    "        agent.epsilon = max(agent.epsilon * 0.99, 0.01)\n",
    "        mean_rw_history.append(evaluate(make_env(), agent, n_games=3))\n",
    "    \n",
    "    if i % 100 == 0:\n",
    "        clear_output(True)\n",
    "        print(\"buffer size = %i, epsilon = %.5f\" % (len(exp_replay), agent.epsilon))\n",
    "        \n",
    "        plt.subplot(1,2,1)\n",
    "        plt.title(\"mean reward per game\")\n",
    "        plt.plot(mean_rw_history)\n",
    "        plt.grid()\n",
    "\n",
    "        assert not np.isnan(loss_t)\n",
    "        plt.figure(figsize=[12, 4])\n",
    "        plt.subplot(1,2,2)\n",
    "        plt.title(\"TD loss history (moving average)\")\n",
    "        plt.plot(moving_average(np.array(td_loss_history), span=100, min_periods=100))\n",
    "        plt.grid()\n",
    "        plt.show()\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assert np.mean(mean_rw_history[-10:]) > 10.\n",
    "print(\"That's good enough for tutorial.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__ How to interpret plots: __\n",
    "\n",
    "\n",
    "This aint no supervised learning so don't expect anything to improve monotonously. \n",
    "* __ TD loss __ is the MSE between agent's current Q-values and target Q-values. It may slowly increase or decrease, it's ok. The \"not ok\" behavior includes going NaN or stayng at exactly zero before agent has perfect performance.\n",
    "* __ mean reward__ is the expected sum of r(s,a) agent gets over the full game session. It will oscillate, but on average it should get higher over time (after a few thousand iterations...). \n",
    " * In basic q-learning implementation it takes 5-10k steps to \"warm up\" agent before it starts to get better.\n",
    "* __ buffer size__ - this one is simple. It should go up and cap at max size.\n",
    "* __ epsilon__ - agent's willingness to explore. If you see that agent's already at 0.01 epsilon before it's average reward is above 0 - __ it means you need to increase epsilon__. Set it back to some 0.2 - 0.5 and decrease the pace at which it goes down.\n",
    "* Also please ignore first 100-200 steps of each plot - they're just oscillations because of the way moving average works.\n",
    "\n",
    "At first your agent will lose quickly. Then it will learn to suck less and at least hit the ball a few times before it loses. Finally it will learn to actually score points.\n",
    "\n",
    "__Training will take time.__ A lot of it actually. An optimistic estimate is to say it's gonna start winning (average reward > 10) after 10k steps. \n",
    "\n",
    "But hey, look on the bright side of things:\n",
    "\n",
    "![img](https://s17.postimg.org/hy2v7r8hr/my_bot_is_training.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "agent.epsilon=0 # Don't forget to reset epsilon back to previous value if you want to go on training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#record sessions\n",
    "import gym.wrappers\n",
    "env_monitor = gym.wrappers.Monitor(make_env(),directory=\"videos\",force=True)\n",
    "sessions = [evaluate(env_monitor, agent, n_games=1) for _ in range(100)]\n",
    "env_monitor.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./videos/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./videos/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More\n",
    "\n",
    "If you want to play with DQN a bit more, here's a list of things you can try with it:\n",
    "\n",
    "### Easy:\n",
    "* Implementing __double q-learning__ shouldn't be a problem if you've already have target networks in place.\n",
    "  * You will probably need `tf.argmax` to select best actions\n",
    "  * Here's an original [article](https://arxiv.org/abs/1509.06461)\n",
    "\n",
    "* __Dueling__ architecture is also quite straightforward if you have standard DQN.\n",
    "  * You will need to change network architecture, namely the q-values layer\n",
    "  * It must now contain two heads: V(s) and A(s,a), both dense layers\n",
    "  * You should then add them up via elemwise sum layer.\n",
    "  * Here's an [article](https://arxiv.org/pdf/1511.06581.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hard: Prioritized experience replay\n",
    "\n",
    "In this section, you're invited to implement prioritized experience replay\n",
    "\n",
    "* You will probably need to provide a custom data structure\n",
    "* Once pool.update is called, collect the pool.experience_replay.observations, actions, rewards and is_alive and store them in your data structure\n",
    "* You can now sample such transitions in proportion to the error (see [article](https://arxiv.org/abs/1511.05952)) for training.\n",
    "\n",
    "It's probably more convenient to explicitly declare inputs for \"sample observations\", \"sample actions\" and so on to plug them into q-learning.\n",
    "\n",
    "Prioritized (and even normal) experience replay should greatly reduce amount of game sessions you need to play in order to achieve good performance. \n",
    "\n",
    "While it's effect on runtime is limited for atari, more complicated envs (further in the course) will certainly benefit for it.\n",
    "\n",
    "There is even more out there - see this [overview article](https://arxiv.org/abs/1710.02298)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from submit import submit_breakout\n",
    "env = make_env()\n",
    "submit_breakout(agent, env, evaluate, <EMAIL>, <TOKEN>)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}