pratical-rl/week5_policy_based/practice_a3c.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deep Kung-Fu with advantage actor-critic\n",
    "\n",
    "In this notebook you'll build a deep reinforcement learning agent for atari [KungFuMaster](https://gym.openai.com/envs/KungFuMaster-v0/) and train it with advantage actor-critic.\n",
    "\n",
    "![http://www.retroland.com/wp-content/uploads/2011/07/King-Fu-Master.jpg](http://www.retroland.com/wp-content/uploads/2011/07/King-Fu-Master.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function, division\n",
    "from IPython.core import display\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "\n",
    "#If you are running on a server, launch xvfb to record game videos\n",
    "#Please make sure you have xvfb installed\n",
    "import os\n",
    "if os.environ.get(\"DISPLAY\") is str and len(os.environ.get(\"DISPLAY\"))!=0:\n",
    "    !bash ../xvfb start\n",
    "    %env DISPLAY=:1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For starters, let's take a look at the game itself:\n",
    "* Image resized to 42x42 and grayscale to run faster\n",
    "* Rewards divided by 100 'cuz they are all divisible by 100\n",
    "* Agent sees last 4 frames of game to account for object velocity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observation shape: (42, 42, 4)\n",
      "Num actions: 14\n",
      "Action names: ['NOOP', 'UP', 'RIGHT', 'LEFT', 'DOWN', 'DOWNRIGHT', 'DOWNLEFT', 'RIGHTFIRE', 'LEFTFIRE', 'DOWNFIRE', 'UPRIGHTFIRE', 'UPLEFTFIRE', 'DOWNRIGHTFIRE', 'DOWNLEFTFIRE']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/robotcator/gym/gym/logger.py:30: UserWarning: \u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "from atari_util import PreprocessAtari\n",
    "\n",
    "def make_env():\n",
    "    env = gym.make(\"KungFuMasterDeterministic-v0\")\n",
    "    env = PreprocessAtari(env, height=42, width=42,\n",
    "                          crop = lambda img: img[60:-30, 5:],\n",
    "                          dim_order = 'tensorflow',\n",
    "                          color=False, n_frames=4,\n",
    "                          reward_scale = 0.01)\n",
    "    return env\n",
    "\n",
    "env = make_env()\n",
    "\n",
    "obs_shape = env.observation_space.shape\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "print(\"Observation shape:\", obs_shape)\n",
    "print(\"Num actions:\", n_actions)\n",
    "print(\"Action names:\", env.env.env.get_action_meanings())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAEICAYAAADBfBG8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFnxJREFUeJztnXm0HGWZxn8PQdADKGFLIGyBEzgDLjEiZkQWcQsZR2QWDM4oKDOEkTB4wCMEFBF1ABWYKCMkaIZFBBkRZTwBRcBlBkEghrAJJCySQBKEsIjISHjnj6qGSqf73uqu7ltVfZ/fOXW6+vu+qnrq3nrqW+rrtxQRGGO6Z72yBRhTd2wiYwpiExlTEJvImILYRMYUxCYypiA20QAiaXtJf5A0pmwtowGbqACSZki6WdJzklal65+QpDJ1RcTvImLjiFhTpo7Rgk3UJZKOA+YAXwHGA+OAI4G9gA1KlGZGmojw0uECvA54DvjbYcr9FfAb4BngEeCUTN6OQAAfS/NWk5jwrcBi4CngnKb9fRy4Jy37Y2CHNsdt7Hv99PvPgC8CNwJ/AP4b2By4JNV2C7BjZvs5qaZngNuAvTN5rwEuTDXcA3waWJbJ3wa4AngceBD417L/X32/HsoWUMcFmAa82LhIhyi3H/AGkhr/jcBK4INpXuNCPw94NfBe4E/AD4CtgAnAKmDftPyBwBLgL4D1gc8AN7Y5bisTLQF2Tm8AdwP3Ae9O93UR8J+Z7f8xNdn6wHHACuDVad7pwM+BscC2qeGXpXnrpaY7maQ23gl4AHhf2f+zvl4PZQuo45JeZCua0m5Ma4/ngX3abPfvwNnpeuNCn5DJfwL4UOb7FcAn0/WrgcMzeesBf6RFbdTGRCdl8s8Ers58/2tg0RDnuxp4U7q+limAf8qY6G3A75q2nZ016CAu7hN1xxPAFpLWbyRExNsjYtM0bz0ASW+TdIOkxyU9TdJc26JpXysz68+3+L5xur4DMEfSU5KeAp4ERFJj5SHvcZD0KUn3SHo6PdbrMrq3IWnqNciu7wBs09CYbnsiSX9xYLGJuuNXwAskTayh+A5wFbBdRLyOpOnW7cjdI8DMiNg0s7wmIm7scn8tkbQ3ST/nYGBsemN4mld0P0bSjGuwXZPGB5s0bhIR03upsWrYRF0QEU8Bnwe+IenvJG0iaT1Jk4GNMkU3AZ6MiD9J2hP4cIHDngfMlrQ7gKTXSfr7AvtrxyYk/b3HgfUlnQy8NpN/eapjrKQJwKxM3q+BZyUdL+k1ksZIer2kt/ZBZ2WwibokIr4MHEty116ZLnOB40n6RwCfAE6V9CxJZ/vyAse7EjgDuEzSM8CdwAFdn0B7fgxcQzLw8DDJYEe2yXYqsIxk5O2nwPdIamUieS71fmBymv974JskzcGBRWnnz5iukPQvwIyI2LdsLWXhmsh0hKStJe2VNl93JRkCv7JsXWWy/vBFjFmLDUiarRNJhvQvA75RqqKS6VtzTtI0kiffY4BvRsTpfTmQMSXTFxOls4fvA95D0gm9BTgkIu7u+cGMKZl+Nef2BJZExAMAki4jeabS0kSSPLphqsjvI2LL4Qr1a2BhAmsPiy6j6cm6pCMk3Srp1j5pMKYoD+cpVNrAQkTMA+aBayJTb/pVEy1n7ekg26Zpxgwc/TLRLcAkSRMlbQDMIJlDZszA0ZfmXES8KGkWyRSSMcD8iLirH8cypmwqMe3HfSJTUW6LiD2GK+RpP8YUpBbTfo455piyJZhRyJw5c3KVc01kTEFqURONFDNnzgRg7ty5bfOyNJdrLtNpvqknrolSWpmkVd7cuXNfvviz6VkDdpNv6otNlOJawXSLTZSDrMFmzpw5ZNOuXb4ZXGwiYwrigYWcDDdI0FzGtdHowTVRDvIYwqYZvdRi2s9IPGztdHg6TxkPcdebOXPm5Jr2YxMZ04a8JnJzzpiC2ETGFMSjcxVi7Oyx66StPm11CUpMJ7gmqggNA60+bfXLSzbdVBebyJiCdG0iSdulL7C6W9Jdko5J00+RtFzSonQZ6HfTGFOkT/QicFxELJS0CXCbpGvTvLMj4qvF5RlTfbo2UUQ8RvLWNCLiWUn3kP/Vh8YMDD3pE0naEXgzcHOaNEvSYknzJbXsGTsC6tpkBxIaSzbdVJfCQ9ySNuaVt1w/I+lc4Askb6/+Asmbqj/evJ0joK6LDVNPCtVEkl5FYqBLIuL7ABGxMiLWRMRLwPkkwe2NGViKjM4J+BZwT0SclUnfOlPsIJJ3ixozsBRpzu0FfAS4Q9KiNO1E4JD0LdoBPAT4NwJmoCkyOvc/gFpkLehejqki/gnH0IzauXN33HvIWt/fsOulHeX3Yh95jlE2M2fObBljwkZ6BU/7MUNiswyPTWRyM1Rwy9GMTWRy46CTrbGJzJDYMMPjGAtmWEbr6FzeGAujdnTO5Ge0mKZb3JwzpiA2kTEFsYmMKcio6RM1v2Oo1ZP4VvnZzyzNaY19zZ59f79OoSecdtqksiUMHKOqJhqug5ynA519SVfebcxgM6pMNNwzj+b8VuXzlDGji1FlouZapFV+83pz+VbbuzYa3YwqEzXTzVvtmrdp1V8yowvPWDCmDSM2Y0HSQ8CzwBrgxYjYQ9JmwHeBHUl+3XpwRDgKhxlIetWce2dETM649gTguoiYBFyXfjdmIOnXc6IDgf3S9QuBnwHH9+lYHdHJ86BW6a22yXLAL385MifSJVfvvXfZEgaOXpgogJ+k/Zq5aTy5cWmEVIAVwLgeHKdnFH1NpDFZetGce0dETAEOAI6StE82M5KRi3UGDsqMgNrp86Juy5jRQWETRcTy9HMVcCVJsMaVjfhz6eeqFtvNi4g98ox+9JpOZy60++7nQwaKR0DdKH0jBJI2At5LEqzxKuDQtNihwA+LHKfXtHrWM1S+MUNR6DmRpJ1Iah9I+lffiYgvSdocuBzYHniYZIj7ySH24+dEpnKMyHOiiHgAeFOL9CeAdxXZtzF1oRYzFowpicGJsTDli1PKlmBGIQs/szBXuVqYaKtttypbgjFtqYWJ1rt8VE82NxWnFiZatO2i4QsZUxK1MNH47ceXLcGMQh7l0Vzl3E4ypiC1qIk8sGCqjJ8TGdOeXM+J3JwzpiA2kTEFqUWf6JopnrFgRp5pC/PNWHBNZExBbCJjCmITGVOQWvSJJi/wjAVTAjkvO9dExhSk65pI0q4kUU4b7AScDGwK/DPweJp+YkQs6Foh8OHDTs5VbvZxRwNw2plfL3K4QlhD9TR0ryPfZdu1iSLiXmAygKQxwHKSeAsfA86OiK92u+9uWXP8mmSlxFlC1lBBDX3W0as+0buApRHxsKQe7bJzxpwxJlk5szQJ1lBFDX3W0SsTzQAuzXyfJemjwK3AcSMVzL5Sdz9rqI6GPusoPLAgaQPgA8B/pUnnAjuTNPUeo809oB8RUMecMWbtu08JWEP1NPRbRy9qogOAhRGxEqDxCSDpfOBHrTZKY3bPS8v1ZBZ3pe5+1lAdDX3W0QsTHUKmKSdp60ww+4NIIqKOCJVqh1tDdTT0WUchE6Whg98DZGPuflnSZJIg9g815fWVSt39rKE6Gvqso2gE1OeAzZvSPlJIUQEqdfezhupo6LOOWkz7yUul7n7WUB0NfdYxUCaq1N3PGqqjoc86BspElbr7WUN1NPRZx0CZqFJ3P2uojoY+6xgoE1Xq7mcN1dHQZx0DZaJK3f2soToa+qxjoExUqbufNVRHQ5911CJ444oV00dKijEvM378AgdvNGYkqEVz7oYpfrWKqS6uiYwpiE1kTEFsImMKUos+0TsXTi5bghmNjPeb8owZEWpRE+WNO2dMb8kXd841kTEFyWUiSfMlrZJ0ZyZtM0nXSro//RybpkvS1yQtkbRYkl8uZAaavDXRBcC0prQTgOsiYhJwXfodkug/k9LlCJIQWsYMLLlMFBG/AJ5sSj4QuDBdvxD4YCb9oki4CdhU0ta9EGtMFSnSJxqXCY21AhiXrk8AHsmUW5amrUU/gjcaUwY9GZ2LiOg0AGM/gjcaUwZFaqKVjWZa+rkqTV8ObJcpt22aZsxAUsREVwGHpuuHAj/MpH80HaWbCjydafYZM3Dkas5JuhTYD9hC0jLgc8DpwOWSDgceBg5Oiy8ApgNLgD+SvK/ImIEll4ki4pA2We9qUTaAo4qIMqZOeMaCMQWxiYwpiE1kTEFsImMKYhMZUxCbyJiC2ETGFMQmMqYgNpExBbGJjCmITWRMQWwiYwpiExlTEJvImILYRMYUxCYypiA2kTEFGdZEbaKffkXSb9MIp1dK2jRN31HS85IWpct5/RRvTBXIUxNdwLrRT68FXh8RbwTuA2Zn8pZGxOR0ObI3Mo2pLsOaqFX004j4SUS8mH69iSQsljGjkl70iT4OXJ35PlHSbyT9XNLe7TZyBFQzKBSKgCrpJOBF4JI06TFg+4h4QtJbgB9I2j0inmnetpcRUK+/ZurL6/tPu6nIrmqtYSiqrq/OdF0TSToMeD/wD2mYLCLihYh4Il2/DVgK7NIDnW3JXhxlUQUNnVA3vVWnKxNJmgZ8GvhARPwxk76lpDHp+k4kr1d5oBdC81KFC6QKGrJUTc+gMWxzrk3009nAhsC1kgBuSkfi9gFOlfRn4CXgyIhofiVLX2g0Ucq8YKqgoR1V1lZ3hjVRm+in32pT9grgiqKiuqFxcZTZ3q+ChlbsP+0mm6eP1OLFx0Ox/7Sb+PrbT335+9E3jk4Nw7H429NfXv/kt/0i6V7iaT/GFGQgTHT0jSev9TlaNQxFo/ZxLdR7at+cA9jl9sUcTbkXR1kazjnrtQDMOnadR3Etyn2Vc9J3uQ9X3uSn9jXRLrcvXutzNGloGKh5fahyecqbzqi9ibKUaaQqaWhwzlmvtVlGgNo256pysZapo9EkaxhlOMM0lze9YSBqovve9MayJZSqIdu/mXXsMy2/NxvIfaLeUduayLSmuZZxrdN/BqImMq1rluZaaaiypntqb6LR3pTL0myOxsBC1kw2UO+pvYmyHfuyLuYqaBiKrJlM76m9icza2CgjT+0HFqpw56+Chiy77bbbOjPJr79mauVmlw8KromMKUhtTbRm/r6smb/vWt/L0lG2huFwLdRfat+cA9j5mLFlS6iEhgb7T7tp7edDZ93tvlIf6TYC6imSlmcinU7P5M2WtETSvZLe1y/hrajChVwFDc3YQP2l2wioAGdnIp0uAJC0GzAD2D3d5huNwCW9Zumc1Syds5qdjxnL0jmr+3GI3DrK1mDKJU+MhV9I2jHn/g4ELouIF4AHJS0B9gR+1bXCHFThIq6CBlMORQYWZqUB7edLarRhJgCPZMosS9PWoVcRUBsXbpnNqCpoMOXRrYnOBXYGJpNEPT2z0x1ExLyI2CMi9uhSwzpU4SKuggZPOh1ZujJRRKyMiDUR8RJwPkmTDWA5sF2m6LZpmikJDyr0n24joG6d+XoQ0Bi5uwqYIWlDSRNJIqD+upjEoanCnb8KGkx5dBsBdT9Jk4EAHgJmAkTEXZIuB+4mCXR/VESs6Y900wo35UaenkZATct/CfhSEVF5qMrdvyo6THnUdtpPK6owxFwFDWZkUfpWlHJFDPN+oqHmfe21Ihm3+N/xLUfSR4QqaMhS1ZjgdeP6a6belmf0uBZz546d0v7Vrzcv+CyQXMhvm/6FkZJUOQ1Zrr8m+Rzq72aGp/F3HI7aN+eqcNFWQUMr3v9vi8qWMCqoRXPOmJIYnObcj06cXLYEMwrJW5PXvjlnTNnYRMYUxCYypiAeWDCmPR5YMKYIHlgwZoSoRXNuxYrpQ2Ub0xfGj18wOM25G6b4ybupLm7OGVMQm8iYgthExhSk2wio381EP31I0qI0fUdJz2fyzuuneGOqQJ6BhQuAc4CLGgkR8aHGuqQzgacz5ZdGRE8f7LxzoZ8TmRIY/2iuYoUioEoScDCwfwfSOmb8+AX93L0xhSg6xL03sDIi7s+kTZT0G+AZ4DMR8ctWG0o6Ajgiz0Eu3WabgjKN6ZxDHu1RTTTccYBLM98fA7aPiCckvQX4gaTdI2KdCIIRMQ+YB547Z+pN1yaStD7wN8BbGmlpIPsX0vXbJC0FdgEKxdvuhGz/qfGQtlWaNZSvYSR0tDteL/8WRYa43w38NiKWNRIkbdl4lYqknUgioD5QSGEXtPqjjPSsB2sYWkNVdPSCPEPcl5K8GmVXScskHZ5mzWDtphzAPsDidMj7e8CREfFkT5QaU1G6jYBKRBzWIu0K4IrisoypD56xYExBBtZE2fZuWbPAraE6GvqpoxY/heiUKsxwsIbqaID+6qjFj/L8sNWUwSGPPprrR3m1MJExJTE4v2xN5r92zsV/+XkAPvKrz/VSjDXUVEPnOmbl2ufADiwYM1LYRMYUxCYypiC16BON32bzUrfvBdZQHQ2QT8eKfL+EcE1kTFFqURNtOb7zN3SfdcZnOfb4iwG4+MLPcuzxI/82O2uojoZudIzqmuiSC05n3LiNXv4+btxGXHLB6dZQIQ1V0dEL6lETbbVpV9s1/5G63U8RrKG1hqro6IWGWsxY6OZV8t+54NS1vn/4sJM73kdRrKE6GrrRcf01Uwdn2k83JjKmKHlNNJB9ImNGkjw/D99O0g2S7pZ0l6Rj0vTNJF0r6f70c2yaLklfk7RE0mJJU/p9EsaUSZ6a6EXguIjYDZgKHCVpN+AE4LqImARcl34HOIAkQMkkkrhy5/ZctTEVYlgTRcRjEbEwXX8WuAeYABwIXJgWuxD4YLp+IHBRJNwEbCpp654rN6YidDTEnYYTfjNwMzAuIh5Ls1YA49L1CcAjmc2WpWmPZdI6ioB6/TVTO5FpzIiSe2BB0sYkkXw+2RzRNJIhvo6G+SJiXkTskWf0w5gqk8tEkl5FYqBLIuL7afLKRjMt/VyVpi8Htstsvm2aZsxAkmd0TsC3gHsi4qxM1lXAoen6ocAPM+kfTUfppgJPZ5p9xgweETHkAryDpKm2GFiULtOBzUlG5e4HfgpslpYX8B/AUuAOYI8cxwgvXiq43DrctRsR9ZixYExJeMaCMSOBTWRMQWwiYwpiExlTkKr8KO/3wHPp56CwBYNzPoN0LpD/fHbIs7NKjM4BSLp1kGYvDNL5DNK5QO/Px805YwpiExlTkCqZaF7ZAnrMIJ3PIJ0L9Ph8KtMnMqauVKkmMqaW2ETGFKR0E0maJuneNLDJCcNvUT0kPSTpDkmLJN2aprUM5FJFJM2XtErSnZm02gaiaXM+p0hanv6PFkmansmbnZ7PvZLe1/EB80z17tcCjCH5ycROwAbA7cBuZWrq8jweArZoSvsycEK6fgJwRtk6h9C/DzAFuHM4/SQ/g7ma5CcvU4Gby9af83xOAT7Vouxu6XW3ITAxvR7HdHK8smuiPYElEfFARPwfcBlJoJNBoF0gl8oREb8AnmxKrm0gmjbn044Dgcsi4oWIeBBYQnJd5qZsE7ULalI3AviJpNvSACzQPpBLXeg0EE0dmJU2QednmteFz6dsEw0K74iIKSQx946StE82M5J2Q22fJdRdf8q5wM7AZJLIU2f2asdlm2gggppExPL0cxVwJUlzoF0gl7owUIFoImJlRKyJiJeA83mlyVb4fMo20S3AJEkTJW0AzCAJdFIbJG0kaZPGOvBe4E7aB3KpCwMViKap33YQyf8IkvOZIWlDSRNJIvf+uqOdV2AkZTpwH8moyEll6+lC/04kozu3A3c1zoE2gVyquACXkjRx/kzSJzi8nX66CERTkfO5ONW7ODXO1pnyJ6Xncy9wQKfH87QfYwpSdnPOmNpjExlTEJvImILYRMYUxCYypiA2kTEFsYmMKcj/A9KvXLIGo6A2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAACDCAYAAACdg+BGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHL9JREFUeJztnXu8XFWV57/rVt1787hJyIsQkpAEDSCOAjZgaLWbVlAEFJyxbRAx2NCMOt0DDtojOtOD3TojtO3YfqY/IqMig4gg8kin7UGaAcEWgYC88oAgBJKQ9+vmeR9Va/5Y69zULW7dW69bVfewvp9PfarqPPZZ+3fO2XvttffZR1SVIAiCYOzT1mwDgiAIgvoQBXoQBEFKiAI9CIIgJUSBHgRBkBKiQA+CIEgJUaAHQRCkhCjQg5KIyAIRURHJNtuWShCRi0TkF6OUdqeIrBSR2VXuP0tEHhKRPSLyd/W2r1b8fL+5yn0H5U2MG0Vkp4g8VmWaj4nIW6vZ943ImLpRxzoi8iBwAnCEqvY06JgKLFLVFxtxvEYjIguAl4F2Ve0HUNVbgFtG6ZCXAw+p6sYiOzqAp4FJqjp3hP23AZM1fQ+BDMqbiLwHOBOYq6r7qkzzG8BfA/+uTjammvDQG4QXPO8BFPhwU41pIdyLG0vX4aeBm4dY/gVgaxn7zwdWlirMx1prqIjivM0H1lZTmBfosBT4IxE5ok42phtVjU8DPsBfAf8KfBNYVrRuOvCPQDfwOPBV4FcF648D7gN2AM8DHytY90PgH4B/AvYAjwJv8nUPYRXIPmAv8CdD2NUG/BfgFWAL8H+AKb5uge9/OfAasBH4fMG+pwLL3e7NwDcL1i0Gfg3swjzX0wvWPQh8zfU4APxnYHmRXZ8Dlvrvc4Df+nHWAdcUbPeq27jXP6cBlxTp9/uu627//v0iW/7GbdkD/AKYUeIcHuX2ZouWLwRWAR8E1g9zDfwQ6AN63dYzgGuAO4Afef4uc10fce02Av8L6ChIR4HPAmvc5r8B3uR6dwO3F21/LvCUp/dr4O3D2KjAfwRewrztvwXafN01wI8Ktk2uj+wQefv3wEEg5/+/MpItwFq/Fp4BehKdsWt/SbPv4bHwaboBb5QP8KLfhL/nF/6sgnU/8c8E4HgvtH7l6yb6/0/5jXOS32jH+/ofAtu9EMhioYafFKStwJuHsetP3bajgS7gTuBmX5fcsLe6HW/DvNAzfP0jwMX+uwtY7L/nuE1nYxXGmf5/pq9/ECuI3+o2T/GCaVGBXY8DF/jv0/3YbcDbscrj/CIbswX7XlKg3zRgJ3CxH+tC/z+9wJbfAccA4/3/10todQ6wYojly4CPuJ0lC/SC8/XVgv/X+PVwvudvvF8ji93eBVhlcWXROb0HmOwa9gD3+zmcAqzEC0DsetkCvBPIAEuwgrOzhH0KPOC6HQW8AFxWYOuQBXqJvA2ch3Js8d9PAfOA8QX7fZsCZyE+pT9jqak7ZhGRd2PNz9tV9QmsAPm4r8tg8cH/pqr7VXUlcFPB7udizdYbVbVfVX8L/Az444Jt7lLVx9RiyLcAJ1Zg3kXYzfKSqu4FrgYuKGr6f0VV96nqs8CNWKEIVhC9WURmqOpeVf2NL/8E8HNV/bmq5lX1PsyTP7sgzR+q6grP026sgLrQNVmEtUqWAqjqg6r6rKf1DFbB/GGZ+TsHWKOqN/uxbgVWAx8q2OZGVX1BVQ9g3m0p/Q7DKp4BROQjQEZV7yrTnqF4RFXv9vwdUNUnVPU3bu9a4Lu8Pr/XqWq3qq4AngN+4edwN/DPWOEJ1rr6rqo+qqo5Vb0JqwAWD2PPtaq6Q1VfBb7FofNdK+XY8m1VXefnImEPpn0wAlGgN4Yl2A23zf//2JcBzMQ8sXUF2xf+ng+8U0R2JR+sEC6MKW4q+L0f85bL5Ugs3JLwitszq4Q9r/g+AJdinu1qEXlcRM4tsPmPi2x+N1A4MqQwTTBNkoLj48DdqrofQETeKSIPiMhWEdmNxbFnVJm/JA9zCv6Xq99OYFLyR0QmAtdhIYrXISLXi8he/3xpGBsHaSEix4jIMhHZJCLdwH/n9fndXPD7wBD/kzzMB64qOhfzOHQOR7Kn8HzXSjm2FF8XYJrvqpMNqWYsd8CMCURkPPAxICMiScHRCRwmIidg3lU/MBdr3oJd5AnrgF+q6pmjZOJr2I2WcJTbs9ltSuxZXbD+NQBVXQNc6J2a/xa4Q0Smu803q+qfDXPc4k7B+4CZInIiVrB/rmDdj7E48gdV9aCIfItDBdxII0WK85fk4f+OsN9QPAMsFJGst4YWYWGHh0UEoAOY4ud5sap+Gqt8RqI4D9/B+gwuVNU9InIl8NEq7AU7F19T1a9VsM88YIX/HjjfWF/MhILtKu2oLMeWoc7nW7A+hmAEwkMffc7HOoaOx5ryJ2IX6MPAJ1U1h8WtrxGRCSJyHPDJgv2XAceIyMUi0u6fU0TkLWUefzMWWy3FrcDnRGShiHRh3uBtXmAl/Fe37a1YLP82ABH5hIjMVNU8hzyoPHbzfUhEPiAiGREZJyKni0jJ4Xyq2gf8FOuEm4YV8AmTgB1emJ+Kh6ucrX7MUnn8Oabfx0UkKyJ/gp2LZcNoUsrG9Vh/w6m+6Dms8EvO62WY3icytKdZLpOwzs29fj18poa0/jfwaW/liIhMFJFzRGTSMPt8QUSmisg84Ar8fGPx7T8QkaNEZAoWnhtVW0RkHNancF+pbYJDRIE++izBYrSvquqm5IN5nBd5rPrPsc6sTdiQuFux2CKqugd4P3AB5iltAq7FvPxyuAa4yZu4Hxti/Q/8mA9h47kPAn9RtM0vsYLsfuAbqpo8tHMWsEJE9gJ/j3ViHlDVdcB5wJewAncdNqxvpOvtx9jIj58WVSifBf5aRPZgo4VuT1Z4WOZrwL96HgfFhlV1O9YPcRXWMfuXwLkF4a9K+S7WwYrHuAvP6Q4g7/9zVaYP8Hms0tqDFYK3Db95aVR1OfBn2PW2EzuPl4yw2z3AE1gB/k/A9z2t+9yWZ3x9RZVilbZ8CHhQVV8bYbsAENW0Pdsw9hGRa7GHj5aMuHHQUESkEwuHvE+LHi4K6o+IPApcqqrPNduWsUAU6C2AN6s7gGeBU7AwwWWqendTDQuCYEwRnaKtwSQszHIkFoP9O6zZGwRBUDY1eegichYWO80A31PVr9fLsCAIgqAyqi7Q/YGYF7CnANdjT/Zd6A/GBEEQBA2mllEupwIv+tNpvdij6+fVx6wgCIKgUmqJoc9h8Fjb9dgcDSXpaBun49u66D9sfA2HDRKyu+zp6NCzdkLL+hJ61pf929dvU9WZI2036p2iInI5NocD42Qip3Wdx/ZzbL76fHTJ1sTMn9nDfKFn7YSW9SX0rC9P/uCq4ukrhqQWmTcw+BH1ub5sEKp6A3ADwGSZprnubqausvmNNBvPNdVCrrsbIPSsA6FlfQk9m0MtKj8OLPJHxjuwJxmX1sesIAiCoFKq9tBVtV9E/hy4Fxu2+AOfyrMkksmQmTyFLSfY1A0azbCamLV6CkDoWQdCy/oSetaZR8rbrCaZVfXn2FONQRAEQZNpbL3ZnoXZh7PrfdYD3tnZB4DNPAq1zkJQazqtZkdCyXQeOhwg9Cxz/4Qh0wktK9o/Ia7N+tiRUDKd68tLL3oqgiAIUkLjI1t5JdeTAaBPrDrKZPMA5PqtflGVofctgbTZ/pmMpdffl6nKtEw2N+j4+Vx19V223dIZ9fzke+w4oeewlJWf0LIs4tocTMP0LJPw0IMgCFJCYz30vCI9vWS2twPQ32mH70+qlXxtydeaTq64cq0yLparNT9uR39iT4l0pGcHQOg5EmXoGVqWSVybg9NpkJ7lEh56EARBSmiJ0aHq8SPJVxZ/GsCrJW2rLZ3X2VFlrV1zfpLDV5mf0LOIGvQMLYuIa3P4dCqlRj2LCQ89CIIgJbSEhy6vC2hViMedhNrSGbCj1rGpTc5Ps4//OjvGsJ6hZRFxbQ6dTrXUKT8J4aEHQRCkhJbw0Fsmrpbsn4wlrbLHudn5afbxB+xIgZ6hZRFxbQ5Op0XyU5RcEARBMNZprIfeJmhnB7npNq9DttOeshIZHMiq9GmrhCSdavevF/WyY6R0tLMDIPSsQzqhZX3TCT2bk0546EEQBCmhsR66KuTz4PM79JfYrHgGsnKSrWX/eqVTaqa0UctP3gN/oWftdoSW9bUj9GyIHcWEhx4EQZASGj7KRXJ5xCcu0Lah65Mah4bWvH+rpVNqf8mZFxR61r5/aFnf/UPP5uwfHnoQBEFKaKyHLoK2Z2Gy9Xx3jutr6OHThrb76Qs9aya0rC+hZ3MIDz0IgiAlNGU+9LYtNka1t7O9oYdPG9KzGyD0rAOhZX0JPZtDeOhBEAQpIQr0IAiClBAFehAEQUqIAj0IgiAlRIEeBEGQEqJAD4IgSAlRoAdBEKSEKNCDIAhSQhToQRAEKSEK9CAIgpQwYoEuIvNE5AERWSkiK0TkCl8+TUTuE5E1/j119M0NgiAISlGOh94PXKWqxwOLgf8gIscDXwTuV9VFwP3+PwiCIGgSIxboqrpRVZ/033uAVcAc4DzgJt/sJuD80TIyCIIgGJmKYugisgA4CXgUmKWqG33VJmBWXS0LgiAIKqLsAl1EuoCfAVeqanfhOlVVSrw9SUQuF5HlIrK8N7e/JmODIAiC0pRVoItIO1aY36Kqd/rizSIy29fPBrYMta+q3qCqJ6vqyR2ZCfWwOQiCIBiCcka5CPB9YJWqfrNg1VJgif9eAtxTf/OCIAiCcinnjUXvAi4GnhWRp3zZl4CvA7eLyKXAK8DHKj148kZwsReEk++o7p3XbT3+ZnHPjWZqfHd23tJr67XvfKcbKLUlO1oM6OjVs7bXpmM+eblMW73ehT4G8LfUo8lJrjDvxdfMuHydDHuDkejoryDNd76BrsE6MGKBrqq/onRR9r76mhMEQRBUS2PfKVrEzOX23b7fauENp3u9UaZnmHhDC+/pAWDzKeMB2LswV5Nd47aaqzvvX/YC8PJ5EwHo72pNr2vOw/0A9EzJALB5sa8os0WR2W/5XXiPdVpv+EPL74Eja9NxLNG/bj0AnTuOAuDAkZXt3/WqaTjrsQMAvHxeJxAeZqVMWWM6Tltp9/TaD1lzUbOhYznEo/9BEAQpoakeekJvl9UrolYLl10Xe7ytf6JlQ+uUG/VqrmeqeVnS4o6qiunQN6G6+G/Sh9E3yb2helXzbobkvI8j6dtowb6ItnHjato/uUZ6ppmGkq8yFj/Scby/ZOAcpayfQ3KWn55pWf9vy+t1b7/ueCnTMzz0IAiClNBUD33LYvfI281FlL7KXLdk9MnaDyf7WSw58QhHxDfTCeYGyH6LQR+cY13s20/aB0Dvxq7K0m0wG063ejk30fPRX5md/RNdx/O9pZR3t6jGLoP2Pd4XcZ/FQ199v7V4+ie1Xl9E22x70LlnRnW27TnaNOs+zv5LX33zmJzTo+6142z7N9YS2De/xZuPFbLrLabbzhOsbJDe0TlOWvUMDz0IgiAlNNVDz+yz+iTpwa52RED7LvOscxPc4y9zHHrimX9+8b0AfG/NuwA4dfYrAFx5+P0AXJT5FAC7XjmsKvtGm2yiY5Xj0JN4b3anJdA/0fevMZ6Y2T+4pdBWYQusoXj/TbUh72SkEDbIhdyEOnvo7jhmeuxHW3/7MFuPQVz35FrOW6Nu1Mbzp1XP8NCDIAhSQnM8dH8ar81C3uSTYHaFHnoS005q24Ee8UyZ+x+wDe987SQA/sdb7wLgoFpt/fCBNwOwd39nRXY1DH+6MdGxzeOCuUo99CL9klEvtY52Oejj2F/6lC9Qd7v2Jo/01pZ+Xen3/odKR6ckA3cGRvTYdy7ZvU6NkuQp6pcv9dZsn2vp91JyLdfa79E0kny4/QNPitY2+KgkadUzPPQgCIKU0NQY+qG5V6pNoCidSr0hr31feslGOPTON0PmZXcAcPuWUwDo22FuQqtGgBP9Ks5/cTp1vhqOfNNWAD4670kAHtx2LABPr5wPVD4aZ1TJeSzVHbXc+DL306H7CQbGN1c5r04xbTPMsNtOu8Hs86vxyQMLALjusbNswz0t8WhJ1SRyZvwp8FyiY52fFE2rnuGhB0EQpITmVD8ecOyfkIym8OUVVsJJrd1foxckPWbAFQ9/HIB5c7YDsO6lmba+RcefkzXXvM9HpeTHJ8HvypJJ4okDs13WyRk6YqK9B+XsrhUArDs4DYCndX59DlBPOjuAgpkmyyW5lruSb1+er9M148m8fe4GAH7P7fzqNhvw/v3l77bN9lXbzG0REh0n+3cSu6612fm649hXWvUMDz0IgiAlNMVDT2K+XUeZB3fwoLlF/VvLDVx6Oj5GddbcnQBsfs3GiQ/0UFeIZC29E6fbzHvrXzy8qnQahbqH3rFwDwC9vXY689srG5Wj4y1+fMQc03HTOvOkk5ZLtWzZPwmA1/rt+x9feJul24ItnnyXvU0r11VZKydpJS5YtBmAjjbTcs3qOZ5wjYa5HRv3TQagR234x12vvB0Yu55kMeqtw2OPMc+5J2fX8iurj6jzgewrrXqGhx4EQZASmuKhJ2N1+/qqHZ7i6fT5HCb55BHJ2jy/SZPtMb8F47bXlE6jkD4bgN7nnrlWG7d1Hftzydjb+njQ61fb6KHPPPJpAHJT3F1twbmtdbzPNDnR5wPaW96tkbQ29vZYq+jYafZq3ReS+YlqbOUkbH5pBgDHrfusLfBz1HptnepIdNzTazrO6doNwFrvH6t0nqeRSKue4aEHQRCkhKYOssxmzVWfMM7GhO7cVt1jYdPG25t2tnf6EIOD1dVTbzt8IwCr9s22BWPkKbGOTosDtvncK/t2dlSVzvQJNrvktg6LL9JXWzyxY7ald+a7VgPwyKaFAGx/eWpN6Y4GSeOufbxp2V+mh56Q90ZHd58/s9CZTJBeH59pylzzWM+Y9zwAD762CGhNLavC77WktX0w58ONOpLJ+usb206rnuGhB0EQpITmjHLxamT6RPOst3R3DbP1MOm4F9SesW/1WHDFcTC3Z1qHeZRPbZ9blT2NRtvt9E2ecBCA7bsnVpWO+CiXjkwyEUl9IolTJ9n5vWzGwwBs67Hz3IpekCQudqVXj1877RnzJLfs66oqmZEY12Eth4unPQLAhgM2oqsVtayGZLSQ+Hj0rfv9WpbR6W9Jq57hoQdBEKSExnroqtDXT0e31SMbnrJYdTLDWrbSmLXPnfz8jgUAtPfXZt6vrz8ZgH5/N2f2iNYbjVGIHLC+hx1P2pvq8TBj5TpazH3VFotx16pj3zTz9JfM/w0Ay7pPAGD1dhvXn93Ten5EZvMuANpemAdAttw7wz3x7c/ZU8UD83oPPOlYm11jUUsAfARW+14fidY7ghCu47Zn/NkP33zgIfA63YpjVs8yGdvWB0EQBAM03EPX3j46dvvB97fWqM++ohB0567Wsq8Y7bEXLo7bZnZWPA/JKKFt1lS4bvkH7H+v920ctOUTW1BXPWitnc6doWU90F6LUSf3er69Newcq3qWS3joQRAEKaGxHrqAZNrI+1HzLTbVcKvZMxKS8fcvtpieHd3u5XQP/VxBq9hZSGhZX0LP5hAeehAEQUposIcu0N5Oz7Si+beD6mi3QG/oWQdCy/oSejaF8NCDIAhSQmM99Lyivb107khGEoztHuVmo702yiX0rJ3Qsr6Ens2hbA9dRDIi8lsRWeb/F4rIoyLyoojcJiLVzQgVBEEQ1IVKPPQrgFWAT8XHtcD/VNWfiMj1wKXAd8pJKHkyVMbIbIatTuhZP0LL+hJ6NpayPHQRmQucA3zP/wvwXuAO3+Qm4PzRMDAIgiAoj3I99G8BfwlM8v/TgV2qmsz6sR6YM2Iq+Ty6Zy+TXrX5FPJZi6v1j7fvgRcOJXNT76usZ7yvy3f03drcukxPeekk9uSKhqhmD/hMcLny7GhUfnTPXoDQs0yGy09oGdcmtK6e5TKihy4i5wJbVPWJilI+tP/lIrJcRJb36sFqkgiCIAjKoBwP/V3Ah0XkbGAcFkP/e+AwEcm6lz4X2DDUzqp6A3ADwJTsDJVslv5xg58i2/kW+853WqBNfYq16cttfoWR4m95747dcZJXq7595zbbf9LL5fWwH5zu74c8tm9QOpNX2pjaju7yastdx9l3bpznp6MoPyPU/qXy07Hd9p/8kr//0KcEDD2H378cPUPLuDahdfUslxE9dFW9WlXnquoC4ALg/6nqRcADwEd9syXAPRUdOQiCIKgrolp+jEZETgc+r6rnisjRwE+AacBvgU+oas8I+28F9gHbqrZ49JlB2FctrWwbhH21EvbVRi32zVfVmSNtVFGBXg9EZLmqntzQg1ZA2Fc9rWwbhH21EvbVRiPsi0f/gyAIUkIU6EEQBCmhGQX6DU04ZiWEfdXTyrZB2FcrYV9tjLp9DY+hB0EQBKNDhFyCIAhSQsMKdBE5S0Se99kZv9io4w5jzzwReUBEVorIChG5wpdPE5H7RGSNf09tsp0tO8uliBwmIneIyGoRWSUip7WSfiLyOT+3z4nIrSIyrpn6icgPRGSLiDxXsGxIvcT4ttv5jIi8o0n2/a2f32dE5C4ROaxg3dVu3/Mi8oFm2Few7ioRURGZ4f8bql8p20TkL1y/FSJyXcHy0dFOVUf9A2SA3wFHAx3A08DxjTj2MDbNBt7hvycBLwDHA9cBX/TlXwSubbKd/wn4MbDM/98OXOC/rwc+00TbbgIu898dwGGtoh82t9DLwPgC3S5ppn7AHwDvAJ4rWDakXsDZwD9js4MsBh5tkn3vB7L++9oC+473+7gTWOj3d6bR9vnyecC9wCvAjGboV0K7PwL+Bej0/4ePtnaNupBPA+4t+H81cHUjjl2BjfcAZwLPA7N92Wzg+SbaNBe4H5vZcplfnNsKbrBBujbYtileYErR8pbQzwv0ddiDb1nX7wPN1g9YUHTTD6kX8F3gwqG2a6R9Res+Atzivwfdw16gntYM+7BZX08A1hYU6A3Xb4hzeztwxhDbjZp2jQq5JDdXQnmzMzYIEVkAnAQ8CsxS1Y2+ahMwq0lmwaFZLpMZLqqb5XJ0WAhsBW70kND3RGQiLaKfqm4AvgG8CmwEdgNP0Dr6JZTSqxXvmT/FvF5oEftE5Dxgg6o+XbSqFew7BniPh/h+KSKnjLZtb/hOURHpAn4GXKmq3YXr1KrPpgwDqnWWywaQxZqY31HVk7ApHQb1jTRZv6nAeVjFcyQwETirGbaUSzP1GgkR+TLQD9zSbFsSRGQC8CXgr5ptSwmyWAtxMfAF4HYRGdV38TWqQN+AxbkSSs7O2EhEpB0rzG9R1Tt98WYRme3rZwNbmmReMsvlWmzOnPdSMMulb9NMHdcD61X1Uf9/B1bAt4p+ZwAvq+pWVe0D7sQ0bRX9Ekrp1TL3jIhcApwLXOSVDrSGfW/CKuyn/T6ZCzwpIke0iH3rgTvVeAxrac8YTdsaVaA/DizyEQYd2KyNSxt07CHxmvL7wCpV/WbBqqXY7JHQxFkktcVnuVTVTcA6ETnWF70PWEmL6IeFWhaLyAQ/14l9LaFfAaX0Wgp80kdrLAZ2F4RmGoaInIWF/T6sqvsLVi0FLhCRThFZCCwCHmukbar6rKoerqoL/D5Zjw102ERr6Hc31jGKiByDDRzYxmhqN9qdGAWB/7OxkSS/A77cqOMOY8+7sebtM8BT/jkbi1PfD6zBeqintYCtp3NolMvRfvJfBH6K96A3ya4TgeWu4d3A1FbSD/gKsBp4DrgZG1XQNP2AW7F4fh9W+FxaSi+sA/wf/H55Fji5Sfa9iMV7k3vk+oLtv+z2PQ98sBn2Fa1fy6FO0YbqV0K7DuBHfv09Cbx3tLWLJ0WDIAhSwhu+UzQIgiAtRIEeBEGQEqJAD4IgSAlRoAdBEKSEKNCDIAhSQhToQRAEKSEK9CAIgpQQBXoQBEFK+P+Tn9FRIUbOvgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "s = env.reset()\n",
    "for _ in range(100):\n",
    "    s, _, _, _ = env.step(env.action_space.sample())\n",
    "\n",
    "plt.title('Game image')\n",
    "plt.imshow(env.render('rgb_array'))\n",
    "plt.show()\n",
    "\n",
    "plt.title('Agent observation (4-frame buffer)')\n",
    "plt.imshow(s.transpose([0,2,1]).reshape([42,-1]))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build an agent\n",
    "\n",
    "We now have to build an agent for actor-critic training - a convolutional neural network that converts states into action probabilities $\\pi$ and state values $V$.\n",
    "\n",
    "Your assignment here is to build and apply a neural network - with any framework you want. \n",
    "\n",
    "For starters, we want you to implement this architecture:\n",
    "![https://s17.postimg.org/orswlfzcv/nnet_arch.png](https://s17.postimg.org/orswlfzcv/nnet_arch.png)\n",
    "\n",
    "After your agent gets mean reward above 50, we encourage you to experiment with model architecture to score even better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/robotcator/miniconda3/envs/rl/lib/python3.6/site-packages/tensorflow/python/client/session.py:1702: UserWarning: An interactive session is already active. This can cause out-of-memory errors in some cases. You must explicitly call `InteractiveSession.close()` to release resources held by the other session(s).\n",
      "  warnings.warn('An interactive session is already active. This can '\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "from keras.layers import Conv2D, Dense, Flatten, Input\n",
    "\n",
    "class Agent:\n",
    "    def __init__(self, name, state_shape, n_actions, reuse=False):\n",
    "        \"\"\"A simple actor-critic agent\"\"\"\n",
    "        \n",
    "        with tf.variable_scope(name, reuse=reuse):\n",
    "            \n",
    "            # Prepare neural network architecture\n",
    "            ### Your code here: prepare any necessary layers, variables, etc.\n",
    "            self.network = keras.models.Sequential()\n",
    "            self.network.add(Conv2D(16, (3, 3), strides=2, activation='relu', input_shape=state_shape))\n",
    "            self.network.add(Conv2D(32, (3, 3), strides=2, activation='relu'))\n",
    "            self.network.add(Conv2D(64, (3, 3), strides=2, activation='relu'))\n",
    "            self.network.add(Flatten())\n",
    "            self.network.add(Dense(256, activation='relu'))\n",
    "            self.network.add(Dense(n_actions, activation='linear'))\n",
    "            \n",
    "            # prepare a graph for agent step\n",
    "            self.state_t = tf.placeholder('float32', [None,] + list(state_shape))\n",
    "            self.agent_outputs = self.symbolic_step(self.state_t)\n",
    "        \n",
    "    def symbolic_step(self, state_t):\n",
    "        \"\"\"Takes agent's previous step and observation, returns next state and whatever it needs to learn (tf tensors)\"\"\"\n",
    "        \n",
    "        # Apply neural network\n",
    "        ### Your code here: apply agent's neural network to get policy logits and state values.\n",
    "        \n",
    "        logits = self.network(state_t)\n",
    "        state_value = tf.reduce_max(logits, axis=-1)\n",
    "        \n",
    "        assert tf.is_numeric_tensor(state_value) and state_value.shape.ndims == 1, \\\n",
    "            \"please return 1D tf tensor of state values [you got %s]\" % repr(state_value)\n",
    "        assert tf.is_numeric_tensor(logits) and logits.shape.ndims == 2, \\\n",
    "            \"please return 2d tf tensor of logits [you got %s]\" % repr(logits)\n",
    "        # hint: if you triggered state_values assert with your shape being [None, 1], \n",
    "        # just select [:, 0]-th element of state values as new state values\n",
    "        \n",
    "        return (logits, state_value)\n",
    "    \n",
    "    def step(self, state_t):\n",
    "        \"\"\"Same as symbolic step except it operates on numpy arrays\"\"\"\n",
    "        sess = tf.get_default_session()\n",
    "        return sess.run(self.agent_outputs, {self.state_t: state_t})\n",
    "    \n",
    "    def sample_actions(self, agent_outputs):\n",
    "        \"\"\"pick actions given numeric agent outputs (np arrays)\"\"\"\n",
    "        logits, state_values = agent_outputs\n",
    "        policy = np.exp(logits) / np.sum(np.exp(logits), axis=-1, keepdims=True)\n",
    "        return np.array([np.random.choice(len(p), p=p) for p in policy])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = Agent(\"agent\", obs_shape, n_actions)\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "action logits:\n",
      " [[ 0.01942591 -0.01527654 -0.0039657  -0.03675861  0.00678585 -0.03386937\n",
      "   0.03030618  0.00120379  0.04759545  0.00669582 -0.02636265 -0.00630975\n",
      "   0.0325189   0.01624631]]\n",
      "state values:\n",
      " [0.04759545]\n"
     ]
    }
   ],
   "source": [
    "state = [env.reset()]\n",
    "logits, value = agent.step(state)\n",
    "print(\"action logits:\\n\", logits)\n",
    "print(\"state values:\\n\", value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's play!\n",
    "Let's build a function that measures agent's average reward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(agent, env, n_games=1):\n",
    "    \"\"\"Plays an a game from start till done, returns per-game rewards \"\"\"\n",
    "\n",
    "    game_rewards = []\n",
    "    for _ in range(n_games):\n",
    "        state = env.reset()\n",
    "        \n",
    "        total_reward = 0\n",
    "        while True:\n",
    "            action = agent.sample_actions(agent.step([state]))[0]\n",
    "            state, reward, done, info = env.step(action)\n",
    "            total_reward += reward\n",
    "            if done: break\n",
    "                \n",
    "        game_rewards.append(total_reward)\n",
    "    return game_rewards\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.0, 1.0, 6.0]\n"
     ]
    }
   ],
   "source": [
    "env_monitor = gym.wrappers.Monitor(env, directory=\"kungfu_videos\", force=True)\n",
    "rw = evaluate(agent, env_monitor, n_games=3,)\n",
    "env_monitor.close()\n",
    "print (rw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"./kungfu_videos/openaigym.video.0.8207.video000001.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./kungfu_videos/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./kungfu_videos/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training on parallel games\n",
    "![img](https://s7.postimg.org/4y36s2b2z/env_pool.png)\n",
    "\n",
    "To make actor-critic training more stable, we shall play several games in parallel. This means ya'll have to initialize several parallel gym envs, send agent's actions there and .reset() each env if it becomes terminated. To minimize learner brain damage, we've taken care of them for ya - just make sure you read it before you use it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EnvBatch:\n",
    "    def __init__(self, n_envs = 10):\n",
    "        \"\"\" Creates n_envs environments and babysits them for ya' \"\"\"\n",
    "        self.envs = [make_env() for _ in range(n_envs)]\n",
    "        \n",
    "    def reset(self):\n",
    "        \"\"\" Reset all games and return [n_envs, *obs_shape] observations \"\"\"\n",
    "        return np.array([env.reset() for env in self.envs])\n",
    "    \n",
    "    def step(self, actions):\n",
    "        \"\"\"\n",
    "        Send a vector[batch_size] of actions into respective environments\n",
    "        :returns: observations[n_envs, *obs_shape], rewards[n_envs], done[n_envs,], info[n_envs]\n",
    "        \"\"\"\n",
    "        results = [env.step(a) for env, a in zip(self.envs, actions)]\n",
    "        new_obs, rewards, done, infos = map(np.array, zip(*results))\n",
    "        \n",
    "        # reset environments automatically\n",
    "        for i in range(len(self.envs)):\n",
    "            if done[i]:\n",
    "                new_obs[i] = self.envs[i].reset()\n",
    "        \n",
    "        return new_obs, rewards, done, infos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Let's try it out:__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/robotcator/gym/gym/logger.py:30: UserWarning: \u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State shape: (10, 42, 42, 4)\n",
      "Actions: [ 1 10  6]\n",
      "Rewards: [0. 0. 0.]\n",
      "Done: [False False False]\n"
     ]
    }
   ],
   "source": [
    "env_batch = EnvBatch(10)\n",
    "\n",
    "batch_states = env_batch.reset()\n",
    "\n",
    "batch_actions = agent.sample_actions(agent.step(batch_states))\n",
    "\n",
    "batch_next_states, batch_rewards, batch_done, _ = env_batch.step(batch_actions)\n",
    "\n",
    "print(\"State shape:\", batch_states.shape)\n",
    "print(\"Actions:\", batch_actions[:3])\n",
    "print(\"Rewards:\", batch_rewards[:3])\n",
    "print(\"Done:\", batch_done[:3])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Actor-critic\n",
    "\n",
    "Here we define a loss functions and learning algorithms as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# These placeholders mean exactly the same as in \"Let's try it out\" section above\n",
    "states_ph = tf.placeholder('float32', [None,] + list(obs_shape))    \n",
    "next_states_ph = tf.placeholder('float32', [None,] + list(obs_shape))\n",
    "actions_ph = tf.placeholder('int32', (None,))\n",
    "rewards_ph = tf.placeholder('float32', (None,))\n",
    "is_done_ph = tf.placeholder('float32', (None,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# logits[n_envs, n_actions] and state_values[n_envs, n_actions]\n",
    "logits, state_values = agent.symbolic_step(states_ph)\n",
    "next_logits, next_state_values = agent.symbolic_step(next_states_ph)\n",
    "next_state_values = next_state_values * (1 - is_done_ph)\n",
    "\n",
    "# probabilities and log-probabilities for all actions\n",
    "probs = tf.nn.softmax(logits)            # [n_envs, n_actions]\n",
    "logprobs = tf.nn.log_softmax(logits)     # [n_envs, n_actions]\n",
    "\n",
    "# log-probabilities only for agent's chosen actions\n",
    "logp_actions = tf.reduce_sum(logprobs * tf.one_hot(actions_ph, n_actions), axis=-1) # [n_envs,]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "# compute advantage using rewards_ph, state_values and next_state_values\n",
    "gamma = 0.99\n",
    "# http://rail.eecs.berkeley.edu/deeprlcourse-fa17/f17docs/lecture_5_actor_critic_pdf.pdf\n",
    "# how to caculate advantage\n",
    "# advantage = next_logits - next_state_values\n",
    "advantage = rewards_ph + gamma * next_state_values - state_values\n",
    "\n",
    "assert advantage.shape.ndims == 1, \"please compute advantage for each sample, vector of shape [n_envs,]\"\n",
    "\n",
    "# compute policy entropy given logits_seq. Mind the \"-\" sign!\n",
    "entropy = - tf.reduce_sum( probs * logprobs, 1, name=\"entropy\")\n",
    "\n",
    "assert entropy.shape.ndims == 1, \"please compute pointwise entropy vector of shape [n_envs,] \"\n",
    "\n",
    "\n",
    "\n",
    "actor_loss =  - tf.reduce_mean(logp_actions * tf.stop_gradient(advantage)) - 0.001 * tf.reduce_mean(entropy)\n",
    "\n",
    "# compute target state values using temporal difference formula. Use rewards_ph and next_step_values\n",
    "target_state_values = rewards_ph + gamma * next_state_values\n",
    "\n",
    "critic_loss = tf.reduce_mean((state_values - tf.stop_gradient(target_state_values))**2 )\n",
    "\n",
    "train_step = tf.train.AdamOptimizer(1e-4).minimize(actor_loss + critic_loss)\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You just might be fine!\n"
     ]
    }
   ],
   "source": [
    "# Sanity checks to catch some errors. Specific to KungFuMaster in assignment's default setup.\n",
    "l_act, l_crit, adv, ent = sess.run([actor_loss, critic_loss, advantage, entropy], feed_dict = {\n",
    "        states_ph: batch_states,\n",
    "        actions_ph: batch_actions,\n",
    "        next_states_ph: batch_states,\n",
    "        rewards_ph: batch_rewards,\n",
    "        is_done_ph: batch_done,\n",
    "    })\n",
    "\n",
    "assert abs(l_act) < 100 and abs(l_crit) < 100, \"losses seem abnormally large\"\n",
    "assert 0 <= ent.mean() <= np.log(n_actions), \"impossible entropy value, double-check the formula pls\"\n",
    "if ent.mean() < np.log(n_actions) / 2: print(\"Entropy is too low for untrained agent\")\n",
    "print(\"You just might be fine!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train \n",
    "\n",
    "Just the usual - play a bit, compute loss, follow the graidents, repeat a few million times.\n",
    "![img](http://images6.fanpop.com/image/photos/38900000/Daniel-san-training-the-karate-kid-38947361-499-288.gif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "from tqdm import trange\n",
    "from pandas import DataFrame\n",
    "ewma = lambda x, span=100: DataFrame({'x':np.asarray(x)}).x.ewm(span=span).mean().values\n",
    "\n",
    "env_batch = EnvBatch(10)\n",
    "batch_states = env_batch.reset()\n",
    "\n",
    "rewards_history = []\n",
    "entropy_history = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in trange(100000): \n",
    "    \n",
    "    batch_actions = agent.sample_actions(agent.step(batch_states))\n",
    "    batch_next_states, batch_rewards, batch_done, _ = env_batch.step(batch_actions)\n",
    "    \n",
    "    feed_dict = {\n",
    "        states_ph: batch_states,\n",
    "        actions_ph: batch_actions,\n",
    "        next_states_ph: batch_next_states,\n",
    "        rewards_ph: batch_rewards,\n",
    "        is_done_ph: batch_done,\n",
    "    }\n",
    "    batch_states = batch_next_states\n",
    "    \n",
    "    _, ent_t = sess.run([train_step, entropy], feed_dict)\n",
    "    entropy_history.append(np.mean(ent_t))\n",
    "\n",
    "    if i % 500 == 0: \n",
    "        if i % 2500 == 0:\n",
    "            rewards_history.append(np.mean(evaluate(agent, env, n_games=3)))\n",
    "            if rewards_history[-1] >= 50:\n",
    "                print(\"Your agent has earned the yellow belt\" % color)\n",
    "\n",
    "        clear_output(True)\n",
    "        plt.figure(figsize=[8,4])\n",
    "        plt.subplot(1,2,1)\n",
    "        plt.plot(rewards_history, label='rewards')\n",
    "        plt.plot(ewma(np.array(rewards_history),span=10), marker='.', label='rewards ewma@10')\n",
    "        plt.title(\"Session rewards\"); plt.grid(); plt.legend()\n",
    "        \n",
    "        plt.subplot(1,2,2)\n",
    "        plt.plot(entropy_history, label='entropy')\n",
    "        plt.plot(ewma(np.array(entropy_history),span=1000), label='entropy ewma@1000')\n",
    "        plt.title(\"Policy entropy\"); plt.grid(); plt.legend()        \n",
    "        plt.show()\n",
    "        \n",
    "        \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Relax and grab some refreshments while your agent is locked in an infinite loop of violence and death.\n",
    "\n",
    "__How to interpret plots:__\n",
    "\n",
    "The session reward is the easy thing: it should in general go up over time, but it's okay if it fluctuates ~~like crazy~~. It's also OK if it reward doesn't increase substantially before some 10k initial steps. However, if reward reaches zero and doesn't seem to get up over 2-3 evaluations, there's something wrong happening.\n",
    "\n",
    "\n",
    "Since we use a policy-based method, we also keep track of __policy entropy__ - the same one you used as a regularizer. The only important thing about it is that your entropy shouldn't drop too low (`< 0.1`) before your agent gets the yellow belt. Or at least it can drop there, but _it shouldn't stay there for long_.\n",
    "\n",
    "If it does, the culprit is likely:\n",
    "* Some bug in entropy computation. Remember that it is $ - \\sum p(a_i) \\cdot log p(a_i) $\n",
    "* Your agent architecture converges too fast. Increase entropy coefficient in actor loss. \n",
    "* Gradient explosion - just [clip gradients](https://stackoverflow.com/a/43486487) and maybe use a smaller network\n",
    "* Us. Or TF developers. Or aliens. Or lizardfolk. Contact us on forums before it's too late!\n",
    "\n",
    "If you're debugging, just run `logits, values = agent.step(batch_states)` and manually look into logits and values. This will reveal the problem 9 times out of 10: you'll likely see some NaNs or insanely large numbers or zeros. Try to catch the moment when this happens for the first time and investigate from there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"Final\" evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env_monitor = gym.wrappers.Monitor(env, directory=\"kungfu_videos\", force=True)\n",
    "final_rewards = evaluate(agent, env_monitor, n_games=20,)\n",
    "env_monitor.close()\n",
    "print(\"Final mean reward:\", np.mean(final_rewards))\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./kungfu_videos/\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./kungfu_videos/\"+video_names[-1])) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./kungfu_videos/\"+video_names[-2])) #try other indices "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# if you don't see videos, just navigate to ./kungfu_videos and download .mp4 files from there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from submit import submit_kungfu\n",
    "env = make_env()\n",
    "submit_kungfu(agent, env, evaluate, <EMAIL>, <TOKEN>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "\n",
    "```\n",
    "```\n",
    "\n",
    "```\n",
    "```\n",
    "\n",
    "```\n",
    "```\n",
    "\n",
    "```\n",
    "```\n",
    "\n",
    "```\n",
    "```\n",
    "\n",
    "```\n",
    "```\n",
    "\n",
    "```\n",
    "```\n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now what?\n",
    "Well, 5k reward is [just the beginning](https://www.buzzfeed.com/mattjayyoung/what-the-color-of-your-karate-belt-actually-means-lg3g). Can you get past 200? With recurrent neural network memory, chances are you can even beat 400!\n",
    "\n",
    "* Try n-step advantage and \"lambda\"-advantage (aka GAE) - see [this article](https://arxiv.org/abs/1506.02438)\n",
    " * This change should improve early convergence a lot\n",
    "* Try recurrent neural network \n",
    " * RNN memory will slow things down initially, but in will reach better final reward at this game\n",
    "* Implement asynchronuous version\n",
    " * Remember [A3C](https://arxiv.org/abs/1602.01783)? The first \"A\" stands for asynchronuous. It means there are several parallel actor-learners out there.\n",
    " * You can write custom code for synchronization, but we recommend using [redis](https://redis.io/)\n",
    "   * You can store full parameter set in redis, along with any other metadate\n",
    "   * Here's a _quick_ way to (de)serialize parameters for redis\n",
    "   ```\n",
    "   import joblib\n",
    "   from six import BytesIO\n",
    "```\n",
    "```\n",
    "   def dumps(data):\n",
    "        \"converts whatever to string\"\n",
    "        s = BytesIO()\n",
    "        joblib.dump(data,s)\n",
    "        return s.getvalue()\n",
    "``` \n",
    "```\n",
    "    def loads(string):\n",
    "        \"converts string to whatever was dumps'ed in it\"\n",
    "        return joblib.load(BytesIO(string))\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}