Captcha.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Probabilistic Programming to break CAPTCHAS\n",
    "\n",
    "Captchas are puzzles usually given as part of a verification procedure to ensure a internet user is indeed a human and not a bit. It involves identifying a string of letters (or numbers)\n",
    "\n",
    "\n",
    "In this example we will write a probabilistic CAPTCHA generator and instrument it using the probabilistic programming library `pyprob` in order to run amortized inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we import some basic libraries we will need later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[31mWarning: Empirical distributions on disk may perform slow because GNU DBM is not available. Please install and configure gdbm library for Python for better speed.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import torch\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import pyprob\n",
    "import pyprob.distributions\n",
    "import IPython\n",
    "\n",
    "from PIL import Image, ImageFilter, ImageDraw, ImageFont\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing a Generator\n",
    "\n",
    "Here we use the `Pillow` library to write a generator. This is a good example how probabilistic programs allow you use general purpose host languages (in this case Python, but could also be C++) with all their nice libraries, instead of restricting you to a statistical modeling framework (try writing this in RooFit :) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare the set of letters from which we \n",
    "\n",
    "alphabetorder = list('abcdefghijklmnopqrstuvw')\n",
    "alphabet = dict(zip(range(len(alphabetorder)),alphabetorder))\n",
    "\n",
    "\n",
    "def message_to_picture(width, height, message, blur):\n",
    "    canvas = Image.new('L', (width, height), \"white\")\n",
    "    font = ImageFont.load_default()\n",
    "    ImageDraw.Draw(canvas).text((2,0), message, 'black', font)\n",
    "    canvas = canvas.filter(ImageFilter.GaussianBlur(blur))\n",
    "    d = np.asarray(list(canvas.getdata()), dtype = np.float)\n",
    "    return d\n",
    "\n",
    "def random_captcha(width, height, blur = 0.5):\n",
    "    letter_distr = pyprob.distributions.Categorical([1/len(alphabet)]*len(alphabet))\n",
    "    length = 3\n",
    "    word =  ''.join([alphabet[letter_distr.sample().item()] for i in range(length)])\n",
    "    d = message_to_picture(width, height, word,blur)\n",
    "    return word, torch.tensor(d, dtype = torch.float).view(-1)/255-0.5\n",
    "\n",
    "def likelihoood(observation,eps):\n",
    "    lhood = pyprob.distributions.Normal(observation,eps*torch.ones(observation.shape))\n",
    "    return lhood"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An Example CAPTCHA\n",
    "\n",
    "Note that our captcha generator is very un-sophisticated, but it has the correct components: sampling latters, adding some blur, etc...\n",
    "\n",
    "Let's generate a CAPTCHA to see how it looks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'ground truth: ian')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAS2ElEQVR4nO3dfbAldX3n8feHGVAeDDADy7OCKZesbhahboxJiKEEXWAjxJSVgmgWHzYTa9dEtpIVUiZqYp40D2WSNTFEEaIskOBDkMIAMRrLrEIGHAQEZUJABwcYB+VBQXn45o/TQy7H+zS3e865v+H9qjp1+3T/uvvbfft+Tt/fOac7VYUkqT27TLsASdLyGOCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywLXTSvK2JB+c8DoPT1JJVi9z/vck+fWh69LOyQDXU1KS45JsGmA5tyc5YYiaAKrq9VX19qGWp52bAa6JWe5Z6bS0Vq+eegxw9ZLkmCSfT/JAkr9JcnGS3+qmHZdkU5KzktwFvL8b//NJNia5N8mlSQ7uxn9P90OSTyX5H93wq5N8JskfJPlGkn9NctKstkck+ceulquA/eapeU/g48DBSR7sHgd3XS6XJPlgkvuBVyc5b9v2zN6mbvgDwDOBj3XLeNOs1bwyyVeSfD3Jm7djfz6xviT7JrksyZZuey9LcujYvnl7kn/qtvnKJHNus3ZOBriWLcluwEeA84A1wIXAy8eaHdhNexawLsmLgd8FfgY4CLgDuGg7VvvDwJcYhfM7gfclSTft/wHXdtPeDpwx1wKq6lvAScDXqmqv7vG1bvKpwCXAPsAFCxVSVT8HfAV4WbeMd86afCxwJHA88JYk/wkgybFJvrnEbd2F0Yvesxi9UDwE/N+xNj8LvAb4D8BuwK8scdnaCfgvovp4IaNj6E9qdFGdDye5ZqzN48Bbq+o7AEleCZxbVdd1z38V+EaSw5e4zjuq6i+7ec8H/gw4oHsx+SHghG5dn07ysWVs02er6qPd8EP//tqw3X6jqh4Crk9yPXAUcHNVfYbRi8Oiqmor8KFtz5P8NvDJsWbvr6ovd9P/GjhluQWrPZ6Bq4+DgTvryVdE++pYmy1V9fDYPHdse1JVDwJbgUOWuM67Zs377W5wr2653+jOrre5g+03Xv9y3TVr+NuMatwuSfZI8hdJ7ui6dD4N7JNk1ZDrUbsMcPWxGTgkTz5NPWyszfjlLr/GqEsAeKI/ei1wJ7AtfPeY1f7A7ahl32552zxzgfbzXYZzfPy3FqlnR17O85cZdcP8cFV9H/Cibvyy/y3QzsUAVx+fBR4D3pBkdZJTgRcsMs+FwGuSPD/J04DfAa6uqturagujIH9VklVJXgt8/1IKqao7gPXAbyTZLcmxwMsWmOVuYG2SvRdZ9Abg5CRrkhwInDnHcp69lBqX4RmM+r2/mWQN8NYdtB41ygDXslXVd4GfBl4HfBN4FXAZ8J0F5vl74NcZ9e1uZhTQp81q8vPA/2HUrfI84P9vR0k/y+hNznsZhd1fLVDHLYxeTG5L8s1tn4SZwweA64HbgSuBi8em/y7wa90yFn0DMcmPJ3lwsXaddwG7A18HPgf83RLn01NEvKGDhpTkauA9VfX+adci7ew8A1cvSX4iyYFdF8oZwH/BM0VpIvwYofo6EvhrYE/gNuAVVbV5uiVJTw12oUhSo+xCkaRGGeCS1KiJ9oHvt2ZVHX7YrpNcpSQ179ovfOfrVbX/+PiJBvjhh+3KNVeMf1FPkrSQVQdtnPOyEHahSFKjDHBJapQBLkmN6hXgSU5M8qXu7ipnD1WUJGlxyw7w7prE72Z0Z5PnAqcnee5QhUmSFtbnDPwFwMaquq27Kt1FjG5HJUmagD4BfghPvnvJJua4q0qSdUnWJ1m/ZetjPVYnSZpth7+JWVXnVNVMVc3sv3bV4jNIkpakT4DfyZNvn3VoN06SNAF9AvyfgeckOaK7I/hpwKXDlCVJWsyyv0pfVY8meQNwBbAKOLeqbhqsMknSgnpdC6WqLgcuH6gWSdJ28JuYktQoA1ySGmWAS1KjvKnxlNz3+EPTLoG9d9l92iVI6sEzcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhrl9cCXYYhreZ9+9CkDVNLPhZ+/tPcyvKa4ND2egUtSowxwSWqUAS5JjTLAJalRyw7wJIcl+WSSLya5KckbhyxMkrSwPp9CeRT45aq6LskzgGuTXFVVXxyoNknSApZ9Bl5Vm6vqum74AeBm4JChCpMkLWyQPvAkhwNHA1cPsTxJ0uJ6B3iSvYAPAWdW1f1zTF+XZH2S9Vu2PtZ3dZKkTq8AT7Iro/C+oKo+PFebqjqnqmaqamb/tav6rE6SNEufT6EEeB9wc1X90XAlSZKWos8Z+I8BPwe8OMmG7nHyQHVJkhax7I8RVtVngAxYiyRpO/hNTElqlAEuSY0ywCWpUU+5GzoMcTOGIZx33UenXQJ777LntEsYzEr5vfY1xA0yVsK+cDsmwzNwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjmruhQ9+LvJ9+9Cm9a6iHH+69DB55pPcisnu/i81feMPlvWsY4oL3mx59sPcy1h31sl7z16OP9q4hq/v/Of3pho/1XsYvzry83wIG2BcrYjug97a85/r+2/HM1Xv1XsZ8PAOXpEYZ4JLUKANckhplgEtSo3oHeJJVST6f5LIhCpIkLc0QZ+BvBG4eYDmSpO3QK8CTHAr8N+C9w5QjSVqqvmfg7wLeBDw+X4Mk65KsT7J+y9bHeq5OkrTNsgM8yU8C91TVtQu1q6pzqmqmqmb2X7tquauTJI3pcwb+Y8ApSW4HLgJenOSDg1QlSVrUsgO8qn61qg6tqsOB04B/qKpXDVaZJGlBfg5ckho1yMWsqupTwKeGWJYkaWk8A5ekRhngktQoA1ySGpWqmtjKZo56el1zxWETW99c+t4QYrSM/l9I+oX/eELvZfR10cZPTrsEYKCbbDzwQK/533HLp3rXcNYPHNd7GX1v0gHwrg39Lkv0xiOP713DStgO6L8tQ2zHEDdOWXPInddW1cz4eM/AJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJatQg98Rsyd679L++L/S/pvjOYoj9+a71H+29jL7XfT7rB1/Su4bHH+53TXIY5oxqnxVwWlYP9f8b2Vm2Y0daAbtIkrQcBrgkNcoAl6RGGeCS1KheAZ5knySXJLklyc1JfmSowiRJC+v7KZQ/Bv6uql6RZDdgjwFqkiQtwbIDPMnewIuAVwNU1XeB7w5TliRpMX26UI4AtgDvT/L5JO9NsudAdUmSFtEnwFcDxwB/XlVHA98Czh5vlGRdkvVJ1m/Z+liP1UmSZusT4JuATVV1dff8EkaB/iRVdU5VzVTVzP5rV/VYnSRptmUHeFXdBXw1yZHdqOOBLw5SlSRpUX0/hfKLwAXdJ1BuA17TvyRJ0lL0CvCq2gDMDFSLJGk7+E1MSWqUAS5JjTLAJalRT7kbOtz3eP8LtN/3+M7xefYhtuO+xx/svYwzn/+y3svoe5ONd9xwVe8KzvqB43ovQ9oenoFLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhrV3A0d+t6Q4fSjT+ldQz38cO9lwCMDLKOfX3jeSdMuYeSR/jfZ6OusH3zJAEvp/zvN7rsPUEf7NQxlZ9qWuXgGLkmNMsAlqVEGuCQ1qleAJ/nfSW5KcmOSC5M8fajCJEkLW3aAJzkE+CVgpqr+M7AKOG2owiRJC+vbhbIa2D3JamAP4Gv9S5IkLcWyA7yq7gT+APgKsBm4r6quHKowSdLC+nSh7AucChwBHAzsmeRVc7Rbl2R9kvVbtj62/EolSU/SpwvlBOBfq2pLVT0CfBj40fFGVXVOVc1U1cz+a1f1WJ0kabY+Af4V4IVJ9kgS4Hjg5mHKkiQtpk8f+NXAJcB1wA3dss4ZqC5J0iJ6XQulqt4KvHWgWiRJ28FvYkpSowxwSWqUAS5JjTLAJalRqaqJrWzmqKfXNVccNrH1zaXvDSGklW7vXfrdxGCl/I303Q5YGdsyxHasOmjjtVU1Mz7eM3BJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIa1eumxi0a4tq80s5sZ/ob2Zm2ZS6egUtSowxwSWqUAS5JjTLAJalRiwZ4knOT3JPkxlnj1iS5Ksmt3c99d2yZkqRxSzkDPw84cWzc2cAnquo5wCe655KkCVo0wKvq08C9Y6NPBc7vhs8HfmrguiRJi1huH/gBVbW5G74LOGCgeiRJS9T7TcyqKqDmm55kXZL1SdZv2fpY39VJkjrLDfC7kxwE0P28Z76GVXVOVc1U1cz+a1ctc3WSpHHLDfBLgTO64TOAvx2mHEnSUi3lY4QXAp8FjkyyKcnrgN8DXpLkVuCE7rkkaYIWvZhVVZ0+z6TjB65FkrQd/CamJDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIalaqa3MqSLcAdCzTZD/j6hMrpwzqHZZ3DaaFGsM7t9ayq2n985EQDfDFJ1lfVzLTrWIx1Dss6h9NCjWCdQ7ELRZIaZYBLUqNWWoCfM+0Clsg6h2Wdw2mhRrDOQayoPnBJ0tKttDNwSdISTSXAk5yY5EtJNiY5e47pT0tycTf96iSHT6HGw5J8MskXk9yU5I1ztDkuyX1JNnSPt0y6zq6O25Pc0NWwfo7pSfIn3f78QpJjplDjkbP204Yk9yc5c6zNVPZnknOT3JPkxlnj1iS5Ksmt3c9955n3jK7NrUnOmHCNv5/klu53+pEk+8wz74LHxwTqfFuSO2f9Xk+eZ94Fc2ECdV48q8bbk2yYZ96J7c9FVdVEH8Aq4F+AZwO7AdcDzx1r8z+B93TDpwEXT6HOg4BjuuFnAF+eo87jgMsmXdsctd4O7LfA9JOBjwMBXghcPeV6VwF3Mfps69T3J/Ai4Bjgxlnj3gmc3Q2fDbxjjvnWALd1P/fthvedYI0vBVZ3w++Yq8alHB8TqPNtwK8s4ZhYMBd2dJ1j0/8QeMu09+dij2mcgb8A2FhVt1XVd4GLgFPH2pwKnN8NXwIcnyQTrJGq2lxV13XDDwA3A4dMsoYBnQr8VY18DtgnyUFTrOd44F+qaqEvdU1MVX0auHds9Oxj8Hzgp+aY9b8CV1XVvVX1DeAq4MRJ1VhVV1bVo93TzwGH7oh1b4959uVSLCUXBrNQnV3W/Axw4Y5a/1CmEeCHAF+d9XwT3xuMT7TpDtD7gLUTqW4OXRfO0cDVc0z+kSTXJ/l4kudNtLB/V8CVSa5Nsm6O6UvZ55N0GvP/cayE/QlwQFVt7obvAg6Yo81K2q+vZfRf1lwWOz4m4Q1dV8+583RHraR9+ePA3VV16zzTV8L+BHwTc1FJ9gI+BJxZVfePTb6OUTfAUcCfAh+ddH2dY6vqGOAk4H8ledGU6lhUkt2AU4C/mWPyStmfT1Kj/5tX7Me1krwZeBS4YJ4m0z4+/hz4fuD5wGZG3RMr2eksfPY97f35hGkE+J3AYbOeH9qNm7NNktXA3sDWiVQ3S5JdGYX3BVX14fHpVXV/VT3YDV8O7JpkvwmXSVXd2f28B/gIo39HZ1vKPp+Uk4Drquru8QkrZX927t7WzdT9vGeONlPfr0leDfwk8MruheZ7LOH42KGq6u6qeqyqHgf+cp71T31fwhN589PAxfO1mfb+nG0aAf7PwHOSHNGdjZ0GXDrW5lJg2zv6rwD+Yb6Dc0fp+sHeB9xcVX80T5sDt/XNJ3kBo/050ReaJHsmeca2YUZvbN041uxS4L93n0Z5IXDfrO6BSZv37GYl7M9ZZh+DZwB/O0ebK4CXJtm36xZ4aTduIpKcCLwJOKWqvj1Pm6UcHzvU2PstL59n/UvJhUk4AbilqjbNNXEl7M8nmcY7p4w+FfFlRu86v7kb95uMDkSApzP6F3sjcA3w7CnUeCyjf5u/AGzoHicDrwde37V5A3ATo3fMPwf86BTqfHa3/uu7Wrbtz9l1Bnh3t79vAGam9Hvfk1Eg7z1r3NT3J6MXlM3AI4z6Xl/H6D2XTwC3An8PrOnazgDvnTXva7vjdCPwmgnXuJFRv/G243PbJ7cOBi5f6PiYcJ0f6I67LzAK5YPG6+yef08uTLLObvx5247HWW2ntj8Xe/hNTElqlG9iSlKjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhr1b/ExnIuZPDTNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "word, img = random_captcha(20,12, blur = 0.2)\n",
    "plt.imshow(img.reshape(12,20))\n",
    "plt.title('ground truth: {}'.format(word))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and another one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'ground truth: ckm')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAVSklEQVR4nO3de5RdZX3G8efJTEJCEnOXXDEJpRQQFToiCCrLpBSogLcqKpZbzXK1VLG1FBet2NpVRdFlbamuKDeRAgqowEIhgmhdSnAIdwLmAoGE3AiEEAhJZubXP84ePDnM5cy8O+fMG76ftWbNPme/73t+Z8+eZ/bssy+OCAEA8jOs2QUAAAaHAAeATBHgAJApAhwAMkWAA0CmCHAAyBQBjj2G7S/Y/n6DX3O27bDdmsO42LMQ4HhNsH2M7dUljPOE7fll1ASkIsCx2+S29ZhbvQABjgGxfZjte22/YPuHtq+1/e/FvGNsr7b9T7bXSbqseP4Ttpfbftb2jbanF8+/ajeB7Ttt/3UxfbrtX9u+yPZzth+3fXxV2zm2f1nUskjS5F5qHi3pp5Km295afE0vdrlcZ/v7trdIOt325d3vp/o9FdNXStpX0k3FGOdWvczHbD9p+xnb5w9geY6y/TXbq2w/X7zfUT20+0Cx9f/GquV2hu2nimXzSdtvtf2A7c22/7veGpAvAhx1sz1C0o8kXS5poqSrJb2vptnUYt4bJC2w/W5JX5L0IUnTJK2SdM0AXvZtkh5TJZy/IukS2y7m/a+ke4p5X5R0Wk8DRMSLko6X9HREjCm+ni5mnyzpOknjJV3VVyER8XFJT0o6sRjjK1Wzj5Z0gKR5kj5v+0BJsn207c19DHuRpD+V9HZVltu5krqqG9g+Q9KFkuZHxENVs94maX9JH5b0DUnnS5ov6WBJH7L9rr7eD/LHv4wYiCNUWWe+GZWL6Nxg++6aNl2SLoiI7ZJk+2OSLo2IJcXjz0l6zvbsOl9zVUR8p+h7haT/kbRP8cfkraqE2nZJv7J90yDe028j4sfF9LY//G0YsH+NiG2S7rd9v6Q3S1oaEb9W5Y/Dq9geJulMSUdExJri6d8U87qbnVO0OSYiavfhfzEiXpZ0m+0XJV0dERuK/v8n6VBJvxzsG8LQxxY4BmK6pDWx6xXQnqpps7EIleo+q7ofRMRWSZskzajzNddV9X2pmBxTjPtcsXXdbZUGrrb+wVpXNf2SKjX2Z7KkkZJW9NHmHyVd3EN4S9L6qultPTyupwZkjADHQKyVNMO7bqbOqmlTe3nLp1XZnSLplf3RkyStkdQdvntXtZ86gFomFON127eP9r1ddrP2+Rf7qafMy3c+I+llSfv10eZYSf9s+wMlvi72EAQ4BuK3kjolnW271fbJkg7vp8/Vks6w/Rbbe0n6D0mLI+KJiNioSpCfarvF9pnqO8xeERGrJLVL+lfbI2wfLenEPrqslzTJ9rh+hr5P0gm2J9qeqsoujNpx5tZTY38iokvSpZK+Xnyo2mL7yGI5dXtY0nGSLrZ9Uhmviz0HAY66RcQOSe+XdJakzZJOlXSzpO199Pm5pH+RdL0qW837STqlqsknVNlNsEmVD99+M4CSPqrKB3nPSrpA0vf6qONRVf6YrCyO0pjeS9MrJd0v6QlJt0m6tmb+l1TZIt5s+7P9FWj7Hba39tHks5IelPS74n1cqJrfy4i4X9J7JH2n+igcwNzQASlsL5b07Yi4rNm1AK81bIFjQGy/y/bUYhfKaZLeJOlnza4LeC3iMEIM1AGSfiBptKSVkj4YEWubWxLw2sQuFADIFLtQACBTBDgAZKqh+8AnT2yJ2bOGN/Il0Yco4ZyUMnbAlVGHNehT4Iv+6VJrAHpzzwPbn4mIKbXPNzTAZ88arrtvrT1xD83SGV39N+pHhzqTx9gZ6WMMd0tS/1al9ZekFvMPLXaPlmnLe7xMBGscAGSKAAeATBHgAJCppAC3fZztx4q7rZxXVlEAgP4NOsBtt0i6WJU7nRwk6SO2DyqrMABA31K2wA+XtDwiVhZXqbtGldtTAQAaICXAZ2jXu5msVg93WbG9wHa77faNm9IPFwMAVOz2DzEjYmFEtEVE25RJ6cfaAgAqUgJ8jXa9ndbM4jkAQAOkBPjvJO1ve05xh/BTJN1YTlkAgP4M+lT6iOiwfbakWyW1SLo0Ih4urTIAQJ+SroUSEbdIuqWkWgAAA8CZmACQKQIcADJFgANAprip8WvYpq5tyWPctHW/5DHW7xyXPMa41peS+v/l2EeTa3h9y+jkMYCBYAscADJFgANApghwAMgUAQ4AmSLAASBTBDgAZIoAB4BMEeAAkCkCHAAyRYADQKYIcADIFAEOAJkiwAEgUwQ4AGSKAAeATL3mrgfeGV3JY3Sos4Q6InmM7dGR1P/ObdOTa/jqD96fPMbUxTuTx9g6PXFV/vvkEvThEq4pvreHJ4/RpfR1fE8x3C1J/VuV1l+SWrz7tpPZAgeATBHgAJApAhwAMkWAA0CmBh3gtmfZ/oXtR2w/bPvTZRYGAOhbykf3HZL+ISKW2B4r6R7biyLikZJqAwD0YdBb4BGxNiKWFNMvSFoqaUZZhQEA+lbKPnDbsyUdKmlxGeMBAPqXHOC2x0i6XtI5EbGlh/kLbLfbbt+4Kf0EGABARVKA2x6uSnhfFRE39NQmIhZGRFtEtE2ZlH5WEwCgIuUoFEu6RNLSiPh6eSUBAOqRsgV+lKSPS3q37fuKrxNKqgsA0I9BH0YYEb+W5BJrAQAMAGdiAkCmCHAAyBQBDgCZyu6GDjsj7VjytZ3bkmu4aeuByWO81DUieYy9h+1I6n/IyKeSa5h65NPJY4y8eWzyGMO3pB2iumTLG5JrKONnWoauSNsue7mrjJtKpH88NkzpNz2ZPPyFpP4njlmaXMO0llHJY/SGLXAAyBQBDgCZIsABIFMEOABkigAHgEwR4ACQKQIcADJFgANApghwAMgUAQ4AmSLAASBTBDgAZIoAB4BMEeAAkCkCHAAyRYADQKayu6FD6g0ZPvX4B5JrWH3l3OQxhnUkD6HOvdL6z/jo48k17Dv22eQx1re8LnmM1s0vJ/W/Y8lByTU80H5I8hhdJfxGpq5br1uVdqMQSWrZVsIK7vSbQmyZMzKp/6JPpK8XX5t9ffIYvWELHAAyRYADQKYIcADIFAEOAJlKDnDbLbbvtX1zGQUBAOpTxhb4pyUtLWEcAMAAJAW47ZmS/kLSd8spBwBQr9Qt8G9IOldSV28NbC+w3W67feOmzsSXAwB0G3SA236PpA0RcU9f7SJiYUS0RUTblEktg305AECNlC3woySdZPsJSddIerft75dSFQCgX4MO8Ij4XETMjIjZkk6RdEdEnFpaZQCAPnEcOABkqpSLWUXEnZLuLGMsAEB92AIHgEwR4ACQKQIcADLV0Bs6dKpLW7vSLrx/09YDk/qvvST9ZgwTV6TdVEKSln18RPIYk+9OO65++c/Tl8WEo9YljzG2o9fzwOoWy9JuTvEn356TXMPmg8cnjzH89PXJY2zdnrZutXw57SYIktS6Of13ROufSR5i/I4ZSf1Xfy/9d+Qnn3lT8hhSz79nbIEDQKYIcADIFAEOAJkiwAEgUwQ4AGSKAAeATBHgAJApAhwAMkWAA0CmCHAAyBQBDgCZIsABIFMEOABkigAHgEwR4ACQqYZeD7wMz3WMTuo/9qnt6UVEJA9x4AGrk8d4dt+9k/q/vrUjuYbX7/1C8hhbnPYzlaTYnvZzHbazM7mGdfPSl+dFc25PHuPuF9OuYf2L/Y9MruGZtvRrik+/c0LyGE8fk9Z/0r3JJej5zlHpg/SCLXAAyBQBDgCZIsABIFMEOABkKinAbY+3fZ3tR20vtZ3+6QcAoC6pR6H8p6SfRcQHbY+QlHZYBACgboMOcNvjJL1T0umSFBE7JO0opywAQH9SdqHMkbRR0mW277X9XbuEA3oBAHVJCfBWSYdJ+lZEHCrpRUnn1TayvcB2u+32TZu6El4OAFAtJcBXS1odEYuLx9epEui7iIiFEdEWEW2TJnHQCwCUZdCJGhHrJD1l+4DiqXmSHimlKgBAv1KPQvk7SVcVR6CslHRGekkAgHokBXhE3CepraRaAAADwE5pAMgUAQ4AmSLAASBT2d3QYULri0n9X5i1V3IN41ZsSx5j6WMzk8eYfHdLUv8NM5xcQ8dR65LHGN2SXsew0WnnkMWI9F+FqXekj3HZHx2VPMbcMc8k9e8o4YIYo2em3+hj26RxJdTxfFL/jqXjk2sYpvQbwPQ+NgAgSwQ4AGSKAAeATBHgAJApAhwAMkWAA0CmCHAAyBQBDgCZIsABIFMEOABkigAHgEwR4ACQKQIcADJFgANApghwAMgUAQ4AmWroDR1aNExjho1MGuPEMUuT+i8666Ck/pK0+sq5yWNMuSt5CHUm3pvij+evSK7hoNetTR7jtkPSb2IwfO4hSf3Xze9IrmHiXek3plhx55zkMUbO25nU/8UZySVov/Gbk8dYsW/6DR1S61g+K72GycPTb27RG7bAASBTBDgAZIoAB4BMJQW47c/Yftj2Q7avtp22gxsAULdBB7jtGZI+JaktIt4oqUXSKWUVBgDoW+oulFZJo2y3Stpb0tPpJQEA6jHoAI+INZIukvSkpLWSno+I28oqDADQt5RdKBMknSxpjqTpkkbbPrWHdgtst9tu37ipc/CVAgB2kbILZb6kxyNiY0TslHSDpLfXNoqIhRHRFhFtUya1JLwcAKBaSoA/KekI23vbtqR5ktJOkwQA1C1lH/hiSddJWiLpwWKshSXVBQDoR9K1UCLiAkkXlFQLAGAAOBMTADJFgANApghwAMgUAQ4AmXJENOzF2t48Mu6+dVbSGDsj7WSgtZ3bkvpL0k1bD0we46WuEcljjB32clL/E8Y8llxDGVsAN289IHmMzsRK3jpqZXINS7bNTh5juNNPdjti1ONJ/R/eMTW5hoNHrEseYyjU8eiOfZJreMeo9JueTJu59p6IaKt9ni1wAMgUAQ4AmSLAASBTBDgAZIoAB4BMEeAAkCkCHAAyRYADQKYIcADIFAEOAJkiwAEgUwQ4AGSKAAeATBHgAJApAhwAMpXd9cBTdUZX8hgdSr9mc2cJy73FTurfqpbkGspQxvJMNayEbZkupa9bZUh9L2W8j6GyPIfCsijj92zE9JVcDxwA9iQEOABkigAHgEwR4ACQqX4D3PaltjfYfqjquYm2F9leVnyfsHvLBADUqmcL/HJJx9U8d56k2yNif0m3F48BAA3Ub4BHxK8kPVvz9MmSriimr5D03pLrAgD0Y7D7wPeJiLXF9DpJ+5RUDwCgTskfYkblTKBez0qxvcB2u+32jZuaf8IGAOwpBhvg621Pk6Ti+4beGkbEwohoi4i2KZOGxpl/ALAnGGyA3yjptGL6NEk/KaccAEC96jmM8GpJv5V0gO3Vts+S9GVJf2Z7maT5xWMAQAO19tcgIj7Sy6x5JdcCABgAzsQEgEwR4ACQKQIcADLV7z7wPU2L0/9mtZTxdy/tXgx7lFKW55CwpxwmO1Tex1CoYyjU0Ls95TcHAF5zCHAAyBQBDgCZIsABIFMEOABkigAHgEwR4ACQKQIcADJFgANApghwAMgUAQ4AmSLAASBTBDgAZIoAB4BMEeAAkCkCHAAy5Yho3IvZGyWt6qPJZEnPNKicFNRZLuosTw41StQ5UG+IiCm1TzY0wPtjuz0i2ppdR3+os1zUWZ4capSosyzsQgGATBHgAJCpoRbgC5tdQJ2os1zUWZ4capSosxRDah84AKB+Q20LHABQp6YEuO3jbD9me7nt83qYv5fta4v5i23PbkKNs2z/wvYjth+2/eke2hxj+3nb9xVfn290nUUdT9h+sKihvYf5tv3NYnk+YPuwJtR4QNVyus/2Ftvn1LRpyvK0fantDbYfqnpuou1FtpcV3yf00ve0os0y26c1uMav2n60+Jn+yPb4Xvr2uX40oM4v2F5T9XM9oZe+feZCA+q8tqrGJ2zf10vfhi3PfkVEQ78ktUhaIWmupBGS7pd0UE2bv5H07WL6FEnXNqHOaZIOK6bHSvp9D3UeI+nmRtfWQ61PSJrcx/wTJP1UkiUdIWlxk+ttkbROlWNbm748Jb1T0mGSHqp67iuSziumz5N0YQ/9JkpaWXyfUExPaGCNx0pqLaYv7KnGetaPBtT5BUmfrWOd6DMXdnedNfO/JunzzV6e/X01Ywv8cEnLI2JlROyQdI2kk2vanCzpimL6OknzbLuBNSoi1kbEkmL6BUlLJc1oZA0lOlnS96LiLknjbU9rYj3zJK2IiL5O6mqYiPiVpGdrnq5eB6+Q9N4euv65pEUR8WxEPCdpkaTjGlVjRNwWER3Fw7skzdwdrz0QvSzLetSTC6Xpq84iaz4k6erd9fplaUaAz5D0VNXj1Xp1ML7SplhBn5c0qSHV9aDYhXOopMU9zD7S9v22f2r74IYW9gch6Tbb99he0MP8epZ5I52i3n85hsLylKR9ImJtMb1O0j49tBlKy/VMVf7L6kl/60cjnF3s6rm0l91RQ2lZvkPS+ohY1sv8obA8JfEhZr9sj5F0vaRzImJLzewlquwGeLOk/5L040bXVzg6Ig6TdLykv7X9zibV0S/bIySdJOmHPcweKstzF1H5v3nIHq5l+3xJHZKu6qVJs9ePb0naT9JbJK1VZffEUPYR9b313ezl+YpmBPgaSbOqHs8snuuxje1WSeMkbWpIdVVsD1clvK+KiBtq50fElojYWkzfImm47ckNLlMRsab4vkHSj1T5d7RaPcu8UY6XtCQi1tfOGCrLs7C+ezdT8X1DD22avlxtny7pPZI+VvyheZU61o/dKiLWR0RnRHRJ+k4vr9/0ZSm9kjfvl3Rtb22avTyrNSPAfydpf9tziq2xUyTdWNPmRkndn+h/UNIdva2cu0uxH+wSSUsj4uu9tJnavW/e9uGqLM+G/qGxPdr22O5pVT7Yeqim2Y2S/qo4GuUISc9X7R5otF63bobC8qxSvQ6eJuknPbS5VdKxticUuwWOLZ5rCNvHSTpX0kkR8VIvbepZP3arms9b3tfL69eTC40wX9KjEbG6p5lDYXnuohmfnKpyVMTvVfnU+fziuX9TZUWUpJGq/Iu9XNLdkuY2ocajVfm3+QFJ9xVfJ0j6pKRPFm3OlvSwKp+Y3yXp7U2oc27x+vcXtXQvz+o6LeniYnk/KKmtST/30aoE8riq55q+PFX5g7JW0k5V9r2epcpnLrdLWibp55ImFm3bJH23qu+ZxXq6XNIZDa5xuSr7jbvXz+4jt6ZLuqWv9aPBdV5ZrHcPqBLK02rrLB6/KhcaWWfx/OXd62NV26Ytz/6+OBMTADLFh5gAkCkCHAAyRYADQKYIcADIFAEOAJkiwAEgUwQ4AGSKAAeATP0/E8oxtNrsWfAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "word, img = random_captcha(20,12, blur = 0.7)\n",
    "plt.imshow(img.reshape(12,20))\n",
    "plt.title('ground truth: {}'.format(word))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrating into `pyprob`\n",
    "\n",
    "Now we will see how we integrate this existing generative model into `pyprob`\n",
    "\n",
    "In `pyprob` we only need to write a `forward()` method and inherit frmo the `Model` class. So integrating it is straight forward!\n",
    "\n",
    "In the end we want to `observe`  the resulting image (we allow the pixel values of the generated image to be sampled frmo the mean determined by the latent rendered image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import pyprob\n",
    "from pyprob import Model\n",
    "from pyprob.distributions import Normal\n",
    "\n",
    "class CaptchaModel(Model):\n",
    "    def __init__(self):\n",
    "        super().__init__(name=\"CAPTCHA model\") # give the model a name\n",
    "\n",
    "    def forward(self): # Needed to specifcy how the generative model is run forward\n",
    "        mu = pyprob.sample(pyprob.distributions.Normal(0,1))\n",
    "        blur = pyprob.sample(pyprob.distributions.Normal(0.5,0.3))\n",
    "        word, d = random_captcha(20,12, blur = blur)\n",
    "        obs_distr = likelihoood(d,1.0)\n",
    "        pyprob.observe(pyprob.distributions.Normal(d,0.5*torch.ones(d.shape)), name = 'obs0')\n",
    "        return {\n",
    "            'word': word,\n",
    "            'image': d,\n",
    "            'blur': blur\n",
    "        }\n",
    "\n",
    "model = CaptchaModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The prior distribution (unconditioned)\n",
    "\n",
    "let's see what type of CAPTCHAS we get if we just run the generator. This is basically the same way we ran it above, but now it's integrated in `pyprob`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time spent  | Time remain.| Progress             | Trace | Traces/sec\n",
      "0d:00:00:00 | 0d:00:00:00 | #################### | 10/10 | 566.42       \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAARsElEQVR4nO3dfZAkdX3H8fcnd6ABLYHjimcEDEVFU1GplaCioYToQRTUsiyMGFAjZSUmkmgMxigaUzEaYyWmfChUBJUAEVGQQh58ilpBdMEDQVBO5PDggBMURI14+M0f04fLsk+30zezP+79qpranu5fT3/3t72f6fnNTHeqCklSe35r3AVIkhbHAJekRhngktQoA1ySGmWAS1KjDHBJapQBrq1Ckrcm+cSIt7lPkkqyfJHrfzDJmxfQ7qYkhy9mG2qbAa6tXpJDk6zr4XF6DdKqenVVvb2vx9PDjwGusVjsUem4tFavtg4GuHqT5MAk30ry0ySfTHJ2kn/qlh2aZF2Sv0tyG/DRbv6rkqxJcleS85Ps3s1/yPBDki8n+bNu+vgkX0vy7iQ/TvKDJEdMabtvkv/parkU2HmWmrcHPgfsnuTe7rZ7N+RyTpJPJLkHOD7JaZt+n6m/Uzf9cWBv4LPdY7xhymZemuTmJD9K8qbN6M8Htpdk5yQXJPlJ11dfTeL/71bOHUC9SLIt8GngNGAn4EzgBdOa7doteyxwQpJnAe8AXgzsBqwFztqMzf4B8F0G4fwu4CNJ0i37L+CKbtnbgeNmeoCq+hlwBHBrVT2qu93aLT4aOAfYAThjrkKq6mXAzcDzusd415TFhwAHAIcBb0nyuwBJDknykwX+rq8D1gErgV2Avwc8D8ZWzpeF6svBDPan99bgBDvnJvnGtDa/Bk6uql8CJHkpcGpVXdndfyPw4yT7LHCba6vqQ926pwPvB3bpnkyeAhzebesrST67iN/psqr6TDf9i988N2y2t1XVL4CrklwFPBG4rqq+xuDJYSF+xeBJ7rFVtQb46mKL0cOHR+Dqy+7ALfXgs6P9cFqbDVX1f9PWWbvpTlXdC9wJ7LHAbd42Zd2fd5OP6h73x93R9SZr2XzT61+s26ZM/5xBjZvrX4E1wCVJbkxyUi+VqWkGuPqyHtgjDz5M3Wtam+kv+W9lMJwCPDAevQK4BdgUvttNab/rZtSyY/d4m+w9R/vZhiKmz//ZPPVssSGNqvppVb2uqvYDjgL+JslhW2p7aoMBrr5cBtwPvCbJ8iRHAwfNs86ZwMuTPCnJI4B/Bi6vqpuqagODID82ybIkrwAet5BCqmotMAm8Lcm2SQ4BnjfHKrcDK5I8Zp6HXg0cmWSnJLsCJ87wOPstpMbNleS5SX6ne4K8m0Ff/3pLbEvtMMDVi6q6D3gh8ErgJ8CxwAXAL+dY5/PAm4FPMThqfhxwzJQmrwL+lsGwyhOA/92Mkv6EwZucdwEnAx+bo47rGTyZ3Nh9ymP3WZp+HLgKuAm4BDh72vJ3AP/QPcbr5yswyTOS3Dtfu87+wOeBexk8Wb6/qr60wHX1MBUv6KAtJcnlwAer6qPjrkV6OPIIXL1J8odJdu2GUI4Dfh+4aNx1SQ9XfoxQfToA+G9ge+BG4EVVtX68JUkPXw6hSFKjHEKRpEYZ4JLUqJGOge+807LaZ69tRrlJSWreFVf/8kdVtXL6/JEG+D57bcM3Lp7+5TxJ0lyW7bZmxlNBOIQiSY0ywCWpUQa4JDVqqABPsirJd7srqnh6S0kaoUUHeJJlwPsYXM3k8cBLkjy+r8IkSXMb5gj8IGBNVd3YnYnuLAaXoJIkjcAwAb4HD75iyTpmuJJKkhOSTCaZ3HDn/UNsTpI01RZ/E7OqTqmqiaqaWLli2ZbenCRtNYYJ8Ft48CWz9uzmSZJGYJgA/yawf5J9u6uAHwOc309ZkqT5LPqr9FW1MclrgIuBZcCpVXVtb5VJkuY01LlQqupC4MKeapEkbQa/iSlJjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1aqhrYmrxVu09Me4SuOjmyXGXoCVoKeybALVx47hL4OJbV4+7hDl5BC5JjTLAJalRBrgkNcoAl6RGLTrAk+yV5EtJvpPk2iSv7bMwSdLchvkUykbgdVV1ZZJHA1ckubSqvtNTbZKkOSz6CLyq1lfVld30T4HrgD36KkySNLdexsCT7AM8Gbi8j8eTJM1v6ABP8ijgU8CJVXXPDMtPSDKZZHLDnfcPuzlJUmeoAE+yDYPwPqOqzp2pTVWdUlUTVTWxcsWyYTYnSZpimE+hBPgIcF1Vvae/kiRJCzHMEfjTgZcBz0qyursd2VNdkqR5LPpjhFX1NSA91iJJ2gx+E1OSGmWAS1KjDHBJapQXdBiT89ZeNu4SgG3GXYCWoKVyoY/n7P6kcZew5HkELkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElq1FZ3QYdVe0+MuwQAauPGcZfAxbeuHncJvVkKJ//P8qXx77RULsigLc8jcElqlAEuSY0ywCWpUQa4JDVq6ABPsizJt5Jc0EdBkqSF6eMI/LXAdT08jiRpMwwV4En2BP4Y+HA/5UiSFmrYI/B/B94A/Hq2BklOSDKZZHLDnfcPuTlJ0iaLDvAkzwXuqKor5mpXVadU1URVTaxcsWyxm5MkTTPMEfjTgaOS3AScBTwrySd6qUqSNK9FB3hVvbGq9qyqfYBjgC9W1bG9VSZJmpOfA5ekRvVy9p2q+jLw5T4eS5K0MB6BS1KjDHBJapQBLkmNWhpnoB+hpXKy+6VwAYKlop+LbIz/Ahnnrb1s3CV0thl3ARoRj8AlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElq1FZ3PnAtPX2co30pnF/9qD2eMu4SALj41tXjLkEj4hG4JDXKAJekRhngktQoA1ySGjVUgCfZIck5Sa5Pcl2Sp/ZVmCRpbsN+CuU/gIuq6kVJtgW266EmSdICLDrAkzwGeCZwPEBV3Qfc109ZkqT5DDOEsi+wAfhokm8l+XCS7XuqS5I0j2ECfDlwIPCBqnoy8DPgpOmNkpyQZDLJ5IY77x9ic5KkqYYJ8HXAuqq6vLt/DoNAf5CqOqWqJqpqYuWKZUNsTpI01aIDvKpuA36Y5IBu1mHAd3qpSpI0r2E/hfKXwBndJ1BuBF4+fEmSpIUYKsCrajUw0VMtkqTN4DcxJalRBrgkNcoAl6RGeUEHPSycf8s3x13Ckrmgw6q9h3tbqo8LbGg0PAKXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDVqq7ugw7Anu+/PxnEXsIT64uFi/H9TgPPWXjbU+qv2fmpPlQxr/P25dP5H1sw41yNwSWqUAS5JjTLAJalRQwV4kr9Ocm2Sa5KcmeSRfRUmSZrbogM8yR7AXwETVfV7wDLgmL4KkyTNbdghlOXAbydZDmwH3Dp8SZKkhVh0gFfVLcC7gZuB9cDdVXVJX4VJkuY2zBDKjsDRwL7A7sD2SY6dod0JSSaTTG648/7FVypJepBhhlAOB35QVRuq6lfAucDTpjeqqlOqaqKqJlauWDbE5iRJUw0T4DcDByfZLkmAw4Dr+ilLkjSfYcbALwfOAa4Evt091ik91SVJmsdQ50KpqpOBk3uqRZK0GfwmpiQ1ygCXpEYZ4JLUKANckhq11V3Q4aKbJ8ddgrSFbTPU2v6PLD3Ldpt5vkfgktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElq1LwBnuTUJHckuWbKvJ2SXJrkhu7njlu2TEnSdAs5Aj8NWDVt3knAF6pqf+AL3X1J0gjNG+BV9RXgrmmzjwZO76ZPB57fc12SpHksdgx8l6pa303fBuzSUz2SpAUa+k3MqiqgZlue5IQkk0kmN9x5/7CbkyR1FhvgtyfZDaD7ecdsDavqlKqaqKqJlSuWLXJzkqTpFhvg5wPHddPHAef1U44kaaEW8jHCM4HLgAOSrEvySuBfgD9KcgNweHdfkjRCy+drUFUvmWXRYT3XIknaDH4TU5IaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjUpVjW5jyQZg7RxNdgZ+NKJyhmGd/bLO/rRQI1jn5npsVa2cPnOkAT6fJJNVNTHuOuZjnf2yzv60UCNYZ18cQpGkRhngktSopRbgp4y7gAWyzn5ZZ39aqBGssxdLagxckrRwS+0IXJK0QGMJ8CSrknw3yZokJ82w/BFJzu6WX55knzHUuFeSLyX5TpJrk7x2hjaHJrk7yeru9pZR19nVcVOSb3c1TM6wPEne2/Xn1UkOHEONB0zpp9VJ7kly4rQ2Y+nPJKcmuSPJNVPm7ZTk0iQ3dD93nGXd47o2NyQ5bsQ1/muS67u/6aeT7DDLunPuHyOo861Jbpnydz1ylnXnzIUR1Hn2lBpvSrJ6lnVH1p/zqqqR3oBlwPeB/YBtgauAx09r8+fAB7vpY4Czx1DnbsCB3fSjge/NUOehwAWjrm2GWm8Cdp5j+ZHA54AABwOXj7neZcBtDD7bOvb+BJ4JHAhcM2Xeu4CTuumTgHfOsN5OwI3dzx276R1HWOOzgeXd9DtnqnEh+8cI6nwr8PoF7BNz5sKWrnPa8n8D3jLu/pzvNo4j8IOANVV1Y1XdB5wFHD2tzdHA6d30OcBhSTLCGqmq9VV1ZTf9U+A6YI9R1tCjo4GP1cDXgR2S7DbGeg4Dvl9Vc32pa2Sq6ivAXdNmT90HTweeP8OqzwEuraq7qurHwKXAqlHVWFWXVNXG7u7XgT23xLY3xyx9uRALyYXezFVnlzUvBs7cUtvvyzgCfA/gh1Pur+OhwfhAm24HvRtYMZLqZtAN4TwZuHyGxU9NclWSzyV5wkgL+40CLklyRZITZli+kD4fpWOY/Z9jKfQnwC5Vtb6bvg3YZYY2S6lfX8HgVdZM5ts/RuE13VDPqbMMRy2lvnwGcHtV3TDL8qXQn4BvYs4ryaOATwEnVtU90xZfyWAY4InAfwKfGXV9nUOq6kDgCOAvkjxzTHXMK8m2wFHAJ2dYvFT680Fq8Lp5yX5cK8mbgI3AGbM0Gff+8QHgccCTgPUMhieWspcw99H3uPvzAeMI8FuAvabc37ObN2ObJMuBxwB3jqS6KZJswyC8z6iqc6cvr6p7qurebvpCYJskO4+4TKrqlu7nHcCnGbwcnWohfT4qRwBXVtXt0xcslf7s3L5pmKn7eccMbcber0mOB54LvLR7onmIBewfW1RV3V5V91fVr4EPzbL9sfclPJA3LwTOnq3NuPtzqnEE+DeB/ZPs2x2NHQOcP63N+cCmd/RfBHxxtp1zS+nGwT4CXFdV75mlza6bxuaTHMSgP0f6RJNk+ySP3jTN4I2ta6Y1Ox/40+7TKAcDd08ZHhi1WY9ulkJ/TjF1HzwOOG+GNhcDz06yYzcs8Oxu3kgkWQW8ATiqqn4+S5uF7B9b1LT3W14wy/YXkgujcDhwfVWtm2nhUujPBxnHO6cMPhXxPQbvOr+pm/ePDHZEgEcyeIm9BvgGsN8YajyEwcvmq4HV3e1I4NXAq7s2rwGuZfCO+deBp42hzv267V/V1bKpP6fWGeB9XX9/G5gY0999ewaB/Jgp88benwyeUNYDv2Iw9vpKBu+5fAG4Afg8sFPXdgL48JR1X9Htp2uAl4+4xjUMxo037Z+bPrm1O3DhXPvHiOv8eLffXc0glHebXmd3/yG5MMo6u/mnbdofp7QdW3/Od/ObmJLUKN/ElKRGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXq/wHaOD560wYBSAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAT40lEQVR4nO3de9AldX3n8ffHZ0Dl4jAMs9yRi4aspLxQs0QjcSlBA6wR41oRVhO8bNjUhl3dTWKw3FWSTSXxErJryo1BRYmyQIKoaIECRrTcKDggKBcVAsPNAcbhJsgKDN/94/TomYfnNk+fOef5Me9X1amnT/evf/09/fTzOf306dOdqkKS1J6nTboASdLiGOCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywPWUk+TUJJ8a8zL3T1JJlo1zudq2GeDapiQ5IskdI+hnbZKjRlGTtFgGuLa61vZKW6tX2y4DXIuS5NAk307y4yT/kOTcJH/aTTsiyR1J/ijJXcDHu/G/k+SmJPcmuSDJXt34Jx1+SHJZkn/fDb8pydeTfCDJfUluSXLMUNsDkny1q+USYLdZat4RuAjYK8lD3WOv7pDLeUk+leRB4E1JPrHp9Qy/pm74k8B+wOe7Pt4xtJg3JLktyY+SvGsL1ufTu9d3W5K7k3w4yTOHX/+09pXkOd3wyiSfT/Jgkm8l+dPp7fXUZIBriyXZHvgM8AlgV+Bs4DemNdujm/Zs4KQkLwf+HPhNYE/gVuCcLVjsLwPfZxDO7wM+liTdtP8DXNlN+x/AiTN1UFUPA8cAP6yqnbrHD7vJxwHnAbsAZ81VSFX9FnAb8OtdH+8bmnw4cDBwJPDuJP8SIMnhSe6fo9u/AH4BeCHwHGBv4N1z1THkQ8DDDNb5iczy+vXU47+KWowXM9h2PliDi+mcn+SKaW2eAN5TVT8FSPIG4Iyquqp7/k7gviT7L3CZt1bVR7p5zwT+N7B792byr4CjumV9LcnnF/GavlFVn+2GH/n5e8MW++OqegS4Jsk1wAuAG6rq6wzeHJ6keyM6CXh+Vd3bjfszBm9M75xrYUmmgH8L/FJV/QS4vls/Ryz2BagdBrgWYy/gztr8Smi3T2uzvqr+37R5rtr0pKoeSrKBwZ7mnQtY5l1D8/6kC9idGOx139ftXW9yK7DvQl7IkOn1L9ZdQ8M/YVDjfFYBOwBXDr1xBJha4LzL2Lz+Ub0WLXEeQtFirAP2zua7qdMDc/plLn/I4HAK8LPj0SsZhPem8N1hqP0eW1DLiq6/Tfabo/1sl9+cPv7heeoZ5WU8fwQ8AhxSVbt0j+VVtSn8N6slyXAt64HHgX2Gxm3pm5caZYBrMb4BbAROTrIsyXHAYfPMczbw5iQvTPJ04M+Ay6tqbVWtZxDkb0wyleQtwEELKaSqbgXWAH+cZPskhwO/PscsdwMrkyyfp+urgWOT7NoF5ttn6OfAhdQ4n6p6AvgI8FdJ/gVAkr2T/FrX5BrgkG7dPQM4dWjejcD5wKlJdkjyi8Bvj6IuLX0GuLZYVT0KvBZ4K3A/8EbgC8BP55jnUuC/A59msNd8EHD8UJPfAf4Q2AAcAvzTFpT07xh8yHkv8B7g7+ao43sM3kxuTnL/pjNhZvBJBsG5FrgYOHfa9D8H/lvXxx/MV2CSX03y0BxN/gi4CfhmdybMpQw+DKWqfgD8STfuRmD6GSYnA8sZHL75ZPf6Zv1d6Kkj3tBBo5DkcuDDVfXxSdeyrUvyXmCPqvJslKc498C1KEn+dZI9ukMoJwLPB7446bq2RUl+McnzM3AYg/+MPjPpurT1eRaKFutg4O+BHYGbgddV1brJlrTN2pnBYZO9GByb/0vgcxOtSGPhIRRJapSHUCSpUQa4JDVqrMfAd9t1qvbfd7txLlKSmnfld376o6paNX38WAN8/32344ov+SUxSdoSU3vedOtM4z2EIkmNMsAlqVEGuCQ1qleAJzk6yfe7u6ycMqqiJEnzW3SAdxeS/xCDO5w8DzghyfNGVZgkaW599sAPA26qqpu7q9Odw+C2VJKkMegT4Huz+Z0/7ujGbSbJSUnWJFmzfsPGHouTJA3b6h9iVtXpVbW6qlavWrmQO0RJkhaiT4Dfyea3btqHhd3bUJI0An0C/FvAc5Mc0N0Z/HjggtGUJUmaz6K/Sl9Vjyc5GfgSg7tnn1FV142sMknSnHpdC6WqLgQuHFEtkqQt4DcxJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjVp0gCfZN8lXklyf5LokbxtlYZKkuS3rMe/jwO9X1VVJdgauTHJJVV0/otokSXNY9B54Va2rqqu64R8DNwB7j6owSdLcRnIMPMn+wIuAy0fRnyRpfr0DPMlOwKeBt1fVgzNMPynJmiRr1m/Y2HdxkqROrwBPsh2D8D6rqs6fqU1VnV5Vq6tq9aqVU30WJ0ka0ucslAAfA26oqtNGV5IkaSH67IG/FPgt4OVJru4ex46oLknSPBZ9GmFVfR3ICGuRJG0Bv4kpSY0ywCWpUQa4JDWqz1fpt1m3Pf5Q7z42Vv86pnp+ArHn1DN717Bu4yO9+1gK62K/ZTv1rmEU28UojOK19LVU1kXfbXypbN+zcQ9ckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktSobe6GDpc90v896/1HvWEElUze6y/6v737OPv4V/buI7ff1b+PZ+3ca/7/eukXetdw2ii2i/S/T/gHv/KpXvOvfXx57xpOe8Ube/fxxLN26N3HCedc3Gv+c489vHcNo/GBGce6By5JjTLAJalRBrgkNcoAl6RG9Q7wJFNJvp2k/6dAkqQFG8Ue+NuAG0bQjyRpC/QK8CT7AP8G+OhoypEkLVTfPfD/CbwDeGK2BklOSrImyZr1Gzb2XJwkaZNFB3iSVwH3VNWVc7WrqtOranVVrV61cmqxi5MkTdNnD/ylwKuTrAXOAV6epN9XwCRJC7boAK+qd1bVPlW1P3A88I9V1f/7s5KkBfE8cElq1EguZlVVlwGXjaIvSdLCuAcuSY0ywCWpUQa4JDWquRs63PLYQ73mf/8ILjRf9z/Qu49fvuzu3n1cfsTuveY/95iX9q7h9Rdd2ruPv3/JIb37qPsf7N3HU6EG6H9DhtOOelXvGkbxN/L6C/+pdx99t/F6sP/v9Pev+GrvPr544Mzj3QOXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUc1dD7yvuq//dYo33ndf7z6uOO45vfvgaf2ujT4K+263YdIlaMSWynXNR7Ft9X4tT0vvGvZf1j9zZuMeuCQ1ygCXpEYZ4JLUKANckhrVK8CT7JLkvCTfS3JDkpeMqjBJ0tz6noXyv4AvVtXrkmwP7DCCmiRJC7DoAE+yHHgZ8CaAqnoUeHQ0ZUmS5tPnEMoBwHrg40m+neSjSXYcUV2SpHn0CfBlwKHA31TVi4CHgVOmN0pyUpI1Sdas37Cxx+IkScP6BPgdwB1VdXn3/DwGgb6Zqjq9qlZX1epVK6d6LE6SNGzRAV5VdwG3Jzm4G3UkcP1IqpIkzavvWSj/CTirOwPlZuDN/UuSJC1ErwCvqquB1SOqRZK0BfwmpiQ1ygCXpEYZ4JLUqG3uhg5Zsbx3H1MjuMj7ay+6oncf+2+/vtf8U1TvGp69rP/F/0fxO+l7o461j67qXcNScftjK3vNn12e1buGUdwU4qn0O9la3AOXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDUqVf0v6r9Qq1/wjLriS/uObXkzueyR/u9Z73/Fq0dQyeRtXL5j7z4+9Nm/7d3H2sf739DhtKNe1buPvurBH0+6BABq3z16zX/CORf3ruHcY17au49RePyWW3vNP7ViRe8a/vrqz/fu4xf2u+vKqnrSDeTdA5ekRhngktQoA1ySGtUrwJP8lyTXJbk2ydlJnjGqwiRJc1t0gCfZG/jPwOqq+iVgCjh+VIVJkubW9xDKMuCZSZYBOwA/7F+SJGkhFh3gVXUn8AHgNmAd8EBV9T//SJK0IH0OoawAjgMOAPYCdkzyxhnanZRkTZI16zdsXHylkqTN9DmEchRwS1Wtr6rHgPOBX5neqKpOr6rVVbV61cqpHouTJA3rE+C3AS9OskOSAEcCN4ymLEnSfPocA78cOA+4Cvhu19fpI6pLkjSPZX1mrqr3AO8ZUS2SpC3gNzElqVEGuCQ1ygCXpEYZ4JLUqF4fYrboiGc+0buPAy/7VO8+No7vPhqzmkr/PvZbtlPvPg7arv8XvJ771bN6zb8Ufh9LRf+/ENj70vt697H20VW9+/j04c/rNX+WP6t3DVuTe+CS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1apu7HvgojOIa2Botfyc/d8tjD/Wa//de8x961zD1wMO9+xiFLO83/x9eckHvGg7abuttm+6BS1KjDHBJapQBLkmNMsAlqVHzBniSM5Lck+TaoXG7JrkkyY3dzxVbt0xJ0nQL2QP/BHD0tHGnAF+uqucCX+6eS5LGaN4Ar6qvAfdOG30ccGY3fCbwmhHXJUmax2KPge9eVeu64buA3UdUjyRpgXp/iFlVBdRs05OclGRNkjXrN2zsuzhJUmexAX53kj0Bup/3zNawqk6vqtVVtXrVyqlFLk6SNN1iA/wC4MRu+ETgc6MpR5K0UAs5jfBs4BvAwUnuSPJW4C+AVyS5ETiqey5JGqN5L2ZVVSfMMunIEdciSdoCfhNTkhplgEtSowxwSWqUN3SQnmIO6HkDgdM/97e9a9g46zdDxmsq/eZf6jcKcQ9ckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktSoVI3vyutJ1gO3ztFkN+BHYyqnD+scLescnRZqBOvcUs+uqlXTR441wOeTZE1VrZ50HfOxztGyztFpoUawzlHxEIokNcoAl6RGLbUAP33SBSyQdY6WdY5OCzWCdY7EkjoGLklauKW2By5JWqCJBHiSo5N8P8lNSU6ZYfrTk5zbTb88yf4TqHHfJF9Jcn2S65K8bYY2RyR5IMnV3ePd466zq2Ntku92NayZYXqSfLBbn99JcugEajx4aD1dneTBJG+f1mYi6zPJGUnuSXLt0Lhdk1yS5Mbu54pZ5j2xa3NjkhPHXOP7k3yv+51+Jskus8w75/YxhjpPTXLn0O/12FnmnTMXxlDnuUM1rk1y9Szzjm19zquqxvoApoB/Bg4EtgeuAZ43rc1/BD7cDR8PnDuBOvcEDu2GdwZ+MEOdRwBfGHdtM9S6FthtjunHAhcBAV4MXD7heqeAuxic2zrx9Qm8DDgUuHZo3PuAU7rhU4D3zjDfrsDN3c8V3fCKMdb4SmBZN/zemWpcyPYxhjpPBf5gAdvEnLmwteucNv0vgXdPen3O95jEHvhhwE1VdXNVPQqcAxw3rc1xwJnd8HnAkUkyxhqpqnVVdVU3/GPgBmDvcdYwQscBf1cD3wR2SbLnBOs5EvjnqprrS11jU1VfA+6dNnp4GzwTeM0Ms/4acElV3VtV9wGXAEePq8aquriqHu+efhPYZ2sse0vMsi4XYiG5MDJz1dllzW8CZ2+t5Y/KJAJ8b+D2oed38ORg/FmbbgN9AFg5lupm0B3CeRFw+QyTX5LkmiQXJTlkrIX9XAEXJ7kyyUkzTF/IOh+n45n9j2MprE+A3atqXTd8F7D7DG2W0np9C4P/smYy3/YxDid3h3rOmOVw1FJal78K3F1VN84yfSmsT8APMeeVZCfg08Dbq+rBaZOvYnAY4AXAXwOfHXd9ncOr6lDgGOD3krxsQnXMK8n2wKuBf5hh8lJZn5upwf/NS/Z0rSTvAh4HzpqlyaS3j78BDgJeCKxjcHhiKTuBufe+J70+f2YSAX4nsO/Q8326cTO2SbIMWA5sGEt1Q5JsxyC8z6qq86dPr6oHq+qhbvhCYLsku425TKrqzu7nPcBnGPw7Omwh63xcjgGuqqq7p09YKuuzc/emw0zdz3tmaDPx9ZrkTcCrgDd0bzRPsoDtY6uqqruramNVPQF8ZJblT3xdws/y5rXAubO1mfT6HDaJAP8W8NwkB3R7Y8cDF0xrcwGw6RP91wH/ONvGubV0x8E+BtxQVafN0maPTcfmkxzGYH2O9Y0myY5Jdt40zOCDrWunNbsA+O3ubJQXAw8MHR4Yt1n3bpbC+hwyvA2eCHxuhjZfAl6ZZEV3WOCV3bixSHI08A7g1VX1k1naLGT72Kqmfd7yG7MsfyG5MA5HAd+rqjtmmrgU1udmJvHJKYOzIn7A4FPnd3Xj/oTBhgjwDAb/Yt8EXAEcOIEaD2fwb/N3gKu7x7HA7wK/27U5GbiOwSfm3wR+ZQJ1Htgt/5qulk3rc7jOAB/q1vd3gdUT+r3vyCCQlw+Nm/j6ZPCGsg54jMGx17cy+Mzly8CNwKXArl3b1cBHh+Z9S7ed3gS8ecw13sTguPGm7XPTmVt7ARfOtX2Muc5PdtvddxiE8p7T6+yePykXxllnN/4Tm7bHobYTW5/zPfwmpiQ1yg8xJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY36//Dg0z1otvM3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "prior = model.prior_distribution()\n",
    "sample = prior.sample()\n",
    "plt.imshow(sample['image'].numpy().reshape(12,-1))\n",
    "plt.title('ground truth: {}'.format(sample['word']))\n",
    "plt.show()\n",
    "\n",
    "sample = prior.sample()\n",
    "plt.imshow(sample['image'].numpy().reshape(12,-1))\n",
    "plt.title('ground truth: {}'.format(sample['word']))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning the inference network\n",
    "\n",
    "\n",
    "During inference we want to steer the simulator with a \"smart ML agent\" such that we only get samples that match our observed captcha. This way we can inspect the latent state of the sampled traces and quickly extract the solution.\n",
    "\n",
    "We will now train the ML agent on 5000 traces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating new inference network...\n",
      "Observable obs0: observe embedding not specified, using the default FEEDFORWARD.\n",
      "Observe embedding dimension: 32\n",
      "Train. time | Epoch| Trace     | Init. loss| Min. loss | Curr. loss| T.since min | Traces/sec\n",
      "New layers, address: 16__forward__mu__Normal__1, distribution: Normal\n",
      "New layers, address: 36__forward__blur__Normal__1, distribution: Normal\n",
      "Total addresses: 2, parameters: 61,870\n",
      "0d:00:00:07 | 1    | 5,056     | +1.44e+00 | +1.31e+00 | \u001b[32m+1.32e+00\u001b[0m | 0d:00:00:00 | 617.0                              \n"
     ]
    }
   ],
   "source": [
    "model.learn_inference_network(\n",
    "    num_traces=5000,\n",
    "    observe_embeddings={'obs0': {'dim': 32, 'depth': 3}}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sampling from the Posterior\n",
    "\n",
    "Now we will generate the posterior distribution. `pyprob` will generate 10,000 weighted traces from which we can sample according to their weights, which gives us unweighted samples frmo the posterior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAVeUlEQVR4nO3de5RdZX3G8efJTC7kYu4NuRoQjAUriCMCcolAKaRCBF0KggJSU1elha6qxWUFtF21WutqtS5dURGQWwqiIgvkpoAXCExCCASQBAiQKyGEhHBNMr/+cfbg4TCXM/PunDMvfD9rzZp99t7ve36zZ89z9rxnn70dEQIA5GdQswsAAPQPAQ4AmSLAASBTBDgAZIoAB4BMEeAAkCkCHG8ots+3fUmDn3Om7bDd2sjnBQhwvGnYnm17VQn9rLR9ZBk1ASkIcOxUuR2V5lYv3twIcPSZ7f1s32P7OdtX2l5g+9+KZbNtr7L9z7bXSfpxMf/TtlfYfsb2NbanFPNfN/xg+1bbf1NMn2b7d7a/aXuT7cdsH1O17m62bytquUnShG5qHiHpeklTbG8tvqYUQy5X2b7E9hZJp9m+sPPnqf6ZiumfSJoh6ZdFH1+oepqTbT9h+2nbX6pzW77P9jrbLVXzjre9tJjurG9B8TMutr1Psex027+sarfc9pVVj5+0vW89dSBPBDj6xPYQST+TdKGkcZIul3R8zWq7FsveKmme7cMlfU3SRyVNlvS4pCv68LTvk/RHVcL5G5J+ZNvFssskLSqW/aukU7vqICKel3SMpDURMbL4WlMsnivpKkljJF3aUyER8QlJT0g6tujjG1WLD5Y0S9IRks61/eeSZPtg2892099CSc9LOrxq9seLn6vTXElXqrJNL5P0c9uDJd0m6RDbg4oXxCGSDiyec3dJIyUt7ennQd4IcPTVAZJaJX07IrZFxNWS7qpZp0PSeRHxckS8KOlkSRdExOKIeFnSFyUdaHtmnc/5eET8ICJ2SLpIlReBSbZnSHqvpC8Xz3W7pF/21FE37oiIn0dER1Fvf30lIl6MiHsl3StpH0mKiN9FxJge2l0u6SRJsj1K0pxiXqdFEXFVRGyT9C1JwyQdEBGPSnpO0r6SDpV0g6Q1tt8h6TBJv42IjoSfBwMcAY6+miJpdbz2KmhP1qyzISJeqmnzeOeDiNgqaaOkqXU+57qqti8UkyOLfjcVR9edHlff1dbfX+uqpl9QpcZ6XCbpBNtDJZ0gaXFEVP8cr9ZXBPIqVX52qXIUPluVAL9N0q2qhPdhxWO8gRHg6Ku1kqZWDWFI0vSadWovcblGleEUSa+OR4+XtFqV4QNJGl61/q59qGVs0V+nGT2s392lN2vnP99LPaVewjMiHlDlhecYvX74RKravrYHSZqmyjaV/hTghxTTt4kAf9MgwNFXd0jaIelM262250rav5c2l0s63fa+xVHmv0taGBErI2KDKkF+iu0W25+S9LZ6CimOUtslfcX2ENsHSzq2hybrJY23PbqXrpdImmN7nO1dJZ3dRT+711NjH1wm6SxVjqSvrFn2HtsnFG/0ni3pZUl3Fstuk/QBSbtExCpJv5V0tCovkPeUXCMGGAIcfRIRr6jyb/4Zkp6VdIqka1UJle7a3Czpy5J+qspR89sknVi1yqclfV6VYZW9Jf2hDyV9XJU3OZ+RdJ6ki3uo4yFVXkwetf1s55kwXfiJKmPYKyXdKGlBzfKvSfqXoo/P9Vag7UNsb+1ltctVOWr+dUQ8XbPsF5I+JmmTpE9IOqEYD1dEPCxpqyrBrYjYIulRSb8v3jPAG5i5oQNS2V4o6fsR8eNm1/JGY/t8SXtExCnNrgUDD0fg6DPbh9netRhCOVXSuyT9qtl1AW82fOoM/TFL0v9JGqHKv+sfiYi1zS0JePNhCAUAMsUQCgBkigAHgEw1dAx8wriWmDl9cCOfEgCyt2jpy09HxMTa+Q0N8JnTB+uuG2o/tAcA6EnL5BVdXiKCIRQAyBQBDgCZIsABIFNJAW77aNt/LO60ck5ZRQEAetfvAC9uAfVdVS6BuZekk2zvVVZhAICepRyB7y9pRUQ8Wlyh7gpVbv0EAGiAlACfqtfeyWSVurjDiu15ttttt2/YyNUtAaAsO/1NzIiYHxFtEdE2cXxL7w0AAHVJCfDVeu2ttKYV8wAADZAS4HdL2tP2braHqHKHlWvKKQsA0Jt+f5Q+IrbbPlPSDZJaJF0QEctKqwwA0KOka6FExHWSriupFgBAH/BJTADIFAEOAJkiwAEgUwQ4AGSKAAeATBHgAJApAhwAMkWAA0CmCHAAyBQBDgCZIsABIFMEOABkigAHgEwR4ACQKQIcADJFgANApghwAMgUAQ4AmSLAASBTSffEbIYd0ZHUfrt2lFRJmkElvHZ2KG1bbIuBsS3eKAa7JbmPgbBflGGg/BypdZRRw878O+MIHAAyRYADQKYIcADIFAEOAJnqd4Dbnm77N7YfsL3M9lllFgYA6FnKWSjbJf1TRCy2PUrSIts3RcQDJdUGAOhBv4/AI2JtRCwupp+T9KCkqWUVBgDoWSlj4LZnSnq3pIVl9AcA6F1ygNseKemnks6OiC1dLJ9nu912+4aNfHAEAMqSFOC2B6sS3pdGxNVdrRMR8yOiLSLaJo5P/6QaAKAi5SwUS/qRpAcj4lvllQQAqEfKEfj7JX1C0uG2lxRfc0qqCwDQi36fRhgRv5PkEmsBAPQBn8QEgEwR4ACQKQIcADKV3Q0dNna8mNT+yufekVzDtkg/HfKQ4Q8n9/HbF96e1P7pbaOSa2hxCRe872j+6aVDB21P7mN06wvJfRywyyPJfSx5aWZS+22RHgtl7N/LXpmS3MfeQ9YktU/9G5OkZ7aPSO5DWtXlXI7AASBTBDgAZIoAB4BMEeAAkCkCHAAyRYADQKYIcADIFAEOAJkiwAEgUwQ4AGSKAAeATBHgAJApAhwAMkWAA0CmCHAAyBQBDgCZaugNHXaoQ1s7XkrqY8GWvZLaX/zNOUntJanllUju4+KPvS+5j2GXjk1qP2LNy8k1vDRhSHIfI1al3wgh1dYZw5P76GhNv8f3t4/ZltzHyKVDk9oP3ZS+f1/20bbkPjb/flJyH6Pfvz6p/Y4r/iy5hhLueSLpmi7ncgQOAJkiwAEgUwQ4AGSKAAeATCUHuO0W2/fYvraMggAA9SnjCPwsSQ+W0A8AoA+SAtz2NEl/LemH5ZQDAKhX6hH4f0v6gqRuz3S0Pc92u+32jRtLOSESAKCEALf9QUlPRcSintaLiPkR0RYRbePH854pAJQlJVHfL+k42yslXSHpcNuXlFIVAKBX/Q7wiPhiREyLiJmSTpT064g4pbTKAAA9YkwDADJVysWsIuJWSbeW0RcAoD4cgQNApghwAMgUAQ4AmWroDR3KsENpF81vfSn9YvVj7ns2uY9H3jkuuY9JyzYltX9x+qjkGlbPTr+JwTu+szW5D3ek/V7Xf3hkcg17XJL2+5CkYZvS61h53I6k9mNuTv8bWfPAxOQ+pi7ZntzH6tFpN2SYsjltW0rS5tOeS+5DF3c9myNwAMgUAQ4AmSLAASBTBDgAZIoAB4BMEeAAkCkCHAAyRYADQKYIcADIFAEOAJkiwAEgUwQ4AGSKAAeATBHgAJApAhwAMtXQ64G3aJBGDhqW1MeckcuS2i/45HuS2kuSPj84uYs9//fx5D46xr4lqf3mz6Rfp/hdY9Kvgf3K0DHJfcS2tGtHb5+4LbkG70i/dvTQp15I72NSS1L7jpbhyTW8/Xvrk/vYseKx5D5mLdo1qf326enXNZ81+YnkPu7vZj5H4ACQKQIcADJFgANApghwAMhUUoDbHmP7KtsP2X7Q9oFlFQYA6FnqWSj/I+lXEfER20Mkpb99DQCoS78D3PZoSYdKOk2SIuIVSa+UUxYAoDcpQyi7Sdog6ce277H9Q9sjSqoLANCLlABvlbSfpO9FxLslPS/pnNqVbM+z3W67fcPG9A86AAAqUgJ8laRVEbGweHyVKoH+GhExPyLaIqJt4vi0T4gBAP6k3wEeEeskPWl7VjHrCEkPlFIVAKBXqWeh/L2kS4szUB6VdHp6SQCAeiQFeEQskdRWUi0AgD7gk5gAkCkCHAAyRYADQKYaekOHMkxpSTsV8dip3V0avX7X7zk7uY9Rt6xJ7mPr/tOT2h8z447kGp7dnn71hMdaxyX3MWjLy0ntB29Iv0lHJO6bkvTcHqOS+/jAzHuT2i88bUZyDSPPTd8vWien3YxBkjo2b0lq//xBb02uYdqw9JuedIcjcADIFAEOAJkiwAEgUwQ4AGSKAAeATBHgAJApAhwAMkWAA0CmCHAAyBQBDgCZIsABIFMEOABkigAHgEwR4ACQKQIcADJFgANAprK7ocPIQcOS2p80elFyDX84a/fkPp7Y/Z3Jfcw49rGk9ieOvSu5hgs2Hpzch7ftSO5j+6rVSe2n3TwpuYZn/2JMch+jP/1kch+fnfibpPZ77LJ3cg0XHHV0ch/Pz9oluY+3LB2S1H76h9L+xiTppDF3J/fx1W7mcwQOAJkiwAEgUwQ4AGQqKcBt/6PtZbbvt3257bQBagBA3fod4LanSvoHSW0R8U5JLZJOLKswAEDPUodQWiXtYrtV0nBJa9JLAgDUo98BHhGrJX1T0hOS1kraHBE3llUYAKBnKUMoYyXNlbSbpCmSRtg+pYv15tlut92+YWP6+b4AgIqUIZQjJT0WERsiYpukqyUdVLtSRMyPiLaIaJs4viXh6QAA1VIC/AlJB9gebtuSjpD0YDllAQB6kzIGvlDSVZIWS7qv6Gt+SXUBAHqRdC2UiDhP0nkl1QIA6AM+iQkAmSLAASBTBDgAZIoAB4BMZXdDh1TTWtMvEv+d3a5M7uPuv52a3Md7h6bdxGCbnFxD+4YZyX28ZUdHch+t09K25/Lj0/8Ujj8g/QYZZ064PbmPKa1Dk9pPHH1fcg3TztiY3MfMwU8n9/HQQZOT2h+yy8rkGsrInO5wBA4AmSLAASBTBDgAZIoAB4BMEeAAkCkCHAAyRYADQKYIcADIFAEOAJkiwAEgUwQ4AGSKAAeATBHgAJApAhwAMkWAA0Cm3nTXAx/sluQ+ZrQOT+5jSmv6tY53xJCk9te+MDG5hk2375rcx/Dxzyf3oXEj0mp4In2/2OvwNcl9TGpJ+51K0lAPTmo/flD6tpg7In3/HlTC8eU+Q9YmtW9V+t96i3fecTJH4ACQKQIcADJFgANApghwAMhUrwFu+wLbT9m+v2reONs32V5efB+7c8sEANSq5wj8QklH18w7R9ItEbGnpFuKxwCABuo1wCPidknP1MyeK+miYvoiSR8quS4AQC/6OwY+KSI6T7BcJ2lSSfUAAOqU/CZmRISk6G657Xm22223b9i4I/XpAACF/gb4etuTJan4/lR3K0bE/Ihoi4i2iePTP+EFAKjob4BfI+nUYvpUSb8opxwAQL3qOY3wckl3SJple5XtMyT9h6S/tL1c0pHFYwBAA/V6MauIOKmbRUeUXAsAoA/4JCYAZIoAB4BMEeAAkKk33Q0dylDGBdpbynjtdFrz2buk34BgzcnXJfex8sMTkvtINWdY+g0Ijh35SHIfwwel3ZiiDANm/y7FG/vU5YGylQEAfUSAA0CmCHAAyBQBDgCZIsABIFMEOABkigAHgEwR4ACQKQIcADJFgANApghwAMgUAQ4AmSLAASBTBDgAZIoAB4BMEeAAkClHROOezN4g6fEeVpkgKf3K+jsfdZaLOsuTQ40SdfbVWyNiYu3MhgZ4b2y3R0Rbs+voDXWWizrLk0ONEnWWhSEUAMgUAQ4AmRpoAT6/2QXUiTrLRZ3lyaFGiTpLMaDGwAEA9RtoR+AAgDo1JcBtH237j7ZX2D6ni+VDbS8oli+0PbMJNU63/RvbD9heZvusLtaZbXuz7SXF17mNrrOoY6Xt+4oa2rtYbtvfLrbnUtv7NaHGWVXbaYntLbbPrlmnKdvT9gW2n7J9f9W8cbZvsr28+D62m7anFusst31qg2v8T9sPFb/Tn9ke003bHvePBtR5vu3VVb/XOd207TEXGlDngqoaV9pe0k3bhm3PXkVEQ78ktUh6RNLukoZIulfSXjXr/J2k7xfTJ0pa0IQ6J0var5geJenhLuqcLenaRtfWRa0rJU3oYfkcSddLsqQDJC1scr0tktapcm5r07enpEMl7Sfp/qp535B0TjF9jqSvd9FunKRHi+9ji+mxDazxKEmtxfTXu6qxnv2jAXWeL+lzdewTPebCzq6zZvl/STq32duzt69mHIHvL2lFRDwaEa9IukLS3Jp15kq6qJi+StIRtt3AGhURayNicTH9nKQHJU1tZA0lmivp4qi4U9IY25ObWM8Rkh6JiJ4+1NUwEXG7pGdqZlfvgxdJ+lAXTf9K0k0R8UxEbJJ0k6SjG1VjRNwYEduLh3dKmrYznrsvutmW9agnF0rTU51F1nxU0uU76/nL0owAnyrpyarHq/T6YHx1nWIH3SxpfEOq60IxhPNuSQu7WHyg7XttX29774YW9ich6Ubbi2zP62J5Pdu8kU5U938cA2F7StKkiFhbTK+TNKmLdQbSdv2UKv9ldaW3/aMRziyGei7oZjhqIG3LQyStj4jl3SwfCNtTEm9i9sr2SEk/lXR2RGypWbxYlWGAfSR9R9LPG11f4eCI2E/SMZI+a/vQJtXRK9tDJB0n6couFg+U7fkaUfm/ecCermX7S5K2S7q0m1WavX98T9LbJO0raa0qwxMD2Unq+ei72dvzVc0I8NWSplc9nlbM63Id262SRkva2JDqqtgerEp4XxoRV9cuj4gtEbG1mL5O0mDbExpcpiJidfH9KUk/U+Xf0Wr1bPNGOUbS4ohYX7tgoGzPwvrOYabi+1NdrNP07Wr7NEkflHRy8ULzOnXsHztVRKyPiB0R0SHpB908f9O3pfRq3pwgaUF36zR7e1ZrRoDfLWlP27sVR2MnSrqmZp1rJHW+o/8RSb/ubufcWYpxsB9JejAivtXNOrt2js3b3l+V7dnQFxrbI2yP6pxW5Y2t+2tWu0bSJ4uzUQ6QtLlqeKDRuj26GQjbs0r1PniqpF90sc4Nko6yPbYYFjiqmNcQto+W9AVJx0XEC92sU8/+sVPVvN9yfDfPX08uNMKRkh6KiFVdLRwI2/M1mvHOqSpnRTysyrvOXyrmfVWVHVGShqnyL/YKSXdJ2r0JNR6syr/NSyUtKb7mSPqMpM8U65wpaZkq75jfKemgJtS5e/H89xa1dG7P6jot6bvF9r5PUluTfu8jVAnk0VXzmr49VXlBWStpmypjr2eo8p7LLZKWS7pZ0rhi3TZJP6xq+6liP10h6fQG17hClXHjzv2z88ytKZKu62n/aHCdPyn2u6WqhPLk2jqLx6/LhUbWWcy/sHN/rFq3aduzty8+iQkAmeJNTADIFAEOAJkiwAEgUwQ4AGSKAAeATBHgAJApAhwAMkWAA0Cm/h/E20iwI8vrogAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "condition = prior.sample()\n",
    "plt.imshow(condition['image'].numpy().reshape(12,-1))\n",
    "plt.title('ground truth: {}'.format(condition['word']))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time spent  | Time remain.| Progress             | Trace       | Traces/sec\n",
      "0d:00:00:53 | 0d:00:00:00 | #################### | 10000/10000 | 188.66       \n"
     ]
    }
   ],
   "source": [
    "posterior = model.posterior_distribution(\n",
    "    num_traces=10000,\n",
    "    inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING_WITH_INFERENCE_NETWORK,\n",
    "    observe={'obs0': condition['image'].numpy()}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making the Animations\n",
    "\n",
    "Now that we have the prior and posterior distributions we can sample a few captchas to observe the effect of the conditioning on the above \"observation\"\n",
    "\n",
    "We will draw 100 captchas each. If everything worked, the prior will be changing very rapidly while the posterior should almost always yield a CAPTCHA that looks much like the condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "10\n",
      "20\n",
      "30\n",
      "40\n",
      "50\n",
      "60\n",
      "70\n",
      "80\n",
      "90\n",
      "make animation\n",
      "display image\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"prior_1572353584.1552958.gif\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in range(100):\n",
    "    if i%10 == 0:\n",
    "        print(i)\n",
    "    sample = prior.sample()\n",
    "    plt.imshow(sample['image'].reshape(12,-1))\n",
    "    plt.title('generated {}'.format(sample['word']))\n",
    "    plt.savefig('foranim_prior_{}.png'.format(str(i).zfill(6)))\n",
    "plt.clf()\n",
    "print('make animation')\n",
    "uniq = time.time()\n",
    "!convert -delay 5 -loop 0 foranim_prior_*.png prior_{uniq}.gif\n",
    "!rm foranim_prior_*png\n",
    "print('display image')\n",
    "IPython.display.Image(url='prior_{}.gif'.format(uniq).format(uniq))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "10\n",
      "20\n",
      "30\n",
      "40\n",
      "50\n",
      "60\n",
      "70\n",
      "80\n",
      "90\n",
      "make animation\n",
      "display image\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"posterior_1572353655.433361.gif\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in range(100):\n",
    "    if i%10 == 0:\n",
    "        print(i)\n",
    "    sample = posterior.sample()\n",
    "    plt.imshow(sample['image'].reshape(12,-1))\n",
    "    plt.title('ground truth: {}, generated: {}'.format(condition['word'],sample['word']))\n",
    "    plt.savefig('foranim_posterior_{}.png'.format(str(i).zfill(6)))\n",
    "plt.clf()\n",
    "print('make animation')\n",
    "uniq = time.time()\n",
    "!convert -delay 5 -loop 0 foranim_posterior_*.png posterior_{uniq}.gif\n",
    "!rm -rf foranim_posterior_*png\n",
    "print('display image')\n",
    "IPython.display.Image(url='posterior_{}.gif'.format(uniq))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspecting the Latent State\n",
    "\n",
    "For evey sample we draw from the posterior we have full access to the trace. We can e.g. construct marginal posterior  distributions for specific letters of the captcha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.LineCollection at 0x7f630fd6fc50>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAATnklEQVR4nO3de7QdZXnH8e9TAoT7LZECST1EWAiVJcQISJRagqsIoWAbKEUEXNRURagoImILlEVpVCxSbFOD2ISL3FKQm6IS5C6BXMAAAY0BJFkIATFc0iiRp3/M0HXAcPbMyTlndsbvZ6299lze2fPsfeb8zrvfPXtOZCaSpHb5o6YLkCQNPMNdklrIcJekFjLcJamFDHdJaqFhTRcAMGLEiOzp6Wm6DElaq8ydO/fZzBy5unVdEe49PT3MmTOn6TIkaa0SEU+82TqHZSSphQx3SWohw12SWshwl6QWMtwlqYUMd0lqIcNdklrIcJekFjLcJamFDHdJquPuu4tbl+uKyw9I0lpj772brqASe+6SVIc9d0lqoVNPLe5vvbXRMjqx5y5JLWS4S1ILGe6S1EKGuyS1kB+oSlIdX/ta0xVUYrhLUh277dZ0BZU4LCNJddx8c3HrcvbcJamOs84q7vfbr9k6OrDnLkktZLhLUgsZ7pLUQoa7JLWQH6hKUh3f+EbTFVRiuEtSHTvt1HQFlTgsI0l1XH99cety9twlqY6vfrW4P+igZuvowJ67JLWQ4S5JLWS4S1ILGe6S1EJ+oCpJdVx8cdMVVGK4S1Ido0c3XUElDstIUh1XXFHculylcI+IEyPioYh4MCIui4jhEbF9RMyOiEURcUVErFe2Xb+cX1Su7xnMJyBJQ2rq1OLW5TqGe0RsB5wAjMvMdwDrAIcDXwLOzcwdgOeBY8tNjgWeL5efW7aTJA2hqsMyw4ANImIYsCHwFLAvMLNcPwM4pJw+uJynXD8hImJgypUkVdEx3DNzKXAO8AuKUF8OzAV+nZmrymZLgO3K6e2AJ8ttV5Xtt3rj40bE5IiYExFzli1btqbPQ5LUS5VhmS0oeuPbA9sCGwH7r+mOM3NaZo7LzHEjR45c04eTJPVS5VTI/YDHMnMZQERcDYwHNo+IYWXvfBSwtGy/FBgNLCmHcTYDnhvwyiWpCTNndm7TBaqMuf8C2CsiNizHzicADwM/AiaVbY4Gri2nryvnKdffkpk5cCVLUoNGjChuXa5jzz0zZ0fETGAesAqYD0wDbgQuj4izymUXlptcCFwcEYuAX1GcWSNJa5WeU25c7fJJC24GYOau+/3eusenHDioNdVR6RuqmXk6cPobFi8G9lhN25XAoWtemiR1n77CvZv4DVVJaiHDXZJayHCXpBYy3CWphbzkryTVcMyhZzRdQiWGuyTVsHLd4U2XUInDMpJUw5HzbuTIeas/B76bGO6SVMPER+5g4iN3NF1GR4a7JLWQ4S5JLWS4S1ILGe6S1EKeCilJNRx+xJSmS6jEnrsktZDhLkk1fGz21Xxs9tVNl9GR4S5JNUz4+b1M+Pm9TZfRkeEuSS1kuEtSCxnuktRCngopSTWsHLZ+0yVUYrhLUg3HHPbPTZdQicMyktRChrsk1XD8XZdx/F2XNV1GR4a7JNUw/okHGP/EA02X0ZHhLkktZLhLUgsZ7pLUQp4KKUk1PL/Bpk2XUInhLkk1fOJDpzZdQiUOy0hSCxnuklTDybdN5+TbpjddRkcOy0hSDWOXPtJ0CZXYc5ekFjLcJamFDHdJaqFK4R4Rm0fEzIh4JCIWRsR7ImLLiPhhRPysvN+ibBsR8e8RsSgifhIRYwf3KUjS0HlqkxE8tcmIpsvoqOoHqucBN2XmpIhYD9gQOBWYlZlTIuIU4BTg88AHgR3L257A1PJektZ6Jx50UtMlVNKx5x4RmwH7ABcCZOZvM/PXwMHAjLLZDOCQcvpg4KIs3ANsHhHbDHjlkqQ3VWVYZntgGfDfETE/Ir4ZERsBW2fmU2WbXwJbl9PbAU/22n5Juex1ImJyRMyJiDnLli3r/zOQpCF02s3TOO3maU2X0VGVcB8GjAWmZubuwMsUQzD/LzMTyDo7zsxpmTkuM8eNHDmyzqaS1JhdnlnMLs8sbrqMjqqE+xJgSWbOLudnUoT9068Nt5T3z5TrlwKje20/qlwmSRoiHcM9M38JPBkRO5WLJgAPA9cBR5fLjgauLaevA44qz5rZC1jea/hGkjQEqp4tczxwaXmmzGLgoxR/GK6MiGOBJ4DDyrbfBQ4AFgEryraSpCFUKdwz835g3GpWTVhN2wSOW8O6JKkrLd7y984P6UpeOEySajh1/+ObLqESLz8gSS1kuEtSDWffdD5n33R+02V05LCMJNUw5ldrx5nd9twlqYUMd0lqIcNdklrIMXdJquHht4xpuoRKDHdJquHM/SY3XUIlDstIUgsZ7pJUw7nXn8O515/TdBkdOSwjSTVs8+KzTZdQiT13SWohw12SWshwl6QWcsxdkmqYt93bmy6hEsNdkmr48p8d03QJlTgsI0ktZLhLUg1Trzmbqdec3XQZHTksI0k1bPG/LzRdQiX23CWphQx3SWohw12SWsgxd0mq4a63vrPpEiox3CWphvPH/23TJVTisIwktZDhLkk1TL/ydKZfeXrTZXTksIwk1TB81W+aLqESe+6S1EKGuyS1kOEuSS3kmLsk1TDrbXs0XUIlhrsk1XDBnn/VdAmVOCwjSS1kuEtSDZd/+xQu//YpTZfRUeVwj4h1ImJ+RNxQzm8fEbMjYlFEXBER65XL1y/nF5XrewandEnSm6nTc/8HYGGv+S8B52bmDsDzwLHl8mOB58vl55btJElDqFK4R8Qo4EDgm+V8APsCM8smM4BDyumDy3nK9RPK9pKkIVK15/414GTg1XJ+K+DXmbmqnF8CbFdObwc8CVCuX162f52ImBwRcyJizrJly/pZviRpdTqeChkRE4FnMnNuRLx/oHacmdOAaQDjxo3LgXpcSRpMN7z9fU2XUEmV89zHA38ZEQcAw4FNgfOAzSNiWNk7HwUsLdsvBUYDSyJiGLAZ8NyAVy5JDbhk7IFNl1BJx2GZzPxCZo7KzB7gcOCWzPww8CNgUtnsaODacvq6cp5y/S2Zac9cUisMf2Ulw19Z2XQZHa3Jee6fBz4TEYsoxtQvLJdfCGxVLv8M0P0nhEpSRdOvOoPpV53RdBkd1br8QGbeCtxaTi8Gfu8iC5m5Ejh0AGqTJPWT31CVpBYy3CWphQx3SWohL/krSTXM3HW/pkuoxHCXpBrWlnB3WEaSathixXK2WLG86TI6sucuSTVM/c6/AnD4EVMarqRv9twlqYUMd0lqIcNdklrIcJekFvIDVUmq4ZLdD2i6hEoMd0mq4Yad92m6hEoclpGkGrZ5YRnbvND9/xrUnrsk1XDuDV8FPM9dktQAw12SWshwl6QWMtwlqYX8QFWSarhgjw81XUIlhrsk1TBrhz2bLqESh2UkqYYxzy1hzHNLmi6jI3vuklTD2d//OuB57pKkBhjuktRChrsktZDhLkkt5AeqklTD+Xsf3nQJlRjuklTDXT27NV1CJQ7LSFINuzy9mF2eXtx0GR0Z7pJUw2mzpnHarGlNl9GR4S5JLWS4S1ILGe6S1EKGuyS1UMdTISNiNHARsDWQwLTMPC8itgSuAHqAx4HDMvP5iAjgPOAAYAVwTGbOG5zyJWlofXmfo5suoZIqPfdVwGczcxdgL+C4iNgFOAWYlZk7ArPKeYAPAjuWt8nA1AGvWpIaMm/UzswbtXPTZXTUMdwz86nXet6Z+SKwENgOOBiYUTabARxSTh8MXJSFe4DNI2KbAa9ckhowdslCxi5Z2HQZHdUac4+IHmB3YDawdWY+Va76JcWwDRTB/2SvzZaUyyRprXfy7TM4+fYZnRs2rHK4R8TGwP8An87MF3qvy8ykGI+vLCImR8SciJizbNmyOptKkjqoFO4RsS5FsF+amVeXi59+bbilvH+mXL4UGN1r81HlstfJzGmZOS4zx40cObK/9UuSVqNjuJdnv1wILMzMf+u16jrgtY+Njwau7bX8qCjsBSzvNXwjSRoCVa4KOR74CLAgIu4vl50KTAGujIhjgSeAw8p136U4DXIRxamQHx3QiiVJHXUM98y8E4g3WT1hNe0TOG4N65KkrnTmhMlNl1CJ13OXpBoe3npM0yVU4uUHJKmG8Y/fz/jH7+/csGH23CWphuPvvhzo/v/IZM9dklrIcJekFjLcJamFDHdJaiE/UJWkGk79i081XUIlhrsk1bB4q1FNl1CJwzKSVMOERbOZsGh202V0ZM9dkmr42L3XADBrhz0brqRv9twlqYUMd0lqIcNdklrIcJekFvIDVUmq4cSJn226hEoMd0mq4alN147/+eywjCTVMHHh7UxceHvTZXRkz10aYD2n3Fh7m8enHDgIlWgwHDn/uwDcsPM+DVfSN3vuktRChrsktZDhLkktZLhLUgv5gaok1fCJQ77QdAmVGO5dxLMspO73/IabNV1CJQ7LSFINkxbczKQFNzddRkeGuyTVYLhLkhpjuEtSCxnuktRChrsktZCnQkpSDcccekbTJVRiuEtSDSvXHd50CZU4LCNJNRw570aOnFf/C4dDzZ671AX8dvLaY+IjdwBwydjufv3tuUtSCw1Kzz0i9gfOA9YBvpmZUwZjP5JUVX/eHa3NBjzcI2Id4D+ADwBLgPsi4rrMfHig9wW+nZWk1RmMnvsewKLMXAwQEZcDBwODEu6qxz+GaoM/tF54f0RmDuwDRkwC9s/MvyvnPwLsmZmfekO7ycDkcnYn4NEBLaQwAnh2ENsP1TbWZV3dto111d9mMLw1M0eudk1mDugNmEQxzv7a/EeArw/0firWMmcw2w/VNtZlXd22jXXV32aob4NxtsxSYHSv+VHlMknSEBmMcL8P2DEito+I9YDDgesGYT+SpDcx4B+oZuaqiPgU8H2KUyG/lZkPDfR+Kpo2yO2Hahvr6r599Gebbq2rP9tYV5cb8A9UJUnN8xuqktRChrsktZDhDkRET0Q8OMT7PCMiThqkxz4hIhZGxKWD9Pj9fr0i4u7B2mYN63qpP9tpcETE5hHxyabrWJsZ7u30SeADmfnhpgt5o8zceyi20dCIwmDkyOYUx7H6qZXhHhHfiYi5EfFQ+U3YKoZFxKVlj3dmRGxYYT9HRcRPIuKBiLi4QvsvRsRPI+JOim/ldhQRR0bEvRFxf0R8o7x2T1/t/wsYA3wvIk6suI9/iohHI+LOiLis4juKdSLigvI1/kFEbFBxX7V7yP3cZkxEzI+Id9fdto/H7ImIRyJievlzvDQi9ouIuyLiZxGxRx/bLaz7ekXEZyLiwfL26Rr1VT6Oex+TVX/25X4ejYiLgAd5/fdaVtd+o4i4sfw9eTAi/qbTPoApwNvK4/4rFWt6sNf8SRFxRh/tp0TEcb3m+3wnHRGfi4gTyulzI+KWcnrfwXqHvMaa/hbVYNyALcv7DSgOvq06tO8BEhhfzn8LOKnDNn8K/BQY0XuffbR/F7AA2BDYFFhUYR87A9cD65bz/wkcVeH5P/5aXRXavhu4HxgObAL8rEJdPcAqYLdy/krgyIr7e6kfP89K25R1PUjxh3M+8M6B3Eev570rRcdobnmsBMX1k74zUK9Xr+NlI2Bj4CFg94E8jvtzTPbaz6vAXhVf278GLug1v1nVn2WNn9/r2gMnAWf00X534LZe8w8Do/tovxdwVTl9B3AvsC5wOvD3dY/pobi1sucOnBARDwD3UPQqdqywzZOZeVc5fQnw3g7t96X4YT8LkJm/6tD+fcA1mbkiM1+g2he7JlD8At4XEfeX82MqbFfHeODazFyZmS9S/DGp4rHMvL+cnkvxy9UNRgLXAh/OzAcG4fEfy8wFmfkqReDOyuI3fgF9vwZ1X6/3UhwvL2fmS8DVFMdQJ3WO4/4ck695IjPvqdh2AfCBiPhSRLwvM5fX2M+gyMz5wFsiYtuIeCfwfGY+2ccmc4F3RcSmwG+AHwPjKF7DOwa94H5o3X9iioj3A/sB78nMFRFxK0WvtJM3nvDfDV8ACGBGZn6h6UJW4ze9pn9H8S6pGywHfkERaoNxJdLez/vVXvOv0vfv01C9XkN1HL9ctWFm/jQixgIHAGdFxKzMPHOA61nF64eZq/zOX0VxLaw/Bq7oq2FmvhIRjwHHAHcDPwH+HNgBWNiPegddG3vum1H8FV4REW+neDtVxZ9ExHvK6SOAOzu0vwU4NCK2AoiILTu0vx04JCI2iIhNgIMq1DQLmBQRb3ltHxHx1grb1XEXcFBEDI+IjYGJA/z4Q+23wIeAoyLiiKaLWQN3UBwvG0bERhTPqUoPsc5x3J9jsraI2BZYkZmXAF8BxlbY7EWKYcKqnqboiW8VEetT7Ti+guLyKJMogr6TOyiGe24vpz8OzC/fuXWd1vXcgZuAj0fEQorLCFd96/gocFxEfIuixze1r8aZ+VBE/AtwW0T8jmKM95g+2s+LiCuAB4BnKK7B06fMfDgi/hH4QRRnJLwCHAc8Ue0pdZaZ90XEdRQ9kacp3kI3/rb5DWr98mTmyxExEfhhRLyUmWvdtY3K42U6xdguFFdanV9h08rHcX+OyX7aFfhKRLxKcQx/otMGmflc+UH1g8D3MvNzHdq/EhFnUrxeS4FHKuzjofKP2tLMfKrC87gD+CLw4/IYW0mXDsmAlx8QEBEbZ+ZL5ZkVtwOTM3Ne03UBlO+M5mXmQL9jaZ2I6AFuyMx39HP7Myg+WD5nAMtSQ9rYc1d90yJiF4pxyhldFOzbArcCho1Ukz13SWqhNn6gKkl/8Ax3SWohw12SWshwl6QWMtwlqYX+Dx++TQqr4YC9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "letter_num = 2\n",
    "\n",
    "reverse_alphabet = {v:k for k,v in alphabet.items()} \n",
    "c,_,_ = plt.hist([reverse_alphabet[posterior.sample()['word'][letter_num]] for i in range(1000)], bins = np.linspace(-0.5,len(alphabet)+0.5,len(alphabet)+2))\n",
    "\n",
    "plt.xticks(range(len(alphabetorder)),list(alphabetorder));\n",
    "plt.vlines(reverse_alphabet[condition['word'][letter_num]],0,1.2*max(c), colors = 'r', linestyles = 'dashed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solving the CAPTCHA\n",
    "\n",
    "A good solution is the MAP, the maximum a posteriori value of the posterior. If we sample traces from the posterior and pick the solution which appears most often, it has a good chance to be the true solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'wpw'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "solutions, counts = np.unique([posterior.sample()['word'] for i in range(1000)], return_counts = True)\n",
    "solutions[np.argmax(counts)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}