homeworks/homework8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Homework 8 - Berkeley STAT 157\n",
    "\n",
    "**Your name: XX, SID YY, teammates A,B,C** (Please add your name, SID and teammates to ease Ryan and Rachel to grade.)\n",
    "\n",
    "**Please submit your homework through [gradescope](http://gradescope.com/)**\n",
    "\n",
    "Handout 4/9/2019, due 4/16/2019 by 4pm.\n",
    "\n",
    "This homework deals with sequence models for text and numbers. Due to the computational cost, we strongly encourage you to implement this on a GPU enabled machine. To make things a bit more interesting we will use a larger text collection here - [Shakespeare's collected works](http://www.gutenberg.org/files/100/100-0.txt) which are freely downloadable at Project Gutenberg. \n",
    "\n",
    "**This is teamwork.**\n",
    "\n",
    "## Prerequisites - Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of characters:  5032359\n",
      "project gutenberg s the complete works of william shakespeare by willi\n"
     ]
    }
   ],
   "source": [
    "import urllib3\n",
    "import collections\n",
    "import re\n",
    "shakespeare = 'http://www.gutenberg.org/files/100/100-0.txt'\n",
    "\n",
    "http = urllib3.PoolManager()\n",
    "text = http.request('GET', shakespeare).data.decode('utf-8')\n",
    "raw_dataset = ' '.join(re.sub('[^A-Za-z]+', ' ', text).lower().split())\n",
    "\n",
    "print('number of characters: ', len(raw_dataset))\n",
    "print(raw_dataset[0:70])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset is quite a bit bigger than the time machine (5 million vs. 160k). For convenience we also include the remaining preprocessing steps. A bigger dataset will allow us to generate more meaningful models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chars: project gutenberg s \n",
      "indices: [21, 19, 22, 10, 26, 24, 13, 3, 12, 9, 13, 26, 20, 4, 26, 19, 12, 3, 5, 3]\n"
     ]
    }
   ],
   "source": [
    "idx_to_char = list(set(raw_dataset))\n",
    "char_to_idx = dict([(char, i) for i, char in enumerate(idx_to_char)])\n",
    "vocab_size = len(char_to_idx)\n",
    "corpus_indices = [char_to_idx[char] for char in raw_dataset]\n",
    "sample = corpus_indices[:20]\n",
    "print('chars:', ''.join([idx_to_char[idx] for idx in sample]))\n",
    "print('indices:', sample)\n",
    "\n",
    "train_indices = corpus_indices[:-100000]\n",
    "test_indices = corpus_indices[-100000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly we import other useful libraries to help you getting started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4932359 100000\n"
     ]
    }
   ],
   "source": [
    "import d2l\n",
    "import math\n",
    "from mxnet import autograd, gluon, init, nd\n",
    "from mxnet.gluon import loss as gloss, nn, rnn\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Train Recurrent Latent Variable Models\n",
    "\n",
    "Train a number of different latent variable models using `train_indices` to assess their performance. By default pick 256 dimensions for the hidden units. You can use the codes provided in the class. Also, we strongly encourage you to use the Gluon implementation since it's a lot faster than building it from scratch.\n",
    "\n",
    "1. Train a single-layer RNN (with latent variables). \n",
    "1. Train a single-layer GRU.\n",
    "1. Train a single-layer LSTM. \n",
    "1. Train a two-layer LSTM. \n",
    "\n",
    "How low can you drive the perplexity? Can you reproduce some of Shakespeare's finest writing (generate 200 characters). Start the sequence generator with `But Brutus is an honorable man`. Experiment with a number of settings:\n",
    "\n",
    "* Number of hidden units. \n",
    "* Embedding length.\n",
    "* Gradient clipping.\n",
    "* Number of iterations. \n",
    "* Learning rate.\n",
    "\n",
    "**Save** the models (at least in memory since you'll need them in the next exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "1"
    },
    "collapsed": true
   },
   "source": [
    "## 2. Test Error\n",
    "\n",
    "So far we measured perplexity only on the training set. \n",
    "\n",
    "1. Implement a perplexity calculator that does not involve training.\n",
    "1. Compute the perplexity of the best models in each of the 4 categories on the test set. By how much does it differ? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. $N$-Gram Model\n",
    "\n",
    "So far we only considered latent variable models. Let's see what happens if we use a regular $N$-gram model and an autoregressive setting. That is, we aim to predict the next character given the current characters one character at a time. For this implement the following:\n",
    "\n",
    "1. Split data into $(x,y)$ pairs as before, just that we now use very short subsequences, e.g. only $5$ characters. That is, `But Brutus` turns into the tuples (`(But B, r)`, `(ut Br, u)`, `(t Bru, t)`, `( Brut, u)`, `(Brutu, s)`). \n",
    "1. Use one-hot encoding for each character separately and combine them all. \n",
    "    * In one case use a sequential encoding to obtain an embedding proportional to the length of the sequence.\n",
    "    * Use a bag of characters encoding that sums over all occurrences.\n",
    "1. Implement an MLP with one hidden layer and 256 hidden units.\n",
    "1. Train it to output the next character.\n",
    "\n",
    "How accurate is the model? How does the number of operations and weights compare to an RNN, a GRU and an LSTM discussed above?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}