vui_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Artificial Intelligence Nanodegree\n",
    "\n",
    "## Voice User Interfaces\n",
    "\n",
    "## Project: Speech Recognition with Neural Networks\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, some template code has already been provided for you, and you will need to implement additional functionality to successfully complete this project. You will not need to modify the included code beyond what is requested. Sections that begin with **'(IMPLEMENTATION)'** in the header indicate that the following blocks of code will require additional functionality which you must provide. Please be sure to read the instructions carefully! \n",
    "\n",
    "> **Note**: Once you have completed all of the code implementations, you need to finalize your work by exporting the Jupyter Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to  \\n\",\n",
    "    \"**File -> Download as -> HTML (.html)**. Include the finished document along with this notebook as your submission.\n",
    "\n",
    "In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a **'Question X'** header. Carefully read each question and provide thorough answers in the following text boxes that begin with **'Answer:'**. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.\n",
    "\n",
    ">**Note:** Code and Markdown cells can be executed using the **Shift + Enter** keyboard shortcut.  Markdown cells can be edited by double-clicking the cell to enter edit mode.\n",
    "\n",
    "The rubric contains _optional_ \"Stand Out Suggestions\" for enhancing the project beyond the minimum requirements. If you decide to pursue the \"Stand Out Suggestions\", you should include the code in this Jupyter notebook.\n",
    "\n",
    "---\n",
    "\n",
    "## Introduction  \n",
    "\n",
    "In this notebook, you will build a deep neural network that functions as part of an end-to-end automatic speech recognition (ASR) pipeline!  Your completed pipeline will accept raw audio as input and return a predicted transcription of the spoken language.  The full pipeline is summarized in the figure below.\n",
    "\n",
    "<img src=\"images/pipeline.png\">\n",
    "\n",
    "- **STEP 1** is a pre-processing step that converts raw audio to one of two feature representations that are commonly used for ASR. \n",
    "- **STEP 2** is an acoustic model which accepts audio features as input and returns a probability distribution over all potential transcriptions.  After learning about the basic types of neural networks that are often used for acoustic modeling, you will engage in your own investigations, to design your own acoustic model!\n",
    "- **STEP 3** in the pipeline takes the output from the acoustic model and returns a predicted transcription.  \n",
    "\n",
    "Feel free to use the links below to navigate the notebook:\n",
    "- [The Data](#thedata)\n",
    "- [**STEP 1**](#step1): Acoustic Features for Speech Recognition\n",
    "- [**STEP 2**](#step2): Deep Neural Networks for Acoustic Modeling\n",
    "    - [Model 0](#model0): RNN\n",
    "    - [Model 1](#model1): RNN + TimeDistributed Dense\n",
    "    - [Model 2](#model2): CNN + RNN + TimeDistributed Dense\n",
    "    - [Model 3](#model3): Deeper RNN + TimeDistributed Dense\n",
    "    - [Model 4](#model4): Bidirectional RNN + TimeDistributed Dense\n",
    "    - [Models 5+](#model5)\n",
    "    - [Compare the Models](#compare)\n",
    "    - [Final Model](#final)\n",
    "- [**STEP 3**](#step3): Obtain Predictions\n",
    "\n",
    "<a id='thedata'></a>\n",
    "## The Data\n",
    "\n",
    "We begin by investigating the dataset that will be used to train and evaluate your pipeline.  [LibriSpeech](http://www.danielpovey.com/files/2015_icassp_librispeech.pdf) is a large corpus of English-read speech, designed for training and evaluating models for ASR.  The dataset contains 1000 hours of speech derived from audiobooks.  We will work with a small subset in this project, since larger-scale data would take a long while to train.  However, after completing this project, if you are interested in exploring further, you are encouraged to work with more of the data that is provided [online](http://www.openslr.org/12/).\n",
    "\n",
    "In the code cells below, you will use the `vis_train_features` module to visualize a training example.  The supplied argument `index=0` tells the module to extract the first example in the training set.  (You are welcome to change `index=0` to point to a different training example, if you like, but please **DO NOT** amend any other code in the cell.)  The returned variables are:\n",
    "- `vis_text` - transcribed text (label) for the training example.\n",
    "- `vis_raw_audio` - raw audio waveform for the training example.\n",
    "- `vis_mfcc_feature` - mel-frequency cepstral coefficients (MFCCs) for the training example.\n",
    "- `vis_spectrogram_feature` - spectrogram for the training example. \n",
    "- `vis_audio_path` - the file path to the training example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from data_generator import vis_train_features\n",
    "\n",
    "# extract label and audio features for a single training example\n",
    "vis_text, vis_raw_audio, vis_mfcc_feature, vis_spectrogram_feature, vis_audio_path = vis_train_features()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code cell visualizes the audio waveform for your chosen example, along with the corresponding transcript.  You also have the option to play the audio in the notebook!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display\n",
    "from data_generator import vis_train_features, plot_raw_audio\n",
    "from IPython.display import Audio\n",
    "%matplotlib inline\n",
    "\n",
    "# plot audio signal\n",
    "plot_raw_audio(vis_raw_audio)\n",
    "# print length of audio signal\n",
    "display(Markdown('**Shape of Audio Signal** : ' + str(vis_raw_audio.shape)))\n",
    "# print transcript corresponding to audio clip\n",
    "display(Markdown('**Transcript** : ' + str(vis_text)))\n",
    "# play the audio file\n",
    "Audio(vis_audio_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step1'></a>\n",
    "## STEP 1: Acoustic Features for Speech Recognition\n",
    "\n",
    "For this project, you won't use the raw audio waveform as input to your model.  Instead, we provide code that first performs a pre-processing step to convert the raw audio to a feature representation that has historically proven successful for ASR models.  Your acoustic model will accept the feature representation as input.\n",
    "\n",
    "In this project, you will explore two possible feature representations.  _After completing the project_, if you'd like to read more about deep learning architectures that can accept raw audio input, you are encouraged to explore this [research paper](https://pdfs.semanticscholar.org/a566/cd4a8623d661a4931814d9dffc72ecbf63c4.pdf).\n",
    "\n",
    "### Spectrograms\n",
    "\n",
    "The first option for an audio feature representation is the [spectrogram](https://www.youtube.com/watch?v=_FatxGN3vAM).  In order to complete this project, you will **not** need to dig deeply into the details of how a spectrogram is calculated; but, if you are curious, the code for calculating the spectrogram was borrowed from [this repository](https://github.com/baidu-research/ba-dls-deepspeech).  The implementation appears in the `utils.py` file in your repository.\n",
    "\n",
    "The code that we give you returns the spectrogram as a 2D tensor, where the first (_vertical_) dimension indexes time, and the second (_horizontal_) dimension indexes frequency.  To speed the convergence of your algorithm, we have also normalized the spectrogram.  (You can see this quickly in the visualization below by noting that the mean value hovers around zero, and most entries in the tensor assume values close to zero.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from data_generator import plot_spectrogram_feature\n",
    "\n",
    "# plot normalized spectrogram\n",
    "plot_spectrogram_feature(vis_spectrogram_feature)\n",
    "# print shape of spectrogram\n",
    "display(Markdown('**Shape of Spectrogram** : ' + str(vis_spectrogram_feature.shape)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mel-Frequency Cepstral Coefficients (MFCCs)\n",
    "\n",
    "The second option for an audio feature representation is [MFCCs](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum).  You do **not** need to dig deeply into the details of how MFCCs are calculated, but if you would like more information, you are welcome to peruse the [documentation](https://github.com/jameslyons/python_speech_features) of the `python_speech_features` Python package.  Just as with the spectrogram features, the MFCCs are normalized in the supplied code.\n",
    "\n",
    "The main idea behind MFCC features is the same as spectrogram features: at each time window, the MFCC feature yields a feature vector that characterizes the sound within the window.  Note that the MFCC feature is much lower-dimensional than the spectrogram feature, which could help an acoustic model to avoid overfitting to the training dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from data_generator import plot_mfcc_feature\n",
    "\n",
    "# plot normalized MFCC\n",
    "plot_mfcc_feature(vis_mfcc_feature)\n",
    "# print shape of MFCC\n",
    "display(Markdown('**Shape of MFCC** : ' + str(vis_mfcc_feature.shape)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you construct your pipeline, you will be able to choose to use either spectrogram or MFCC features.  If you would like to see different implementations that make use of MFCCs and/or spectrograms, please check out the links below:\n",
    "- This [repository](https://github.com/baidu-research/ba-dls-deepspeech) uses spectrograms.\n",
    "- This [repository](https://github.com/mozilla/DeepSpeech) uses MFCCs.\n",
    "- This [repository](https://github.com/buriburisuri/speech-to-text-wavenet) also uses MFCCs.\n",
    "- This [repository](https://github.com/pannous/tensorflow-speech-recognition/blob/master/speech_data.py) experiments with raw audio, spectrograms, and MFCCs as features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step2'></a>\n",
    "## STEP 2: Deep Neural Networks for Acoustic Modeling\n",
    "\n",
    "In this section, you will experiment with various neural network architectures for acoustic modeling.  \n",
    "\n",
    "You will begin by training five relatively simple architectures.  **Model 0** is provided for you.  You will write code to implement **Models 1**, **2**, **3**, and **4**.  If you would like to experiment further, you are welcome to create and train more models under the **Models 5+** heading.  \n",
    "\n",
    "All models will be specified in the `sample_models.py` file.  After importing the `sample_models` module, you will train your architectures in the notebook.\n",
    "\n",
    "After experimenting with the five simple architectures, you will have the opportunity to compare their performance.  Based on your findings, you will construct a deeper architecture that is designed to outperform all of the shallow models.\n",
    "\n",
    "For your convenience, we have designed the notebook so that each model can be specified and trained on separate occasions.  That is, say you decide to take a break from the notebook after training **Model 1**.  Then, you need not re-execute all prior code cells in the notebook before training **Model 2**.  You need only re-execute the code cell below, that is marked with **`RUN THIS CODE CELL IF YOU ARE RESUMING THE NOTEBOOK AFTER A BREAK`**, before transitioning to the code cells corresponding to **Model 2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#####################################################################\n",
    "# RUN THIS CODE CELL IF YOU ARE RESUMING THE NOTEBOOK AFTER A BREAK #\n",
    "#####################################################################\n",
    "\n",
    "# allocate 50% of GPU memory (if you like, feel free to change this)\n",
    "from keras.backend.tensorflow_backend import set_session\n",
    "import tensorflow as tf \n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.per_process_gpu_memory_fraction = 0.5\n",
    "set_session(tf.Session(config=config))\n",
    "\n",
    "# watch for any changes in the sample_models module, and reload it automatically\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "# import NN architectures for speech recognition\n",
    "from sample_models import *\n",
    "# import function for training acoustic model\n",
    "from train_utils import train_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='model0'></a>\n",
    "### Model 0: RNN\n",
    "\n",
    "Given their effectiveness in modeling sequential data, the first acoustic model you will use is an RNN.  As shown in the figure below, the RNN we supply to you will take the time sequence of audio features as input.\n",
    "\n",
    "<img src=\"images/simple_rnn.png\" width=\"50%\">\n",
    "\n",
    "At each time step, the speaker pronounces one of 28 possible characters, including each of the 26 letters in the English alphabet, along with a space character (\" \"), and an apostrophe (').\n",
    "\n",
    "The output of the RNN at each time step is a vector of probabilities with 29 entries, where the $i$-th entry encodes the probability that the $i$-th character is spoken in the time sequence.  (The extra 29th character is an empty \"character\" used to pad training examples within batches containing uneven lengths.)  If you would like to peek under the hood at how characters are mapped to indices in the probability vector, look at the `char_map.py` file in the repository.  The figure below shows an equivalent, rolled depiction of the RNN that shows the output layer in greater detail. \n",
    "\n",
    "<img src=\"images/simple_rnn_unrolled.png\" width=\"60%\">\n",
    "\n",
    "The model has already been specified for you in Keras.  To import it, you need only run the code cell below.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_0 = simple_rnn_model(input_dim=161) # change to 13 if you would like to use MFCC features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As explored in the lesson, you will train the acoustic model with the [CTC loss](http://www.cs.toronto.edu/~graves/icml_2006.pdf) criterion.  Custom loss functions take a bit of hacking in Keras, and so we have implemented the CTC loss function for you, so that you can focus on trying out as many deep learning architectures as possible :).  If you'd like to peek at the implementation details, look at the `add_ctc_loss` function within the `train_utils.py` file in the repository.\n",
    "\n",
    "To train your architecture, you will use the `train_model` function within the `train_utils` module; it has already been imported in one of the above code cells.  The `train_model` function takes three **required** arguments:\n",
    "- `input_to_softmax` - a Keras model instance.\n",
    "- `pickle_path` - the name of the pickle file where the loss history will be saved.\n",
    "- `save_model_path` - the name of the HDF5 file where the model will be saved.\n",
    "\n",
    "If we have already supplied values for `input_to_softmax`, `pickle_path`, and `save_model_path`, please **DO NOT** modify these values. \n",
    "\n",
    "There are several **optional** arguments that allow you to have more control over the training process.  You are welcome to, but not required to, supply your own values for these arguments.\n",
    "- `minibatch_size` - the size of the minibatches that are generated while training the model (default: `20`).\n",
    "- `spectrogram` - Boolean value dictating whether spectrogram (`True`) or MFCC (`False`) features are used for training (default: `True`).\n",
    "- `mfcc_dim` - the size of the feature dimension to use when generating MFCC features (default: `13`).\n",
    "- `optimizer` - the Keras optimizer used to train the model (default: `SGD(lr=0.02, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5)`).  \n",
    "- `epochs` - the number of epochs to use to train the model (default: `20`).  If you choose to modify this parameter, make sure that it is *at least* 20.\n",
    "- `verbose` - controls the verbosity of the training output in the `model.fit_generator` method (default: `1`).\n",
    "- `sort_by_duration` - Boolean value dictating whether the training and validation sets are sorted by (increasing) duration before the start of the first epoch (default: `False`).\n",
    "\n",
    "The `train_model` function defaults to using spectrogram features; if you choose to use these features, note that the acoustic model in `simple_rnn_model` should have `input_dim=161`.  Otherwise, if you choose to use MFCC features, the acoustic model should have `input_dim=13`.\n",
    "\n",
    "We have chosen to use `GRU` units in the supplied RNN.  If you would like to experiment with `LSTM` or `SimpleRNN` cells, feel free to do so here.  If you change the `GRU` units to `SimpleRNN` cells in `simple_rnn_model`, you may notice that the loss quickly becomes undefined (`nan`) - you are strongly encouraged to check this for yourself!  This is due to the [exploding gradients problem](http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/).  We have already implemented [gradient clipping](https://arxiv.org/pdf/1211.5063.pdf) in your optimizer to help you avoid this issue.\n",
    "\n",
    "__IMPORTANT NOTE:__ If you notice that your gradient has exploded in any of the models below, feel free to explore more with gradient clipping (the `clipnorm` argument in your optimizer) or swap out any `SimpleRNN` cells for `LSTM` or `GRU` cells.  You can also try restarting the kernel to restart the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_model(input_to_softmax=model_0, \n",
    "            pickle_path='model_0.pickle', \n",
    "            save_model_path='model_0.h5',\n",
    "            spectrogram=True) # change to False if you would like to use MFCC features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='model1'></a>\n",
    "### (IMPLEMENTATION) Model 1: RNN + TimeDistributed Dense\n",
    "\n",
    "Read about the [TimeDistributed](https://keras.io/layers/wrappers/) wrapper and the [BatchNormalization](https://keras.io/layers/normalization/) layer in the Keras documentation.  For your next architecture, you will add [batch normalization](https://arxiv.org/pdf/1510.01378.pdf) to the recurrent layer to reduce training times.  The `TimeDistributed` layer will be used to find more complex patterns in the dataset.  The unrolled snapshot of the architecture is depicted below.\n",
    "\n",
    "<img src=\"images/rnn_model.png\" width=\"60%\">\n",
    "\n",
    "The next figure shows an equivalent, rolled depiction of the RNN that shows the (`TimeDistrbuted`) dense and output layers in greater detail.  \n",
    "\n",
    "<img src=\"images/rnn_model_unrolled.png\" width=\"60%\">\n",
    "\n",
    "Use your research to complete the `rnn_model` function within the `sample_models.py` file.  The function should specify an architecture that satisfies the following requirements:\n",
    "- The first layer of the neural network should be an RNN (`SimpleRNN`, `LSTM`, or `GRU`) that takes the time sequence of audio features as input.  We have added `GRU` units for you, but feel free to change `GRU` to `SimpleRNN` or `LSTM`, if you like!\n",
    "- Whereas the architecture in `simple_rnn_model` treated the RNN output as the final layer of the model, you will use the output of your RNN as a hidden layer.  Use `TimeDistributed` to apply a `Dense` layer to each of the time steps in the RNN output.  Ensure that each `Dense` layer has `output_dim` units.\n",
    "\n",
    "Use the code cell below to load your model into the `model_1` variable.  Use a value for `input_dim` that matches your chosen audio features, and feel free to change the values for `units` and `activation` to tweak the behavior of your recurrent layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_1 = rnn_model(input_dim=161, # change to 13 if you would like to use MFCC features\n",
    "                    units=200,\n",
    "                    activation='relu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please execute the code cell below to train the neural network you specified in `input_to_softmax`.  After the model has finished training, the model is [saved](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) in the HDF5 file `model_1.h5`.  The loss history is [saved](https://wiki.python.org/moin/UsingPickle) in `model_1.pickle`.  You are welcome to tweak any of the optional parameters while calling the `train_model` function, but this is not required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_model(input_to_softmax=model_1, \n",
    "            pickle_path='model_1.pickle', \n",
    "            save_model_path='model_1.h5',\n",
    "            spectrogram=True) # change to False if you would like to use MFCC features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='model2'></a>\n",
    "### (IMPLEMENTATION) Model 2: CNN + RNN + TimeDistributed Dense\n",
    "\n",
    "The architecture in `cnn_rnn_model` adds an additional level of complexity, by introducing a [1D convolution layer](https://keras.io/layers/convolutional/#conv1d).  \n",
    "\n",
    "<img src=\"images/cnn_rnn_model.png\" width=\"100%\">\n",
    "\n",
    "This layer incorporates many arguments that can be (optionally) tuned when calling the `cnn_rnn_model` module.  We provide sample starting parameters, which you might find useful if you choose to use spectrogram audio features.  \n",
    "\n",
    "If you instead want to use MFCC features, these arguments will have to be tuned.  Note that the current architecture only supports values of `'same'` or `'valid'` for the `conv_border_mode` argument.\n",
    "\n",
    "When tuning the parameters, be careful not to choose settings that make the convolutional layer overly small.  If the temporal length of the CNN layer is shorter than the length of the transcribed text label, your code will throw an error.\n",
    "\n",
    "Before running the code cell below, you must modify the `cnn_rnn_model` function in `sample_models.py`.  Please add batch normalization to the recurrent layer, and provide the same `TimeDistributed` layer as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_2 = cnn_rnn_model(input_dim=161, # change to 13 if you would like to use MFCC features\n",
    "                        filters=200,\n",
    "                        kernel_size=11, \n",
    "                        conv_stride=2,\n",
    "                        conv_border_mode='valid',\n",
    "                        units=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please execute the code cell below to train the neural network you specified in `input_to_softmax`.  After the model has finished training, the model is [saved](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) in the HDF5 file `model_2.h5`.  The loss history is [saved](https://wiki.python.org/moin/UsingPickle) in `model_2.pickle`.  You are welcome to tweak any of the optional parameters while calling the `train_model` function, but this is not required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_model(input_to_softmax=model_2, \n",
    "            pickle_path='model_2.pickle', \n",
    "            save_model_path='model_2.h5', \n",
    "            spectrogram=True) # change to False if you would like to use MFCC features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='model3'></a>\n",
    "### (IMPLEMENTATION) Model 3: Deeper RNN + TimeDistributed Dense\n",
    "\n",
    "Review the code in `rnn_model`, which makes use of a single recurrent layer.  Now, specify an architecture in `deep_rnn_model` that utilizes a variable number `recur_layers` of recurrent layers.  The figure below shows the architecture that should be returned if `recur_layers=2`.  In the figure, the output sequence of the first recurrent layer is used as input for the next recurrent layer.\n",
    "\n",
    "<img src=\"images/deep_rnn_model.png\" width=\"80%\">\n",
    "\n",
    "Feel free to change the supplied values of `units` to whatever you think performs best.  You can change the value of `recur_layers`, as long as your final value is greater than 1. (As a quick check that you have implemented the additional functionality in `deep_rnn_model` correctly, make sure that the architecture that you specify here is identical to `rnn_model` if `recur_layers=1`.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_3 = deep_rnn_model(input_dim=161, # change to 13 if you would like to use MFCC features\n",
    "                         units=200,\n",
    "                         recur_layers=2) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please execute the code cell below to train the neural network you specified in `input_to_softmax`.  After the model has finished training, the model is [saved](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) in the HDF5 file `model_3.h5`.  The loss history is [saved](https://wiki.python.org/moin/UsingPickle) in `model_3.pickle`.  You are welcome to tweak any of the optional parameters while calling the `train_model` function, but this is not required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_model(input_to_softmax=model_3, \n",
    "            pickle_path='model_3.pickle', \n",
    "            save_model_path='model_3.h5', \n",
    "            spectrogram=True) # change to False if you would like to use MFCC features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='model4'></a>\n",
    "### (IMPLEMENTATION) Model 4: Bidirectional RNN + TimeDistributed Dense\n",
    "\n",
    "Read about the [Bidirectional](https://keras.io/layers/wrappers/) wrapper in the Keras documentation.  For your next architecture, you will specify an architecture that uses a single bidirectional RNN layer, before a (`TimeDistributed`) dense layer.  The added value of a bidirectional RNN is described well in [this paper](http://www.cs.toronto.edu/~hinton/absps/DRNN_speech.pdf).\n",
    "> One shortcoming of conventional RNNs is that they are only able to make use of previous context. In speech recognition, where whole utterances are transcribed at once, there is no reason not to exploit future context as well.  Bidirectional RNNs (BRNNs) do this by processing the data in both directions with two separate hidden layers which are then fed forwards to the same output layer.\n",
    "\n",
    "<img src=\"images/bidirectional_rnn_model.png\" width=\"80%\">\n",
    "\n",
    "Before running the code cell below, you must complete the `bidirectional_rnn_model` function in `sample_models.py`.  Feel free to use `SimpleRNN`, `LSTM`, or `GRU` units.  When specifying the `Bidirectional` wrapper, use `merge_mode='concat'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_4 = bidirectional_rnn_model(input_dim=161, # change to 13 if you would like to use MFCC features\n",
    "                                  units=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please execute the code cell below to train the neural network you specified in `input_to_softmax`.  After the model has finished training, the model is [saved](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) in the HDF5 file `model_4.h5`.  The loss history is [saved](https://wiki.python.org/moin/UsingPickle) in `model_4.pickle`.  You are welcome to tweak any of the optional parameters while calling the `train_model` function, but this is not required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_model(input_to_softmax=model_4, \n",
    "            pickle_path='model_4.pickle', \n",
    "            save_model_path='model_4.h5', \n",
    "            spectrogram=True) # change to False if you would like to use MFCC features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='model5'></a>\n",
    "### (OPTIONAL IMPLEMENTATION) Models 5+\n",
    "\n",
    "If you would like to try out more architectures than the ones above, please use the code cell below.  Please continue to follow the same convention for saving the models; for the $i$-th sample model, please save the loss at **`model_i.pickle`** and saving the trained model at **`model_i.h5`**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## (Optional) TODO: Try out some more models!\n",
    "### Feel free to use as many code cells as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='compare'></a>\n",
    "### Compare the Models\n",
    "\n",
    "Execute the code cell below to evaluate the performance of the drafted deep learning models.  The training and validation loss are plotted for each model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from glob import glob\n",
    "import numpy as np\n",
    "import _pickle as pickle\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "sns.set_style(style='white')\n",
    "\n",
    "# obtain the paths for the saved model history\n",
    "all_pickles = sorted(glob(\"results/*.pickle\"))\n",
    "# extract the name of each model\n",
    "model_names = [item[8:-7] for item in all_pickles]\n",
    "# extract the loss history for each model\n",
    "valid_loss = [pickle.load( open( i, \"rb\" ) )['val_loss'] for i in all_pickles]\n",
    "train_loss = [pickle.load( open( i, \"rb\" ) )['loss'] for i in all_pickles]\n",
    "# save the number of epochs used to train each model\n",
    "num_epochs = [len(valid_loss[i]) for i in range(len(valid_loss))]\n",
    "\n",
    "fig = plt.figure(figsize=(16,5))\n",
    "\n",
    "# plot the training loss vs. epoch for each model\n",
    "ax1 = fig.add_subplot(121)\n",
    "for i in range(len(all_pickles)):\n",
    "    ax1.plot(np.linspace(1, num_epochs[i], num_epochs[i]), \n",
    "            train_loss[i], label=model_names[i])\n",
    "# clean up the plot\n",
    "ax1.legend()  \n",
    "ax1.set_xlim([1, max(num_epochs)])\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Training Loss')\n",
    "\n",
    "# plot the validation loss vs. epoch for each model\n",
    "ax2 = fig.add_subplot(122)\n",
    "for i in range(len(all_pickles)):\n",
    "    ax2.plot(np.linspace(1, num_epochs[i], num_epochs[i]), \n",
    "            valid_loss[i], label=model_names[i])\n",
    "# clean up the plot\n",
    "ax2.legend()  \n",
    "ax2.set_xlim([1, max(num_epochs)])\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Validation Loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Question 1:__ Use the plot above to analyze the performance of each of the attempted architectures.  Which performs best?  Provide an explanation regarding why you think some models perform better than others. \n",
    "\n",
    "__Answer:__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='final'></a>\n",
    "### (IMPLEMENTATION) Final Model\n",
    "\n",
    "Now that you've tried out many sample models, use what you've learned to draft your own architecture!  While your final acoustic model should not be identical to any of the architectures explored above, you are welcome to merely combine the explored layers above into a deeper architecture.  It is **NOT** necessary to include new layer types that were not explored in the notebook.\n",
    "\n",
    "However, if you would like some ideas for even more layer types, check out these ideas for some additional, optional extensions to your model:\n",
    "\n",
    "- If you notice your model is overfitting to the training dataset, consider adding **dropout**!  To add dropout to [recurrent layers](https://faroit.github.io/keras-docs/1.0.2/layers/recurrent/), pay special attention to the `dropout_W` and `dropout_U` arguments.  This [paper](http://arxiv.org/abs/1512.05287) may also provide some interesting theoretical background.\n",
    "- If you choose to include a convolutional layer in your model, you may get better results by working with **dilated convolutions**.  If you choose to use dilated convolutions, make sure that you are able to accurately calculate the length of the acoustic model's output in the `model.output_length` lambda function.  You can read more about dilated convolutions in Google's [WaveNet paper](https://arxiv.org/abs/1609.03499).  For an example of a speech-to-text system that makes use of dilated convolutions, check out this GitHub [repository](https://github.com/buriburisuri/speech-to-text-wavenet).  You can work with dilated convolutions [in Keras](https://keras.io/layers/convolutional/) by paying special attention to the `padding` argument when you specify a convolutional layer.\n",
    "- If your model makes use of convolutional layers, why not also experiment with adding **max pooling**?  Check out [this paper](https://arxiv.org/pdf/1701.02720.pdf) for example architecture that makes use of max pooling in an acoustic model.\n",
    "- So far, you have experimented with a single bidirectional RNN layer.  Consider stacking the bidirectional layers, to produce a [deep bidirectional RNN](https://www.cs.toronto.edu/~graves/asru_2013.pdf)!\n",
    "\n",
    "All models that you specify in this repository should have `output_length` defined as an attribute.  This attribute is a lambda function that maps the (temporal) length of the input acoustic features to the (temporal) length of the output softmax layer.  This function is used in the computation of CTC loss; to see this, look at the `add_ctc_loss` function in `train_utils.py`.  To see where the `output_length` attribute is defined for the models in the code, take a look at the `sample_models.py` file.  You will notice this line of code within most models:\n",
    "```\n",
    "model.output_length = lambda x: x\n",
    "```\n",
    "The acoustic model that incorporates a convolutional layer (`cnn_rnn_model`) has a line that is a bit different:\n",
    "```\n",
    "model.output_length = lambda x: cnn_output_length(\n",
    "        x, kernel_size, conv_border_mode, conv_stride)\n",
    "```\n",
    "\n",
    "In the case of models that use purely recurrent layers, the lambda function is the identity function, as the recurrent layers do not modify the (temporal) length of their input tensors.  However, convolutional layers are more complicated and require a specialized function (`cnn_output_length` in `sample_models.py`) to determine the temporal length of their output.\n",
    "\n",
    "You will have to add the `output_length` attribute to your final model before running the code cell below.  Feel free to use the `cnn_output_length` function, if it suits your model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# specify the model\n",
    "model_end = final_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please execute the code cell below to train the neural network you specified in `input_to_softmax`.  After the model has finished training, the model is [saved](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model) in the HDF5 file `model_end.h5`.  The loss history is [saved](https://wiki.python.org/moin/UsingPickle) in `model_end.pickle`.  You are welcome to tweak any of the optional parameters while calling the `train_model` function, but this is not required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_model(input_to_softmax=model_end, \n",
    "            pickle_path='model_end.pickle', \n",
    "            save_model_path='model_end.h5', \n",
    "            spectrogram=True) # change to False if you would like to use MFCC features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Question 2:__ Describe your final model architecture and your reasoning at each step.  \n",
    "\n",
    "__Answer:__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='step3'></a>\n",
    "## STEP 3: Obtain Predictions\n",
    "\n",
    "We have written a function for you to decode the predictions of your acoustic model.  To use the function, please execute the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from data_generator import AudioGenerator\n",
    "from keras import backend as K\n",
    "from utils import int_sequence_to_text\n",
    "from IPython.display import Audio\n",
    "\n",
    "def get_predictions(index, partition, input_to_softmax, model_path):\n",
    "    \"\"\" Print a model's decoded predictions\n",
    "    Params:\n",
    "        index (int): The example you would like to visualize\n",
    "        partition (str): One of 'train' or 'validation'\n",
    "        input_to_softmax (Model): The acoustic model\n",
    "        model_path (str): Path to saved acoustic model's weights\n",
    "    \"\"\"\n",
    "    # load the train and test data\n",
    "    data_gen = AudioGenerator()\n",
    "    data_gen.load_train_data()\n",
    "    data_gen.load_validation_data()\n",
    "    \n",
    "    # obtain the true transcription and the audio features \n",
    "    if partition == 'validation':\n",
    "        transcr = data_gen.valid_texts[index]\n",
    "        audio_path = data_gen.valid_audio_paths[index]\n",
    "        data_point = data_gen.normalize(data_gen.featurize(audio_path))\n",
    "    elif partition == 'train':\n",
    "        transcr = data_gen.train_texts[index]\n",
    "        audio_path = data_gen.train_audio_paths[index]\n",
    "        data_point = data_gen.normalize(data_gen.featurize(audio_path))\n",
    "    else:\n",
    "        raise Exception('Invalid partition!  Must be \"train\" or \"validation\"')\n",
    "        \n",
    "    # obtain and decode the acoustic model's predictions\n",
    "    input_to_softmax.load_weights(model_path)\n",
    "    prediction = input_to_softmax.predict(np.expand_dims(data_point, axis=0))\n",
    "    output_length = [input_to_softmax.output_length(data_point.shape[0])] \n",
    "    pred_ints = (K.eval(K.ctc_decode(\n",
    "                prediction, output_length)[0][0])+1).flatten().tolist()\n",
    "    \n",
    "    # play the audio file, and display the true and predicted transcriptions\n",
    "    print('-'*80)\n",
    "    Audio(audio_path)\n",
    "    print('True transcription:\\n' + '\\n' + transcr)\n",
    "    print('-'*80)\n",
    "    print('Predicted transcription:\\n' + '\\n' + ''.join(int_sequence_to_text(pred_ints)))\n",
    "    print('-'*80)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the code cell below to obtain the transcription predicted by your final model for the first example in the training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_predictions(index=0, \n",
    "                partition='train',\n",
    "                input_to_softmax=final_model(), \n",
    "                model_path='model_end.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the next code cell to visualize the model's prediction for the first example in the validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_predictions(index=0, \n",
    "                partition='validation',\n",
    "                input_to_softmax=final_model(), \n",
    "                model_path='model_end.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One standard way to improve the results of the decoder is to incorporate a language model.  We won't pursue this in the notebook, but you are welcome to do so as an _optional extension_. \n",
    "\n",
    "If you are interested in creating models that provide improved transcriptions, you are encouraged to download [more data](http://www.openslr.org/12/) and train bigger, deeper models.  But beware - the model will likely take a long while to train.  For instance, training this [state-of-the-art](https://arxiv.org/pdf/1512.02595v1.pdf) model would take 3-6 weeks on a single GPU!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}