NLPCW2_main_task_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.1"
    },
    "colab": {
      "name": "NLPCW2 main task_2.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Bgiq_D7pCIV"
      },
      "source": [
        "### Coursework coding instructions (please also see full coursework spec)\n",
        "\n",
        "Please choose if you want to do either Task 1 or Task 2. You should write your report about one task only.\n",
        "\n",
        "For the task you choose you will need to do two approaches:\n",
        "  - Approach 1, which can use use pre-trained embeddings / models\n",
        "  - Approach 2, which should not use any pre-trained embeddings or models\n",
        "We should be able to run both approaches from the same colab file\n",
        "\n",
        "#### Running your code:\n",
        "  - Your models should run automatically when running your colab file without further intervention\n",
        "  - For each task you should automatically output the performance of both models\n",
        "  - Your code should automatically download any libraries required\n",
        "\n",
        "#### Structure of your code:\n",
        "  - You are expected to use the 'train', 'eval' and 'model_performance' functions, although you may edit these as required\n",
        "  - Otherwise there are no restrictions on what you can do in your code\n",
        "\n",
        "#### Documentation:\n",
        "  - You are expected to produce a .README file summarising how you have approached both tasks\n",
        "\n",
        "#### Reproducibility:\n",
        "  - Your .README file should explain how to replicate the different experiments mentioned in your report\n",
        "\n",
        "Good luck! We are really looking forward to seeing your reports and your model code!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ctZiGPdWpCJI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f15b1091-f235-4161-d5c8-1d6d8d24ba5d"
      },
      "source": [
        "# You will need to download any word embeddings required for your code, e.g.:\n",
        "\n",
        "!wget http://nlp.stanford.edu/data/glove.6B.zip\n",
        "!unzip glove.6B.zip\n",
        "\n",
        "# For any packages that Colab does not provide auotmatically you will also need to install these below, e.g.:\n",
        "\n",
        "#! pip install torch"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-02-18 12:05:48--  http://nlp.stanford.edu/data/glove.6B.zip\n",
            "Resolving nlp.stanford.edu (nlp.stanford.edu)... 171.64.67.140\n",
            "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:80... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://nlp.stanford.edu/data/glove.6B.zip [following]\n",
            "--2021-02-18 12:05:48--  https://nlp.stanford.edu/data/glove.6B.zip\n",
            "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:443... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip [following]\n",
            "--2021-02-18 12:05:49--  http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip\n",
            "Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 171.64.64.22\n",
            "Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|171.64.64.22|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 862182613 (822M) [application/zip]\n",
            "Saving to: ‘glove.6B.zip’\n",
            "\n",
            "glove.6B.zip        100%[===================>] 822.24M  1.92MB/s    in 6m 52s  \n",
            "\n",
            "2021-02-18 12:12:41 (2.00 MB/s) - ‘glove.6B.zip’ saved [862182613/862182613]\n",
            "\n",
            "Archive:  glove.6B.zip\n",
            "  inflating: glove.6B.50d.txt        \n",
            "  inflating: glove.6B.100d.txt       \n",
            "  inflating: glove.6B.200d.txt       \n",
            "  inflating: glove.6B.300d.txt       \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0iA-C2dcpCJN"
      },
      "source": [
        "# Imports\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from torch.utils.data import Dataset, random_split\n",
        "from sklearn.feature_extraction.text import TfidfTransformer\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "import torch.optim as optim\n",
        "import codecs\n",
        "import tqdm"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uWs9IykGpCJQ"
      },
      "source": [
        "# Setting random seed and device\n",
        "SEED = 1\n",
        "\n",
        "torch.manual_seed(SEED)\n",
        "torch.cuda.manual_seed(SEED)\n",
        "torch.backends.cudnn.deterministic = True\n",
        "\n",
        "use_cuda = torch.cuda.is_available()\n",
        "device = torch.device(\"cuda:0\" if use_cuda else \"cpu\")"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ITI-7C13pCJS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 402
        },
        "outputId": "7c1e5d1b-1830-45be-ec5a-44752db6e0ad"
      },
      "source": [
        "# Load data\n",
        "train_df = pd.read_csv('task-2/train.csv')\n",
        "test_df = pd.read_csv('data/task-2/dev.csv')"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "error",
          "ename": "FileNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-7-95c93d73b0fd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Load data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtrain_df\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'task-2/train.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mtest_df\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'data/task-2/dev.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36mread_csv\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)\u001b[0m\n\u001b[1;32m    686\u001b[0m     )\n\u001b[1;32m    687\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 688\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    689\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    690\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    452\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    453\u001b[0m     \u001b[0;31m# Create the parser.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 454\u001b[0;31m     \u001b[0mparser\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTextFileReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfp_or_buf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    455\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    456\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mchunksize\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0miterator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, f, engine, **kwds)\u001b[0m\n\u001b[1;32m    946\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"has_index_names\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    947\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 948\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    949\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    950\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m_make_engine\u001b[0;34m(self, engine)\u001b[0m\n\u001b[1;32m   1178\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_make_engine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"c\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1179\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"c\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1180\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCParserWrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1181\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1182\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"python\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, src, **kwds)\u001b[0m\n\u001b[1;32m   2008\u001b[0m         \u001b[0mkwds\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"usecols\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0musecols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2009\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2010\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTextReader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2011\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munnamed_cols\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munnamed_cols\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2012\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.__cinit__\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32mpandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._setup_parser_source\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'task-2/train.csv'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dC4ZVEmBpCJT"
      },
      "source": [
        "# Number of epochs\n",
        "epochs = 10\n",
        "\n",
        "# Proportion of training data for train compared to dev\n",
        "train_proportion = 0.8\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q3FGvU_UpCJU"
      },
      "source": [
        "#### Approach 1: Using pre-trained representations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u9mZ3V13pCJW"
      },
      "source": [
        "# We define our training loop\n",
        "def train(train_iter, dev_iter, model, number_epoch):\n",
        "    \"\"\"\n",
        "    Training loop for the model, which calls on eval to evaluate after each epoch\n",
        "    \"\"\"\n",
        "\n",
        "    print(\"Training model.\")\n",
        "\n",
        "    for epoch in range(1, number_epoch+1):\n",
        "        \n",
        "        model.train()\n",
        "        \n",
        "        epoch_loss = 0\n",
        "        epoch_correct = 0\n",
        "        no_observations = 0  # Observations used for training so far\n",
        "\n",
        "        for batch in train_iter:\n",
        "            feature, target = batch\n",
        "            feature, target = feature.to(device), target.to(device)\n",
        "\n",
        "            # for RNN:\n",
        "            model.batch_size = target.shape[0]\n",
        "            no_observations = no_observations + target.shape[0]\n",
        "            model.hidden = model.init_hidden()\n",
        "\n",
        "            predictions = model(feature).squeeze(1)\n",
        "            optimizer.zero_grad()\n",
        "            loss = loss_fn(predictions, target)\n",
        "\n",
        "            correct, __ = model_performance(np.argmax(predictions.detach().cpu().numpy(), axis=1), target.detach().cpu().numpy())\n",
        "\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "\n",
        "            epoch_loss += loss.item()*target.shape[0]\n",
        "            epoch_correct += correct\n",
        "\n",
        "        valid_loss, valid_acc, __, __ = eval(dev_iter, model)\n",
        "\n",
        "        epoch_loss, epoch_acc = epoch_loss / no_observations, epoch_correct / no_observations\n",
        "        print(f'| Epoch: {epoch:02} | Train Loss: {epoch_loss:.2f} | Train Accuracy: {epoch_acc:.2f} | \\\n",
        "        Val. Loss: {valid_loss:.2f} | Val. Accuracy: {valid_acc:.2f} |')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h4TIRKC0pCJZ"
      },
      "source": [
        "# We evaluate performance on our dev set\n",
        "def eval(data_iter, model):\n",
        "    \"\"\"\n",
        "    Evaluating model performance on the dev set\n",
        "    \"\"\"\n",
        "    model.eval()\n",
        "    epoch_loss = 0\n",
        "    epoch_correct = 0\n",
        "    pred_all = []\n",
        "    trg_all = []\n",
        "    no_observations = 0\n",
        "\n",
        "    with torch.no_grad():\n",
        "        for batch in data_iter:\n",
        "            feature, target = batch\n",
        "\n",
        "            feature, target = feature.to(device), target.to(device)\n",
        "\n",
        "            # for RNN:\n",
        "            model.batch_size = target.shape[0]\n",
        "            no_observations = no_observations + target.shape[0]\n",
        "            model.hidden = model.init_hidden()\n",
        "\n",
        "            predictions = model(feature).squeeze(1)\n",
        "            loss = loss_fn(predictions, target)\n",
        "\n",
        "            # We get the mse\n",
        "            pred, trg = predictions.detach().cpu().numpy(), target.detach().cpu().numpy()\n",
        "            correct, __ = model_performance(np.argmax(pred, axis=1), trg)\n",
        "\n",
        "            epoch_loss += loss.item()*target.shape[0]\n",
        "            epoch_correct += correct\n",
        "            pred_all.extend(pred)\n",
        "            trg_all.extend(trg)\n",
        "\n",
        "    return epoch_loss/no_observations, epoch_correct/no_observations, np.array(pred_all), np.array(trg_all)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iI3iXWnMpCJb"
      },
      "source": [
        "# How we print the model performance\n",
        "def model_performance(output, target, print_output=False):\n",
        "    \"\"\"\n",
        "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
        "    \"\"\"\n",
        "\n",
        "    correct_answers = (output == target)\n",
        "    correct = sum(correct_answers)\n",
        "    acc = np.true_divide(correct,len(output))\n",
        "\n",
        "    if print_output:\n",
        "        print(f'| Acc: {acc:.2f} ')\n",
        "\n",
        "    return correct, acc"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-opM9TDlpCJd"
      },
      "source": [
        "# To create our vocab\n",
        "def create_vocab(data):\n",
        "    \"\"\"\n",
        "    Creating a corpus of all the tokens used\n",
        "    \"\"\"\n",
        "    tokenized_corpus = [] # Let us put the tokenized corpus in a list\n",
        "\n",
        "    for sentence in data:\n",
        "\n",
        "        tokenized_sentence = []\n",
        "\n",
        "        for token in sentence.split(' '): # simplest split is\n",
        "\n",
        "            tokenized_sentence.append(token)\n",
        "\n",
        "        tokenized_corpus.append(tokenized_sentence)\n",
        "\n",
        "    # Create single list of all vocabulary\n",
        "    vocabulary = []  # Let us put all the tokens (mostly words) appearing in the vocabulary in a list\n",
        "\n",
        "    for sentence in tokenized_corpus:\n",
        "\n",
        "        for token in sentence:\n",
        "\n",
        "            if token not in vocabulary:\n",
        "\n",
        "                if True:\n",
        "                    vocabulary.append(token)\n",
        "\n",
        "    return vocabulary, tokenized_corpus"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1oHN7atmpCJi"
      },
      "source": [
        "# Used for collating our observations into minibatches:\n",
        "def collate_fn_padd(batch):\n",
        "    '''\n",
        "    We add padding to our minibatches and create tensors for our model\n",
        "    '''\n",
        "\n",
        "    batch_labels = [l for f, l in batch]\n",
        "    batch_features = [f for f, l in batch]\n",
        "\n",
        "    batch_features_len = [len(f) for f, l in batch]\n",
        "\n",
        "    seq_tensor = torch.zeros((len(batch), max(batch_features_len))).long()\n",
        "\n",
        "    for idx, (seq, seqlen) in enumerate(zip(batch_features, batch_features_len)):\n",
        "        seq_tensor[idx, :seqlen] = torch.LongTensor(seq)\n",
        "\n",
        "    batch_labels = torch.LongTensor(batch_labels)\n",
        "\n",
        "    return seq_tensor, batch_labels\n",
        "\n",
        "# We create a Dataset so we can create minibatches\n",
        "class Task2Dataset(Dataset):\n",
        "\n",
        "    def __init__(self, train_data, labels):\n",
        "        self.x_train = train_data\n",
        "        self.y_train = labels\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.y_train)\n",
        "\n",
        "    def __getitem__(self, item):\n",
        "        return self.x_train[item], self.y_train[item]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8C1uu91ZpCJm"
      },
      "source": [
        "\n",
        "class BiLSTM_classification(nn.Module):\n",
        "\n",
        "    def __init__(self, embedding_dim, hidden_dim, vocab_size, batch_size, device):\n",
        "        super(BiLSTM_classification, self).__init__()\n",
        "        self.hidden_dim = hidden_dim\n",
        "        self.embedding_dim = embedding_dim\n",
        "        self.device = device\n",
        "        self.batch_size = batch_size\n",
        "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)\n",
        "\n",
        "        # The LSTM takes word embeddings as inputs, and outputs hidden states\n",
        "        # with dimensionality hidden_dim.\n",
        "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, bidirectional=True)\n",
        "\n",
        "        # The linear layer that maps from hidden state space to tag space\n",
        "        self.hidden2label = nn.Linear(hidden_dim * 2, 3)\n",
        "        self.hidden = self.init_hidden()\n",
        "\n",
        "    def init_hidden(self):\n",
        "        # Before we've done anything, we dont have any hidden state.\n",
        "        # Refer to the Pytorch documentation to see exactly why they have this dimensionality.\n",
        "        # The axes semantics are (num_layers * num_directions, minibatch_size, hidden_dim)\n",
        "        return torch.zeros(2, self.batch_size, self.hidden_dim).to(self.device), \\\n",
        "               torch.zeros(2, self.batch_size, self.hidden_dim).to(self.device)\n",
        "\n",
        "    def forward(self, sentence):\n",
        "        embedded = self.embedding(sentence)\n",
        "        embedded = embedded.permute(1, 0, 2)\n",
        "\n",
        "        lstm_out, self.hidden = self.lstm(\n",
        "            embedded.view(len(embedded), self.batch_size, self.embedding_dim), self.hidden)\n",
        "\n",
        "        out = self.hidden2label(lstm_out[-1])\n",
        "        return out"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3tvo0qlzpCJq",
        "outputId": "3f169f1e-b516-4639-ddb8-6d3442065d52"
      },
      "source": [
        "## Approach 1 code, using functions defined above:\n",
        "\n",
        "# We set our training data and test data\n",
        "training_data = train_df['original1']\n",
        "test_data = test_df['original1']\n",
        "\n",
        "# Creating word vectors\n",
        "training_vocab, training_tokenized_corpus = create_vocab(training_data)\n",
        "test_vocab, test_tokenized_corpus = create_vocab(test_data)\n",
        "\n",
        "# Creating joint vocab from test and train:\n",
        "joint_vocab, joint_tokenized_corpus = create_vocab(pd.concat([training_data, test_data]))\n",
        "\n",
        "print(\"Vocab created.\")\n",
        "\n",
        "# We create representations for our tokens\n",
        "wvecs = [] # word vectors\n",
        "word2idx = [] # word2index\n",
        "idx2word = []\n",
        "\n",
        "# This is a large file, it will take a while to load in the memory!\n",
        "with codecs.open('glove.6B.100d.txt', 'r','utf-8') as f:\n",
        "  index = 1\n",
        "  for line in f.readlines():\n",
        "    # Ignore the first line - first line typically contains vocab, dimensionality\n",
        "    if len(line.strip().split()) > 3:\n",
        "      word = line.strip().split()[0]\n",
        "      if word in joint_vocab:\n",
        "          (word, vec) = (word,\n",
        "                     list(map(float,line.strip().split()[1:])))\n",
        "          wvecs.append(vec)\n",
        "          word2idx.append((word, index))\n",
        "          idx2word.append((index, word))\n",
        "          index += 1\n",
        "\n",
        "\n",
        "wvecs = np.array(wvecs)\n",
        "word2idx = dict(word2idx)\n",
        "idx2word = dict(idx2word)\n",
        "\n",
        "vectorized_seqs = [[word2idx[tok] for tok in seq if tok in word2idx] for seq in training_tokenized_corpus]\n",
        "\n",
        "INPUT_DIM = len(word2idx)\n",
        "EMBEDDING_DIM = 100\n",
        "BATCH_SIZE = 32\n",
        "\n",
        "model = BiLSTM_classification(EMBEDDING_DIM, 50, INPUT_DIM, BATCH_SIZE, device)\n",
        "print(\"Model initialised.\")\n",
        "\n",
        "model.to(device)\n",
        "# We provide the model with our embeddings\n",
        "model.embedding.weight.data.copy_(torch.from_numpy(wvecs))\n",
        "\n",
        "feature = vectorized_seqs\n",
        "\n",
        "# 'feature' is a list of lists, each containing embedding IDs for word tokens\n",
        "train_and_dev = Task2Dataset(feature, train_df['label'])\n",
        "\n",
        "train_examples = round(len(train_and_dev)*train_proportion)\n",
        "dev_examples = len(train_and_dev) - train_examples\n",
        "\n",
        "train_dataset, dev_dataset = random_split(train_and_dev,\n",
        "                                           (train_examples,\n",
        "                                            dev_examples))\n",
        "\n",
        "\n",
        "train_loader = torch.utils.data.DataLoader(train_dataset, shuffle=True, batch_size=BATCH_SIZE, collate_fn=collate_fn_padd)\n",
        "dev_loader = torch.utils.data.DataLoader(dev_dataset, batch_size=BATCH_SIZE, collate_fn=collate_fn_padd)\n",
        "\n",
        "print(\"Dataloaders created.\")\n",
        "\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "loss_fn = loss_fn.to(device)\n",
        "\n",
        "optimizer = torch.optim.Adam(model.parameters())\n",
        "\n",
        "train(train_loader, dev_loader, model, epochs)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Vocab created.\n",
            "Model initialised.\n",
            "Dataloaders created.\n",
            "Training model.\n",
            "| Epoch: 01 | Train Loss: 0.96 | Train Accuracy: 0.44 |         Val. Loss: 0.98 | Val. Accuracy: 0.43 |\n",
            "| Epoch: 02 | Train Loss: 0.96 | Train Accuracy: 0.45 |         Val. Loss: 0.97 | Val. Accuracy: 0.45 |\n",
            "| Epoch: 03 | Train Loss: 0.96 | Train Accuracy: 0.45 |         Val. Loss: 0.97 | Val. Accuracy: 0.43 |\n",
            "| Epoch: 04 | Train Loss: 0.95 | Train Accuracy: 0.46 |         Val. Loss: 0.98 | Val. Accuracy: 0.43 |\n",
            "| Epoch: 05 | Train Loss: 0.94 | Train Accuracy: 0.52 |         Val. Loss: 0.99 | Val. Accuracy: 0.46 |\n",
            "| Epoch: 06 | Train Loss: 0.87 | Train Accuracy: 0.60 |         Val. Loss: 1.02 | Val. Accuracy: 0.48 |\n",
            "| Epoch: 07 | Train Loss: 0.80 | Train Accuracy: 0.62 |         Val. Loss: 1.05 | Val. Accuracy: 0.49 |\n",
            "| Epoch: 08 | Train Loss: 0.75 | Train Accuracy: 0.65 |         Val. Loss: 1.16 | Val. Accuracy: 0.47 |\n",
            "| Epoch: 09 | Train Loss: 0.70 | Train Accuracy: 0.66 |         Val. Loss: 1.18 | Val. Accuracy: 0.48 |\n",
            "| Epoch: 10 | Train Loss: 0.66 | Train Accuracy: 0.67 |         Val. Loss: 1.27 | Val. Accuracy: 0.48 |\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeHtP2bipCJ0"
      },
      "source": [
        "#### Approach 2: No pre-trained representations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "79iNJHTvpCJ2",
        "outputId": "b42ea40a-377e-42e9-8de4-195fcff9939e"
      },
      "source": [
        "train_and_dev = train_df['edit1']\n",
        "\n",
        "training_data, dev_data, training_y, dev_y = train_test_split(train_df['edit1'], train_df['label'],\n",
        "                                                                        test_size=(1-train_proportion),\n",
        "                                                                        random_state=42)\n",
        "\n",
        "# We train a Tf-idf model\n",
        "count_vect = CountVectorizer(stop_words='english')\n",
        "train_counts = count_vect.fit_transform(training_data)\n",
        "transformer = TfidfTransformer().fit(train_counts)\n",
        "train_counts = transformer.transform(train_counts)\n",
        "naive_model = MultinomialNB().fit(train_counts, training_y)\n",
        "\n",
        "# Train predictions\n",
        "predicted_train = naive_model.predict(train_counts)\n",
        "\n",
        "# Calculate Tf-idf using train and dev, and validate on dev:\n",
        "test_and_test_counts = count_vect.transform(train_and_dev)\n",
        "transformer = TfidfTransformer().fit(test_and_test_counts)\n",
        "\n",
        "test_counts = count_vect.transform(dev_data)\n",
        "\n",
        "test_counts = transformer.transform(test_counts)\n",
        "\n",
        "# Dev predictions\n",
        "predicted = naive_model.predict(test_counts)\n",
        "\n",
        "# We run the evaluation:\n",
        "print(\"\\nTrain performance:\")\n",
        "\n",
        "sse, mse = model_performance(predicted_train, training_y, True)\n",
        "\n",
        "print(\"\\nDev performance:\")\n",
        "sse, mse = model_performance(predicted, dev_y, True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Train performance:\n",
            "| Acc: 0.73 \n",
            "\n",
            "Dev performance:\n",
            "| Acc: 0.52 \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1BAoHfeOpCJ3"
      },
      "source": [
        "#### Baseline for task 2"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p0MwFSbSpCJ4",
        "outputId": "1de243d0-2f12-4dc6-ac64-333d4510637e"
      },
      "source": [
        "# Baseline for the task\n",
        "pred_baseline = torch.zeros(len(dev_y)) + 1  # 1 is most common class\n",
        "print(\"\\nBaseline performance:\")\n",
        "sse, mse = model_performance(pred_baseline, torch.tensor(dev_y.values), True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Baseline performance:\n",
            "| Acc: 0.45 \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eVK4P6UtpCJ6"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-i5xBiQWpCJ7"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}