ch16/ch16-part3-bert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Machine Learning with PyTorch and Scikit-Learn  \n",
    "# -- Code Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Package version checks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add folder to path in order to load from the check_packages.py script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check recommended package versions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] Your Python version is 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) \n",
      "[GCC 9.3.0]\n",
      "[OK] pandas 1.3.5\n",
      "[OK] torch 1.10.0\n",
      "[OK] torchtext 0.11.0\n",
      "[OK] datasets 1.11.0\n",
      "[OK] transformers 4.9.1\n"
     ]
    }
   ],
   "source": [
    "from python_environment_check import check_packages\n",
    "\n",
    "\n",
    "d = {\n",
    "    'pandas': '1.3.2',\n",
    "    'torch': '1.9.0',\n",
    "    'torchtext': '0.11.0',\n",
    "    'datasets': '1.11.0',\n",
    "    'transformers': '4.9.1',\n",
    "}\n",
    "check_packages(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 16: Transformers – Improving Natural Language Processing with Attention Mechanisms (Part 3/3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Outline**\n",
    "\n",
    "- [Fine-tuning a BERT model in PyTorch](#Fine-tuning-a-BERT-model-in-PyTorch)\n",
    "  - [Loading the IMDb movie review dataset](#Loading-the-IMDb-movie-review-dataset)\n",
    "  - [Tokenizing the dataset](#Tokenizing-the-dataset)\n",
    "  - [Loading and fine-tuning a pre-trained BERT model](#[Loading-and-fine-tuning-a-pre-trained-BERT-model)\n",
    "  - [Fine-tuning a transformer more conveniently using the Trainer API](#Fine-tuning-a-transformer-more-conveniently-using-the-Trainer-API)\n",
    "- [Summary](#Summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Quote from https://huggingface.co/transformers/custom_datasets.html:\n",
    "\n",
    "> DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than bert-base-uncased , runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning a BERT model in PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vY4SK0xKAJgm"
   },
   "source": [
    "### Loading the IMDb movie review dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gzip\n",
    "import shutil\n",
    "import time\n",
    "\n",
    "import pandas as pd\n",
    "import requests\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import torchtext\n",
    "\n",
    "import transformers\n",
    "from transformers import DistilBertTokenizerFast\n",
    "from transformers import DistilBertForSequenceClassification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GSRL42Qgy8I8"
   },
   "source": [
    "**General Settings**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OvW1RgfepCBq"
   },
   "outputs": [],
   "source": [
    "torch.backends.cudnn.deterministic = True\n",
    "RANDOM_SEED = 123\n",
    "torch.manual_seed(RANDOM_SEED)\n",
    "DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "NUM_EPOCHS = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mQMmKUEisW4W"
   },
   "source": [
    "**Download Dataset**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells will download the IMDB movie review dataset (http://ai.stanford.edu/~amaas/data/sentiment/) for positive-negative sentiment classification in as CSV-formatted file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://github.com/rasbt/machine-learning-book/raw/main/ch08/movie_data.csv.gz\"\n",
    "filename = url.split(\"/\")[-1]\n",
    "\n",
    "with open(filename, \"wb\") as f:\n",
    "    r = requests.get(url)\n",
    "    f.write(r.content)\n",
    "\n",
    "with gzip.open('movie_data.csv.gz', 'rb') as f_in:\n",
    "    with open('movie_data.csv', 'wb') as f_out:\n",
    "        shutil.copyfileobj(f_in, f_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that the dataset looks okay:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In 1974, the teenager Martha Moxley (Maggie Gr...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OK... so... I really like Kris Kristofferson a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***SPOILER*** Do not read this, if you think a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>hi for all the people who have seen this wonde...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I recently bought the DVD, forgetting just how...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  sentiment\n",
       "0  In 1974, the teenager Martha Moxley (Maggie Gr...          1\n",
       "1  OK... so... I really like Kris Kristofferson a...          0\n",
       "2  ***SPOILER*** Do not read this, if you think a...          0\n",
       "3  hi for all the people who have seen this wonde...          1\n",
       "4  I recently bought the DVD, forgetting just how...          0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('movie_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000, 2)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Split Dataset into Train/Validation/Test**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_texts = df.iloc[:35000]['review'].values\n",
    "train_labels = df.iloc[:35000]['sentiment'].values\n",
    "\n",
    "valid_texts = df.iloc[35000:40000]['review'].values\n",
    "valid_labels = df.iloc[35000:40000]['sentiment'].values\n",
    "\n",
    "test_texts = df.iloc[40000:]['review'].values\n",
    "test_labels = df.iloc[40000:]['sentiment'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizing the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_encodings = tokenizer(list(train_texts), truncation=True, padding=True)\n",
    "valid_encodings = tokenizer(list(valid_texts), truncation=True, padding=True)\n",
    "test_encodings = tokenizer(list(test_texts), truncation=True, padding=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Encoding(num_tokens=512, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_encodings[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dataset Class and Loaders**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class IMDbDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self, encodings, labels):\n",
    "        self.encodings = encodings\n",
    "        self.labels = labels\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}\n",
    "        item['labels'] = torch.tensor(self.labels[idx])\n",
    "        return item\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.labels)\n",
    "\n",
    "\n",
    "train_dataset = IMDbDataset(train_encodings, train_labels)\n",
    "valid_dataset = IMDbDataset(valid_encodings, valid_labels)\n",
    "test_dataset = IMDbDataset(test_encodings, test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, shuffle=True)\n",
    "valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=16, shuffle=False)\n",
    "test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=16, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading and fine-tuning a pre-trained BERT model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_projector.bias', 'vocab_transform.weight', 'vocab_projector.weight', 'vocab_transform.bias', 'vocab_layer_norm.bias', 'vocab_layer_norm.weight']\n",
      "- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'classifier.weight', 'classifier.bias', 'pre_classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')\n",
    "model.to(DEVICE)\n",
    "model.train()\n",
    "\n",
    "optim = torch.optim.Adam(model.parameters(), lr=5e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Train Model -- Manual Training Loop**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_accuracy(model, data_loader, device):\n",
    "    with torch.no_grad():\n",
    "        correct_pred, num_examples = 0, 0\n",
    "        \n",
    "        for batch_idx, batch in enumerate(data_loader):\n",
    "        \n",
    "        ### Prepare data\n",
    "            input_ids = batch['input_ids'].to(device)\n",
    "            attention_mask = batch['attention_mask'].to(device)\n",
    "            labels = batch['labels'].to(device)\n",
    "            outputs = model(input_ids, attention_mask=attention_mask)\n",
    "            logits = outputs['logits']\n",
    "            predicted_labels = torch.argmax(logits, 1)\n",
    "            num_examples += labels.size(0)\n",
    "            correct_pred += (predicted_labels == labels).sum()\n",
    "        \n",
    "        return correct_pred.float()/num_examples * 100\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001/0003 | Batch 0000/2188 | Loss: 0.6771\n",
      "Epoch: 0001/0003 | Batch 0250/2188 | Loss: 0.3006\n",
      "Epoch: 0001/0003 | Batch 0500/2188 | Loss: 0.3678\n",
      "Epoch: 0001/0003 | Batch 0750/2188 | Loss: 0.1487\n",
      "Epoch: 0001/0003 | Batch 1000/2188 | Loss: 0.6674\n",
      "Epoch: 0001/0003 | Batch 1250/2188 | Loss: 0.3264\n",
      "Epoch: 0001/0003 | Batch 1500/2188 | Loss: 0.4358\n",
      "Epoch: 0001/0003 | Batch 1750/2188 | Loss: 0.2579\n",
      "Epoch: 0001/0003 | Batch 2000/2188 | Loss: 0.2474\n",
      "Training accuracy: 96.32%\n",
      "Valid accuracy: 92.34%\n",
      "Time elapsed: 20.67 min\n",
      "Epoch: 0002/0003 | Batch 0000/2188 | Loss: 0.0850\n",
      "Epoch: 0002/0003 | Batch 0250/2188 | Loss: 0.3433\n",
      "Epoch: 0002/0003 | Batch 0500/2188 | Loss: 0.0793\n",
      "Epoch: 0002/0003 | Batch 0750/2188 | Loss: 0.0061\n",
      "Epoch: 0002/0003 | Batch 1000/2188 | Loss: 0.1536\n",
      "Epoch: 0002/0003 | Batch 1250/2188 | Loss: 0.0816\n",
      "Epoch: 0002/0003 | Batch 1500/2188 | Loss: 0.0786\n",
      "Epoch: 0002/0003 | Batch 1750/2188 | Loss: 0.1395\n",
      "Epoch: 0002/0003 | Batch 2000/2188 | Loss: 0.0344\n",
      "Training accuracy: 98.35%\n",
      "Valid accuracy: 92.46%\n",
      "Time elapsed: 41.41 min\n",
      "Epoch: 0003/0003 | Batch 0000/2188 | Loss: 0.0403\n",
      "Epoch: 0003/0003 | Batch 0250/2188 | Loss: 0.0036\n",
      "Epoch: 0003/0003 | Batch 0500/2188 | Loss: 0.0156\n",
      "Epoch: 0003/0003 | Batch 0750/2188 | Loss: 0.0114\n",
      "Epoch: 0003/0003 | Batch 1000/2188 | Loss: 0.1227\n",
      "Epoch: 0003/0003 | Batch 1250/2188 | Loss: 0.0125\n",
      "Epoch: 0003/0003 | Batch 1500/2188 | Loss: 0.0074\n",
      "Epoch: 0003/0003 | Batch 1750/2188 | Loss: 0.0202\n",
      "Epoch: 0003/0003 | Batch 2000/2188 | Loss: 0.0746\n",
      "Training accuracy: 99.08%\n",
      "Valid accuracy: 91.84%\n",
      "Time elapsed: 62.15 min\n",
      "Total Training Time: 62.15 min\n",
      "Test accuracy: 92.50%\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    \n",
    "    model.train()\n",
    "    \n",
    "    for batch_idx, batch in enumerate(train_loader):\n",
    "        \n",
    "        ### Prepare data\n",
    "        input_ids = batch['input_ids'].to(DEVICE)\n",
    "        attention_mask = batch['attention_mask'].to(DEVICE)\n",
    "        labels = batch['labels'].to(DEVICE)\n",
    "\n",
    "        ### Forward\n",
    "        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)\n",
    "        loss, logits = outputs['loss'], outputs['logits']\n",
    "        \n",
    "        ### Backward\n",
    "        optim.zero_grad()\n",
    "        loss.backward()\n",
    "        optim.step()\n",
    "        \n",
    "        ### Logging\n",
    "        if not batch_idx % 250:\n",
    "            print (f'Epoch: {epoch+1:04d}/{NUM_EPOCHS:04d} | '\n",
    "                   f'Batch {batch_idx:04d}/{len(train_loader):04d} | '\n",
    "                   f'Loss: {loss:.4f}')\n",
    "            \n",
    "    model.eval()\n",
    "\n",
    "    with torch.set_grad_enabled(False):\n",
    "        print(f'Training accuracy: '\n",
    "              f'{compute_accuracy(model, train_loader, DEVICE):.2f}%'\n",
    "              f'\\nValid accuracy: '\n",
    "              f'{compute_accuracy(model, valid_loader, DEVICE):.2f}%')\n",
    "        \n",
    "    print(f'Time elapsed: {(time.time() - start_time)/60:.2f} min')\n",
    "    \n",
    "print(f'Total Training Time: {(time.time() - start_time)/60:.2f} min')\n",
    "print(f'Test accuracy: {compute_accuracy(model, test_loader, DEVICE):.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "del model # free memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fine-tuning a transformer more conveniently using the Trainer API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reload pretrained model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_projector.bias', 'vocab_transform.weight', 'vocab_projector.weight', 'vocab_transform.bias', 'vocab_layer_norm.bias', 'vocab_layer_norm.weight']\n",
      "- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'classifier.weight', 'classifier.bias', 'pre_classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')\n",
    "model.to(DEVICE)\n",
    "model.train();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import Trainer, TrainingArguments\n",
    "\n",
    "\n",
    "optim = torch.optim.Adam(model.parameters(), lr=5e-5)\n",
    "training_args = TrainingArguments(\n",
    "    output_dir='./results', \n",
    "    num_train_epochs=3,     \n",
    "    per_device_train_batch_size=16, \n",
    "    per_device_eval_batch_size=16,   \n",
    "    logging_dir='./logs',\n",
    "    logging_steps=10,\n",
    ")\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=train_dataset,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install dataset via pip install datasets\n",
    "from datasets import load_metric\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "metric = load_metric(\"accuracy\")\n",
    "\n",
    "def compute_metrics(eval_pred):\n",
    "    logits, labels = eval_pred # logits are a numpy array, not pytorch tensor\n",
    "    predictions = np.argmax(logits, axis=-1)\n",
    "    return metric.compute(\n",
    "               predictions=predictions, references=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "PyTorch: setting up devices\n",
      "The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code and make this info disappear :-).\n"
     ]
    }
   ],
   "source": [
    "optim = torch.optim.Adam(model.parameters(), lr=5e-5)\n",
    "\n",
    "\n",
    "training_args = TrainingArguments(\n",
    "    output_dir='./results', \n",
    "    num_train_epochs=3,     \n",
    "    per_device_train_batch_size=16, \n",
    "    per_device_eval_batch_size=16,   \n",
    "    logging_dir='./logs',\n",
    "    logging_steps=10\n",
    ")\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    compute_metrics=compute_metrics,\n",
    "    args=training_args,\n",
    "    train_dataset=train_dataset,\n",
    "    eval_dataset=test_dataset,\n",
    "    optimizers=(optim, None) # optimizer and learning rate scheduler\n",
    ")\n",
    "\n",
    "# force model to only use 1 GPU (even if multiple are availabe)\n",
    "# to compare more fairly to previous code\n",
    "\n",
    "trainer.args._n_gpu = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running training *****\n",
      "  Num examples = 35000\n",
      "  Num Epochs = 3\n",
      "  Instantaneous batch size per device = 16\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 16\n",
      "  Gradient Accumulation steps = 1\n",
      "  Total optimization steps = 6564\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='6564' max='6564' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [6564/6564 45:20, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.705800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>0.684100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>0.681500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>40</td>\n",
       "      <td>0.591600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>0.328600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>0.478300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>70</td>\n",
       "      <td>0.426300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>80</td>\n",
       "      <td>0.356900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>90</td>\n",
       "      <td>0.359900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>0.268200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>110</td>\n",
       "      <td>0.470800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>120</td>\n",
       "      <td>0.319600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>130</td>\n",
       "      <td>0.355500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>140</td>\n",
       "      <td>0.346000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>0.291500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>160</td>\n",
       "      <td>0.405700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>170</td>\n",
       "      <td>0.317200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>180</td>\n",
       "      <td>0.260100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>190</td>\n",
       "      <td>0.307400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>0.271200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>210</td>\n",
       "      <td>0.294300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>220</td>\n",
       "      <td>0.310300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>230</td>\n",
       "      <td>0.313700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>240</td>\n",
       "      <td>0.254800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>0.190200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>260</td>\n",
       "      <td>0.367500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>270</td>\n",
       "      <td>0.221200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>280</td>\n",
       "      <td>0.369700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>290</td>\n",
       "      <td>0.192000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>0.270900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>310</td>\n",
       "      <td>0.385000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>320</td>\n",
       "      <td>0.274900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>330</td>\n",
       "      <td>0.300300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>340</td>\n",
       "      <td>0.223900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>0.269700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>360</td>\n",
       "      <td>0.249200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>370</td>\n",
       "      <td>0.269200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>380</td>\n",
       "      <td>0.242800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>390</td>\n",
       "      <td>0.309700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>0.343000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>410</td>\n",
       "      <td>0.262100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>420</td>\n",
       "      <td>0.291000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>430</td>\n",
       "      <td>0.229900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>440</td>\n",
       "      <td>0.368100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>0.255200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>460</td>\n",
       "      <td>0.295800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>470</td>\n",
       "      <td>0.305200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>480</td>\n",
       "      <td>0.218600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>490</td>\n",
       "      <td>0.221100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>0.268100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>510</td>\n",
       "      <td>0.252100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>520</td>\n",
       "      <td>0.224700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>530</td>\n",
       "      <td>0.227200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>540</td>\n",
       "      <td>0.275900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>550</td>\n",
       "      <td>0.301900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>560</td>\n",
       "      <td>0.250800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>570</td>\n",
       "      <td>0.316900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>580</td>\n",
       "      <td>0.255700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>590</td>\n",
       "      <td>0.324200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>0.246900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>610</td>\n",
       "      <td>0.285000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>620</td>\n",
       "      <td>0.194800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>630</td>\n",
       "      <td>0.257100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>640</td>\n",
       "      <td>0.240800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>650</td>\n",
       "      <td>0.335300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>660</td>\n",
       "      <td>0.252100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>670</td>\n",
       "      <td>0.321400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>680</td>\n",
       "      <td>0.158000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>690</td>\n",
       "      <td>0.247900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>700</td>\n",
       "      <td>0.429300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>710</td>\n",
       "      <td>0.364000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>720</td>\n",
       "      <td>0.331900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>730</td>\n",
       "      <td>0.259800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>740</td>\n",
       "      <td>0.344300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>750</td>\n",
       "      <td>0.153000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>760</td>\n",
       "      <td>0.169400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>770</td>\n",
       "      <td>0.409000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>780</td>\n",
       "      <td>0.270300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>790</td>\n",
       "      <td>0.299100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>800</td>\n",
       "      <td>0.270900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>810</td>\n",
       "      <td>0.255800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>820</td>\n",
       "      <td>0.253800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>830</td>\n",
       "      <td>0.233500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>840</td>\n",
       "      <td>0.291900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>850</td>\n",
       "      <td>0.316100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>860</td>\n",
       "      <td>0.202000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>870</td>\n",
       "      <td>0.236200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>880</td>\n",
       "      <td>0.300700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>890</td>\n",
       "      <td>0.288600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>900</td>\n",
       "      <td>0.274700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>910</td>\n",
       "      <td>0.263100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>920</td>\n",
       "      <td>0.208900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>930</td>\n",
       "      <td>0.203900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>940</td>\n",
       "      <td>0.150200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>950</td>\n",
       "      <td>0.254300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>960</td>\n",
       "      <td>0.205200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>970</td>\n",
       "      <td>0.252700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>980</td>\n",
       "      <td>0.147900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>990</td>\n",
       "      <td>0.239200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>0.215100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1010</td>\n",
       "      <td>0.212400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1020</td>\n",
       "      <td>0.289600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1030</td>\n",
       "      <td>0.268400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1040</td>\n",
       "      <td>0.320700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1050</td>\n",
       "      <td>0.244900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1060</td>\n",
       "      <td>0.138700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1070</td>\n",
       "      <td>0.238200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1080</td>\n",
       "      <td>0.278000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1090</td>\n",
       "      <td>0.266900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1100</td>\n",
       "      <td>0.253200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1110</td>\n",
       "      <td>0.229300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1120</td>\n",
       "      <td>0.227700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1130</td>\n",
       "      <td>0.235800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1140</td>\n",
       "      <td>0.151200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1150</td>\n",
       "      <td>0.284900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1160</td>\n",
       "      <td>0.263400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1170</td>\n",
       "      <td>0.279100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1180</td>\n",
       "      <td>0.216900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1190</td>\n",
       "      <td>0.221800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1200</td>\n",
       "      <td>0.217400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1210</td>\n",
       "      <td>0.299200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1220</td>\n",
       "      <td>0.341800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1230</td>\n",
       "      <td>0.304300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1240</td>\n",
       "      <td>0.227100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1250</td>\n",
       "      <td>0.245300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1260</td>\n",
       "      <td>0.169900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1270</td>\n",
       "      <td>0.353700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1280</td>\n",
       "      <td>0.182200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1290</td>\n",
       "      <td>0.354300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1300</td>\n",
       "      <td>0.260600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1310</td>\n",
       "      <td>0.274300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1320</td>\n",
       "      <td>0.205000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1330</td>\n",
       "      <td>0.239700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1340</td>\n",
       "      <td>0.222100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1350</td>\n",
       "      <td>0.287200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1360</td>\n",
       "      <td>0.420300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1370</td>\n",
       "      <td>0.134100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1380</td>\n",
       "      <td>0.364400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1390</td>\n",
       "      <td>0.247400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1400</td>\n",
       "      <td>0.239600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1410</td>\n",
       "      <td>0.211200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1420</td>\n",
       "      <td>0.315900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1430</td>\n",
       "      <td>0.150800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1440</td>\n",
       "      <td>0.178800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1450</td>\n",
       "      <td>0.144900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1460</td>\n",
       "      <td>0.306700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1470</td>\n",
       "      <td>0.079000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1480</td>\n",
       "      <td>0.216600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1490</td>\n",
       "      <td>0.262500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>0.335500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1510</td>\n",
       "      <td>0.280900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1520</td>\n",
       "      <td>0.315300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1530</td>\n",
       "      <td>0.145300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1540</td>\n",
       "      <td>0.220900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1550</td>\n",
       "      <td>0.196700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1560</td>\n",
       "      <td>0.263500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1570</td>\n",
       "      <td>0.231500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1580</td>\n",
       "      <td>0.166600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1590</td>\n",
       "      <td>0.142700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1600</td>\n",
       "      <td>0.154200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1610</td>\n",
       "      <td>0.215400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1620</td>\n",
       "      <td>0.191000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1630</td>\n",
       "      <td>0.257600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1640</td>\n",
       "      <td>0.195500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1650</td>\n",
       "      <td>0.284700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1660</td>\n",
       "      <td>0.234900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1670</td>\n",
       "      <td>0.276400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1680</td>\n",
       "      <td>0.130500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1690</td>\n",
       "      <td>0.307100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1700</td>\n",
       "      <td>0.204100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1710</td>\n",
       "      <td>0.186800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1720</td>\n",
       "      <td>0.217100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1730</td>\n",
       "      <td>0.243300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1740</td>\n",
       "      <td>0.234800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1750</td>\n",
       "      <td>0.185400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1760</td>\n",
       "      <td>0.227400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1770</td>\n",
       "      <td>0.231300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1780</td>\n",
       "      <td>0.219100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1790</td>\n",
       "      <td>0.204300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1800</td>\n",
       "      <td>0.164700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1810</td>\n",
       "      <td>0.102400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1820</td>\n",
       "      <td>0.178300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1830</td>\n",
       "      <td>0.268000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1840</td>\n",
       "      <td>0.175100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1850</td>\n",
       "      <td>0.230600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1860</td>\n",
       "      <td>0.208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1870</td>\n",
       "      <td>0.179400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1880</td>\n",
       "      <td>0.208800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1890</td>\n",
       "      <td>0.202300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1900</td>\n",
       "      <td>0.214800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1910</td>\n",
       "      <td>0.343700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1920</td>\n",
       "      <td>0.359200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1930</td>\n",
       "      <td>0.382100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1940</td>\n",
       "      <td>0.239300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1950</td>\n",
       "      <td>0.137500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1960</td>\n",
       "      <td>0.275200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1970</td>\n",
       "      <td>0.179000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1980</td>\n",
       "      <td>0.204500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1990</td>\n",
       "      <td>0.310900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2000</td>\n",
       "      <td>0.182200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2010</td>\n",
       "      <td>0.127800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2020</td>\n",
       "      <td>0.146700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2030</td>\n",
       "      <td>0.203600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2040</td>\n",
       "      <td>0.210500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2050</td>\n",
       "      <td>0.286100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2060</td>\n",
       "      <td>0.165600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2070</td>\n",
       "      <td>0.114200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2080</td>\n",
       "      <td>0.304400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2090</td>\n",
       "      <td>0.272100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2100</td>\n",
       "      <td>0.178300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2110</td>\n",
       "      <td>0.238400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2120</td>\n",
       "      <td>0.226300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2130</td>\n",
       "      <td>0.243300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2140</td>\n",
       "      <td>0.218600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2150</td>\n",
       "      <td>0.240300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2160</td>\n",
       "      <td>0.196600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2170</td>\n",
       "      <td>0.176800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2180</td>\n",
       "      <td>0.204600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2190</td>\n",
       "      <td>0.206400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2200</td>\n",
       "      <td>0.135700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2210</td>\n",
       "      <td>0.100200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2220</td>\n",
       "      <td>0.158400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2230</td>\n",
       "      <td>0.193600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2240</td>\n",
       "      <td>0.156200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2250</td>\n",
       "      <td>0.119800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2260</td>\n",
       "      <td>0.203500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2270</td>\n",
       "      <td>0.075600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2280</td>\n",
       "      <td>0.138300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2290</td>\n",
       "      <td>0.200100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2300</td>\n",
       "      <td>0.113700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2310</td>\n",
       "      <td>0.145700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2320</td>\n",
       "      <td>0.076900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2330</td>\n",
       "      <td>0.162000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2340</td>\n",
       "      <td>0.119600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2350</td>\n",
       "      <td>0.228300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2360</td>\n",
       "      <td>0.070000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2370</td>\n",
       "      <td>0.192600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2380</td>\n",
       "      <td>0.102300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2390</td>\n",
       "      <td>0.081700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2400</td>\n",
       "      <td>0.129500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2410</td>\n",
       "      <td>0.100300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2420</td>\n",
       "      <td>0.128800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2430</td>\n",
       "      <td>0.254700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2440</td>\n",
       "      <td>0.113200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2450</td>\n",
       "      <td>0.124200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2460</td>\n",
       "      <td>0.160800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2470</td>\n",
       "      <td>0.131000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2480</td>\n",
       "      <td>0.123300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2490</td>\n",
       "      <td>0.041700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2500</td>\n",
       "      <td>0.095600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2510</td>\n",
       "      <td>0.100300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2520</td>\n",
       "      <td>0.142300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2530</td>\n",
       "      <td>0.191700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2540</td>\n",
       "      <td>0.147800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2550</td>\n",
       "      <td>0.163400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2560</td>\n",
       "      <td>0.093700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2570</td>\n",
       "      <td>0.066600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2580</td>\n",
       "      <td>0.200100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2590</td>\n",
       "      <td>0.166200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2600</td>\n",
       "      <td>0.230400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2610</td>\n",
       "      <td>0.233600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2620</td>\n",
       "      <td>0.097000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2630</td>\n",
       "      <td>0.108300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2640</td>\n",
       "      <td>0.074100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2650</td>\n",
       "      <td>0.132100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2660</td>\n",
       "      <td>0.144500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2670</td>\n",
       "      <td>0.044000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2680</td>\n",
       "      <td>0.115600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2690</td>\n",
       "      <td>0.110700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2700</td>\n",
       "      <td>0.197900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2710</td>\n",
       "      <td>0.035000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2720</td>\n",
       "      <td>0.131700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2730</td>\n",
       "      <td>0.145600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2740</td>\n",
       "      <td>0.169800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2750</td>\n",
       "      <td>0.232100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2760</td>\n",
       "      <td>0.175400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2770</td>\n",
       "      <td>0.078600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2780</td>\n",
       "      <td>0.126200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2790</td>\n",
       "      <td>0.135200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2800</td>\n",
       "      <td>0.127300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2810</td>\n",
       "      <td>0.126600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2820</td>\n",
       "      <td>0.179700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2830</td>\n",
       "      <td>0.114100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2840</td>\n",
       "      <td>0.167100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2850</td>\n",
       "      <td>0.111600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2860</td>\n",
       "      <td>0.124500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2870</td>\n",
       "      <td>0.147700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2880</td>\n",
       "      <td>0.052600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2890</td>\n",
       "      <td>0.138600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2900</td>\n",
       "      <td>0.078600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2910</td>\n",
       "      <td>0.143300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2920</td>\n",
       "      <td>0.146400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2930</td>\n",
       "      <td>0.101300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2940</td>\n",
       "      <td>0.094100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2950</td>\n",
       "      <td>0.039700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2960</td>\n",
       "      <td>0.216700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2970</td>\n",
       "      <td>0.124600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2980</td>\n",
       "      <td>0.070600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2990</td>\n",
       "      <td>0.054100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3000</td>\n",
       "      <td>0.226500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3010</td>\n",
       "      <td>0.095900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3020</td>\n",
       "      <td>0.056000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3030</td>\n",
       "      <td>0.111600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3040</td>\n",
       "      <td>0.062800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3050</td>\n",
       "      <td>0.063100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3060</td>\n",
       "      <td>0.202300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3070</td>\n",
       "      <td>0.043300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3080</td>\n",
       "      <td>0.118100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3090</td>\n",
       "      <td>0.101000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3100</td>\n",
       "      <td>0.111400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3110</td>\n",
       "      <td>0.115700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3120</td>\n",
       "      <td>0.144200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3130</td>\n",
       "      <td>0.076500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3140</td>\n",
       "      <td>0.180900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3150</td>\n",
       "      <td>0.054400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3160</td>\n",
       "      <td>0.184200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3170</td>\n",
       "      <td>0.075900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3180</td>\n",
       "      <td>0.058900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3190</td>\n",
       "      <td>0.017800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3200</td>\n",
       "      <td>0.148900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3210</td>\n",
       "      <td>0.199700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3220</td>\n",
       "      <td>0.303600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3230</td>\n",
       "      <td>0.135400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3240</td>\n",
       "      <td>0.139500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3250</td>\n",
       "      <td>0.169700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3260</td>\n",
       "      <td>0.110100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3270</td>\n",
       "      <td>0.147300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3280</td>\n",
       "      <td>0.106500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3290</td>\n",
       "      <td>0.125700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3300</td>\n",
       "      <td>0.092800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3310</td>\n",
       "      <td>0.201200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3320</td>\n",
       "      <td>0.140500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3330</td>\n",
       "      <td>0.102900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3340</td>\n",
       "      <td>0.112700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3350</td>\n",
       "      <td>0.164300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3360</td>\n",
       "      <td>0.236400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3370</td>\n",
       "      <td>0.154800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3380</td>\n",
       "      <td>0.093300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3390</td>\n",
       "      <td>0.092200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3400</td>\n",
       "      <td>0.133100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3410</td>\n",
       "      <td>0.051100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3420</td>\n",
       "      <td>0.133300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3430</td>\n",
       "      <td>0.133000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3440</td>\n",
       "      <td>0.250200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3450</td>\n",
       "      <td>0.261200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3460</td>\n",
       "      <td>0.167300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3470</td>\n",
       "      <td>0.123800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3480</td>\n",
       "      <td>0.071300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3490</td>\n",
       "      <td>0.105300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3500</td>\n",
       "      <td>0.142000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3510</td>\n",
       "      <td>0.086900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3520</td>\n",
       "      <td>0.129500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3530</td>\n",
       "      <td>0.125300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3540</td>\n",
       "      <td>0.174700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3550</td>\n",
       "      <td>0.115400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3560</td>\n",
       "      <td>0.036700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3570</td>\n",
       "      <td>0.125900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3580</td>\n",
       "      <td>0.097200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3590</td>\n",
       "      <td>0.140800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3600</td>\n",
       "      <td>0.118000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3610</td>\n",
       "      <td>0.068900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3620</td>\n",
       "      <td>0.096500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3630</td>\n",
       "      <td>0.107800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3640</td>\n",
       "      <td>0.083700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3650</td>\n",
       "      <td>0.199500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3660</td>\n",
       "      <td>0.112200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3670</td>\n",
       "      <td>0.213000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3680</td>\n",
       "      <td>0.144700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3690</td>\n",
       "      <td>0.124100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3700</td>\n",
       "      <td>0.103700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3710</td>\n",
       "      <td>0.141800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3720</td>\n",
       "      <td>0.216300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3730</td>\n",
       "      <td>0.203600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3740</td>\n",
       "      <td>0.091600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3750</td>\n",
       "      <td>0.121200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3760</td>\n",
       "      <td>0.071500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3770</td>\n",
       "      <td>0.157000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3780</td>\n",
       "      <td>0.113700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3790</td>\n",
       "      <td>0.071200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3800</td>\n",
       "      <td>0.221000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3810</td>\n",
       "      <td>0.080000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3820</td>\n",
       "      <td>0.089100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3830</td>\n",
       "      <td>0.175100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3840</td>\n",
       "      <td>0.218000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3850</td>\n",
       "      <td>0.080400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3860</td>\n",
       "      <td>0.099900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3870</td>\n",
       "      <td>0.100900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3880</td>\n",
       "      <td>0.129800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3890</td>\n",
       "      <td>0.089300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3900</td>\n",
       "      <td>0.065100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3910</td>\n",
       "      <td>0.223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3920</td>\n",
       "      <td>0.102400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3930</td>\n",
       "      <td>0.122500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3940</td>\n",
       "      <td>0.097800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3950</td>\n",
       "      <td>0.057000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3960</td>\n",
       "      <td>0.118300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3970</td>\n",
       "      <td>0.104000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3980</td>\n",
       "      <td>0.114400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3990</td>\n",
       "      <td>0.060300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4000</td>\n",
       "      <td>0.127200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4010</td>\n",
       "      <td>0.117500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4020</td>\n",
       "      <td>0.138800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4030</td>\n",
       "      <td>0.096700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4040</td>\n",
       "      <td>0.158500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4050</td>\n",
       "      <td>0.031500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4060</td>\n",
       "      <td>0.101200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4070</td>\n",
       "      <td>0.044900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4080</td>\n",
       "      <td>0.040500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4090</td>\n",
       "      <td>0.176400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4100</td>\n",
       "      <td>0.098300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4110</td>\n",
       "      <td>0.152000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4120</td>\n",
       "      <td>0.146600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4130</td>\n",
       "      <td>0.120000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4140</td>\n",
       "      <td>0.166600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4150</td>\n",
       "      <td>0.068200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4160</td>\n",
       "      <td>0.112100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4170</td>\n",
       "      <td>0.250300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4180</td>\n",
       "      <td>0.104400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4190</td>\n",
       "      <td>0.180000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4200</td>\n",
       "      <td>0.074200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4210</td>\n",
       "      <td>0.129800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4220</td>\n",
       "      <td>0.218700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4230</td>\n",
       "      <td>0.064900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4240</td>\n",
       "      <td>0.081900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4250</td>\n",
       "      <td>0.176500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4260</td>\n",
       "      <td>0.151200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4270</td>\n",
       "      <td>0.186100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4280</td>\n",
       "      <td>0.132200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4290</td>\n",
       "      <td>0.198600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4300</td>\n",
       "      <td>0.122500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4310</td>\n",
       "      <td>0.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4320</td>\n",
       "      <td>0.028600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4330</td>\n",
       "      <td>0.105500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4340</td>\n",
       "      <td>0.088200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4350</td>\n",
       "      <td>0.081800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4360</td>\n",
       "      <td>0.124400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4370</td>\n",
       "      <td>0.135100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4380</td>\n",
       "      <td>0.159300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4390</td>\n",
       "      <td>0.075500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4400</td>\n",
       "      <td>0.022200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4410</td>\n",
       "      <td>0.123500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4420</td>\n",
       "      <td>0.048000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4430</td>\n",
       "      <td>0.033400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4440</td>\n",
       "      <td>0.003100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4450</td>\n",
       "      <td>0.006700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4460</td>\n",
       "      <td>0.003200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4470</td>\n",
       "      <td>0.124000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4480</td>\n",
       "      <td>0.019000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4490</td>\n",
       "      <td>0.031200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4500</td>\n",
       "      <td>0.061200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4510</td>\n",
       "      <td>0.041000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4520</td>\n",
       "      <td>0.169400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4530</td>\n",
       "      <td>0.005700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4540</td>\n",
       "      <td>0.036900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4550</td>\n",
       "      <td>0.121600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4560</td>\n",
       "      <td>0.047700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4570</td>\n",
       "      <td>0.002400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4580</td>\n",
       "      <td>0.113500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4590</td>\n",
       "      <td>0.003800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4600</td>\n",
       "      <td>0.066100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4610</td>\n",
       "      <td>0.031800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4620</td>\n",
       "      <td>0.092700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4630</td>\n",
       "      <td>0.062700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4640</td>\n",
       "      <td>0.009900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4650</td>\n",
       "      <td>0.069600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4660</td>\n",
       "      <td>0.001300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4670</td>\n",
       "      <td>0.057400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4680</td>\n",
       "      <td>0.035400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4690</td>\n",
       "      <td>0.153800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4700</td>\n",
       "      <td>0.091200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4710</td>\n",
       "      <td>0.105600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4720</td>\n",
       "      <td>0.033400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4730</td>\n",
       "      <td>0.066500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4740</td>\n",
       "      <td>0.091500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4750</td>\n",
       "      <td>0.046200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4760</td>\n",
       "      <td>0.018100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4770</td>\n",
       "      <td>0.024300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4780</td>\n",
       "      <td>0.099700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4790</td>\n",
       "      <td>0.063600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4800</td>\n",
       "      <td>0.120300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4810</td>\n",
       "      <td>0.094300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4820</td>\n",
       "      <td>0.047000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4830</td>\n",
       "      <td>0.086400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4840</td>\n",
       "      <td>0.095200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4850</td>\n",
       "      <td>0.031000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4860</td>\n",
       "      <td>0.080100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4870</td>\n",
       "      <td>0.039700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4880</td>\n",
       "      <td>0.158400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4890</td>\n",
       "      <td>0.072000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4900</td>\n",
       "      <td>0.091200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4910</td>\n",
       "      <td>0.004900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4920</td>\n",
       "      <td>0.023700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4930</td>\n",
       "      <td>0.013800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4940</td>\n",
       "      <td>0.002500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4950</td>\n",
       "      <td>0.002000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4960</td>\n",
       "      <td>0.089600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4970</td>\n",
       "      <td>0.091700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4980</td>\n",
       "      <td>0.055700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4990</td>\n",
       "      <td>0.058700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5000</td>\n",
       "      <td>0.042500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5010</td>\n",
       "      <td>0.093800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5020</td>\n",
       "      <td>0.042100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5030</td>\n",
       "      <td>0.050900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5040</td>\n",
       "      <td>0.045900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5050</td>\n",
       "      <td>0.143700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5060</td>\n",
       "      <td>0.040500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5070</td>\n",
       "      <td>0.103100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5080</td>\n",
       "      <td>0.033200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5090</td>\n",
       "      <td>0.031400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5100</td>\n",
       "      <td>0.026400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5110</td>\n",
       "      <td>0.009400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5120</td>\n",
       "      <td>0.110200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5130</td>\n",
       "      <td>0.027900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5140</td>\n",
       "      <td>0.031700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5150</td>\n",
       "      <td>0.159000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5160</td>\n",
       "      <td>0.006300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5170</td>\n",
       "      <td>0.043300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5180</td>\n",
       "      <td>0.030800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5190</td>\n",
       "      <td>0.080600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5200</td>\n",
       "      <td>0.206100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5210</td>\n",
       "      <td>0.140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5220</td>\n",
       "      <td>0.059400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5230</td>\n",
       "      <td>0.025800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5240</td>\n",
       "      <td>0.051100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5250</td>\n",
       "      <td>0.002700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5260</td>\n",
       "      <td>0.003000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5270</td>\n",
       "      <td>0.083500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5280</td>\n",
       "      <td>0.048800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5290</td>\n",
       "      <td>0.114500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5300</td>\n",
       "      <td>0.057500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5310</td>\n",
       "      <td>0.046900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5320</td>\n",
       "      <td>0.081800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5330</td>\n",
       "      <td>0.031700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5340</td>\n",
       "      <td>0.079300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5350</td>\n",
       "      <td>0.038000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5360</td>\n",
       "      <td>0.030300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5370</td>\n",
       "      <td>0.099600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5380</td>\n",
       "      <td>0.054500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5390</td>\n",
       "      <td>0.051100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5400</td>\n",
       "      <td>0.094900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5410</td>\n",
       "      <td>0.060700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5420</td>\n",
       "      <td>0.055400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5430</td>\n",
       "      <td>0.080400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5440</td>\n",
       "      <td>0.120200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5450</td>\n",
       "      <td>0.071300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5460</td>\n",
       "      <td>0.041500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5470</td>\n",
       "      <td>0.091500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5480</td>\n",
       "      <td>0.072800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5490</td>\n",
       "      <td>0.039200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5500</td>\n",
       "      <td>0.032500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5510</td>\n",
       "      <td>0.145400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5520</td>\n",
       "      <td>0.039300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5530</td>\n",
       "      <td>0.002800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5540</td>\n",
       "      <td>0.118200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5550</td>\n",
       "      <td>0.110000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5560</td>\n",
       "      <td>0.050800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5570</td>\n",
       "      <td>0.007900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5580</td>\n",
       "      <td>0.003800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5590</td>\n",
       "      <td>0.039500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5600</td>\n",
       "      <td>0.137000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5610</td>\n",
       "      <td>0.007000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5620</td>\n",
       "      <td>0.002900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5630</td>\n",
       "      <td>0.031800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5640</td>\n",
       "      <td>0.035400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5650</td>\n",
       "      <td>0.008300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5660</td>\n",
       "      <td>0.029100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5670</td>\n",
       "      <td>0.042600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5680</td>\n",
       "      <td>0.103500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5690</td>\n",
       "      <td>0.022800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5700</td>\n",
       "      <td>0.196400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5710</td>\n",
       "      <td>0.002000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5720</td>\n",
       "      <td>0.058700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5730</td>\n",
       "      <td>0.081200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5740</td>\n",
       "      <td>0.042700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5750</td>\n",
       "      <td>0.036100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5760</td>\n",
       "      <td>0.014900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5770</td>\n",
       "      <td>0.008600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5780</td>\n",
       "      <td>0.072200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5790</td>\n",
       "      <td>0.043400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5800</td>\n",
       "      <td>0.042300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5810</td>\n",
       "      <td>0.054400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5820</td>\n",
       "      <td>0.002500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5830</td>\n",
       "      <td>0.038000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5840</td>\n",
       "      <td>0.069600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5850</td>\n",
       "      <td>0.032300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5860</td>\n",
       "      <td>0.071800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5870</td>\n",
       "      <td>0.089100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5880</td>\n",
       "      <td>0.051900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5890</td>\n",
       "      <td>0.016100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5900</td>\n",
       "      <td>0.072500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5910</td>\n",
       "      <td>0.077100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5920</td>\n",
       "      <td>0.064200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5930</td>\n",
       "      <td>0.170300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5940</td>\n",
       "      <td>0.010100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5950</td>\n",
       "      <td>0.078100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5960</td>\n",
       "      <td>0.101000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5970</td>\n",
       "      <td>0.007900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5980</td>\n",
       "      <td>0.002200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5990</td>\n",
       "      <td>0.078600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6000</td>\n",
       "      <td>0.111400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6010</td>\n",
       "      <td>0.124700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6020</td>\n",
       "      <td>0.024700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6030</td>\n",
       "      <td>0.048300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6040</td>\n",
       "      <td>0.077900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6050</td>\n",
       "      <td>0.122700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6060</td>\n",
       "      <td>0.043600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6070</td>\n",
       "      <td>0.093100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6080</td>\n",
       "      <td>0.074000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6090</td>\n",
       "      <td>0.025400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6100</td>\n",
       "      <td>0.008500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6110</td>\n",
       "      <td>0.114600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6120</td>\n",
       "      <td>0.014500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6130</td>\n",
       "      <td>0.038400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6140</td>\n",
       "      <td>0.054000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6150</td>\n",
       "      <td>0.012300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6160</td>\n",
       "      <td>0.043600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6170</td>\n",
       "      <td>0.093800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6180</td>\n",
       "      <td>0.017400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6190</td>\n",
       "      <td>0.099600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6200</td>\n",
       "      <td>0.045300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6210</td>\n",
       "      <td>0.018800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6220</td>\n",
       "      <td>0.112600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6230</td>\n",
       "      <td>0.070800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6240</td>\n",
       "      <td>0.070900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6250</td>\n",
       "      <td>0.017900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6260</td>\n",
       "      <td>0.045500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6270</td>\n",
       "      <td>0.054800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6280</td>\n",
       "      <td>0.027100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6290</td>\n",
       "      <td>0.035800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6300</td>\n",
       "      <td>0.071200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6310</td>\n",
       "      <td>0.150600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6320</td>\n",
       "      <td>0.066400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6330</td>\n",
       "      <td>0.051200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6340</td>\n",
       "      <td>0.007700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6350</td>\n",
       "      <td>0.072000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6360</td>\n",
       "      <td>0.005200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6370</td>\n",
       "      <td>0.104200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6380</td>\n",
       "      <td>0.013400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6390</td>\n",
       "      <td>0.023500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6400</td>\n",
       "      <td>0.029200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6410</td>\n",
       "      <td>0.038300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6420</td>\n",
       "      <td>0.056000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6430</td>\n",
       "      <td>0.001900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6440</td>\n",
       "      <td>0.045300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6450</td>\n",
       "      <td>0.009300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6460</td>\n",
       "      <td>0.015000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6470</td>\n",
       "      <td>0.062600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6480</td>\n",
       "      <td>0.014300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6490</td>\n",
       "      <td>0.077200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6500</td>\n",
       "      <td>0.032500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6510</td>\n",
       "      <td>0.041000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6520</td>\n",
       "      <td>0.003800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6530</td>\n",
       "      <td>0.071300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6540</td>\n",
       "      <td>0.004900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6550</td>\n",
       "      <td>0.050400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6560</td>\n",
       "      <td>0.020400</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Saving model checkpoint to ./results/checkpoint-500\n",
      "Configuration saved in ./results/checkpoint-500/config.json\n",
      "Model weights saved in ./results/checkpoint-500/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-1000\n",
      "Configuration saved in ./results/checkpoint-1000/config.json\n",
      "Model weights saved in ./results/checkpoint-1000/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-1500\n",
      "Configuration saved in ./results/checkpoint-1500/config.json\n",
      "Model weights saved in ./results/checkpoint-1500/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-2000\n",
      "Configuration saved in ./results/checkpoint-2000/config.json\n",
      "Model weights saved in ./results/checkpoint-2000/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-2500\n",
      "Configuration saved in ./results/checkpoint-2500/config.json\n",
      "Model weights saved in ./results/checkpoint-2500/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-3000\n",
      "Configuration saved in ./results/checkpoint-3000/config.json\n",
      "Model weights saved in ./results/checkpoint-3000/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-3500\n",
      "Configuration saved in ./results/checkpoint-3500/config.json\n",
      "Model weights saved in ./results/checkpoint-3500/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-4000\n",
      "Configuration saved in ./results/checkpoint-4000/config.json\n",
      "Model weights saved in ./results/checkpoint-4000/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-4500\n",
      "Configuration saved in ./results/checkpoint-4500/config.json\n",
      "Model weights saved in ./results/checkpoint-4500/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-5000\n",
      "Configuration saved in ./results/checkpoint-5000/config.json\n",
      "Model weights saved in ./results/checkpoint-5000/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-5500\n",
      "Configuration saved in ./results/checkpoint-5500/config.json\n",
      "Model weights saved in ./results/checkpoint-5500/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-6000\n",
      "Configuration saved in ./results/checkpoint-6000/config.json\n",
      "Model weights saved in ./results/checkpoint-6000/pytorch_model.bin\n",
      "Saving model checkpoint to ./results/checkpoint-6500\n",
      "Configuration saved in ./results/checkpoint-6500/config.json\n",
      "Model weights saved in ./results/checkpoint-6500/pytorch_model.bin\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Training Time: 45.36 min\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "trainer.train()\n",
    "print(f'Total Training Time: {(time.time() - start_time)/60:.2f} min')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 10000\n",
      "  Batch size = 16\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='625' max='625' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [625/625 01:26]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.30534815788269043,\n",
       " 'eval_accuracy': 0.9327,\n",
       " 'eval_runtime': 87.1161,\n",
       " 'eval_samples_per_second': 114.789,\n",
       " 'eval_steps_per_second': 7.174,\n",
       " 'epoch': 3.0}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 93.27%\n"
     ]
    }
   ],
   "source": [
    "model.eval()\n",
    "model.to(DEVICE)\n",
    "print(f'Test accuracy: {compute_accuracy(model, test_loader, DEVICE):.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Readers may ignore the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] WARNING | Config option `kernel_spec_manager_class` not recognized by `NbConvertApp`.\n",
      "[NbConvertApp] Converting notebook ch16-part3-bert.ipynb to script\n",
      "[NbConvertApp] Writing 9089 bytes to ch16-part3-bert.py\n"
     ]
    }
   ],
   "source": [
    "! python ../.convert_notebook_to_script.py --input ch16-part3-bert.ipynb --output ch16-part3-bert.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "rnn_lstm_packed_imdb.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}