1_FeedforwardNet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSE6250BDH Deep Learning Labs\n",
    "## 1. Feed-forward Neural Network\n",
    "\n",
    "In this chapter, we will learn how to implement a feed-forward neural network by using PyTorch.\n",
    "Also, we will use the dataset generated from the previous lab [Spark-mllib](http://www.sunlab.org/teaching/cse6250/fall2017/lab/spark-mllib/#Scikit-learn). If you have not completed that part, please complete it first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, import packages we need and load data generated from the previous lab series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import load_svmlight_file\n",
    "from sklearn.preprocessing import MaxAbsScaler\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "\n",
    "X, y = load_svmlight_file(\"patients.svmlight\")\n",
    "X = X.toarray() # make it dense\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=6250)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(240, 9978)\n",
      "(240,)\n",
      "(60, 9978)\n",
      "(60,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(y_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, We will use features scaled into values between 0 and 1. &#8251;`MaxAbsScaler` scales the values between -1 and 1, but we have positive values only.\n",
    "\n",
    "Since we calcaulte the scale using training set only, we have to, the features in the test set may have different scales. It applies same for different scalers even if you want to use another one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = MaxAbsScaler().fit(X_train)\n",
    "X_train_transformed = scaler.transform(X_train)\n",
    "X_test_transformed = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train - Min: 0.0, Max: 1.0\n",
      "Test  - Min: 0.0, Max: 360.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "print(\"Train - Min: {}, Max: {}\".format(np.min(X_train_transformed), np.max(X_train_transformed)))\n",
    "print(\"Test  - Min: {}, Max: {}\".format(np.min(X_test_transformed), np.max(X_test_transformed)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feedforward Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will train a feed-forward neural network by using PyTorch. We will do the following steps in order:\n",
    "\n",
    "1. Load the training and test datasets using DataLoader\n",
    "2. Define a Feedforwad Neural Network\n",
    "3. Define a loss function\n",
    "4. Train the network on the training data\n",
    "5. Test the network on the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Loading datasets\n",
    "We will use DataLoader and TensorDataset (from [torch.utils.data](http://pytorch.org/docs/master/data.html#)) for convinience in data handling. You can create your custom dataset class by inheriting Dataset with some required member functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "\n",
    "# lets fix the random seeds for reproducibility.\n",
    "torch.manual_seed(6250)\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.manual_seed(6250)\n",
    "\n",
    "trainset = TensorDataset(torch.from_numpy(X_train_transformed.astype('float32')), torch.from_numpy(y_train.astype('float32')).view(-1,1))\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)\n",
    "\n",
    "testset = TensorDataset(torch.from_numpy(X_test_transformed.astype('float32')), torch.from_numpy(y_test.astype('float32')).view(-1,1))\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check some training samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0., 0., 0.,  ..., 0., 0., 0.],\n",
      "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
      "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
      "        [0., 0., 0.,  ..., 1., 0., 0.]])\n",
      "tensor([[0.],\n",
      "        [1.],\n",
      "        [1.],\n",
      "        [0.]])\n"
     ]
    }
   ],
   "source": [
    "# get some random training samples\n",
    "dataiter = iter(trainloader)\n",
    "records, labels = dataiter.next()\n",
    "\n",
    "print(records)\n",
    "print(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Define a Feed-forward Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will define a model, feed-forward neural network for this chapter..\n",
    "For simplicity, we will use 3-layer, 2 hidden layers and 1 hidden-to-output layer, feed-forward net. Each layer is a fully-connected layer where the module `torch.nn.Linear` is the implementation of it. Also, we will apply ReLU activation for each layer.\n",
    "\n",
    "Basically, we are required to define a member method of `forward(self, x)` when we define a class for any customized network. It represents a forward pass of a computational graph and a backward pass (back-propagation) with automatic differentiation will be performed later based on this forward definition.\n",
    "\n",
    "Usually, we define layers of entire network structure at the constructor of the class `__init__` with some arguments. Then, define `forward` function for forward computation based on the layers defined in the constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "\n",
    "class FeedForwardNet(nn.Module):\n",
    "    def __init__(self, n_input, n_hidden, n_output):\n",
    "        super(FeedForwardNet, self).__init__()\n",
    "        self.hidden1 = nn.Linear(n_input, n_hidden)\n",
    "        self.hidden2 = nn.Linear(n_hidden, n_hidden)\n",
    "        self.out = nn.Linear(n_hidden, n_output)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.hidden1(x))\n",
    "        x = F.relu(self.hidden2(x))\n",
    "        x = self.out(x)\n",
    "        return x\n",
    "\n",
    "net = FeedForwardNet(n_input=9978, n_hidden=256, n_output=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Define a Loss function and Optimizer\n",
    "We will use Binary Cross Entropy loss and SGD with momentum as our optimizer.\n",
    "PyTorch provide BCEWithLogitsLoss loss function which combines a Sigmoid layer and the BCEloss together and it is more numerically stable than using them separately. **Keep in mind that you should not apply sigmoid activation after the output layer in the model class definition to use this combined loss.** See the last computation in `forward` function above.\n",
    "\n",
    "When we create an optimizer in PyTorch, we need to pass parameters that we want to optimize (train) as input arguments. We can retrieve all trainable parameters of the model by calling `MODEL.parameters()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "criterion = nn.BCEWithLogitsLoss()\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Train the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will actually train the model.\n",
    "For each full coverage of train dataset, we just need to do a forward pass computation with a mini-batch of dataset and a backward pass to compute gradients followed by a step of optimization.\n",
    "We need to do this for a reasonable number of iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1,    10] loss: 0.688\n",
      "[1,    20] loss: 0.684\n",
      "[1,    30] loss: 0.682\n",
      "[1,    40] loss: 0.677\n",
      "[1,    50] loss: 0.666\n",
      "[1,    60] loss: 0.688\n",
      "[2,    10] loss: 0.675\n",
      "[2,    20] loss: 0.647\n",
      "[2,    30] loss: 0.655\n",
      "[2,    40] loss: 0.660\n",
      "[2,    50] loss: 0.670\n",
      "[2,    60] loss: 0.680\n",
      "[3,    10] loss: 0.680\n",
      "[3,    20] loss: 0.643\n",
      "[3,    30] loss: 0.639\n",
      "[3,    40] loss: 0.629\n",
      "[3,    50] loss: 0.664\n",
      "[3,    60] loss: 0.662\n",
      "[4,    10] loss: 0.685\n",
      "[4,    20] loss: 0.644\n",
      "[4,    30] loss: 0.670\n",
      "[4,    40] loss: 0.645\n",
      "[4,    50] loss: 0.591\n",
      "[4,    60] loss: 0.631\n",
      "[5,    10] loss: 0.660\n",
      "[5,    20] loss: 0.636\n",
      "[5,    30] loss: 0.658\n",
      "[5,    40] loss: 0.638\n",
      "[5,    50] loss: 0.615\n",
      "[5,    60] loss: 0.623\n",
      "[6,    10] loss: 0.632\n",
      "[6,    20] loss: 0.654\n",
      "[6,    30] loss: 0.595\n",
      "[6,    40] loss: 0.633\n",
      "[6,    50] loss: 0.607\n",
      "[6,    60] loss: 0.680\n",
      "[7,    10] loss: 0.666\n",
      "[7,    20] loss: 0.578\n",
      "[7,    30] loss: 0.627\n",
      "[7,    40] loss: 0.641\n",
      "[7,    50] loss: 0.642\n",
      "[7,    60] loss: 0.627\n",
      "[8,    10] loss: 0.613\n",
      "[8,    20] loss: 0.599\n",
      "[8,    30] loss: 0.608\n",
      "[8,    40] loss: 0.637\n",
      "[8,    50] loss: 0.682\n",
      "[8,    60] loss: 0.623\n",
      "[9,    10] loss: 0.697\n",
      "[9,    20] loss: 0.578\n",
      "[9,    30] loss: 0.651\n",
      "[9,    40] loss: 0.636\n",
      "[9,    50] loss: 0.578\n",
      "[9,    60] loss: 0.607\n",
      "[10,    10] loss: 0.694\n",
      "[10,    20] loss: 0.663\n",
      "[10,    30] loss: 0.666\n",
      "[10,    40] loss: 0.604\n",
      "[10,    50] loss: 0.532\n",
      "[10,    60] loss: 0.574\n",
      "Finished Training\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(10):  # loop over the dataset multiple times\n",
    "\n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        # get the inputs\n",
    "        inputs, labels = data\n",
    "\n",
    "        # wrap them in Variable (deprecated)\n",
    "        # inputs, labels = Variable(inputs), Variable(labels)\n",
    "\n",
    "        # zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # forward\n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        # backward\n",
    "        loss.backward()\n",
    "        # optimize\n",
    "        optimizer.step()\n",
    "\n",
    "        # print statistics\n",
    "        running_loss += loss.item()\n",
    "        \n",
    "        if i % 10 == 9:    # print every 10 mini-batches\n",
    "            print('[%d, %5d] loss: %.3f' %\n",
    "                  (epoch + 1, i + 1, running_loss / 10))\n",
    "            running_loss = 0.0\n",
    "\n",
    "print('Finished Training')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Test the network on the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we do always, we will calculate a test set performance.\n",
    "To utilize scikit-learn pacakges, we need to convert PyTorch Tensor to Numpy ndarray by simply calling `TENSOR.numpy()`. **Note again, Tensor and corresponding ndarray share the memory.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true = []\n",
    "y_scores = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "for data in testloader:\n",
    "    inputs, labels = data\n",
    "    outputs = net(inputs)\n",
    "    outputs = torch.sigmoid(outputs)  # since we have no activation at the end of output layer\n",
    "    y_true.extend(labels.numpy().flatten().tolist())\n",
    "    y_scores.extend(outputs.data.numpy().flatten().tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7344322344322345"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fpr, tpr, _ = roc_curve(y_true, y_scores)\n",
    "auc_ffnet = auc(fpr, tpr)\n",
    "auc_ffnet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have finished a simple example to be familiar with PyTorch. Let's try other types of neural networks in the following chapters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1: Try to use GPU if you have one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2: How is the result comparing to the previous lab, e.g. SVM? Is it better or worse? If it is worse, can you improve the performance of the network?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3: How could you check whether the network underfit, overfit or well-fit?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch1]",
   "language": "python",
   "name": "conda-env-pytorch1-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}