5-on_CC_cluster.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning on Compute Canada Systems\n",
    "\n",
    "Now that we've seem how to do Machine Learning with scikit-learn and PyTorch, next we'll look at the best practices for running ML code on Compute Canada Systems. This tutorial will be divided in two parts, namely:\n",
    "\n",
    "* Developing/testing/debugging your code\n",
    "\n",
    "* Resources: Do you really need a GPU?\n",
    "\n",
    "\n",
    "## Developing/testing/debugging your code\n",
    "\n",
    "So you are just getting started exploring a new dataset, testing out different algorithms/model architectures and trying to figure out how you will predict some variable Y from a set of variables X. \n",
    "\n",
    "We recommend that, unless you are exploring a really huge dataset, you should do the first steps outilned in the **scikit-learn** notebook on your own computer, or on an interactive session on Compute Canada with very low resource requests. You generally won't need more than a single CPU and a few GBs of memory to carry out some exploratory analysis, and maybe try out some simple algorithms to establish a performance baseline that you will try to improve on with more complex ones.\n",
    "\n",
    "In fact, the same applies for more complicated models - unless you are trying to develop a HUGE model (as in, the model itself is so big that it does not fit in most laptops' memory), you should do development on your own computer, or an interactive session on Compute Canada with low resource requests. By \"doing development\" we mean the process figuring out the steps of work your code needs to perform, writing the (serial) code and making sure that, given a small input dataset, your code runs without errors.\n",
    "\n",
    "Once your code runs without errors on a small dataset, you are ready to start thinking about tweaking its performance, parallelizing work and/or preparing to crunch large amounts of data on a Compute Canada cluster.\n",
    "\n",
    "Before we do that however, let's look into what are the best practices if you decide to do your development on an interactive session on a Compute Canada cluster.\n",
    "\n",
    "### Developing/Testing/Debugging on an Interactive Session\n",
    "\n",
    "The first step is to log into a compute canada cluster with:\n",
    "```shell\n",
    "\n",
    "ssh -Y username@clustername.computecanada.ca\n",
    "\n",
    "```\n",
    "\n",
    "Here is what you should see after logging in:\n",
    "\n",
    "![](./images/beluga.png)\n",
    "\n",
    "This location is known as the **Login Node**: it is your gateway to the cluster and it is **NOT** where you should run your machine learning code! Instead, we will request **Compute Resources** on the actual cluster to do our work. In this example, imagine we are dealing with a very small dataset like the Iris dataset - so we will request a single CPU and 4 GB memory for an hour with:\n",
    "\n",
    "```shell\n",
    "\n",
    "salloc --time=1:0:0 --ntasks=1 --cpus-per-task=1 --mem-per-cpu=4096M\n",
    "\n",
    "```\n",
    "\n",
    "Here is what you will see after running this:\n",
    "\n",
    "![](./images/salloc.png)\n",
    "\n",
    "Now you are on a **Compute Node** - this is the ideal place to do your work!\n",
    "\n",
    "The first thing we will do is setup a **Python Virtual Environment**. With a **Python Virtual Environment** you will be able to install your own Python libraries to use in your code. To get this done, run the following command:\n",
    "\n",
    "```shell\n",
    "\n",
    "module load python/3.6\n",
    "\n",
    "virtualenv --no-download $SLURM_TMPDIR/my_virtual_environment\n",
    "\n",
    "```\n",
    "\n",
    "Here is what you should see after running this command:\n",
    "\n",
    "![](./images/virtualenv.png)\n",
    "\n",
    "What we did here is: load Python as a module, so we are able to run the command ```python``` on the command line and execute python scripts; then we created a virtual environment at a special location called **$SLURM_TMPDIR**. This special location allows you to write to the local disk of the **Compute Node**. We will also write our data to this location later on.\n",
    "\n",
    "Now that we've created our **Python Virtual Environment** we are ready to install our libraries and start writing our code. Here is how you do that:\n",
    "\n",
    "First, activate your **Python Virtual Environment:**\n",
    "\n",
    "```shell\n",
    "source $SLURM_TMPDIR/my_virtual_environment/bin/activate\n",
    "```\n",
    "\n",
    "Here's what you should see after activating your virtual environment:\n",
    "\n",
    "![](./images/start_venv.png)\n",
    "\n",
    "Note how the name of your virtual environment is now showing up on the left-hand side of the shell console. This means your are now inside your virtual environment.\n",
    "\n",
    "Then you can install your libraries with:\n",
    "\n",
    "```shell\n",
    "pip install --no-index <libraryname>\n",
    "```\n",
    "\n",
    "Notice the use of the ```--no-index``` option. If you had typed simply ```pip install <libraryname>``` then ```pip``` will search for your library on the internet. **Compute Nodes** are not connected to the internet however, so this command will fail. Adding the ```--no-index``` option, tells ```pip``` to not search for your library on the internet and search for it on Compute Canada's internal repository of Python libraries (called a Wheelhouse) instead.\n",
    "\n",
    "If the library you are attempting to install does not exist on Compute Canada's wheelhouse, write to **support@computecanada.ca** and let us know. If you are in a hurry, you can follow the steps above to create a virtual environment on the **Login Node** (i.e., before running the ```salloc``` step), where there is a connection to the internet and you should be able to install your library with ```pip install <libraryname>```. Then once on a **Compute Node**, activate the virtual environment and try to use your library normally.\n",
    "\n",
    "Once your libraries have been installed, you can copy your data from wherever it is located (we are assuming you have already loaded your data on the **Login Node** either in /home, /project or /scratch) to ```$SLURM_TMPDIR``` with:\n",
    "\n",
    "```shell\n",
    "mkdir $SLURM_TMPDIR/work\n",
    "cp /path/to/dataset $SLURM_TMPDIR/work\n",
    "```\n",
    "\n",
    "Copying your data to the local disk on the **Compute Node** is an especially good idea if you are developing on an image dataset. These datasets are usually made up of a large number of small files, even when the total size of the dataset on disk is small. Moving them to the local disk will make loading them from inside your code a lot more efficient and it is a good habit to acquire for when you will start running experiments on actual huge datasets!\n",
    "\n",
    "## Developing your code on JupyterHub\n",
    "\n",
    "Another, presumably easier, option to use Compute Canada to develop your ML code is **JupyterHub on Beluga**. You can access this at: <a href=https://jupyterhub.beluga.computecanada.ca>https://jupyterhub.beluga.computecanada.ca<a>\n",
    "    \n",
    "Use your Compute Canada credentials to log in:\n",
    "    \n",
    "![](./images/JH_beluga.png)\n",
    "    \n",
    "\n",
    "Then keep your **resource requests low**... and **NO GPUs** at this point!\n",
    "    \n",
    "![](./images/JH_options.png) \n",
    "    \n",
    "    \n",
    "To install your libraries on a JupyterHub session, run a cell with:\n",
    "    \n",
    "```python\n",
    "   !pip install --no-index <libraryname> \n",
    "```\n",
    "    \n",
    "Then restart the Python kernel and you will be able to import the libraries you just installed.\n",
    "    \n",
    "Before you start working, open a terminal tab on JupyterHub by clicking on \"file > new > terminal\" and follow the steps outlined above to move your data to the local disk of the **Compute Node** where your notebook will be running.\n",
    "    \n",
    "Now that you've seen options to develop your code, let's turn to evaluating whether or not your code would benefit from requesting a larger amount of resources! \n",
    "    \n",
    "## Resources: GPU or no GPU?\n",
    "    \n",
    "So you've finished writing your code and you've made sure it runs without errors on a small subset of your potentially large dataset. Now it is time to look into making your code more performant and asking whether or not you can achieve this by making more resources available to it!\n",
    "    \n",
    "Let's start by answering the question \"Do you need a GPU?\"\n",
    "    \n",
    "As we've warned in the **PyTorch** notebook, sometimes attempting to use one or more GPUs can introduce overhead that results in your code taking *more* time, not *less*, to run. That happens whenever you try to use a GPU to train:\n",
    "    \n",
    "    a) a model that is so small that the time it would take to get it loaded on a GPU and then train it is greater than the time a CPU can would take to train it. \n",
    "    \n",
    "    b) models using small datasets.\n",
    "    \n",
    "Here are some quick rules-of-thumb to check if your work falls in one of these two categories:\n",
    "    \n",
    "    a) your model has less than several hundred thousand trainable parameters\n",
    "    \n",
    "    b) the examples in your dataset, when combined into your desired batch size (or even the entire dataset), add up to just a few MBs.\n",
    "    \n",
    "If you think your code falls in one of these categories, then you should not use a GPU.\n",
    "        \n",
    "That being said, these are just rules-of-thumb. The best way to verify whether or not your code could benefit from a GPU is to profile it. Let's look at an example profiling our code from the **PyTorch** notebook that trains a neural network on the Iris Dataset to see what that means:\n",
    "    \n",
    "    \n",
    "First we use the ```%%timeit``` decorator to get the average execution time of the code block below using a GPU:\n",
    "    \n",
    "![](./images/profiling_1.png) \n",
    "    \n",
    "    \n",
    "Then we time the execution of the same block using only the CPU:\n",
    "    \n",
    "![](./images/profiling_2.png) \n",
    "    \n",
    "    \n",
    "As you can see, in our case training on the CPU is actually faster than on a GPU. This is a strong indication that this model will not benefit from using a GPU.\n",
    "    \n",
    "Another test you can run to decide whether or not your code would benefit from a GPU is shown below, using the ```profiler``` method to compute how much time in total both the CPU and the GPU spend running your code:\n",
    "    \n",
    "![](./images/profiling_3.png) \n",
    "    \n",
    "\n",
    "The output of the profiler is a potentially very long table describing the CPU and GPU times of each line of your code. The result we are interested in is at the very bottom:\n",
    "    \n",
    "![](./images/profiling_out.png) \n",
    "    \n",
    "    \n",
    "This tells us that the CPU and the GPU spend a similar amount of time doing work on our code. Ideally, we would like to see most of the time being spent on the GPU as this would mean it is doing most of the work. In our case, the similar times are another indication that this code would not use a GPU efficiently.\n",
    "    \n",
    "Finally, you can check GPU utilisation by your code by opening a terminal tab on JupyterHub and running the ```nvidia-smi``` command *while your Python code is running on the other tab* :\n",
    "    \n",
    "![](./images/nvidia-smi.png) \n",
    "\n",
    "This shows us the GPU utilisation is only around 19%, with about only 1 out of 16GB of its memory in use. This is another indication that our code does not use a GPU efficiently.\n",
    "\n",
    "In summary, you should run these three tests using a **small sample from your dataset** to get a sense of whether or not you should consider using a GPU when running your code on your full dataset. Consider using a GPU if:\n",
    "    \n",
    "    a) Execution time is **at the very least** more than 2x faster on GPU than on CPU.\n",
    "    \n",
    "    b) GPU time is **considerably longer** than CPU time.\n",
    "    \n",
    "    c) Utilisation % and/or memory usage are high.\n",
    "    \n",
    "    \n",
    "And if your model is big, but efficiency is still not great based on the tests above, try tweaking your code to increase GPU efficiency and increase your performance gains. You can try:\n",
    "    \n",
    "    * Increasing the batch size.\n",
    "    \n",
    "    * Increasing the number of workers on your DataLoader (the number of CPUs you request for this should be 1 more than the number of workers you wish to use on your DataLoader).\n",
    "    \n",
    "    * Move your data to $SLURM_TMPDIR in case you forgot to do it\n",
    "    \n",
    "\n",
    "The reasoning for deciding whether or not to use multiple GPUs is similar and you can run the tests above on an interactive session where you require multiple GPUs. **Make sure your code is GPU efficient on ONE GPU before moving to trying it on multiple GPUs!**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}