1_multithread_vs_multiprocess.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1. Embarrassingly parallel: multiple threads vs. multiple processes\n",
    "\n",
    "In this example, you will investigate the speed-ups (or lack of speed-up) for an embarrssingly parallel workflow with a CPU-bound function that is executed using different paradigms (single thread, multiple threads and multiple processes). Here is a diagram of the workflow:\n",
    "<img src='figures/embarrassingly_parallel.png'>\n",
    "\n",
    "**Your task**:\n",
    "1. Modify the code inside `func` function in `1_problem.py` so that it takes approximately 1 second to execute. (You choose what it does.)\n",
    "1. Load it into this notebook using the cell below that has the `%load` magic.\n",
    "1. Use the cell with the `%%time` magic to see how long it takes 4 instances of the `func` function to execute with 1) a single thread, 2) multiple threads and 3) multiple processes.\n",
    "1. Note down the times for all executions and any other interesting observations.\n",
    "1. Submit your work (see instructions in last cell).\n",
    "\n",
    "**Hints**:\n",
    "* If your `func` function is not reproducing the multithread/multiprocessing behaviour you expect, make sure it is CPU-bound. Ask a tutor if you have questions. :)\n",
    "\n",
    "### Step 0: Import package(s) we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask  # we'll explain this package later"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Modify 1_problem.py, load it and execute the cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load 1_problem.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Make predictions\n",
    "\n",
    "Write down how long you think it will take your function to run with  \n",
    "a) a single thread  \n",
    "b) multiple threads  \n",
    "c) multiple processes  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Test your predictions\n",
    "\n",
    "The cell below is where you can execute the computations with different schedulers (i.e., multithreaded vs. multiprocesses). We will explain later what the code is acutally doing, but for now all you need to change is the `scheduler` variable. The options for `scheduler` are\n",
    "* `single-threaded`: a single thread\n",
    "* `threads`: multiple threads\n",
    "* `processes`: multiple processes\n",
    "\n",
    "Try with all three scheduler options and write down your results. **Consider the following questions**. Do they match the times you predicted? Why/why not? Any other interesting observations?\n",
    "\n",
    "**Last note**: I recommend copying the cell into three different cells, one for each scheduler option, to more easily compare the different paradigms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "scheduler = 'single-threaded'  # <--- ***CHANGE ME!***\n",
    "\n",
    "# don't change the code below\n",
    "res = dask.compute(lazy, scheduler=scheduler)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Submit your work\n",
    "\n",
    "Please follow these instructions carefully! Otherwise I won't be able to load your solution into my notebook.\n",
    "\n",
    "In short, create a branch, add/commit **only** the text file you touched and then push the branch.\n",
    "\n",
    "```\n",
    "git checkout -b <unique branch name>\n",
    "git add 1_problem.py\n",
    "git commit -m \"uploading our solution\"\n",
    "git push origin <unique branch name>\n",
    "\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}