10 Steps to Become a Data Scientist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "9f550639a0754f42a243e5785d895d24ba655515"
   },
   "source": [
    "\n",
    "<img src=\"http://s9.picofile.com/file/8338833934/DS.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e02d495da0fb0ad24e0341e91848f4c4cfc35bdb"
   },
   "source": [
    " ## <div align=\"center\">  10 Steps to Become a Data Scientist</div>\n",
    "\n",
    "How to become a **Data Scientist**?\n",
    "\n",
    "in my point of view, It is necessary to 10 steps, in this kernel I want to cover all of 10 steps, **quite practical and far from any theoretical concepts**.\n",
    "<div style=\"text-align:center\">last update: <b>10/09/2018</b></div>\n",
    "\n",
    "---------------------------------------------------------------------\n",
    "you can follow me on:\n",
    "> ###### [ GitHub](https://github.com/mjbahmani)\n",
    "> ###### [LinkedIn](https://www.linkedin.com/in/bahmani/)\n",
    "> ###### [Kaggle](https://www.kaggle.com/mjbahmani/)\n",
    ">###### you may  be interested have a look at it: [A Comprehensive Machine Learning Workflow with Python](http://https://www.kaggle.com/mjbahmani/a-comprehensive-ml-workflow-with-python)\n",
    "\n",
    "-------------------------------------------------------------------------------------------------------------\n",
    " **I hope you find this kernel helpful and some upvotes would be very much appreciated**\n",
    " \n",
    " -----------\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "85b27cf82d3023fd69c338df2be7afb2d7afaf32"
   },
   "source": [
    " <a id=\"0\"></a> <br>\n",
    "**Notebook Content**\n",
    "1. [Python](#1)\n",
    "    1. [Basics](#2)\n",
    "    1. [Functions](#3)\n",
    "    1. [Types and Sequences](#4)\n",
    "    1. [More on Strings](#5)\n",
    "    1. [Reading and Writing CSV files](#6)\n",
    "    1. [Dates and Times](#7)\n",
    "    1. [Objects and map()](#8)\n",
    "    1. [Lambda and List Comprehensions](#9)\n",
    "    1. [OOP](#10)\n",
    "1. [Python Packages](#11)\n",
    "    1. [Numpy](#12)\n",
    "        1. [Creating Arrays](#13)\n",
    "        1. [Combining Arrays](#14)\n",
    "        1. [Operations](#15)\n",
    "        1. [Math Functions](#16)\n",
    "        1. [Indexing / Slicing](#17)\n",
    "        1. [Copying Data](#18)\n",
    "        1. [Iterating Over Arrays](#19)\n",
    "        1. [The Series Data Structure](#20)\n",
    "        1. [Querying a Series](#21)\n",
    "    1. [Pandas](#22)\n",
    "        1. [The DataFrame Data Structure](#22)\n",
    "        1. [Dataframe Indexing and Loading](#23)\n",
    "        1. [Missing values](#24)\n",
    "        1. [Merging Dataframes](#25)\n",
    "        1. [Making Code Pandorable](#26)\n",
    "        1. [Group by](#27)\n",
    "        1. [Scales](#28)\n",
    "        1. [Pivot Tables](#29)\n",
    "        1. [Date Functionality](#30)\n",
    "        1. [Distributions in Pandas](#31)\n",
    "        1. [Hypothesis Testing](#32)\n",
    "    1. [Matplotlib](#33)\n",
    "        1. [Scatterplots](#34)\n",
    "        1. [Line Plots](#35)\n",
    "        1. [Bar Charts](#36)\n",
    "        1. [Histograms](#37)\n",
    "        1. [Box and Whisker Plots](#38)\n",
    "        1. [Heatmaps](#39)\n",
    "        1. [Animations](#40)\n",
    "        1. [Interactivity](#41)\n",
    "        1. [DataFrame.plot](#42)\n",
    "    1. [seaborn](#43)\n",
    "    1. [SKlearn](#44)\n",
    "        1. [Hypothesis Testing](#45)\n",
    "1. [Mathematics and Linear Algebra](#46)\n",
    "1. [Programming & Analysis Tools](#47)\n",
    "    1. [Hypothesis Testing](#48)\n",
    "1. [Big Data](#49)\n",
    "1. [Data visualization](#50)\n",
    "1. [Data Cleaning](#51)\n",
    "1. [How to solve Problem?](#52)\n",
    "1. [Machine Learning](#53)\n",
    "1. [Deep Learning](#54)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5efeff35ad9951e40551d0763eaf26f08bb4119e"
   },
   "source": [
    " <a id=\"1\"></a> <br>\n",
    "# 1-Python\n",
    "Python is a modern, robust, high level programming language. It is very easy to pick up even if you are completely new to programming.\n",
    "\n",
    "- The __open source__ Python ecosystem provides __a standalone, versatile and powerful scientific working environment__, including: [NumPy](http://numpy.org), [SciPy](http://scipy.org), [IPython](http://ipython.org), [Matplotlib](http://matplotlib.org), [Pandas](http://pandas.pydata.org/), _and many others..._\n",
    "\n",
    "\n",
    "\n",
    "- Scikit-Learn builds upon NumPy and SciPy and __complements__ this scientific environment with machine learning algorithms;\n",
    "- By design, Scikit-Learn is __non-intrusive__, easy to use and easy to combine with other libraries;\n",
    "- Core algorithms are implemented in low-level languages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a276b26993c902362e522d8a29a67ef670db103d"
   },
   "source": [
    "<a id=\"2\"></a> <br>\n",
    "# 1-2 Python: Basics\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Zen of Python, by Tim Peters\n",
      "\n",
      "Beautiful is better than ugly.\n",
      "Explicit is better than implicit.\n",
      "Simple is better than complex.\n",
      "Complex is better than complicated.\n",
      "Flat is better than nested.\n",
      "Sparse is better than dense.\n",
      "Readability counts.\n",
      "Special cases aren't special enough to break the rules.\n",
      "Although practicality beats purity.\n",
      "Errors should never pass silently.\n",
      "Unless explicitly silenced.\n",
      "In the face of ambiguity, refuse the temptation to guess.\n",
      "There should be one-- and preferably only one --obvious way to do it.\n",
      "Although that way may not be obvious at first unless you're Dutch.\n",
      "Now is better than never.\n",
      "Although never is often better than *right* now.\n",
      "If the implementation is hard to explain, it's a bad idea.\n",
      "If the implementation is easy to explain, it may be a good idea.\n",
      "Namespaces are one honking great idea -- let's do more of those!\n"
     ]
    }
   ],
   "source": [
    "import this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables\n",
    "A name that is used to denote something or a value is called a variable. In python, variables can be declared and values can be assigned to it as follows,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = 2\n",
    "y = 5\n",
    "xy = 'Hey'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Symbol | Task Performed |\n",
    "|----|---|\n",
    "| +  | Addition |\n",
    "| -  | Subtraction |\n",
    "| /  | division |\n",
    "| %  | mod |\n",
    "| *  | multiplication |\n",
    "| //  | floor division |\n",
    "| **  | to the power of |\n",
    "\n",
    "### Relational Operators\n",
    "| Symbol | Task Performed |\n",
    "|----|---|\n",
    "| == | True, if it is equal |\n",
    "| !=  | True, if not equal to |\n",
    "| < | less than |\n",
    "| > | greater than |\n",
    "| <=  | less than or equal to |\n",
    "| >=  | greater than or equal to |\n",
    "### Bitwise Operators\n",
    "| Symbol | Task Performed |\n",
    "|----|---|\n",
    "| &  | Logical And |\n",
    "| l  | Logical OR |\n",
    "| ^  | XOR |\n",
    "| ~  | Negate |\n",
    "| >>  | Right shift |\n",
    "| <<  | Left shift |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "281c637be82f68519f371353afa603fe842bd5fa"
   },
   "source": [
    "<a id=\"3\"></a> <br>\n",
    "# 1-3 Python : Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b513caa4821bc20e6b833577ba4d894bce011e32"
   },
   "source": [
    "<br>\n",
    "`add_numbers` is a function that takes two numbers and adds them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "700ebb9c4d8c7c57948194e3d848b4b480e06f9a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_numbers(x, y):\n",
    "    return x + y\n",
    "\n",
    "add_numbers(1, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "0fb081fa2f33e01e616a82aad306ab6490f48a03"
   },
   "source": [
    "<br>\n",
    "`add_numbers` updated to take an optional 3rd parameter. Using `print` allows printing of multiple expressions within a single cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b7dd7bc4f7f52acac95ac94586c74f4b59a22acb",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_numbers(x,y,z=None):\n",
    "    if (z==None):\n",
    "        return x+y\n",
    "    else:\n",
    "        return x+y+z\n",
    "\n",
    "print(add_numbers(1, 2))\n",
    "print(add_numbers(1, 2, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "51d91959b8088baa0884eaf191edc9520e012625"
   },
   "source": [
    "<br>\n",
    "`add_numbers` updated to take an optional flag parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "8f109910dd7dacbac0212d03b3a0d55e8d92b4bf",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_numbers(x, y, z=None, flag=False):\n",
    "    if (flag):\n",
    "        print('Flag is true!')\n",
    "    if (z==None):\n",
    "        return x + y\n",
    "    else:\n",
    "        return x + y + z\n",
    "    \n",
    "print(add_numbers(1, 2, flag=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "052a1ebfb11cba627a4ed7ae13360e15cfd2f5bc"
   },
   "source": [
    "<br>\n",
    "Assign function `add_numbers` to variable `a`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "774773afde862fefb51b22d9e676b44744410c6f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_numbers(x,y):\n",
    "    return x+y\n",
    "\n",
    "a = add_numbers\n",
    "a(1,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "54244074f574884d9c241727f559466cdc646793"
   },
   "source": [
    "<a id=\"4\"></a> <br>\n",
    "# 1-4 Python : Types and Sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "fe3bfa4b99b20b534a24cd4b04953de4c81a2032"
   },
   "source": [
    "<br>\n",
    "Use `type` to return the object's type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ea5486e825425487854b66e135142787a8696547",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type('This is a string')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6b1b6307a4b2363c5f13c86a94a6c6ed83cf05d1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type(None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1bfe51959705a633d7ae10b8e807ba752809f8a2",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9c6ea01bf5fac295d6ca5dea7a867fd769d9854f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type(1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "08e7ecceb2d59d5964d039b1307736a106abef30",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type(add_numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "843a9232374917b366c4afadc94fd123e13bc2cf"
   },
   "source": [
    "<br>\n",
    "Tuples are an immutable data structure (cannot be altered)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "02c0f949470c05c302705b8cbde9dbc4c5e9ec58",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = (1, 'a', 2, 'b')\n",
    "type(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "32e3d57ead5741f615aaf0ec276583163a6fc1a7"
   },
   "source": [
    "<br>\n",
    "Lists are a mutable data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e559da4fdd4adcf955e36ee33209728c7f04c873",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = [1, 'a', 2, 'b']\n",
    "type(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1c25590aa57c37e383007ea43e9aebd08dbb7ef8"
   },
   "source": [
    "<br>\n",
    "Use `append` to append an object to a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "46b3996bb68391469648cfaf732d9f891aad8e76",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x.append(3.3)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a0a9f9ad271ff5cab43f7d95c00f917dfa42cda3"
   },
   "source": [
    "<br>\n",
    "This is an example of how to loop through each item in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "fdbcbc6f491284146fb7ec2288a48dca70d028f0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for item in x:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a863bf6c33d37823812c0d373bc685f8018328c3"
   },
   "source": [
    "<br>\n",
    "Or using the indexing operator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "34ebdc7a81ca55c6d96576e8e1032bb2e2a8c38d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "i=0\n",
    "while( i != len(x) ):\n",
    "    print(x[i])\n",
    "    i = i + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7accd0a14f6f94b5f8f87c28e772fd7e70f5aa89"
   },
   "source": [
    "<br>\n",
    "Use `+` to concatenate lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2e2d33c027d2d2e66444ea8c5bc16ac182604699",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[1,2] + [3,4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8dac836d3bfd72d4ba18e0ad35e918fd5578e0c4"
   },
   "source": [
    "<br>\n",
    "Use `*` to repeat lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "33fbab44facf2994b9a1f4f842b5b68db790c60c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "[1]*3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1bf6aa261fae9ee56cc8e2ec7ba789818619bfc7"
   },
   "source": [
    "<br>\n",
    "Use the `in` operator to check if something is inside a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9dcaf39b71a922a3a61f6565cb9025fe89291cc8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "1 in [1, 2, 3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2fdc7e3a4da86bf2fc3daef510d521ff4f9e780a"
   },
   "source": [
    "<br>\n",
    "Now let's look at strings. Use bracket notation to slice a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "31a527ad3d4ac52203a81f2aa05d6ccbcbefd7d0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = 'This is a string'\n",
    "print(x[0]) #first character\n",
    "print(x[0:1]) #first character, but we have explicitly set the end character\n",
    "print(x[0:2]) #first two characters\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a2da025858712f120a1708fa8de0da26c6b8b1de"
   },
   "source": [
    "<br>\n",
    "This will return the last element of the string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "24cb16eaabaa7e13b0a4a69dfc6cdf71ffb0238b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "892cb74f3b1ba32053e1b854681e9520d9ae7578"
   },
   "source": [
    "<br>\n",
    "This will return the slice starting from the 4th element from the end and stopping before the 2nd element from the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b947154182b9eb7aa08f5986eb768020e9eefdcb",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x[-4:-2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7d88463b1e8c9555030af32e1e99cdd522c40610"
   },
   "source": [
    "<br>\n",
    "This is a slice from the beginning of the string and stopping before the 3rd element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "961b4829e231286702b78c13f6ce68c780df7c56",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d365c72e42116bbd6c7ae960d77af0d89d35f201"
   },
   "source": [
    "<br>\n",
    "And this is a slice starting from the 3rd element of the string and going all the way to the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "82c83cb0bfe09b269037a97ef0b00cd66254ab89",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x[3:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b3914291465c80995a05d30f2842f38b1e2d7255",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "firstname = 'MJ'\n",
    "lastname = 'Bahmani'\n",
    "\n",
    "print(firstname + ' ' + lastname)\n",
    "print(firstname*3)\n",
    "print('mj' in firstname)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5e92142f11b79daaa369a0a031bab1979602457e"
   },
   "source": [
    "<br>\n",
    "`split` returns a list of all the words in a string, or a list split on a specific character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "63a7aea0881d73a1c2caab2b2bc0bf09d46e0402",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "firstname = 'Mr Dr Mj Bahmani'.split(' ')[0] # [0] selects the first element of the list\n",
    "lastname = 'Mr Dr Mj Bahmani'.split(' ')[-1] # [-1] selects the last element of the list\n",
    "print(firstname)\n",
    "print(lastname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a46edc198922ad4957bcdc3d990f43aaae35a464",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'MJ' + str(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "50a1372d759bcc6e0b4d8852588ba9fe1c3a1200"
   },
   "source": [
    "<br>\n",
    "Dictionaries associate keys with values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b6dd088e810aca788fc74f76d23777b2115e12d0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = {'MJ Bahmani': 'Mohamadjavad.bahmani@gmail.com', 'irmatlab': 'irmatlab.ir@gmail.com'}\n",
    "x['MJ Bahmani'] # Retrieve a value by using the indexing operator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6ec5106de063d06587a532835173cb3e15c4c074",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x['MJ Bahmani'] = None\n",
    "x['MJ Bahmani']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "909deaebc50f94808fbdbe831ffbe1137066a2fe"
   },
   "source": [
    "<br>\n",
    "Iterate over all of the keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "993e0c88238de00475ddd22a2f4594ce14aa936d",
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for name in x:\n",
    "    print(x[name])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "9248f6bfe90a9c867ca248b08ad81c8626eaf689"
   },
   "source": [
    "<br>\n",
    "Iterate over all of the values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9f995195b06da4b3e64cabbe950d19570fbb910c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for email in x.values():\n",
    "    print(email)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "147f360dd1e8f878ae393c46648ec5f630d8d80c"
   },
   "source": [
    "<br>\n",
    "Iterate over all of the items in the list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ab973ccad15b9f3af27e6a2cbd2425ede962be86",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for name, email in x.items():\n",
    "    print(name)\n",
    "    print(email)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "3a909b20788d5df862adec16c2d6d15e021a74dc"
   },
   "source": [
    "<br>\n",
    "You can unpack a sequence into different variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "296600127a5b164c23ad4aad0e622883f6ce56b1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = ('MJ', 'Bahmani', 'Mohamadjavad.bahmani@gmail.com')\n",
    "fname, lname, email = x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b1d783512fd2e8d76b8d1c147ed51f8584d203b7",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fname"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a69c79b2a267725470ff843205f11c8a6e2a820e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lname"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6b069e526dd64ce6efba8cb51854ef2636e308b9"
   },
   "source": [
    "<a id=\"5\"></a> <br>\n",
    "# 1-5 Python: More on Strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2aa934d05afb73cd7cdcb9ff223b2014798f3b18",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print('MJ' + str(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2a9b5f81e7ae92846e4bce5c8c34ff23490b83ef"
   },
   "source": [
    "<br>\n",
    "Python has a built in method for convenient string formatting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a8fa5e6a70f8a2e642f0799b514d96f5e8b4cf67",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sales_record = {\n",
    "'price': 3.24,\n",
    "'num_items': 4,\n",
    "'person': 'MJ'}\n",
    "\n",
    "sales_statement = '{} bought {} item(s) at a price of {} each for a total of {}'\n",
    "\n",
    "print(sales_statement.format(sales_record['person'],\n",
    "                             sales_record['num_items'],\n",
    "                             sales_record['price'],\n",
    "                             sales_record['num_items']*sales_record['price']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7fee00e11e78abee9a0397711ca813c60b1b8c57"
   },
   "source": [
    "<a id=\"6\"></a> <br>\n",
    "# 1-6 Python:Reading and Writing CSV files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "3f79954fc298a34c3c231d35161dfb3bf743576d"
   },
   "source": [
    "<br>\n",
    "Let's import our datafile train.csv \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ab5ab332ba1dc0fb1668d18df1b1771232cc4e35",
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "%precision 2\n",
    "\n",
    "with open('../input/train.csv') as csvfile:\n",
    "    train = list(csv.DictReader(csvfile))\n",
    "    \n",
    "train[:1] # The first three dictionaries in our list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ac3ffa35fd8725c3b1d81ab92470f1e5de19061b"
   },
   "source": [
    "<br>\n",
    "`csv.Dictreader` has read in each row of our csv file as a dictionary. `len` shows that our list is comprised of 234 dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "3f51c4e51000c69c3b3113f4cc8ed1e5e5fa005f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "59457b14190fe59e900f644450b66f542b2320c4"
   },
   "source": [
    "<br>\n",
    "`keys` gives us the column names of our csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "de6d753e74e0345bcc292cb6cfded49241b696f8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train[0].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "391e4c865b352e70ae7bc5cbdef9559ccf396cb8"
   },
   "source": [
    "<br>\n",
    "How to do some math action on the data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c654cdc458ff6f8d960caa80268f61e9f213e9e8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sum(float(d['Fare']) for d in train) / len(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a12ec3e5c23e0660c9cfce427333afabe5a4384c"
   },
   "source": [
    "<br>\n",
    "Use `set` to return the unique values for the type of Sex  in our dataset have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "3541bdc6bd01a88c58e408bcc99411136a953538",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Sex = set(d['Sex'] for d in train)\n",
    "Sex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "51bd65ce547fceba323660a1370a407d004e9abe"
   },
   "source": [
    "<a id=\"7\"></a> <br>\n",
    "# 1-7 Python: Dates and Times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "77aea246af2c06fe57b72bf45e424a4a6ae2e8c5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "import time as tm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f14fbbf5385512cc6d6631677a62ff620d51f9ae"
   },
   "source": [
    "<br>\n",
    "`time` returns the current time in seconds since the Epoch. (January 1st, 1970)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "73ffa7423c31ed5efb8bab44ead79a4162711920",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tm.time()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "83a2aceba5919534adf74e59c16baa61d6e24bda"
   },
   "source": [
    "<br>\n",
    "Convert the timestamp to datetime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b0726eaf4ebb492d3da2396520a7328bbb00263c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dtnow = dt.datetime.fromtimestamp(tm.time())\n",
    "dtnow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "df2a8c0e8237c6c88ea5d8f446604d6c60435b72"
   },
   "source": [
    "<br>\n",
    "Handy datetime attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "3ca2dd821f01874e23fc1eccf13b6584ba21e795",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dtnow.year, dtnow.month, dtnow.day, dtnow.hour, dtnow.minute, dtnow.second # get year, month, day, etc.from a datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "85160de1a52833048050277d7df6713172b367c9"
   },
   "source": [
    "<br>\n",
    "`timedelta` is a duration expressing the difference between two dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c466ccf8a9f33641fec51525f0e8a03dbec8fd62",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "delta = dt.timedelta(days = 100) # create a timedelta of 100 days\n",
    "delta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c09e3bd1c17b5a67b8e340956fdb27b2e6b32b2d"
   },
   "source": [
    "<br>\n",
    "`date.today` returns the current local date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "539408df43c0aa1b449551e14b84b49110b04beb",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "today = dt.date.today()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4022bf4458c6495ab1c4ddfcf82c155742641823",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "today - delta # the date 100 days ago"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "38399dc2975b4f44e083ae319f37217286d9533a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "today > today-delta # compare dates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a8bcac1b08550c3f143e21cf7ca9049449304a5c"
   },
   "source": [
    "<a id=\"8\"></a> <br>\n",
    "# 1-8 Python: Objects and map()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "28be3905d37e6b7d2fb09a5f1a0e0c9df7e48559"
   },
   "source": [
    "<br>\n",
    "An example of a class in python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2dc3faf4892b932cb1015902e83a13bd47125b20",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Person:\n",
    "    department = 'School of Information' #a class variable\n",
    "\n",
    "    def set_name(self, new_name): #a method\n",
    "        self.name = new_name\n",
    "    def set_location(self, new_location):\n",
    "        self.location = new_location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1da9aaf6d97f50fff89e9d08b658adb3c691a262",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "person = Person()\n",
    "person.set_name('MJ Bahmani')\n",
    "person.set_location('MI, Berlin, Germany')\n",
    "print('{} live in {} and works in the department {}'.format(person.name, person.location, person.department))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8422c57273a71b4a74b6b4a9eb5535444356823d"
   },
   "source": [
    "<br>\n",
    "Here's an example of mapping the `min` function between two lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "3052442e823ee0a450e404607e44cb3c321b508c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store1 = [10.00, 11.00, 12.34, 2.34]\n",
    "store2 = [9.00, 11.10, 12.34, 2.01]\n",
    "cheapest = map(min, store1, store2)\n",
    "cheapest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "787b960d3c824c764a0eb7c2dab69d073eacfa9d"
   },
   "source": [
    "<br>\n",
    "Now let's iterate through the map object to see the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a9d6751e7c95241dbd10a44f0179a40df8ee3a7a",
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for item in cheapest:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e9c6c70988cd1459c7078f31fb9a43a275965d46"
   },
   "source": [
    "<a id=\"9\"></a> <br>\n",
    "# 1-9-Python : Lambda and List Comprehensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b6257386671f84333b4cfae3a979a2c1f8100e76"
   },
   "source": [
    "<br>\n",
    "Here's an example of lambda that takes in three parameters and adds the first two."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a306ae5687e20ae64431306035359f33aa465382",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_function = lambda a, b, c : a + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6bf98de2fab3b4239e41a093386d5fb9acad6367",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_function(1, 2, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5db80d702a02a1873aa09fa3acd26d922d74ff46"
   },
   "source": [
    "<br>\n",
    "Let's iterate from 0 to 999 and return the even numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "90e7eb5e7689d6ce448b8c0c801c2caef70e6d44",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_list = []\n",
    "for number in range(0, 1000):\n",
    "    if number % 2 == 0:\n",
    "        my_list.append(number)\n",
    "my_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1bfe68a8553e5718a04b14a321262e6e88407b2c"
   },
   "source": [
    "<br>\n",
    "Now the same thing but with list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ee501b9820296d1452845b8da201dbd69d4e0637",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_list = [number for number in range(0,1000) if number % 2 == 0]\n",
    "my_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e99f98ecb3393b8846fca621cdbad5dcaf1c60d1"
   },
   "source": [
    "<a id=\"10\"></a> <br>\n",
    "# 1-10 OOP\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class FirstClass:\n",
    "    test = 'test'\n",
    "    def __init__(self,name,symbol):\n",
    "        self.name = name\n",
    "        self.symbol = symbol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "eg3 = FirstClass('Three',3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print eg3.test, eg3.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class FirstClass:\n",
    "    def __init__(self,name,symbol):\n",
    "        self.name = name\n",
    "        self.symbol = symbol\n",
    "    def square(self):\n",
    "        return self.symbol * self.symbol\n",
    "    def cube(self):\n",
    "        return self.symbol * self.symbol * self.symbol\n",
    "    def multiply(self, x):\n",
    "        return self.symbol * x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "eg4 = FirstClass('Five',5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print eg4.square()\n",
    "print eg4.cube()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "eg4.multiply(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FirstClass.multiply(eg4,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1-10-1 Inheritance\n",
    "\n",
    "There might be cases where a new class would have all the previous characteristics of an already defined class. So the new class can \"inherit\" the previous class and add it's own methods to it. This is called as inheritance.\n",
    "\n",
    "Consider class SoftwareEngineer which has a method salary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class SoftwareEngineer:\n",
    "    def __init__(self,name,age):\n",
    "        self.name = name\n",
    "        self.age = age\n",
    "    def salary(self, value):\n",
    "        self.money = value\n",
    "        print (self.name,\"earns\",self.money)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = SoftwareEngineer('Kartik',26)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a.salary(40000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dir(SoftwareEngineer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Artist:\n",
    "    def __init__(self,name,age):\n",
    "        self.name = name\n",
    "        self.age = age\n",
    "    def money(self,value):\n",
    "        self.money = value\n",
    "        print self.name,\"earns\",self.money\n",
    "    def artform(self, job):\n",
    "        self.job = job\n",
    "        print self.name,\"is a\", self.job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "b = Artist('Nitin',20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "b.money(50000)\n",
    "b.artform('Musician')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dir(Artist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1a8697f93952e076f6f949997676d40518d7b5a6"
   },
   "source": [
    "<a id=\"11\"></a> <br>\n",
    "# 2-Python Packages\n",
    "* Numpy\n",
    "* Pandas\n",
    "* Matplotlib\n",
    "* Seaborn\n",
    "* Sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "bfb701e45e93aea0b3ed64e148ca2fdb53559038",
    "collapsed": true
   },
   "source": [
    "<a id=\"12\"></a> <br>\n",
    "# 2-1 Numerical Python (NumPy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "db9a850ebb440ca960a0713d822e20090bc10601",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "79699175b4559f509181d359393167f801735485"
   },
   "source": [
    "<a id=\"13\"></a> <br>\n",
    "## 2-1-1 NumPy :Creating Arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "fb5123cfa4687a819758ea82810984fa69d631e3"
   },
   "source": [
    "Create a list and convert it to a numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1cdc9404b31261269891723ddc59064802063041",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mylist = [1, 2, 3]\n",
    "x = np.array(mylist)\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "85123a6bf0589918ff03fe9916b06635fa32b776"
   },
   "source": [
    "<br>\n",
    "Or just pass in a list directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "dca7dd9319716e863760bb7c4e1e47a1c17d7b1b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = np.array([4, 5, 6])\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5a29d1e6bb19131b3bde9ae197b562cf5c905f2a"
   },
   "source": [
    "<br>\n",
    "Pass in a list of lists to create a multidimensional array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e18e77b6d1becf1b7ded3a4daa361dbe3f985d96",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m = np.array([[7, 8, 9], [10, 11, 12]])\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "956c870e985074aadd2aa97b8a4c820c64bd7d2b"
   },
   "source": [
    "<br>\n",
    "Use the shape method to find the dimensions of the array. (rows, columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7e5472233313b1eb806b7e9dfd2478f4155b23d0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "m.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d20433435f865bf4593ee8b5eae4cb2173794a57"
   },
   "source": [
    "<br>\n",
    "`arange` returns evenly spaced values within a given interval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "87aded8cbe232bfcb4ce246ecf1bac0b70d9477e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n = np.arange(0, 30, 2) # start at 0 count up by 2, stop before 30\n",
    "n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ef6d0651972cb576188cd986830aae2d53bb20b8"
   },
   "source": [
    "<br>\n",
    "`reshape` returns an array with the same data with a new shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6722026f830d545bfd21a2760f80c20e33c0d757",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n = n.reshape(3, 5) # reshape array to be 3x5\n",
    "n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a47a0e9b094d81ae9ec235cd33d395ca34c379b3"
   },
   "source": [
    "<br>\n",
    "`linspace` returns evenly spaced numbers over a specified interval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "099d0b4b7f3c6e5aef6cebd760e3c6b8111205d8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "o = np.linspace(0, 4, 9) # return 9 evenly spaced values from 0 to 4\n",
    "o"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e5abb93fd66c344624ad2bb5e91ec1cd8a9ab220"
   },
   "source": [
    "<br>\n",
    "`resize` changes the shape and size of array in-place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "66de4ad6fbb8a2ffe7251b667de732fe0048c39c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "o.resize(3, 3)\n",
    "o"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "95af33fa252b8ccc4afad8666e0a7ced2d83aee9"
   },
   "source": [
    "<br>\n",
    "`ones` returns a new array of given shape and type, filled with ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "015e5a604fceb5b335760d85f090812416ab0edd",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.ones((3, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "415ba7eb6ffcca82d1ad73b6e652d77c2b40c4c7"
   },
   "source": [
    "<br>\n",
    "`zeros` returns a new array of given shape and type, filled with zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4833b4502eac60ec53ca4ab55f10653103be3584",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.zeros((2, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "acbef4f6ab93b0ea5ed8bea30ec0eddbce8415f1"
   },
   "source": [
    "<br>\n",
    "`eye` returns a 2-D array with ones on the diagonal and zeros elsewhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "5e4e8112e5829b290385bd076041d9d0c43bdd4e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.eye(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "bb258a18b84c56c26c19e5e467f8886a3209eb6d"
   },
   "source": [
    "<br>\n",
    "`diag` extracts a diagonal or constructs a diagonal array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0167bbb06dc5175a14e2d1844b3e92c3f601e156",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.diag(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "05f0ea0d69c70181a0ddee0b0a0670afc1a5761c"
   },
   "source": [
    "<br>\n",
    "Create an array using repeating list (or see `np.tile`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "76f7c4dd56d1f9ba9c8dcc3bf3b0e67bc62326de",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.array([1, 2, 3] * 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e9d6935951d0df2a4d60f66b302da7b876006c71"
   },
   "source": [
    "<br>\n",
    "Repeat elements of an array using `repeat`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7299b4990895d4ea8a59447489fa6fbc0cde5ea6",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.repeat([1, 2, 3], 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "01250c2ea726f387db5f3fab5a004e741e574a35"
   },
   "source": [
    "<a id=\"14\"></a> <br>\n",
    "## 2-1-2 Numpy:Combining Arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0dc57919256360d1ac309813fb6e836f75d17484",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "p = np.ones([2, 3], int)\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "98aa86e98478bcad2ce35eded8e2adb825bbf709"
   },
   "source": [
    "<br>\n",
    "Use `vstack` to stack arrays in sequence vertically (row wise)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "48b0f8194df6a4f63932d97b24883d6a5d69d0df",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.vstack([p, 2*p])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a2cf2eb6fdf3deccca31df9979dd978241adadb8"
   },
   "source": [
    "<br>\n",
    "Use `hstack` to stack arrays in sequence horizontally (column wise)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "834984b63056d7161f261d868d85b83f24403287",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.hstack([p, 2*p])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b82f24eec48ab5206a47d04a36e5e639dad7f9a1"
   },
   "source": [
    "<a id=\"15\"></a> <br>\n",
    "## 2-1-3 Numpy:Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d58a4bb35b9f1525736b28ffc7e2b6ead4035266"
   },
   "source": [
    "Use `+`, `-`, `*`, `/` and `**` to perform element wise addition, subtraction, multiplication, division and power."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7ba8b42fcdcdc83e6f43df83b73665aa289e9786",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(x + y) # elementwise addition     [1 2 3] + [4 5 6] = [5  7  9]\n",
    "print(x - y) # elementwise subtraction  [1 2 3] - [4 5 6] = [-3 -3 -3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "cd42ade6bd206750b5dde1086b06bde519f2b1df",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(x * y) # elementwise multiplication  [1 2 3] * [4 5 6] = [4  10  18]\n",
    "print(x / y) # elementwise divison         [1 2 3] / [4 5 6] = [0.25  0.4  0.5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "3370d5b9b8ca04ed5be48331c3d6d4d08738ab5d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(x**2) # elementwise power  [1 2 3] ^2 =  [1 4 9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e665d9db501694e29f54b02de56befaf69305629"
   },
   "source": [
    "<br>\n",
    "**Dot Product:**  \n",
    "\n",
    "$ \\begin{bmatrix}x_1 \\ x_2 \\ x_3\\end{bmatrix}\n",
    "\\cdot\n",
    "\\begin{bmatrix}y_1 \\\\ y_2 \\\\ y_3\\end{bmatrix}\n",
    "= x_1 y_1 + x_2 y_2 + x_3 y_3$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b45a94567cf5801e05869bde614e831603b1599a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x.dot(y) # dot product  1*4 + 2*5 + 3*6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b27ce9c3b1328461ba14f5efdd36079e3f827951",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z = np.array([y, y**2])\n",
    "print(len(z)) # number of rows of array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "30ab350aa70687cbdd087d86386b96504aec4479"
   },
   "source": [
    "<br>\n",
    "Let's look at transposing arrays. Transposing permutes the dimensions of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2630bf81db6d00483f7e0abd3a3c3af28144d55f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z = np.array([y, y**2])\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8bb90454186cab0c688be9a01ff84c7ca67fa6a9"
   },
   "source": [
    "<br>\n",
    "The shape of array `z` is `(2,3)` before transposing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "bdcbfc2e1c9b985a83aefa1a5972aa919acb8365",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "3b796b3d6c4e9a90e4fb5332d708398bb848c2e4"
   },
   "source": [
    "<br>\n",
    "Use `.T` to get the transpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "8ca46bf46ed15a5fa3af7e2d325ac68bc8785f05",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d2949665fb40cdb932376219b6d78156265d5ebf"
   },
   "source": [
    "<br>\n",
    "The number of rows has swapped with the number of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e48e8eff60259a9c865cf2dae88a3ce1641e826c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z.T.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "08fac1d78a66fdd513372cd31eaa5f973655e80a"
   },
   "source": [
    "<br>\n",
    "Use `.dtype` to see the data type of the elements in the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6e1ea214c877f23d670b72f06060c2dee1bdaee5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "70da3bcabbc0631bb11b48796106c07f4de41ba8"
   },
   "source": [
    "<br>\n",
    "Use `.astype` to cast to a specific type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "60684ed5f18c88c4ef4e2fe4207fa94eede9d993",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "z = z.astype('f')\n",
    "z.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e1199442dd07a4a4eef965c584d9e0f443c40013"
   },
   "source": [
    "<a id=\"16\"></a> <br>\n",
    "## 2-1-4 Numpy: Math Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "01121300b7b5b8a83d213d4f065383da19d5f7d8"
   },
   "source": [
    "Numpy has many built in math functions that can be performed on arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e1eaeb06cf68d055f6a5536ea72a17606b6762c1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = np.array([-4, -2, 1, 3, 5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2918e83be55935fa03fd24924bc7f07a271c40d3",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2e86a6ea98c6c86dbe7a96612af4f37452b23670",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1e8e48425f65e90a18a29fb7983dc6aa424f8445",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "27131855b2c26a5f90c6524d4a8c7ed0266bf378",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a020531df7f87af15577e986699e80f892300773",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6d26c78ddd00e9387ba4214470debcc8147fd2bc"
   },
   "source": [
    "<br>\n",
    "`argmax` and `argmin` return the index of the maximum and minimum values in the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "dd6f6aee91fc8dd99cc5a8359dc81ce1d443e77f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a.argmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "59aeec1ba92a0eb4c6928564e64445e6ce46cc3c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a.argmin()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "cf3ad4800506fd903882f60d811ca2548756e7c8"
   },
   "source": [
    "<a id=\"17\"></a> <br>\n",
    "\n",
    "## 2-1-5 Numpy:Indexing / Slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "82ca4b616a46de3280b5a50df4c0114298d07aea",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s = np.arange(13)**2\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "417c1b56b9d3eca21021422bde199036c3f08ec3"
   },
   "source": [
    "<br>\n",
    "Use bracket notation to get the value at a specific index. Remember that indexing starts at 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6f90f5dee19a73afde4e9455b47c0dbe86c9ce6b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s[0], s[4], s[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "9d0cf8a30f9b1bd1f8dc60a46d3271e5cac235d0"
   },
   "source": [
    "<br>\n",
    "Use `:` to indicate a range. `array[start:stop]`\n",
    "\n",
    "\n",
    "Leaving `start` or `stop` empty will default to the beginning/end of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1ea11cc5be369a751b250e7085cf507b122bcf88",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s[1:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "dd42e9a4274baebf747c627767a273ef8ca9a26f"
   },
   "source": [
    "<br>\n",
    "Use negatives to count from the back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ce51a60f59516b22174f97a4f7c6da6c75322a8b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s[-4:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "0fdd94c22e7ceb4cac00001bd98f09d5d879613e"
   },
   "source": [
    "<br>\n",
    "A second `:` can be used to indicate step-size. `array[start:stop:stepsize]`\n",
    "\n",
    "Here we are starting 5th element from the end, and counting backwards by 2 until the beginning of the array is reached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f28245bea6ab9a1bed7a76859ddfbb295ea95038",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s[-5::-2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2f2295df96d9e7da01cde82bf87d5c7157bd9d21"
   },
   "source": [
    "<br>\n",
    "Let's look at a multidimensional array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "582e15f695a2891826ef4eb8e913a0dc61913898",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r = np.arange(36)\n",
    "r.resize((6, 6))\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "26fd89ebfe4b92cedad89152753918608b68eafe"
   },
   "source": [
    "<br>\n",
    "Use bracket notation to slice: `array[row, column]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7bf0f1e77c5243c3e2b846f7b268c4891786621b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r[2, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "fcbf322288c6671ef799299b5ddb828b68b8cab9"
   },
   "source": [
    "<br>\n",
    "And use : to select a range of rows or columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "64d430ac4626977723c6357865a5297699e5323d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r[3, 3:6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e109144f7316b357450c4b3cae3cb61ad327f5d9"
   },
   "source": [
    "<br>\n",
    "Here we are selecting all the rows up to (and not including) row 2, and all the columns up to (and not including) the last column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7b80dc3ba5c970afe2db37c23599d476399b1c68",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r[:2, :-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6c5126aa54a1e6b9d0d078f2226b68d3fb42c4a2"
   },
   "source": [
    "<br>\n",
    "This is a slice of the last row, and only every other element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "054f80d47b0e2d249a9140321846ff7b50daba1f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r[-1, ::2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "62e5354b353e1fe2e60036f2c3f94909a8af1b7b"
   },
   "source": [
    "<br>\n",
    "We can also perform conditional indexing. Here we are selecting values from the array that are greater than 30. (Also see `np.where`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "5476c43a5e7d195804cac7517abbdbcdbc234829",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r[r > 30]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "77ae22731783157aaecdd019be77004504b943af"
   },
   "source": [
    "<br>\n",
    "Here we are assigning all values in the array that are greater than 30 to the value of 30."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6651ea99641526b7df83e7a9d9c1efe4eab9dc99",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r[r > 30] = 30\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6189d2e7dc9038dc1c4f2ed95a47fad76d700890"
   },
   "source": [
    "<a id=\"18\"></a> <br>\n",
    "## 2-1-6 Numpy :Copying Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ad2453d83a8db6de311e9265672e9b10b0337284"
   },
   "source": [
    "Be careful with copying and modifying arrays in NumPy!\n",
    "\n",
    "\n",
    "`r2` is a slice of `r`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e6bb5dab7586dd4cfbc93b2cee05c57aae1e8518",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r2 = r[:3,:3]\n",
    "r2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "65d42e0a4cade74ae9b2b587f182efaa8a4b1dbf"
   },
   "source": [
    "<br>\n",
    "Set this slice's values to zero ([:] selects the entire array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9026a103a457054124258eefca3492008f884ef4",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r2[:] = 0\n",
    "r2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f73d4004c9181b5cffddf6f13b452d7e40cea3f9"
   },
   "source": [
    "<br>\n",
    "`r` has also been changed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "80c0d26d5cb0374f82929e3d79c183f5e116f4ea",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c8a21fd94e5dfcd322ec0017e89302533ac3cf2d"
   },
   "source": [
    "<br>\n",
    "To avoid this, use `r.copy` to create a copy that will not affect the original array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "d2b2e17295f75a5a3b8dacf5ca65b4c4f3b6ca47",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r_copy = r.copy()\n",
    "r_copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2ef3568f7ac932cc45931854410cbb9e4d909df8"
   },
   "source": [
    "<br>\n",
    "Now when r_copy is modified, r will not be changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4f7fa9e1b65eea1e9514fbd6f91a2720822430ba",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r_copy[:] = 10\n",
    "print(r_copy, '\\n')\n",
    "print(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "752b09b0f8f1606d55bf0423d3f2d9cb162d3ce9"
   },
   "source": [
    "<a id=\"19\"></a> <br>\n",
    "## 2-1-7 Numpy: Iterating Over Arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6a7f8c639e5f8cbdbd0cf2fe4c78504955ec2ccb"
   },
   "source": [
    "Let's create a new 4 by 3 array of random numbers 0-9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c1496dab8d8f90434e8c44e63c225bfb3ca9713f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test = np.random.randint(0, 10, (4,3))\n",
    "test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "bf971c58819a069592b3d5c195cf9e21faa4797b"
   },
   "source": [
    "<br>\n",
    "Iterate by row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "993bb7b7f1be5a0b088caac550c32257dd1c9297",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for row in test:\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f5b7fa4289acc25093efbbc99fe36355073bcd02"
   },
   "source": [
    "<br>\n",
    "Iterate by index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9de357edca10cf7708e1c5b37ef5a0ad337fcbbf",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in range(len(test)):\n",
    "    print(test[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "00ed3013376dff60b43f6f1eb5091580279063ad"
   },
   "source": [
    "<br>\n",
    "Iterate by row and index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a75a7881baf87a49a3f48236fc9a9281f2ace310",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i, row in enumerate(test):\n",
    "    print('row', i, 'is', row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "08f51011e8d4da3ab72a33ccb6daca3c97832eb4"
   },
   "source": [
    "<br>\n",
    "Use `zip` to iterate over multiple iterables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "63709cf63fc6e5596bc540055a287d56a57c55df",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test2 = test**2\n",
    "test2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "88776121d1063744cdb0d1df15320af35a40690f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i, j in zip(test, test2):\n",
    "    print(i,'+',j,'=',i+j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c2ec9941ed71b0d102881252688723804c536b65"
   },
   "source": [
    "<a id=\"20\"></a> <br>\n",
    "## 2-1-8  Numpy: The Series Data Structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "83019fe62ffed6257a1f1fa975464742b020e135",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ff60c47c0ee85b3534fa0eeb1fc6c18951e13a93",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "animals = ['Tiger', 'Bear', 'Moose']\n",
    "pd.Series(animals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "3b9d3593c2f04eb52439d1e2f6eaced42103b385",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "numbers = [1, 2, 3]\n",
    "pd.Series(numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ac6b145a659c5c6e143e47a726be2d2bc904ea05",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "animals = ['Tiger', 'Bear', None]\n",
    "pd.Series(animals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "608363b045521c88d96135d4651624753a0a97f8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "numbers = [1, 2, None]\n",
    "pd.Series(numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "81c84dd1739a442c3eca83911b9e9cd146beccf1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.nan == None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ee829b3241dc2b99e7aeb8b16daa61f43516a08e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.nan == np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c39a1b7d020fa502a055e4befa580529da3d9206",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.isnan(np.nan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "703803f890a0c351d5122b4f509c0835b949481d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sports = {'Archery': 'Bhutan',\n",
    "          'Golf': 'Scotland',\n",
    "          'Sumo': 'Japan',\n",
    "          'Taekwondo': 'South Korea'}\n",
    "s = pd.Series(sports)\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6a4a03374c688db5bc1cb49cf0ae67a166f33ab5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "16a45c1d82eb06da0e940aa9304455d5b6629723",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s = pd.Series(['Tiger', 'Bear', 'Moose'], index=['India', 'America', 'Canada'])\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "d5c017e5124d3c8c62bbe91b16ae5e2fb76f2cd5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sports = {'Archery': 'Bhutan',\n",
    "          'Golf': 'Scotland',\n",
    "          'Sumo': 'Japan',\n",
    "          'Taekwondo': 'South Korea'}\n",
    "s = pd.Series(sports, index=['Golf', 'Sumo', 'Hockey'])\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "91a03b68b0698ad0f0736d0a8106bb4c2023437d"
   },
   "source": [
    "<a id=\"21\"></a> <br>\n",
    "# 2-1-9 Numpy: Querying a Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6c44f80eadf6e60f8d1d966594169e6e579fd91a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sports = {'Archery': 'Bhutan',\n",
    "          'Golf': 'Scotland',\n",
    "          'Sumo': 'Japan',\n",
    "          'Taekwondo': 'South Korea'}\n",
    "s = pd.Series(sports)\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "560e0377aa2d53cc14f61a42a6492f99932a6eab",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s.iloc[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2c0e34116ab3b36363b7d7ffde48d3520b493d50",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s.loc['Golf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "12dce7548346ce489c21f50e02a30e4a7dad6a81",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a44e0fd0b5ea294bfde2383f3ed660bb4dc7c032",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s['Golf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "97f106d843bd9da560c4aaa36e9a2baf0fd5f820",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sports = {99: 'Bhutan',\n",
    "          100: 'Scotland',\n",
    "          101: 'Japan',\n",
    "          102: 'South Korea'}\n",
    "s = pd.Series(sports)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6c8d10a115a42956c1bf95a925006f1a8f44ac77",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s = pd.Series([100.00, 120.00, 101.00, 3.00])\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a877d83b4ea29606ab220884b742b9436640a87b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "total = 0\n",
    "for item in s:\n",
    "    total+=item\n",
    "print(total)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "eb4aaac9d42d6ad0df5b0d766bf142cec13ca640",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "total = np.sum(s)\n",
    "print(total)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c5ec441b291558581a79c6bda7d67ee7df640ac4",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#this creates a big series of random numbers\n",
    "s = pd.Series(np.random.randint(0,1000,10000))\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "cafe2341bfb1b9419fcff2624e9973a12beb579a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b0028ee6c78f4715a1848458c3b5ef2ea75e601b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%timeit -n 100\n",
    "summary = 0\n",
    "for item in s:\n",
    "    summary+=item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "bbc4da9b2ba74a04d797ff837bddf7553b4625d9",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%timeit -n 100\n",
    "summary = np.sum(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a9ef0f8d0db28ffb3f5c92c86dbcffd8b9d01840",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s+=2 #adds two to each item in s using broadcasting\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f092ba84abb6a9172a23030b25caa23b7dea1c3f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for label, value in s.iteritems():\n",
    "    s.set_value(label, value+2)\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "979c2ea48abace1804d37a316ecf8a7cb0b53aa2",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%timeit -n 10\n",
    "s = pd.Series(np.random.randint(0,1000,100))\n",
    "for label, value in s.iteritems():\n",
    "    s.loc[label]= value+2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "84c6c0358bf55fd26254ae784ae2b60b3d7c3526",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%timeit -n 10\n",
    "s = pd.Series(np.random.randint(0,1000,100))\n",
    "s+=2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "35dabb561b6b3aaf6520311c68004880cadf5f7d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s = pd.Series([1, 2, 3])\n",
    "s.loc['Animal'] = 'Bears'\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1f40d7bded3fd8cea73b8ab31e945929d75a57f4",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "original_sports = pd.Series({'Archery': 'Bhutan',\n",
    "                             'Golf': 'Scotland',\n",
    "                             'Sumo': 'Japan',\n",
    "                             'Taekwondo': 'South Korea'})\n",
    "cricket_loving_countries = pd.Series(['Australia',\n",
    "                                      'Barbados',\n",
    "                                      'Pakistan',\n",
    "                                      'England'], \n",
    "                                   index=['Cricket',\n",
    "                                          'Cricket',\n",
    "                                          'Cricket',\n",
    "                                          'Cricket'])\n",
    "all_countries = original_sports.append(cricket_loving_countries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "95fa6b69a0927865a896c7a518ba848c3b994cad",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "original_sports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "bb617195f684747e2a79b87eb7307e832f2bfe50",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cricket_loving_countries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "5274031be8720d734db1866f20ad048c4c2ea7da",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_countries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "33c9d9f54962decaa00a7328b124252ddcf2b661",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_countries.loc['Cricket']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7653d84bad68f6370b1bdf484a2e9b6fb5982977"
   },
   "source": [
    "<a id=\"22\"></a> <br>\n",
    "## 2-2 Pandas:The DataFrame Data Structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4e82246f590c37992f9190583cdb0035d93c0dcd",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "purchase_1 = pd.Series({'Name': 'Chris',\n",
    "                        'Item Purchased': 'Dog Food',\n",
    "                        'Cost': 22.50})\n",
    "purchase_2 = pd.Series({'Name': 'Kevyn',\n",
    "                        'Item Purchased': 'Kitty Litter',\n",
    "                        'Cost': 2.50})\n",
    "purchase_3 = pd.Series({'Name': 'Vinod',\n",
    "                        'Item Purchased': 'Bird Seed',\n",
    "                        'Cost': 5.00})\n",
    "df = pd.DataFrame([purchase_1, purchase_2, purchase_3], index=['Store 1', 'Store 1', 'Store 2'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "835c6bb2dba772d11345bb2d5e40a992999d31b6",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.loc['Store 2']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0b48c39ad30936af6ce6932597073451c6b2bac9",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type(df.loc['Store 2'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c929565dc25461fd914c756431df400e6cdf058b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.loc['Store 1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a11d22702f1be476a4d443cf811fe0a07a5dbbe4",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.loc['Store 1', 'Cost']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "adb35d3ac7e00c00ced7236aa7c3eaab3c85e675",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0e478aabe2d50b04f5bd9cca220150476adc8b1f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.T.loc['Cost']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2ed1c53b43bb00bc07a95642071be8ef2b5aa779",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Cost']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "084454d6fcc47a738808b0c832d59aebc49be70c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.loc['Store 1']['Cost']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "fdbddb30cb59ed8462a07795aaadbcd9ad2b1aa4",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.loc[:,['Name', 'Cost']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2541e9071816bd8f496ddf62b77cccb8fe325fbc",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.drop('Store 1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "5fa5a22b81b92bae274d3a9afd76283ded17b478",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "918e58d124508bd5edff0ed84ba6a4c252cdec3d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "copy_df = df.copy()\n",
    "copy_df = copy_df.drop('Store 1')\n",
    "copy_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0e3830ddb755492607166e3975ac6e18c9436422",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "copy_df.drop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "946e48754fa8ed9914ef62b1c7049260861098db",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "del copy_df['Name']\n",
    "copy_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "de93b5c96c8b546bfb01b57d75347f5045ea01d1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Location'] = None\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "42414bc5b478108d59aaf9b5dff463c95904097d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "costs = df['Cost']\n",
    "costs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "76922dc612283caa9821f793abac91dad9328c75",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "costs+=2\n",
    "costs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "3ea1881948d05207d7dc2e1805c446adfa544959",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4d339cee9608b148762d7ad3068c362bbc9454f7"
   },
   "source": [
    "<a id=\"23\"></a> <br>\n",
    "# 2-2-1 Pandas:Dataframe Indexing and Loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "53fa2f4cb18784de6d077871a606dcf5b1511862",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('../input/train.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "01a8c7bc0a10635dc10dd56ba6edcbe595013772",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ffb1b09a5a8953e7dae65425890b652c214b1fb5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Querying a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "258b2c7201efba77d84b92286bbe69a6af240ca8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Survived'] > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "739f5037a2fcdd548abc5e68f5abcba3fcdb68e4",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "only_Survived = df.where(df['Survived'] > 0)\n",
    "only_Survived.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e420645f2daa14d2bf12b3370438b5c1741f5c52",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "only_Survived['Survived'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2c3561ac4d86a22f3984b11ebe1200100fc95417",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Survived'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "86e547ab11dacd87ccfe4657f8eb11fd9fcf3fef",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "only_Survived = only_Survived.dropna()\n",
    "only_Survived.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "cf0829126fff5c075151fcc5418bbe9b945c14c9",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "only_Survived = df[df['Survived'] > 0]\n",
    "only_Survived.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "5b957eae6c82a982ff9672d321deadf637aa421c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(df[(df['Survived'] > 0) | (df['Survived'] > 0)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "d076fe5f6dade0e49c2a35c7f9c64baeaf42a59d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df[(df['Survived'] > 0) & (df['Survived'] == 0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "717b27412a1a852c84f820272d8bf94a45022aca",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Indexing Dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "822efde2bbb058575dea289d057368d1af7d1394",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b11ed5fbe0e8d35b303125afc78b04abf4dc0190",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['PassengerId'] = df.index\n",
    "df = df.set_index('Survived')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4848977e538e02e0444862e632101a9d6bc97742",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "df = df.reset_index()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "5da1a958ccd43f5f5427415dc8682ccbbd589b3d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('../input/train.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9c818adf02056d59d534e4cb790dd6ce74c2b861",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Age'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c1adb2169bf24831daaa59655083e069d5fda4a5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df=df[df['Age'] == 50]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "94d2eb99802e00e342e3a046f9b26a06a3c501a7"
   },
   "source": [
    "<a id=\"24\"></a> <br>\n",
    "# 2-2-2 Pandas:Missing values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6946487c3ba7a29af57472c6fe03cde0ababd341",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('../input/train.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "30cc0a09aa17b60a69ddccebbc0b6ceaf6077bfb",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.fillna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "be537539e67066ad45f9217988aa7ca7c23a370b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = df.set_index('PassengerId')\n",
    "df = df.sort_index()\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "d2681f382b87e0eb47c41745576c2d35a8f55f5b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = df.reset_index()\n",
    "df = df.set_index(['PassengerId', 'Survived'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "bea1dfdc973fe52315d701fecb6abb28edaecb81",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = df.fillna(method='ffill')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d79a17c1a5930de30ef9c238bf143cfc9962d24f"
   },
   "source": [
    "<a id=\"25\"></a> <br>\n",
    "# 2-2-3 Pandas :Merging Dataframes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4809bb7be74c5ef657c069446ecffb409937f952",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame([{'Name': 'MJ', 'Item Purchased': 'Sponge', 'Cost': 22.50},\n",
    "                   {'Name': 'Kevyn', 'Item Purchased': 'Kitty Litter', 'Cost': 2.50},\n",
    "                   {'Name': 'Filip', 'Item Purchased': 'Spoon', 'Cost': 5.00}],\n",
    "                  index=['Store 1', 'Store 1', 'Store 2'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f30d304abf7b7345a4e6e7c1105e190dd1a621d2",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Date'] = ['December 1', 'January 1', 'mid-May']\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "fbedab0057046e0510dd1331f03ffc18c9ba520b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Delivered'] = True\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "8ed708570d219bad3637b3a907bb0a00be33b939",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Feedback'] = ['Positive', None, 'Negative']\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "fc549de9e14ccf0504553ee8960442180ba895b0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "adf = df.reset_index()\n",
    "adf['Date'] = pd.Series({0: 'December 1', 2: 'mid-May'})\n",
    "adf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "80e056750b87aa3d692e6f3aa07ca4e40ce05512",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "staff_df = pd.DataFrame([{'Name': 'Kelly', 'Role': 'Director of HR'},\n",
    "                         {'Name': 'Sally', 'Role': 'Course liasion'},\n",
    "                         {'Name': 'James', 'Role': 'Grader'}])\n",
    "staff_df = staff_df.set_index('Name')\n",
    "student_df = pd.DataFrame([{'Name': 'James', 'School': 'Business'},\n",
    "                           {'Name': 'Mike', 'School': 'Law'},\n",
    "                           {'Name': 'Sally', 'School': 'Engineering'}])\n",
    "student_df = student_df.set_index('Name')\n",
    "print(staff_df.head())\n",
    "print()\n",
    "print(student_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c0e141f46ea59c406f9c75a501139d808720bea6",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.merge(staff_df, student_df, how='outer', left_index=True, right_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "117f0b5ad0687b45deead65bfd2cd2e2b42aec7a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.merge(staff_df, student_df, how='inner', left_index=True, right_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0fcc1d0780a0b786ffbb77e88a1e1bdc5f415a4a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.merge(staff_df, student_df, how='left', left_index=True, right_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "d2492a6a8108c115d1f9c980a8b01242cc695a37",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.merge(staff_df, student_df, how='right', left_index=True, right_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7876e8102392731c7d48123c8c5dced6693a32d2",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "staff_df = staff_df.reset_index()\n",
    "student_df = student_df.reset_index()\n",
    "pd.merge(staff_df, student_df, how='left', left_on='Name', right_on='Name')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "08c724f0bf11154a1244924bd7ca0b195fff3a21",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "staff_df = pd.DataFrame([{'Name': 'Kelly', 'Role': 'Director of HR', 'Location': 'State Street'},\n",
    "                         {'Name': 'Sally', 'Role': 'Course liasion', 'Location': 'Washington Avenue'},\n",
    "                         {'Name': 'James', 'Role': 'Grader', 'Location': 'Washington Avenue'}])\n",
    "student_df = pd.DataFrame([{'Name': 'James', 'School': 'Business', 'Location': '1024 Billiard Avenue'},\n",
    "                           {'Name': 'Mike', 'School': 'Law', 'Location': 'Fraternity House #22'},\n",
    "                           {'Name': 'Sally', 'School': 'Engineering', 'Location': '512 Wilson Crescent'}])\n",
    "pd.merge(staff_df, student_df, how='left', left_on='Name', right_on='Name')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "da1dc7a2a97543ef9d338dd45fc74b8e66f6221e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "staff_df = pd.DataFrame([{'First Name': 'Kelly', 'Last Name': 'Desjardins', 'Role': 'Director of HR'},\n",
    "                         {'First Name': 'Sally', 'Last Name': 'Brooks', 'Role': 'Course liasion'},\n",
    "                         {'First Name': 'James', 'Last Name': 'Wilde', 'Role': 'Grader'}])\n",
    "student_df = pd.DataFrame([{'First Name': 'James', 'Last Name': 'Hammond', 'School': 'Business'},\n",
    "                           {'First Name': 'Mike', 'Last Name': 'Smith', 'School': 'Law'},\n",
    "                           {'First Name': 'Sally', 'Last Name': 'Brooks', 'School': 'Engineering'}])\n",
    "staff_df\n",
    "student_df\n",
    "pd.merge(staff_df, student_df, how='inner', left_on=['First Name','Last Name'], right_on=['First Name','Last Name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c47476edc934d4d851254db98b156de91018a0c8"
   },
   "source": [
    "<a id=\"26\"></a> <br>\n",
    "# 2-2-4 Idiomatic Pandas: Making Code Pandorable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2cd537dc9b3bd93a924d808ef8d6377853dae984",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('../input/train.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6ec456b0ba8db3f621911ad4c3a36008b04cfc7f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = df[df['Age']==50]\n",
    "df.set_index(['PassengerId','Survived'], inplace=True)\n",
    "df.rename(columns={'Pclass': 'pclass'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ef246335b484cd563b29cafd6c178820c04a6f0f"
   },
   "source": [
    "<a id=\"27\"></a> <br>\n",
    "## 2-2-5 Pandas :Group by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b45f3ee990bce0101774749b4b81e24b81911ad6",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df = pd.read_csv('../input/train.csv')\n",
    "df = df[df['Age']==50]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1c9afa54368039e6439950f57d39e5e0ae1faf7a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "49994af42a822d8e0ad579866d12fdcd3a7b65ba"
   },
   "source": [
    "<a id=\"28\"></a> <br>\n",
    "## 2-2-6 Pandas:Scales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ef0c2c453afcd5f43e37f27dd3dccd01aa7e33c0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(['A+', 'A', 'A-', 'B+', 'B', 'B-', 'C+', 'C', 'C-', 'D+', 'D'],\n",
    "                  index=['excellent', 'excellent', 'excellent', 'good', 'good', 'good', 'ok', 'ok', 'ok', 'poor', 'poor'])\n",
    "df.rename(columns={0: 'Grades'}, inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "cc490b8a1f851253430185eaab0d8a5ac1b843b8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['Grades'].astype('category').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "388f3454ed9fcc1b88c898e329c0c2d4b062df1f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grades = df['Grades'].astype('category',\n",
    "                             categories=['D', 'D+', 'C-', 'C', 'C+', 'B-', 'B', 'B+', 'A-', 'A', 'A+'],\n",
    "                             ordered=True)\n",
    "grades.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "51a0c5d18dbc6c395b900c34e9e590591d671676",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grades > 'C'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d205d889ce4d23e00f2ced7864c044e9d3d3ec84"
   },
   "source": [
    "<a id=\"30\"></a> <br>\n",
    "## 2-2-8 Pandas:Date Functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "aec2fc2ae67e3c567fb6ad840aa46ff62f7d69f1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a466f75d44bdabf52ddb21d0c173d9421afce7d9"
   },
   "source": [
    "### 2-2-8-1 Timestamp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "80a449e3cac139ac3b9697dba331363538a1a65f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.Timestamp('9/1/2016 10:05AM')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c39545f2495d3f837c6b75dfd57b1a53c3d27d75"
   },
   "source": [
    "### 2-2-8-2 Period"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "3c160f45b74a5e4faecbf6661978e9c59e933e14",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.Period('1/2016')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "124c5a5ba7872ab55b3cd1fbbe18669747124eea",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.Period('3/5/2016')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f19b87e81ba7aa0d8adb850d75d9452cf3a73ddf"
   },
   "source": [
    "### 2-10-3 DatetimeIndex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "68a99def5b8fbd9839cf35667a3481ed23c476c0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "t1 = pd.Series(list('abc'), [pd.Timestamp('2016-09-01'), pd.Timestamp('2016-09-02'), pd.Timestamp('2016-09-03')])\n",
    "t1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "073eeeb51e53b50660a822eb7484d4e8b72a7dfa",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type(t1.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c891236e512838ed6088e02c88dd888029b226a3"
   },
   "source": [
    "### 2-10-4 PeriodIndex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "5a19abe4e4a7324a8f7565c66f8270ab9eb3cae6",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "t2 = pd.Series(list('def'), [pd.Period('2016-09'), pd.Period('2016-10'), pd.Period('2016-11')])\n",
    "t2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "522200c3bbb47e10177f1c63a0ed3bfb49cbcf47",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type(t2.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "50e147a4abff8fd3014f3fb2a105f516b4a5ea2f"
   },
   "source": [
    "### 2-10-5 Pandas: Converting to Datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "862d2bc5c5b430ed2be1292b9d3b5efe8a3c9cc1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d1 = ['2 June 2013', 'Aug 29, 2014', '2015-06-26', '7/12/16']\n",
    "ts3 = pd.DataFrame(np.random.randint(10, 100, (4,2)), index=d1, columns=list('ab'))\n",
    "ts3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "840963eeb8ca7bfe0a5d3167211e74d48446fe3a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ts3.index = pd.to_datetime(ts3.index)\n",
    "ts3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e2482002fae0947549bdd81a3da93c8e0cde40fe",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.to_datetime('4.7.12', dayfirst=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "cae5847092e33cba7b657a90810a7ef35ae307e4",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.Timestamp('9/3/2016')-pd.Timestamp('9/1/2016')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c54a7d88507df86710fc585582b2074cb8d5aa5a"
   },
   "source": [
    "### 2-10-6 Timedeltas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "fe333509538c1ec2bf81fda4613344cfa699b410",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.Timestamp('9/3/2016')-pd.Timestamp('9/1/2016')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "08e10cce4428eaa329eb84677c755b0307488bfa",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.Timestamp('9/2/2016 8:10AM') + pd.Timedelta('12D 3H')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a73485dcf7310c754e69ab5ed802d7e466684242"
   },
   "source": [
    "### 2-10-7 Working with Dates in a Dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f5bcbc00ce23e50d346495588428e5d6f430a8df",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dates = pd.date_range('10-01-2016', periods=9, freq='2W-SUN')\n",
    "dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "41df9aeeb10a2d404f1eada62b4a4066e0f37af0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.index.ravel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a73af20a36d6b19a5833d835c5a0f5fa8d0ab6a8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.asfreq('W', method='ffill')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "018c262936f3e048c43c2d9a38068d4f0b67db77",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "df.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2f7c5d5041dc630abeaff47ff5a96a0dd53db8e5"
   },
   "source": [
    "<a id=\"31\"></a> <br>\n",
    "## 2-2-9 Distributions in Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a226de16a7d77afa240411ef9b9260fb06cea8a7",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "96be284ba6d63fd0b1db5641a21d75aacdfb7da4",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.binomial(1, 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4ae2c7ff2cf941bae62be23864a1685a196551d0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.binomial(1000, 0.5)/1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c5e89f6f0c7376c164f80dc4c5d582b0a639e254",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chance_of_tornado = 0.01/100\n",
    "np.random.binomial(100000, chance_of_tornado)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7804824638e0e6e97ebe5a252806d51a4e5cac2c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chance_of_tornado = 0.01\n",
    "\n",
    "tornado_events = np.random.binomial(1, chance_of_tornado, 1000000)\n",
    "    \n",
    "two_days_in_a_row = 0\n",
    "for j in range(1,len(tornado_events)-1):\n",
    "    if tornado_events[j]==1 and tornado_events[j-1]==1:\n",
    "        two_days_in_a_row+=1\n",
    "\n",
    "print('{} tornadoes back to back in {} years'.format(two_days_in_a_row, 1000000/365))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4583c0a0ec05e914af1c01e2c901da41828ad653",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.uniform(0, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "365cfd6e6602f46bc98c13d10a47ad1d98af978e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.normal(0.75)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "8091adda4fbdccffb73424476c68a7aa0fb53c9a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "distribution = np.random.normal(0.75,size=1000)\n",
    "\n",
    "np.sqrt(np.sum((np.mean(distribution)-distribution)**2)/len(distribution))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9eed1ac016763a69de465e82736d30d7e5b1d028",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.std(distribution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ab89e527fcd900577d879b50272d400ae0bdbaa0",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import scipy.stats as stats\n",
    "stats.kurtosis(distribution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c974e2563460f9b4942ab8c9b1d1783479779fa5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stats.skew(distribution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "de01fc7f82016eeb5f3b206788e7213ad373a441",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chi_squared_df2 = np.random.chisquare(2, size=10000)\n",
    "stats.skew(chi_squared_df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "d8cce6278eaf4b4865efebf34dfb4ec2d9b684ca",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chi_squared_df5 = np.random.chisquare(5, size=10000)\n",
    "stats.skew(chi_squared_df5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1c30298fc2dcd63dface93cb1626c5e7367d4698",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "output = plt.hist([chi_squared_df2,chi_squared_df5], bins=50, histtype='step', \n",
    "                  label=['2 degrees of freedom','5 degrees of freedom'])\n",
    "plt.legend(loc='upper right')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4f02c235284ef14f0007400125119ba36b81e325"
   },
   "source": [
    "<a id=\"33\"></a> <br>\n",
    "##  2-3 Matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "857d9a02e72211b4fde6f06debd3a57c8a0b1849"
   },
   "source": [
    "You can show matplotlib figures directly in the notebook by using the `%matplotlib notebook` and `%matplotlib inline` magic commands. \n",
    "\n",
    "`%matplotlib notebook` provides an interactive environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4ede24326f52afac77ee6571ce4f9722ca3e47df",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "fe647fec40a9aa6c57fe43c77c1c07fa3151d7db",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib as mpl\n",
    "mpl.get_backend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "54db662283ceee1087373f8ede796a832acfb57e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0d66eabe3f4f7288f12f8f0160b2f8e914d16ba9",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# because the default is the line style '-', \n",
    "# nothing will be shown if we only pass in one point (3,2)\n",
    "plt.plot(3, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "dad68ac5985444a4458259c448d1b6ffd9bf1710",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we can pass in '.' to plt.plot to indicate that we want\n",
    "# the point (3,2) to be indicated with a marker '.'\n",
    "plt.plot(3, 2, '.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "18c4b411903becff1b3612730bc0287cb6ac14ad"
   },
   "source": [
    "Let's see how to make a plot without using the scripting layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "8f1bd139edb5cf1cc73f9f956d2d34844a726566",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# First let's set the backend without using mpl.use() from the scripting layer\n",
    "from matplotlib.backends.backend_agg import FigureCanvasAgg\n",
    "from matplotlib.figure import Figure\n",
    "\n",
    "# create a new figure\n",
    "fig = Figure()\n",
    "\n",
    "# associate fig with the backend\n",
    "canvas = FigureCanvasAgg(fig)\n",
    "\n",
    "# add a subplot to the fig\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot the point (3,2)\n",
    "ax.plot(3, 2, '.')\n",
    "\n",
    "# save the figure to test.png\n",
    "# you can see this figure in your Jupyter workspace afterwards by going to\n",
    "# https://hub.coursera-notebooks.org/\n",
    "canvas.print_png('test.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "66c8f302e0afffb4652ff2f80212610fd37ae126"
   },
   "source": [
    "We can use html cell magic to display the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "d43abba30465cee9dc1364fd2a27334c70da8598",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%html\n",
    "<img src='test.png' />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2c76e50e3ecda1fe2144f16b9c9fdaa6bca37c61",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a new figure\n",
    "plt.figure()\n",
    "\n",
    "# plot the point (3,2) using the circle marker\n",
    "plt.plot(3, 2, 'o')\n",
    "\n",
    "# get the current axes\n",
    "ax = plt.gca()\n",
    "\n",
    "# Set axis properties [xmin, xmax, ymin, ymax]\n",
    "ax.axis([0,6,0,10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0cc553845bb24ead8e57b5ccc59c6753a847f136",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a new figure\n",
    "plt.figure()\n",
    "\n",
    "# plot the point (1.5, 1.5) using the circle marker\n",
    "plt.plot(1.5, 1.5, 'o')\n",
    "# plot the point (2, 2) using the circle marker\n",
    "plt.plot(2, 2, 'o')\n",
    "# plot the point (2.5, 2.5) using the circle marker\n",
    "plt.plot(2.5, 2.5, 'o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "41cc8f299ccc6825cf1ac49a8bc2039496bcc9a9",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get current axes\n",
    "ax = plt.gca()\n",
    "# get all the child objects the axes contains\n",
    "ax.get_children()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "0f4decd6f93319534a203748fd8a2b9bcc154ddb"
   },
   "source": [
    "<a id=\"34\"></a> <br>\n",
    "## 2-3-1 Scatterplots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "bf892e30bb09f0dc5fdf039d827ac55560b6ec06",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "x = np.array([1,2,3,4,5,6,7,8])\n",
    "y = x\n",
    "\n",
    "plt.figure()\n",
    "plt.scatter(x, y) # similar to plt.plot(x, y, '.'), but the underlying child objects in the axes are not Line2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9d4e979c8b366aa6e511a6d7316f63586346c649",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "x = np.array([1,2,3,4,5,6,7,8])\n",
    "y = x\n",
    "\n",
    "# create a list of colors for each point to have\n",
    "# ['green', 'green', 'green', 'green', 'green', 'green', 'green', 'red']\n",
    "colors = ['green']*(len(x)-1)\n",
    "colors.append('red')\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "# plot the point with size 100 and chosen colors\n",
    "plt.scatter(x, y, s=100, c=colors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "935be30b9a6a7c78b58fe1a9c6e4690c494fb697",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# convert the two lists into a list of pairwise tuples\n",
    "zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])\n",
    "\n",
    "print(list(zip_generator))\n",
    "# the above prints:\n",
    "# [(1, 6), (2, 7), (3, 8), (4, 9), (5, 10)]\n",
    "\n",
    "zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])\n",
    "# The single star * unpacks a collection into positional arguments\n",
    "print(*zip_generator)\n",
    "# the above prints:\n",
    "# (1, 6) (2, 7) (3, 8) (4, 9) (5, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "d6f848fc8d0879c658a6a4e90ca533bddf3be6e3",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# use zip to convert 5 tuples with 2 elements each to 2 tuples with 5 elements each\n",
    "print(list(zip((1, 6), (2, 7), (3, 8), (4, 9), (5, 10))))\n",
    "# the above prints:\n",
    "# [(1, 2, 3, 4, 5), (6, 7, 8, 9, 10)]\n",
    "\n",
    "\n",
    "zip_generator = zip([1,2,3,4,5], [6,7,8,9,10])\n",
    "# let's turn the data back into 2 lists\n",
    "x, y = zip(*zip_generator) # This is like calling zip((1, 6), (2, 7), (3, 8), (4, 9), (5, 10))\n",
    "print(x)\n",
    "print(y)\n",
    "# the above prints:\n",
    "# (1, 2, 3, 4, 5)\n",
    "# (6, 7, 8, 9, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b9c79e1558adbb3b19de2564376a24191766ab1f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "# plot a data series 'Tall students' in red using the first two elements of x and y\n",
    "plt.scatter(x[:2], y[:2], s=100, c='red', label='Tall students')\n",
    "# plot a second data series 'Short students' in blue using the last three elements of x and y \n",
    "plt.scatter(x[2:], y[2:], s=100, c='blue', label='Short students')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "70731835527ce5a16e4a8bd2a0017e3bbe8065ee",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# add a label to the x axis\n",
    "plt.xlabel('The number of times the child kicked a ball')\n",
    "# add a label to the y axis\n",
    "plt.ylabel('The grade of the student')\n",
    "# add a title\n",
    "plt.title('Relationship between ball kicking and grades')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "691db571df12646a4fe2740e55a43942ea779063",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# add a legend (uses the labels from plt.scatter)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0908d7c94c14feb08f953e5cd10f31cb0081f5d3",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# add the legend to loc=4 (the lower right hand corner), also gets rid of the frame and adds a title\n",
    "plt.legend(loc=4, frameon=False, title='Legend')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0eb79626b7b55559e40ca71ba5c3542466b629c9",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get children from current axes (the legend is the second to last item in this list)\n",
    "plt.gca().get_children()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f2b5d97457e3ebd63311aef5b92fc137bd1701f5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get the legend from the current axes\n",
    "legend = plt.gca().get_children()[-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "5334176f4177a64cb1726accb26502beb9b36626",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import the artist class from matplotlib\n",
    "from matplotlib.artist import Artist\n",
    "\n",
    "def rec_gc(art, depth=0):\n",
    "    if isinstance(art, Artist):\n",
    "        # increase the depth for pretty printing\n",
    "        print(\"  \" * depth + str(art))\n",
    "        for child in art.get_children():\n",
    "            rec_gc(child, depth+2)\n",
    "\n",
    "# Call this function on the legend artist to see what the legend is made up of\n",
    "rec_gc(plt.legend())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c372afa727e992b9ec097c1f172becfd06bff4ef"
   },
   "source": [
    "<a id=\"35\"></a> <br>\n",
    "## 2-3-2 Line Plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "163942ca710e3c67e8479cd02518ca37eae8ead7",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "linear_data = np.array([1,2,3,4,5,6,7,8])\n",
    "exponential_data = linear_data**2\n",
    "\n",
    "plt.figure()\n",
    "# plot the linear data and the exponential data\n",
    "plt.plot(linear_data, '-o', exponential_data, '-o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7a4c6677b81fb82e1e7a1164ced70c4168080dbf",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# plot another series with a dashed red line\n",
    "plt.plot([22,44,55], '--r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e3dedbff4cf4110154395ac3e4dea7c8059d4154",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.xlabel('Some data')\n",
    "plt.ylabel('Some other data')\n",
    "plt.title('A title')\n",
    "# add a legend with legend entries (because we didn't have labels when we plotted the data series)\n",
    "plt.legend(['Baseline', 'Competition', 'Us'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1a4024330f4b9c2a1223a7223173bb0cb3ffc290",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# fill the area between the linear data and exponential data\n",
    "plt.gca().fill_between(range(len(linear_data)), \n",
    "                       linear_data, exponential_data, \n",
    "                       facecolor='blue', \n",
    "                       alpha=0.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "92dd946d09a9ac10700f9245abab61810d05ad1d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "\n",
    "observation_dates = np.arange('2017-01-01', '2017-01-09', dtype='datetime64[D]')\n",
    "\n",
    "plt.plot(observation_dates, linear_data, '-o',  observation_dates, exponential_data, '-o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "08dcc7d6fed62bbf6d63755ecaa4006b44ed83e7",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "plt.figure()\n",
    "observation_dates = np.arange('2017-01-01', '2017-01-09', dtype='datetime64[D]')\n",
    "observation_dates = map(pd.to_datetime, observation_dates) # trying to plot a map will result in an error\n",
    "plt.plot(observation_dates, linear_data, '-o',  observation_dates, exponential_data, '-o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "58a9979b9d76e9d839bbaa1febbb0ea07f32b68e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "observation_dates = np.arange('2017-01-01', '2017-01-09', dtype='datetime64[D]')\n",
    "observation_dates = list(map(pd.to_datetime, observation_dates)) # convert the map to a list to get rid of the error\n",
    "plt.plot(observation_dates, linear_data, '-o',  observation_dates, exponential_data, '-o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b8ce468f668ca7dd57b899b623d8525c2cb565f9",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = plt.gca().xaxis\n",
    "\n",
    "# rotate the tick labels for the x axis\n",
    "for item in x.get_ticklabels():\n",
    "    item.set_rotation(45)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "aace2f89a0a58533af0aa9b70b554496f071100a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# adjust the subplot so the text doesn't run off the image\n",
    "plt.subplots_adjust(bottom=0.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7332d743ea723fb314daa0cc421b6b5528d5e301",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ax = plt.gca()\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Units')\n",
    "ax.set_title('Exponential vs. Linear performance')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "23db0f659f776bb67e749fe0791e784cf9cd81a8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# you can add mathematical expressions in any text element\n",
    "ax.set_title(\"Exponential ($x^2$) vs. Linear ($x$) performance\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "517e72eb500a3ba4db76467919f59ded9d3683cb"
   },
   "source": [
    "<a id=\"36\"></a> <br>\n",
    "## 2-3-3 Bar Charts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "8a5ee722c74c9228af2bcece24831e1539fe97b2",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "xvals = range(len(linear_data))\n",
    "plt.bar(xvals, linear_data, width = 0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "11a85a29ca9b1574536bfee91cc63122148d996e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "new_xvals = []\n",
    "\n",
    "# plot another set of bars, adjusting the new xvals to make up for the first set of bars plotted\n",
    "for item in xvals:\n",
    "    new_xvals.append(item+0.3)\n",
    "\n",
    "plt.bar(new_xvals, exponential_data, width = 0.3 ,color='red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f6597010276961362ce22b84d49a9ef5f46521f6",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from random import randint\n",
    "linear_err = [randint(0,15) for x in range(len(linear_data))] \n",
    "\n",
    "# This will plot a new set of bars with errorbars using the list of random error values\n",
    "plt.bar(xvals, linear_data, width = 0.3, yerr=linear_err)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "8db2b0bb5d4160f7954bbcd7d9e53d29c68ef342",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# stacked bar charts are also possible\n",
    "plt.figure()\n",
    "xvals = range(len(linear_data))\n",
    "plt.bar(xvals, linear_data, width = 0.3, color='b')\n",
    "plt.bar(xvals, exponential_data, width = 0.3, bottom=linear_data, color='r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ee208f2f850122b6e01ff1236b2915914fe59466",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# or use barh for horizontal bar charts\n",
    "plt.figure()\n",
    "xvals = range(len(linear_data))\n",
    "plt.barh(xvals, linear_data, height = 0.3, color='b')\n",
    "plt.barh(xvals, exponential_data, height = 0.3, left=linear_data, color='r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "fc3641e49ef8b3866ff9bad66c11d293e127451f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "db3db00430b8e5ddb633f3c7c82a6bf63ccb0cbc",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "# subplot with 1 row, 2 columns, and current axis is 1st subplot axes\n",
    "plt.subplot(1, 2, 1)\n",
    "\n",
    "linear_data = np.array([1,2,3,4,5,6,7,8])\n",
    "\n",
    "plt.plot(linear_data, '-o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ce5372a8e09fd5379b1b33c7b451ec1b983f8616",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "exponential_data = linear_data**2 \n",
    "\n",
    "# subplot with 1 row, 2 columns, and current axis is 2nd subplot axes\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.plot(exponential_data, '-o')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "cc4de0125429bdcd3227e894d083c0425934b228",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# plot exponential data on 1st subplot axes\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.plot(exponential_data, '-x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0b565e5356d99d5875de88a853710ee2dd3c4a53",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "ax1 = plt.subplot(1, 2, 1)\n",
    "plt.plot(linear_data, '-o')\n",
    "# pass sharey=ax1 to ensure the two subplots share the same y axis\n",
    "ax2 = plt.subplot(1, 2, 2, sharey=ax1)\n",
    "plt.plot(exponential_data, '-x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ec35c2bab0a1debaa6420f15b929d127d0d5f8cd",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "# the right hand side is equivalent shorthand syntax\n",
    "plt.subplot(1,2,1) == plt.subplot(121)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "43bb9cbc7acd94927be086a66ad913bd0a796fc5",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a 3x3 grid of subplots\n",
    "fig, ((ax1,ax2,ax3), (ax4,ax5,ax6), (ax7,ax8,ax9)) = plt.subplots(3, 3, sharex=True, sharey=True)\n",
    "# plot the linear_data on the 5th subplot axes \n",
    "ax5.plot(linear_data, '-')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ec95c476865ba15ca11e2ef4a1ba466a9212f9b9",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# set inside tick labels to visible\n",
    "for ax in plt.gcf().get_axes():\n",
    "    for label in ax.get_xticklabels() + ax.get_yticklabels():\n",
    "        label.set_visible(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "161be784e0c413060709d21a160985ad798b40f8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# necessary on some systems to update the plot\n",
    "plt.gcf().canvas.draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "487fb5e77983c55ae891c3a99537d6cef0450b74"
   },
   "source": [
    "<a id=\"37\"></a> <br>\n",
    "## 2-3-4 Histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "42551850b478e274b9f78d4f6ef717c636242616",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create 2x2 grid of axis subplots\n",
    "fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, sharex=True)\n",
    "axs = [ax1,ax2,ax3,ax4]\n",
    "\n",
    "# draw n = 10, 100, 1000, and 10000 samples from the normal distribution and plot corresponding histograms\n",
    "for n in range(0,len(axs)):\n",
    "    sample_size = 10**(n+1)\n",
    "    sample = np.random.normal(loc=0.0, scale=1.0, size=sample_size)\n",
    "    axs[n].hist(sample)\n",
    "    axs[n].set_title('n={}'.format(sample_size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "39b84012b223069dd4cc6f1441d2ad0f585218bf",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# repeat with number of bins set to 100\n",
    "fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, sharex=True)\n",
    "axs = [ax1,ax2,ax3,ax4]\n",
    "\n",
    "for n in range(0,len(axs)):\n",
    "    sample_size = 10**(n+1)\n",
    "    sample = np.random.normal(loc=0.0, scale=1.0, size=sample_size)\n",
    "    axs[n].hist(sample, bins=100)\n",
    "    axs[n].set_title('n={}'.format(sample_size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4def4012769663b5665f7b70a077cf839b7793f1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "Y = np.random.normal(loc=0.0, scale=1.0, size=10000)\n",
    "X = np.random.random(size=10000)\n",
    "plt.scatter(X,Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "f6304a7d71b6edf6894e93bc62ad9ee4ffb0cce1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# use gridspec to partition the figure into subplots\n",
    "import matplotlib.gridspec as gridspec\n",
    "\n",
    "plt.figure()\n",
    "gspec = gridspec.GridSpec(3, 3)\n",
    "\n",
    "top_histogram = plt.subplot(gspec[0, 1:])\n",
    "side_histogram = plt.subplot(gspec[1:, 0])\n",
    "lower_right = plt.subplot(gspec[1:, 1:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2a8f66c5169fcabafafa104e87897f175624245e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y = np.random.normal(loc=0.0, scale=1.0, size=10000)\n",
    "X = np.random.random(size=10000)\n",
    "lower_right.scatter(X, Y)\n",
    "top_histogram.hist(X, bins=100)\n",
    "s = side_histogram.hist(Y, bins=100, orientation='horizontal')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "97062a938d44626006bd9e24b5893fdf9d98155d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# clear the histograms and plot normed histograms\n",
    "top_histogram.clear()\n",
    "top_histogram.hist(X, bins=100, normed=True)\n",
    "side_histogram.clear()\n",
    "side_histogram.hist(Y, bins=100, orientation='horizontal', normed=True)\n",
    "# flip the side histogram's x axis\n",
    "side_histogram.invert_xaxis()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e6b1208bb887180b8f0363c9e82cc6fcbf387698",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# change axes limits\n",
    "for ax in [top_histogram, lower_right]:\n",
    "    ax.set_xlim(0, 1)\n",
    "for ax in [side_histogram, lower_right]:\n",
    "    ax.set_ylim(-5, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2bc0c6fc0bb19748e9e87603e3207f75ffa9b565"
   },
   "source": [
    "<a id=\"38\"></a> <br>\n",
    "## 2-3-5 Box and Whisker Plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "94dad21ec08e2633dacb64a89e5c807145042994",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "normal_sample = np.random.normal(loc=0.0, scale=1.0, size=10000)\n",
    "random_sample = np.random.random(size=10000)\n",
    "gamma_sample = np.random.gamma(2, size=10000)\n",
    "\n",
    "df = pd.DataFrame({'normal': normal_sample, \n",
    "                   'random': random_sample, \n",
    "                   'gamma': gamma_sample})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "87c08a9f914647f1735cb4b835b80f645685ef1e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9f4b288fe4b8ab78e6ad788e4bcfb5931920fcf2",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "# create a boxplot of the normal data, assign the output to a variable to supress output\n",
    "_ = plt.boxplot(df['normal'], whis='range')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9dc56a6415be6584fba51630ced26b0aaa486a09",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# clear the current figure\n",
    "plt.clf()\n",
    "# plot boxplots for all three of df's columns\n",
    "_ = plt.boxplot([ df['normal'], df['random'], df['gamma'] ], whis='range')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "0f44453d7022928d2aeed3c3e0126cbd7118cdd9",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "_ = plt.hist(df['gamma'], bins=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4dba4f705171f002fc94429ec80f7b6a2fe67ff3",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import mpl_toolkits.axes_grid1.inset_locator as mpl_il\n",
    "\n",
    "plt.figure()\n",
    "plt.boxplot([ df['normal'], df['random'], df['gamma'] ], whis='range')\n",
    "# overlay axis on top of another \n",
    "ax2 = mpl_il.inset_axes(plt.gca(), width='60%', height='40%', loc=2)\n",
    "ax2.hist(df['gamma'], bins=100)\n",
    "ax2.margins(x=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "90b1e8ffe23e39ec54414c1fd63b5d5c4e72be6f",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# switch the y axis ticks for ax2 to the right side\n",
    "ax2.yaxis.tick_right()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "cb3652a440484d391d27122878456a642c58d804",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# if `whis` argument isn't passed, boxplot defaults to showing 1.5*interquartile (IQR) whiskers with outliers\n",
    "plt.figure()\n",
    "_ = plt.boxplot([ df['normal'], df['random'], df['gamma'] ] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "fbf8bf0a67f6c49d78911c5f37be531ebbcd9edb"
   },
   "source": [
    "<a id=\"39\"></a> <br>\n",
    "## 2-3-6 Heatmaps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ebfc0dcb8e85aa540f6568fa96431a4e9707f3c1",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "\n",
    "Y = np.random.normal(loc=0.0, scale=1.0, size=10000)\n",
    "X = np.random.random(size=10000)\n",
    "_ = plt.hist2d(X, Y, bins=25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "fdbcf35950f94a4d0f1ce10efee6a4502f6ecfc8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "_ = plt.hist2d(X, Y, bins=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "26dfc6c0d1c88f2bfa5d80483c728ecd4c0b8522",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# add a colorbar legend\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "139f44a4deb043128c8c7254eb60c33e0fc26e68"
   },
   "source": [
    "<a id=\"40\"></a> <br>\n",
    "## 2-3-7 Animations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b3676970195153dc6056600f024f55c1b6f0ba12",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.animation as animation\n",
    "\n",
    "n = 100\n",
    "x = np.random.randn(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ea3eb5835e4acd53d43da483e17d79c32228cad6",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create the function that will do the plotting, where curr is the current frame\n",
    "def update(curr):\n",
    "    # check if animation is at the last frame, and if so, stop the animation a\n",
    "    if curr == n: \n",
    "        a.event_source.stop()\n",
    "    plt.cla()\n",
    "    bins = np.arange(-4, 4, 0.5)\n",
    "    plt.hist(x[:curr], bins=bins)\n",
    "    plt.axis([-4,4,0,30])\n",
    "    plt.gca().set_title('Sampling the Normal Distribution')\n",
    "    plt.gca().set_ylabel('Frequency')\n",
    "    plt.gca().set_xlabel('Value')\n",
    "    plt.annotate('n = {}'.format(curr), [3,27])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "fb2314b3b1735c5e191c8427c5abe6429e4ff767",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "a = animation.FuncAnimation(fig, update, interval=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7e702e5e0a876f9fa0b2e4fe497a56b91e00a95d"
   },
   "source": [
    "<a id=\"41\"></a> <br>\n",
    "## 2-3-8 Interactivity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "51fefb947daf8ca558cbc153e2ddbf39bcb7d4b2",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "data = np.random.rand(10)\n",
    "plt.plot(data)\n",
    "\n",
    "def onclick(event):\n",
    "    plt.cla()\n",
    "    plt.plot(data)\n",
    "    plt.gca().set_title('Event at pixels {},{} \\nand data {},{}'.format(event.x, event.y, event.xdata, event.ydata))\n",
    "\n",
    "# tell mpl_connect we want to pass a 'button_press_event' into onclick when the event is detected\n",
    "plt.gcf().canvas.mpl_connect('button_press_event', onclick)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "1d64a25dc386a30bd895ca8f58ca86d632f05d74",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from random import shuffle\n",
    "origins = ['China', 'Brazil', 'India', 'USA', 'Canada', 'UK', 'Germany', 'Iraq', 'Chile', 'Mexico']\n",
    "\n",
    "shuffle(origins)\n",
    "\n",
    "df = pd.DataFrame({'height': np.random.rand(10),\n",
    "                   'weight': np.random.rand(10),\n",
    "                   'origin': origins})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e13dd7e002938af1a52d7520004d839a3d7d2011",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "# picker=5 means the mouse doesn't have to click directly on an event, but can be up to 5 pixels away\n",
    "plt.scatter(df['height'], df['weight'], picker=5)\n",
    "plt.gca().set_ylabel('Weight')\n",
    "plt.gca().set_xlabel('Height')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "e926ab1d2dc2098a6af48526e9f980bf594c79cd",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def onpick(event):\n",
    "    origin = df.iloc[event.ind[0]]['origin']\n",
    "    plt.gca().set_title('Selected item came from {}'.format(origin))\n",
    "\n",
    "# tell mpl_connect we want to pass a 'pick_event' into onpick when the event is detected\n",
    "plt.gcf().canvas.mpl_connect('pick_event', onpick)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "2d60c1f7f19fa489291f13d5a4565bf0c43e19aa",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# see the pre-defined styles provided.\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib notebook\n",
    "plt.style.available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "bb968584c16b9acc6466a3897c9415c57f3a7404",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# use the 'seaborn-colorblind' style\n",
    "plt.style.use('seaborn-colorblind')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "84742816a16280f7bca7c43879d6762e10e0a440"
   },
   "source": [
    "<a id=\"42\"></a> <br>\n",
    "### DataFrame.plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "97a3554f9640a2b77e07c861b5a5b6c814a3b276",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(123)\n",
    "\n",
    "df = pd.DataFrame({'A': np.random.randn(365).cumsum(0), \n",
    "                   'B': np.random.randn(365).cumsum(0) + 20,\n",
    "                   'C': np.random.randn(365).cumsum(0) - 20}, \n",
    "                  index=pd.date_range('1/1/2017', periods=365))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a28530c9001950c828e678178ba3b87bf7f41297",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.plot(); # add a semi-colon to the end of the plotting call to suppress unwanted output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "c42d2bb7be9791cfc03494a16461815d02fa27c8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "We can select which plot we want to use by passing it into the 'kind' parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "4d64fa4bc8b62fe1f4d2c2de0869bb49c8f7fc3d",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.plot('A','B', kind = 'scatter');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "857cecae1e2c9eb59c1a9d136ef1c5422d86d5ba"
   },
   "source": [
    "You can also choose the plot kind by using the `DataFrame.plot.kind` methods instead of providing the `kind` keyword argument.\n",
    "\n",
    "`kind` :\n",
    "- `'line'` : line plot (default)\n",
    "- `'bar'` : vertical bar plot\n",
    "- `'barh'` : horizontal bar plot\n",
    "- `'hist'` : histogram\n",
    "- `'box'` : boxplot\n",
    "- `'kde'` : Kernel Density Estimation plot\n",
    "- `'density'` : same as 'kde'\n",
    "- `'area'` : area plot\n",
    "- `'pie'` : pie plot\n",
    "- `'scatter'` : scatter plot\n",
    "- `'hexbin'` : hexbin plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "74997530957394a96f0aed15c21e65f54911159c",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create a scatter plot of columns 'A' and 'C', with changing color (c) and size (s) based on column 'B'\n",
    "df.plot.scatter('A', 'C', c='B', s=df['B'], colormap='viridis')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "6299f8dddb909c7850620499edc49afdfd909f75",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ax = df.plot.scatter('A', 'C', c='B', s=df['B'], colormap='viridis')\n",
    "ax.set_aspect('equal')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "91a25480397c8759047100da9ebc6c0264d8a918",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.plot.box();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b7bfa0ce17ea260d75eb97a0161af3dbd700f780",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.plot.hist(alpha=0.7);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "21a68cb3d0111753d29df2b402011daff81c5ff4"
   },
   "source": [
    "[Kernel density estimation plots](https://en.wikipedia.org/wiki/Kernel_density_estimation) are useful for deriving a smooth continuous function from a given sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7b4d0f65af26e55acaf9a06da13dc71eb21a408b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.plot.kde();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "8c8d4d76f51dfa4b573804a99d102594a0d3600e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### pandas.tools.plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "68f7c4b8c85e6e28d978f11adbd9b54223d58538"
   },
   "source": [
    "<a id=\"43\"></a> <br>\n",
    "# Seaborn\n",
    "As you have just read, **Seaborn** is complimentary to Matplotlib and it specifically targets statistical data visualization. But it goes even further than that: Seaborn extends Matplotlib and that’s why it can address the two biggest frustrations of working with Matplotlib. Or, as Michael Waskom says in the “introduction to Seaborn”: “If matplotlib “tries to make easy things easy and hard things possible”, seaborn tries to make a well-defined set of hard things easy too.”\n",
    "\n",
    "One of these hard things or frustrations had to do with the default Matplotlib parameters. Seaborn works with different parameters, which undoubtedly speaks to those users that don’t use the default looks of the Matplotlib plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "33a2102c45ea66e672bf8113baebeaf71b1e33ed",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "a78ee6a9554ff6a42f4d584d00f700c63b3945cb",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(1234)\n",
    "\n",
    "v1 = pd.Series(np.random.normal(0,10,1000), name='v1')\n",
    "v2 = pd.Series(2*v1 + np.random.normal(60,15,1000), name='v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ee4e031c263014b7d16c96351bbec7503ea16fbf",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "plt.hist(v1, alpha=0.7, bins=np.arange(-50,150,5), label='v1');\n",
    "plt.hist(v2, alpha=0.7, bins=np.arange(-50,150,5), label='v2');\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "67dcace37ccbc4ebbb58297ad5b09c4edcbc32e8",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "# we can pass keyword arguments for each individual component of the plot\n",
    "sns.distplot(v2, hist_kws={'color': 'Teal'}, kde_kws={'color': 'Navy'});"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "7bc165c0ea4d4f20a5975bd71a81e203afae0feb",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sns.jointplot(v1, v2, alpha=0.4);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "ee8e1e256786a02ada1f55233695ac507c05558e",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grid = sns.jointplot(v1, v2, alpha=0.4);\n",
    "grid.ax_joint.set_aspect('equal')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "b25a62cb3ddf0be742e2465d20c9f69bc8062a95",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sns.jointplot(v1, v2, kind='hex');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "28893e60d141b295029b9414d6f4ac3fece7c43a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# set the seaborn style for all the following plots\n",
    "sns.set_style('white')\n",
    "\n",
    "sns.jointplot(v1, v2, kind='kde', space=0);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "9eef5ff46fbf313f209a35ea44bce3b0d85e5966",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train = pd.read_csv('../input/train.csv')\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4524fb4d3bdeca34f1a526bbf2fc2caa9bfbf51a",
    "collapsed": true
   },
   "source": [
    "<a id=\"44\"></a> <br>\n",
    "## 2-4 SKlearn\n",
    "for Reading this section **please** fork and upvote  this kernel:\n",
    "\n",
    "[a-journey-with-scikit-learn](https://www.kaggle.com/mjbahmani/a-journey-with-scikit-learn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ad8fa54ba57aa4a336080eb044109702c743d7a0"
   },
   "source": [
    "<a id=\"45\"></a> <br>\n",
    "##  3- Mathematics and Linear Algebra\n",
    "for Reading this section **please** fork and upvote  this kernel:\n",
    "\n",
    "[Linear Algebra in 60 Minutes](https://www.kaggle.com/mjbahmani/linear-algebra-in-60-minutes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "697ba206ad7adf4d99814cb1d89375b745eaba19"
   },
   "source": [
    "<a id=\"46\"></a> <br>\n",
    "## 4- Programming & Analysis Tools\n",
    "we will release the full version of Deep Learning **Coming Soon**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "00f5c5ce80c7e302e83f0ea9b451dfaae7aa52cf"
   },
   "source": [
    "## 5- Big Data\n",
    "we will release the full version of Deep Learning **Coming Soon**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "33bb9c265bef5e4474dcac0638cc632b5532f1ce"
   },
   "source": [
    "## 6- Data Visualization\n",
    "we will release the full version of Deep Learning **Coming Soon**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "9bf1d9444651e2756c4fa4d71914ec20d621305e"
   },
   "source": [
    "## 7- Data Cleaning\n",
    "we will release the full version of Deep Learning **Coming Soon**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8720a4ddaab64e4bff226bed9e4e200dc9b94913"
   },
   "source": [
    "## 8- How to solve Problem?\n",
    "**Data Science has so many techniques and procedures that can confuse anyone.**\n",
    "\n",
    "**Step 1**: Translate your business problem statement into technical one\n",
    "\n",
    "Analogous to any other software problem, data science aims at solving a business problem. Most of the times, business problem statements are vague and can be interpreted in multiple ways. This occurs mostly because we generally use qualitative words in our language which cannot be directly translated into a machine readable code.\n",
    "\n",
    "Eg. Let’s say we need to develop a solution to reduce crime rate of a city. The term “reduce” can be interpreted as:\n",
    "\n",
    "Decreasing crime rate of areas with high crime rate\n",
    "Decreasing crime rate of the most common type of crime\n",
    "It is a good practice to circle back with the client or the business team who define the problem to decide on the right interpretation.\n",
    "\n",
    "**Step 2**: Decide on the supervised learning technique\n",
    "\n",
    "The end goal of almost every data science problem is usually classification or regression. Deciding the supervised technique for the problem will help you get more clarity on the business statement.\n",
    "\n",
    "Eg. Let’s look at our problem of reducing crime rate. While the problem of reducing crime rate is more of a policy decision, depending on the choice above, we would have to decide if we need to do classification or regression.\n",
    "\n",
    "If we need to decrease crime rate of areas with high crime rate, we would need to determine the crime rate rate of an area. This is a regression problem.\n",
    "If we need to decrease crime rate of most common type of crime, we would need to determine the most common type of crime in an area. This is a classification problem.\n",
    "Again it is a good practice to circle back with the client or the business team who define the problem requirements to clarify on the exact requirement.\n",
    "\n",
    "**Step 3**: Literature survey\n",
    "\n",
    "Literature Survey is one of the most important step (and often most ignored step) to approach any problem. If you read any article about components of Data Science, you will find computer science, statistics / math and domain knowledge. As it is quite inhuman for someone to have subject expertise in all possible fields, literature survey can often help in bridging the gaps of inadequate subject expertise.\n",
    "\n",
    "After going through existing literature related to a problem, I usually try to come up with a set of hypotheses that could form my potential set of features. Going through existing literature helps you understand existing proofs in the domain serving as a guide to take the right direction in your problem. It also helps in interpretation of the results obtained from the prediction models.\n",
    "\n",
    "Eg. Going back to our problem of reducing crime rate, if you want to predict crime rate of an area, you would consider factors from general knowledge like demographics, neighboring areas, law enforcement rules etc. Literature survey will help you consider additional variables like climate, mode of transportation, divorce rate etc.\n",
    "\n",
    "**Step 4**: Data cleaning\n",
    "\n",
    "If you speak with anyone who has spent some time in data science, they will always say that most of their time is spent on cleaning the data. Real world data is always messy. Here are a few common discrepancies in most data-sets and some techniques of how to clean them:\n",
    "\n",
    "Missing values\n",
    "Missing values are values that are blank in the data-set. This can be due to various reasons like value being unknown, unrecorded, confidential etc. Since the reason for a value being missing is not clear, it is hard to guess the value.\n",
    "\n",
    "You could try different techniques to impute missing values starting with simple methods like column mean, median etc. and complex methods like using machine leaning models to estimate missing values.\n",
    "\n",
    "Duplicate records\n",
    "The challenge with duplicate records is identifying a record being duplicate. Duplicate records often occur while merging data from multiple sources. It could also occur due to human error. To identify duplicates, you could approximate a numeric values to certain decimal places and for text values, fuzzy matching could be a good start. Identification of duplicates could help the data engineering team to improve collection of data to prevent such errors.\n",
    "\n",
    "Incorrect values\n",
    "Incorrect values are mostly due to human error. For Eg. If there is a field called age and the value is 500, it is clearly wrong. Having domain knowledge of the data will help identify such values. A good technique to identify incorrect values for numerical columns could be to manually look at values beyond 3 standard deviations from the mean to check for correctness.\n",
    "\n",
    "**Step 5**: Feature engineering\n",
    "\n",
    "Feature Engineering is one of the most important step in any data science problem. Good set of features might make simple models work for your data. If features are not good enough, you might need to go for complex models. Feature Engineering mostly involves:\n",
    "\n",
    "Removing redundant features\n",
    "If a feature is not contributing a lot to the output value or is a function of other features, you can remove the feature. There are various metrics like AIC and BIC to identify redundant features. There are built in packages to perform operations like forward selection, backward selection etc. to remove redundant features.\n",
    "\n",
    "Transforming a feature\n",
    "A feature might have a non linear relationship with the output column. While complex models can capture this with enough data, simple models might not be able to capture this. I usually try to visualize different functions of each column like log, inverse, quadratic, cubic etc. and choose the transformation that looks closest to a normal curve.\n",
    "\n",
    "**Step 6**: Data modification\n",
    "\n",
    "Once the data is cleaned, there are a few modifications that might be needed before applying machine learning models. One of the most common modification would be scaling every column to the same range in order to give same weight to all columns. Some of the other required modifications might be data specific Eg. If output column is skewed, you might need to up-sample or down-sample.\n",
    "\n",
    "Steps 7 through 9 are iterative.\n",
    "\n",
    "**Step 7**: Modelling\n",
    "\n",
    "Once I have the data ready, I usually start with trying all the standard machine learning models. If it is a classification problem, a good start will beLogistic Regression, Naive Bayes, k-Nearest Neighbors, Decision Tree etc. If it is a regression problem, you could try linear regression, regression tree etc. The reason for starting with simple models is that simple models have lesser parameters to alter. If we start with a complex model like Neural Network orSupport Vector Machines, there are so many parameters that you could change that trying all options exhaustively might be time consuming.\n",
    "\n",
    "Each of the machine learning models make some underlying assumptions about the data. For Eg. Linear Regression / Logistic Regression assumes that the data comes from a linear combination of input parameters. Naive Bayes makes an assumption that the input parameters are independent of each other. Having the knowledge of these assumptions can help you judge the results of the different models. It is often helpful to visualize the actual vs predicted values to see these differences.\n",
    "\n",
    "**Step 8**: Model comparison\n",
    "\n",
    "One of the most standard technique to evaluate different machine learning models would be through the process of cross validation. I usually choose 10-fold cross validation but you may choose the right cross validation split based on the size of the data. Cross validation basically brings out an average performance of a model. This can help eliminate choosing a model that performs good specific to the data or in other words avoid over-fitting. It is often a good practice to randomize data before cross validation.\n",
    "\n",
    "A good technique to compare performance of different models is ROC curves. ROC curves help you visualize performance of different models across different thresholds. While ROC curves give a holistic sense of model performance, based on the business decision, you must choose the performance metric like Accuracy, True Positive Rate, False Positive Rate, F1-Score etc.\n",
    "\n",
    "**Step 9**: Error analysis\n",
    "\n",
    "At this point, you have tried a bunch of machine learning models and got the results. It is a good usage of time to not just look at the results like accuracy or True Positive Rate but to look at the set of data points that failed in some of the models. This will help you understand the data better and improve the models faster than trying all possible combinations of models. This is the time to try ensemble models like Random Forest, Gradient Boosting or a meta model of your own [Eg. Decision tree + Logistic Regression]. Ensemble models are almost always guaranteed to perform better than any standard model.\n",
    "\n",
    "**Step 10**: Improving your best model\n",
    "\n",
    "Once I have the best model, I usually plot training vs testing accuracy [or the right metric] against the number of parameters. Usually, it is easy to check training and testing accuracy against number of data points. Basically this plot will tell you whether your model is over-fitting or under-fitting. This articleDetecting over-fitting vs under-fitting explains this concept clearly.\n",
    "\n",
    "Understanding if your model is over-fitting or under-fitting will tell you how to proceed with the next steps. If the model is over-fitting, you might consider collecting more data. If the model is under-fitting, you might consider making the models more complex. [Eg. Adding higher order terms to a linear / logistic regression]\n",
    "\n",
    "**Step 11**: Deploying the model\n",
    "\n",
    "Once you have your final model, you would want the model to be deployed so that it automatically predicts output for new data point without retraining. While you can derive a formula for simple models like Linear Regression, Logistic Regression, Decision Tree etc. , it is not so straight forward for complex models like SVM, Neural Networks, Random Forest etc. I’m not very familiar with other languages but Python has a library called pickle which allows you to save models and use it to predict output for new data.\n",
    "\n",
    "**Step 12**: Adding feedback\n",
    "\n",
    "Usually, data for any data science problem is historical data. While this might be similar to the current data up-to a certain degree, it might not be able to capture the current trends or changes. For Eg. If you are using population as an input parameter, while population from 2015–2016 might vary slightly, if you use the model after 5 years, it might give incorrect results.\n",
    "\n",
    "One way to deal with this problem is to keep retraining your model with additional data. This might be a good option but retraining a model might be time consuming. Also, if you have applications in which data inflow is huge, this might need to be done at regular intervals. An alternative and a better option would be to use active learning. Active learning basically tries to use real time data as feedback and automatically update the model. The most common approaches to do this are Batch Gradient Descent and Stochastic Gradient Descent. It might be appropriate to use the right approach based on the application.\n",
    "\n",
    "Concluding remarks\n",
    "\n",
    "The field of data science is really vast. People spend their lifetime researching on individual topics discussed above. As a data scientist, you would mostly have to solve business problems than researching on individual subtopics. Additionally, you will have to explain the technical process and results to business teams who might not have enough technical knowledge. Thus, while you might not need a very in-depth knowledge of every technique, you need to have enough clarity to abstract the technical process and results and explain it in business terms.[3]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "d4f8718cc7e1a8fc60a3815b55a2ab9a5eeef4f9"
   },
   "source": [
    "## 9- Machine learning  \n",
    "for Reading this section **please** fork and upvote  this kernel:\n",
    "\n",
    "[A Comprehensive ML Workflow with Python](http://https://www.kaggle.com/mjbahmani/a-comprehensive-ml-workflow-with-python)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "3544d2fd1490f646f2f1c0fd4271f9a8745d2e36"
   },
   "source": [
    "##  10- Deep Learning\n",
    "for Reading this section **please** fork and upvote  this kernel:\n",
    "\n",
    "[a-comprehensive-deep-learning-workflow-with-python](https://www.kaggle.com/mjbahmani/a-comprehensive-deep-learning-workflow-with-python)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "3065412feed4f072e90154bb3eaed0fc3504d88d",
    "collapsed": true
   },
   "source": [
    "## References:\n",
    "1. [Coursera](https://www.coursera.org/specializations/data-science-python)\n",
    "2. [Hands-On Machine Learning with Scikit-Learn and TensorFlow](http://shop.oreilly.com/product/0636920052289.do)\n",
    "3. [How to solve Problem](https://www.linkedin.com/pulse/how-i-approach-data-science-problem-ganesh-n-prasad/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "edb768e0b3390ec29acab20593948c3f3bbf5bba",
    "collapsed": true
   },
   "source": [
    "---------------------------------------------------------------------\n",
    "you can follow me on:\n",
    "> ###### [ GitHub](https://github.com/mjbahmani)\n",
    "> ###### [LinkedIn](https://www.linkedin.com/in/bahmani/)\n",
    "> ###### [Kaggle](https://www.kaggle.com/mjbahmani/)\n",
    "\n",
    "-------------------------------------------------------------------------------------------------------------\n",
    " **I hope you find this kernel helpful and some upvotes would be very much appreciated**\n",
    " \n",
    " -----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e07313484155d573b97e7d21e6be7a60dc6768e3",
    "collapsed": true
   },
   "source": [
    "**Not completed yet!!!**\n",
    "\n",
    "**Update every two days**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_uuid": "15278765f89f04a99d9ae1003f35ff06b577af0e",
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}