SparkifyDraft.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sparkify Project Workspace\n",
    "This workspace contains a tiny subset (128MB) of the full dataset available (12GB). Feel free to use this workspace to build your project, or to explore a smaller subset with Spark before deploying your cluster on the cloud. Instructions for setting up your Spark cluster is included in the last lesson of the Extracurricular Spark Course content.\n",
    "\n",
    "You can follow the steps below to guide your data analysis and model building portion of this project.\n",
    "\n",
    "### Metrics to consider:\n",
    "- Monthly active users\n",
    "- Daily active users in past month\n",
    "- Total paid and unpaid users\n",
    "- Total ads served in the past month\n",
    "- Cohort per Month - % of users cancelled, % of users upgrades\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "        <script type=\"text/javascript\">\n",
       "        window.PlotlyConfig = {MathJaxConfig: 'local'};\n",
       "        if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}\n",
       "        if (typeof require !== 'undefined') {\n",
       "        require.undef(\"plotly\");\n",
       "        requirejs.config({\n",
       "            paths: {\n",
       "                'plotly': ['https://cdn.plot.ly/plotly-latest.min']\n",
       "            }\n",
       "        });\n",
       "        require(['plotly'], function(Plotly) {\n",
       "            window._Plotly = Plotly;\n",
       "        });\n",
       "        }\n",
       "        </script>\n",
       "        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# import libraries\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import datetime # missing and together with \"pip install --upgrade pyspark\" were causing issues with time series\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "%matplotlib inline\n",
    "from plotly.offline import iplot\n",
    "import plotly.graph_objects as go\n",
    "import plotly.express as px\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import cufflinks as cf\n",
    "cf.go_offline()\n",
    "from pyspark.ml.feature import RegexTokenizer, VectorAssembler, Normalizer, StandardScaler, MinMaxScaler\n",
    "from pyspark.sql.functions import isnan, count, when, col, desc, udf, col, sort_array, asc, avg\n",
    "from pyspark.sql.functions import sum as Fsum\n",
    "from pyspark.sql.functions import *\n",
    "from pyspark.sql.window import Window\n",
    "from pyspark.sql.types import IntegerType\n",
    "from pyspark.context import SparkContext\n",
    "from pyspark.sql.session import SparkSession\n",
    "from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler\n",
    "from pyspark.ml.classification import LogisticRegression, RandomForestClassifier, LinearSVC, GBTClassifier, DecisionTreeClassifier\n",
    "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator, RegressionEvaluator\n",
    "import pyspark.ml.evaluation as evals\n",
    "import pyspark.ml.tuning as tune\n",
    "from pyspark.ml import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load and Clean Dataset\n",
    "In this workspace, the mini-dataset file is `mini_sparkify_event_data.json`. Load and clean the dataset, checking for invalid or missing data - for example, records without userids or sessionids. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a Spark session\n",
    "# spark = SparkSession.builder.getOrCreate()\n",
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .appName(\"Sparkify\") \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <div>\n",
       "                <p><b>SparkSession - in-memory</b></p>\n",
       "                \n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://192.168.1.97:4040\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v3.1.1</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>local[*]</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>Sparkify</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        \n",
       "            </div>\n",
       "        "
      ],
      "text/plain": [
       "<pyspark.sql.session.SparkSession at 0x7f779afc1cd0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "print(spark.catalog.listTables())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/freemo/Projects/SparkProject'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "mini = '/home/freemo/Projects/largeData/mini_sparkify_event_data.json'\n",
    "medium = '/home/freemo/Projects/largeData/medium_sparkify_event_data.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.json(mini)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "286500"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(artist='Martha Tilston', auth='Logged In', firstName='Colin', gender='M', itemInSession=50, lastName='Freeman', length=277.89016, level='paid', location='Bakersfield, CA', method='PUT', page='NextSong', registration=1538173362000, sessionId=29, song='Rockpools', status=200, ts=1538352117000, userAgent='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0', userId='30')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------+---------+---------+------+-------------+--------+---------+-----+--------------------+------+--------+-------------+---------+--------------------+------+-------------+--------------------+------+\n",
      "|          artist|     auth|firstName|gender|itemInSession|lastName|   length|level|            location|method|    page| registration|sessionId|                song|status|           ts|           userAgent|userId|\n",
      "+----------------+---------+---------+------+-------------+--------+---------+-----+--------------------+------+--------+-------------+---------+--------------------+------+-------------+--------------------+------+\n",
      "|  Martha Tilston|Logged In|    Colin|     M|           50| Freeman|277.89016| paid|     Bakersfield, CA|   PUT|NextSong|1538173362000|       29|           Rockpools|   200|1538352117000|Mozilla/5.0 (Wind...|    30|\n",
      "|Five Iron Frenzy|Logged In|    Micah|     M|           79|    Long|236.09424| free|Boston-Cambridge-...|   PUT|NextSong|1538331630000|        8|              Canada|   200|1538352180000|\"Mozilla/5.0 (Win...|     9|\n",
      "|    Adam Lambert|Logged In|    Colin|     M|           51| Freeman| 282.8273| paid|     Bakersfield, CA|   PUT|NextSong|1538173362000|       29|   Time For Miracles|   200|1538352394000|Mozilla/5.0 (Wind...|    30|\n",
      "|          Enigma|Logged In|    Micah|     M|           80|    Long|262.71302| free|Boston-Cambridge-...|   PUT|NextSong|1538331630000|        8|Knocking On Forbi...|   200|1538352416000|\"Mozilla/5.0 (Win...|     9|\n",
      "|       Daft Punk|Logged In|    Colin|     M|           52| Freeman|223.60771| paid|     Bakersfield, CA|   PUT|NextSong|1538173362000|       29|Harder Better Fas...|   200|1538352676000|Mozilla/5.0 (Wind...|    30|\n",
      "+----------------+---------+---------+------+-------------+--------+---------+-----+--------------------+------+--------+-------------+---------+--------------------+------+-------------+--------------------+------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- artist: string (nullable = true)\n",
      " |-- auth: string (nullable = true)\n",
      " |-- firstName: string (nullable = true)\n",
      " |-- gender: string (nullable = true)\n",
      " |-- itemInSession: long (nullable = true)\n",
      " |-- lastName: string (nullable = true)\n",
      " |-- length: double (nullable = true)\n",
      " |-- level: string (nullable = true)\n",
      " |-- location: string (nullable = true)\n",
      " |-- method: string (nullable = true)\n",
      " |-- page: string (nullable = true)\n",
      " |-- registration: long (nullable = true)\n",
      " |-- sessionId: long (nullable = true)\n",
      " |-- song: string (nullable = true)\n",
      " |-- status: long (nullable = true)\n",
      " |-- ts: long (nullable = true)\n",
      " |-- userAgent: string (nullable = true)\n",
      " |-- userId: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop missing values\n",
    "df = df.dropna(how = \"any\", subset = [\"userId\", \"sessionId\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+\n",
      "|userId|\n",
      "+------+\n",
      "|      |\n",
      "|    10|\n",
      "|   100|\n",
      "|100001|\n",
      "|100002|\n",
      "|100003|\n",
      "|100004|\n",
      "|100005|\n",
      "|100006|\n",
      "|100007|\n",
      "|100008|\n",
      "|100009|\n",
      "|100010|\n",
      "|100011|\n",
      "|100012|\n",
      "|100013|\n",
      "|100014|\n",
      "|100015|\n",
      "|100016|\n",
      "|100017|\n",
      "+------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(\"userId\").dropDuplicates().sort(\"userId\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+\n",
      "|sessionId|\n",
      "+---------+\n",
      "|     1336|\n",
      "|      287|\n",
      "|     2175|\n",
      "|     2878|\n",
      "|     2403|\n",
      "|     1840|\n",
      "|     3009|\n",
      "|      385|\n",
      "|     1599|\n",
      "|      736|\n",
      "|      558|\n",
      "|     1371|\n",
      "|     1277|\n",
      "|     3199|\n",
      "|     1921|\n",
      "|     3327|\n",
      "|      191|\n",
      "|     3280|\n",
      "|     4126|\n",
      "|     3603|\n",
      "+---------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(\"sessionId\").dropDuplicates().sort(\"userId\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop userIds with \"\" \n",
    "df = df.filter(df[\"userId\"] != \"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "278154"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get feauture candidates for modeling\n",
    "df.select('userId', 'gender', \"ts\") \\\n",
    "    .where(df.userId == \"179\").count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+----+---------+------+-------------+--------+------+-----+--------+------+----+------------+---------+-----+------+---+---------+------+\n",
      "|artist|auth|firstName|gender|itemInSession|lastName|length|level|location|method|page|registration|sessionId| song|status| ts|userAgent|userId|\n",
      "+------+----+---------+------+-------------+--------+------+-----+--------+------+----+------------+---------+-----+------+---+---------+------+\n",
      "| 58392|   0|     8346|  8346|            0|    8346| 58392|    0|    8346|     0|   0|        8346|        0|58392|     0|  0|     8346|     0|\n",
      "+------+----+---------+------+-------------+--------+------+-----+--------+------+----+------------+---------+-----+------+---+---------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# check columns with Null values\n",
    "df.select([count(when(isnan(c) | col(c).isNull(), c)).alias(c) for c in df.columns]).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------------------------------+\n",
      "|count(CASE WHEN isnan(userID) THEN true END)|\n",
      "+--------------------------------------------+\n",
      "|                                           0|\n",
      "+--------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# missing values in userID\n",
    "df.select([count(when(isnan('userID'),True))]).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------------------------------------+\n",
      "|count(CASE WHEN isnan(sessionID) THEN true END)|\n",
      "+-----------------------------------------------+\n",
      "|                                              0|\n",
      "+-----------------------------------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# missing values in sessionID\n",
    "df.select([count(when(isnan('sessionID'),True))]).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get_hour = udf(lambda x: datetime.datetime.fromtimestamp(x / 1000.0). hour)\n",
    "# df = df.withColumn(\"hour\", get_hour(df.ts))\n",
    "# songs_in_hour = df.filter(df.page == \"NextSong\").groupBy(df.hour).count().orderBy(df.hour.cast(\"float\"))\n",
    "# songs_in_hour.show()\n",
    "\n",
    "# songs_in_hour_pd = songs_in_hour.toPandas()\n",
    "# songs_in_hour_pd.hour = pd.to_numeric(songs_in_hour_pd.hour)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create udf (user defined functions) for workdays and hour columns\n",
    "day_name= ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday','Sunday']\n",
    "day_ind = [1,2,3,4,5,6,7]\n",
    "\n",
    "# Convert to hours\n",
    "get_hour = udf(lambda x: datetime.datetime.fromtimestamp(x/1000).hour, IntegerType() )\n",
    "\n",
    "# Convert to days\n",
    "get_day = udf(lambda x: datetime.datetime.fromtimestamp(x/1000).day, IntegerType() )\n",
    "\n",
    "# Convert to Workdays in string format\n",
    "get_Workday = udf(lambda x: day_name[datetime.datetime.fromtimestamp(x/1000).weekday()] )\n",
    "\n",
    "# Create in workdays in integer format\n",
    "get_workday = udf(lambda x: day_ind[datetime.datetime.fromtimestamp(x/1000).weekday()], IntegerType() )\n",
    "\n",
    "# Create months\n",
    "get_month = udf(lambda x: datetime.datetime.fromtimestamp(x/1000).month, IntegerType() )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new columns based on ts column for analyses\n",
    "df = df.withColumn('hour', get_hour('ts'))\n",
    "df = df.withColumn('day', get_day('ts'))\n",
    "df = df.withColumn('workday_', get_Workday('ts'))\n",
    "df = df.withColumn('workday', get_workday('ts'))\n",
    "df = df.withColumn('month', get_month('ts'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+------+----+---+--------+-----+\n",
      "|userId|gender|hour|day|workday_|month|\n",
      "+------+------+----+---+--------+-----+\n",
      "+------+------+----+---+--------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get feauture candidates for modeling\n",
    "df.select('userId', 'gender', 'hour', 'day', 'workday_', 'month') \\\n",
    "    .where(df.userId == \"179\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.dropDuplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(artist=None, auth='Logged In', firstName='Daelin', gender='M', itemInSession=0, lastName='Turner', length=None, level='free', location='New York-Newark-Jersey City, NY-NJ-PA', method='GET', page='Home', registration=1538227408000, sessionId=125, song=None, status=200, ts=1538387004000, userAgent='\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36\"', userId='126', hour=12, day=1, workday_='Monday', workday=1, month=10)]"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+---+---------+-----+\n",
      "|hour|day| workday_|month|\n",
      "+----+---+---------+-----+\n",
      "|  12|  1|   Monday|   10|\n",
      "|  13|  1|   Monday|   10|\n",
      "|  13|  1|   Monday|   10|\n",
      "|  14|  1|   Monday|   10|\n",
      "|  15|  1|   Monday|   10|\n",
      "|  16|  1|   Monday|   10|\n",
      "|  19|  1|   Monday|   10|\n",
      "|   5|  2|  Tuesday|   10|\n",
      "|   8|  2|  Tuesday|   10|\n",
      "|  12|  2|  Tuesday|   10|\n",
      "|  17|  2|  Tuesday|   10|\n",
      "|  20|  2|  Tuesday|   10|\n",
      "|  20|  2|  Tuesday|   10|\n",
      "|   1|  3|Wednesday|   10|\n",
      "|   2|  3|Wednesday|   10|\n",
      "|   2|  3|Wednesday|   10|\n",
      "|   5|  3|Wednesday|   10|\n",
      "|   8|  3|Wednesday|   10|\n",
      "|   8|  3|Wednesday|   10|\n",
      "|  10|  3|Wednesday|   10|\n",
      "+----+---+---------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select('hour', 'day', 'workday_', 'month').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.groupBy('ts_hour').count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.groupBy('ts_day').count().show(31)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_log_valid = df.dropna(how = \"any\", subset = [\"userId\", \"sessionId\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "286500"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_log_valid.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+\n",
      "|userId|\n",
      "+------+\n",
      "|      |\n",
      "|    10|\n",
      "|   100|\n",
      "|100001|\n",
      "|100002|\n",
      "+------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select(\"userId\").dropDuplicates().sort(\"userId\").show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_log_valid = user_log_valid.filter(user_log_valid[\"userId\"] != \"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-8346"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_log_valid.count() - 286500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "##### df1 = df.dropDuplicates(['userId'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Data Analysis\n",
    "When you're working with the full dataset, perform EDA by loading a small subset of the data and doing basic manipulations within Spark. In this workspace, you are already provided a small subset of data you can explore."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Churn\n",
    "\n",
    "Once you've done some preliminary analysis, create a column `Churn` to use as the label for your model. I suggest using the `Cancellation Confirmation` events to define your churn, which happen for both paid and free users. As a bonus task, you can also look into the `Downgrade` events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|                page|\n",
      "+--------------------+\n",
      "|               About|\n",
      "|          Add Friend|\n",
      "|     Add to Playlist|\n",
      "|              Cancel|\n",
      "|Cancellation Conf...|\n",
      "|           Downgrade|\n",
      "|               Error|\n",
      "|                Help|\n",
      "|                Home|\n",
      "|               Login|\n",
      "|              Logout|\n",
      "|            NextSong|\n",
      "|            Register|\n",
      "|         Roll Advert|\n",
      "|       Save Settings|\n",
      "|            Settings|\n",
      "|    Submit Downgrade|\n",
      "| Submit Registration|\n",
      "|      Submit Upgrade|\n",
      "|         Thumbs Down|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# check the page column\n",
    "df.select(\"page\").dropDuplicates().sort(\"page\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+\n",
      "|avg(count(period))|\n",
      "+------------------+\n",
      "| 23.60389072847682|\n",
      "+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# How many songs do users listen to on average between visiting the home page? \n",
    "\n",
    "function = udf(lambda ishome : int(ishome == 'Home'), IntegerType())\n",
    "\n",
    "user_window = Window \\\n",
    "    .partitionBy('userID') \\\n",
    "    .orderBy(desc('ts')) \\\n",
    "    .rangeBetween(Window.unboundedPreceding, 0)\n",
    "\n",
    "cusum = df.filter((df.page == 'NextSong') | (df.page == 'Home')) \\\n",
    "    .select('userID', 'page', 'ts') \\\n",
    "    .withColumn('homevisit', function(col('page'))) \\\n",
    "    .withColumn('period', Fsum('homevisit').over(user_window))\n",
    "\n",
    "cusum.filter((cusum.page == 'NextSong')) \\\n",
    "    .groupBy('userID', 'period') \\\n",
    "    .agg({'period':'count'}) \\\n",
    "    .agg({'count(period)':'avg'}).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----------+\n",
      "|              Artist|Artistcount|\n",
      "+--------------------+-----------+\n",
      "|       Kings Of Leon|       1841|\n",
      "|            Coldplay|       1813|\n",
      "|Florence + The Ma...|       1236|\n",
      "|       Dwight Yoakam|       1135|\n",
      "|            BjÃÂ¶rk|       1133|\n",
      "+--------------------+-----------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# top 5 played artist\n",
    "df.filter(df.page == 'NextSong') \\\n",
    "    .select('Artist') \\\n",
    "    .groupBy('Artist') \\\n",
    "    .agg({'Artist':'count'}) \\\n",
    "    .withColumnRenamed('count(Artist)', 'Artistcount') \\\n",
    "    .sort(desc('Artistcount')) \\\n",
    "    .show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+---------+\n",
      "|                song|SongCount|\n",
      "+--------------------+---------+\n",
      "|      You're The One|     1153|\n",
      "|                Undo|     1026|\n",
      "|             Revelry|      854|\n",
      "|       Sehr kosmisch|      728|\n",
      "|Horn Concerto No....|      641|\n",
      "+--------------------+---------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# top 5 played songs\n",
    "df.filter(df.page == 'NextSong') \\\n",
    "    .select('song') \\\n",
    "    .groupBy('song') \\\n",
    "    .agg({'song':'count'}) \\\n",
    "    .withColumnRenamed('count(song)', 'SongCount') \\\n",
    "    .sort(desc('Songcount')) \\\n",
    "    .show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"They Can't Take That Away From Me (1999 Digital Remaster)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Little Yellow Spider'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Walk Through Hell (featuring Max Bemis Acoustic Exclusive)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Baby Girl_ I'm A Blur\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"You Don't Have A Clue\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Conspiracy (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sehr kosmisch'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Naughty Girl (single version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Artist In The Ambulance'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Trash'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Mr. Jones'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='In Denial (Live Acoustic)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Intensified'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='My Descent Into Madness'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='What If (Film Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"You'll See It\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Guilty As Charged [feat. Estelle] (Explicit Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Wie Jetzt'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='She Is My Sin'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='16 Candles'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Short Tales Of The Black Forest'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Never Let Me Go'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='So Lonely'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='This Is How We Do'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"It's Bad You Know\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Love Like Winter'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Free'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='This Is Halloween'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Greece 2000'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Estados'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Inhale (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Cuarto movimiento: La realidad'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='True Love Will Find You In The End'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Val's Blues (Digitally Remastered)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Creep (Explicit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Nutshell'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Home Entertainment'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Billionaire [feat. Bruno Mars]  (Explicit Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Hymn'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Big Girl'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Tres mil millones de latidos'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='sillyworld (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='And No Matches'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sexy Can I'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Cold As You'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='What If?'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Fight'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sample Track 2'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Black Is White'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='My Own Way'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Bright Lights (Remastered Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Some Kinda Rush'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Forks And Knives (La FÃ\\x83Âªte)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Check-It'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Panic'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='My Delirium'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Three Times A Lady'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Zealots'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Neighborhoods'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Visions Of Endless Hopes'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"What I've Done (Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='I Give You To His Heart'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Minor Earth Major Sky'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Feel Good Inc (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Lullaby In Three/Four'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Future Reflections'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Nemo Egg (Main Title)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Make Love To Your Mind'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Vagabond'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Out Of Sight'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Novio Imaginario'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Play Girl'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Condemned'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Hollywood's Not America\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Happy Song'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Paperweight'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Invalid'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Ghost Song'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Come On Eileen'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Here_ There And Everywhere'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Worried Down With the Blues'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sleepyhead'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Mr. Brownstone'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Rosemary'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Here We Go Again (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='One And Only'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Bent (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Monster (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Holiday [feat. Green Day & The Cast Of American Idiot] (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Blues Shaker'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='One Time'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Fire & Fast Bullets'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='DÃ\\x83Â³nde jugarÃ\\x83Â¡n los niÃ\\x83Â±os'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Riot In Cell Block Number Nine'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Freak Train'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Disappear (Oubliette)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sayonara-Nostalgia'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Be My Escape (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Voices On A String (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Holding Out For A Hero'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Elephant Gun'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Green Door'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Use Somebody'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Hanging With The Wrong Crowd'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Rorol'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='World in darkness'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Mr. Jones'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Read My Mind'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Undo'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Strut (1993 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sei LÃ\\x83Â¡ Mangueira'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Bachelor and the Bride'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Empire Ants (Feat. Little Dragon)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='End Of The Road'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Cheat'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"We Don't Have To Take Our Clothes Off (7'' Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Back On The Road'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Wounder'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Esperame'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Radioland (2009 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Miss Murder'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Next Time (Explicit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Gift'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Screaming For Vengeance'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Absurd'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"I'm But A Wave To...(2004 remix) (Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Morning Dance'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Tipsy'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='No Seas FanÃ\\x83Â¡tica'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Para Tu Amor'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Scientist'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"What's A Broken Heart\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='If You Want To Sing Out_ Sing Out'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Live And Let Die'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='ReprÃ\\x83Â©sente'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Float On'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Spectrum'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Soraya'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Geishas En Madrid'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Ballad Of Miss Kate'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Guillotine (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Esfera'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Drive Away'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='NiÃ\\x83Â±a Candela'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Extraordinary Girl/Letterbomb (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Undo'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Black 47'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Holiday [feat. Green Day & The Cast Of American Idiot] (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='New Zandu'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Kind Hearted Light'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Life's A Dream\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Rising Moon'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Fopp (Album)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Incentive (Bonus Track)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Ready For The Weekend'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Hi Ho Silver Lining'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Float On'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='22'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='History'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Undo'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='One More Time (Short Radio Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"I'm Still Breathing\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Every Time I Close My Eyes'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='U Don\\x19t Like Rock N Roll'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='My moon my man (Grizzy Bear Remix)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Crossfire'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sleigh Ride'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='People Help The People'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Reason'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Last Night A DJ Saved My Life'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Riot In Cell Block Number Nine'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Billionaire [feat. Bruno Mars]  (Explicit Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Away With Murder'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='You Can Do It (Explicit) (Feat. Mack 10 And Ms Toi)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Memories'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Just One Smile'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Lengths'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Familiar Taste Of Poison (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Again & Again'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Ambush (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Wish'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='I Know Where I\\'ve Been (\"Hairspray\")'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The American'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Invalid'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='No Celebration'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='A Bit Patchy'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Mile End'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Hard Rock Bottom Of Your Heart (Single Remix Remastered Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Love Train'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Ave Maria No Morro'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Onward 'Til Morning\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Uz'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Blindfolded'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"I Can't Get Started\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Neural Aggressor'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Mad Season (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Under Fire'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Ecstasy'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Flash Pan Hunter'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Guitarra y vos'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Something (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Snowbound (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Bummer In The Summer'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Witchcult Today'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Father Cannot Yell (Pete Shelley/Black Radio Mix)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Around The World / Harder Better Faster Stronger'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Fear'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Lookin' At Me [feat. Puff Daddy] (Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Fireworks (bonus track)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Midlife Crisis'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Show Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='After All (Single Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Lost'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Betcha Gon' Know (the prologue)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Calendar girl'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Footloose And Fancy Free'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Alma'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='This Joint'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Floating Vibes'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Deathzone'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Don't You Forget About Me\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Another (Featuring Lil' Kim) (Amended Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='1234_ 1234 (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Loaded For Bear'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Delirium Trigger'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Feel Good Inc (Single Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Help', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Help', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Hand Of Doom'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Fireworks'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"I'm Made Of Wax_ Larry_ What Are You Made Of?\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Dog Days Are Over (Radio Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Tempest (album version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='That Home'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Adrift'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='LoveStoned/I Think She Knows'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sleep After Toyle_ Port After Stormie Seas'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Blue Orchid'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Pieces Of Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Photographs'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Ain't That Peculiar\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Hello'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"la possibilitÃ\\x83Â© d'une Ã\\x83Â®le\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Love Story'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"That Ain't The Way To Behave (2006 Digital Remaster)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='I Can Go Deep (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Funeral (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Alejandro'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Daylight'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Higher Messiah'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Madre Tierra'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='1517'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Hide & Seek'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='I Gotta Feeling'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Jack & Diane'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Ice Is Getting Thinner (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Head Over Heels'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Christmas Is My Favorite Time Of The Year'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Dog Days Are Over (Demo)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Colourful'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Taste The Pain'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Jefferson Aeroplane [Demo]'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='I Hate Everything About You'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Slave To Love (1999 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Hey_ Soul Sister'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Every Lasting Light'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='To Speak Your Name (Split Trax)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='A Mile In My Shoes ft. Sir Adrian'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='I Second That Emotion'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Nothin' On You [feat. Bruno Mars] (Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Mia'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Sugar_ We're Goin Down\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Billionaire [feat. Bruno Mars]  (Explicit Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Love Story'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Robot Rock'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Amigo (7'' Edit)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Roger That'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Pela Janela (Through the Window) featuring Gigi Rezende'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='My Boy Builds Coffins'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Woodstock'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Murano'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Lasso'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Smoke Some Weed (edited version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Gears'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Creep (Explicit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Error', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Easy'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"There's a Better Way\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Without Lions'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Rolodex Propaganda'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Revelry'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='El Cantor De Fonseca'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"It's Not The End Of The World?\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Everything In Its Right Place'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sacrifice'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Forgot About Dre'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Scientist'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Summer'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Rehab'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Mother Of Abominations (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Kansas City'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='BreÃ\\x83Â±a'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Composure'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Adam's Song\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Wavin'  Flag\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Soft Shock'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='SinceritÃ\\x83Â© Et Jalousie'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Pursuit Of Happiness (nightmare)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Flagpole Sitta'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Blue'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Take A Bow'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Catch You Baby (Steve Pitron & Max Sanna Radio Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Pure Funk'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Don't Think I Don't Think About It\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Help', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Stricken'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Someone Great'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Bubble Toes'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='A Thousand Miles'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Find The Reason (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Troubled Waters'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Rain Is A Good Thing'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Dust In The Wind'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Up Up & Away'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Drop The World'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Outshined'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sooner Surrender'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='La Pared'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Joy Ride'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Map Of The Problematique'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Do You Wanna Dance? (9-11-01)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Emnete'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='You And Me Jesus'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Corazon Espinado'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='You Gave Me A Promise'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Crash'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Everything's Ruined\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='!@*$%#'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Dance Stance'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Frisch und g'sund\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Permanent'),\n",
       " Row(userId='98', firstname='Sawyer', page='Upgrade', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Only Superstition'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Blind'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Heiterkeit'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sehr kosmisch'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Unwell (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Cacca Nello Spazio'),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Real Slim Shady'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Never Before'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Game (Amended Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='White Flag Warrior'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Melanor'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Not Sorry'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Someday'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Fly High Michelle (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='The Divided Sky (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"The Remedy (I Won't Worry) (New EQ'd LP Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Stargazer'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Kissin' Ears [feat. The-Dream] (Explicit Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Edge Of Love'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Money Straight (Explicit Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='No You Girls'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Daisies Of The Galaxy'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='No Llores'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"I've Got The World On A String (1995 Digital Remaster)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Roll Advert', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Hazey Jane I'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Vix Tape Tappeh'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Leaving In The Morning'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"'Round Midnight\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Error', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song=\"Don't Pull Your Love\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Sweet And Wild'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='free', song='Dissident'),\n",
       " Row(userId='98', firstname='Sawyer', page='Upgrade', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Submit Upgrade', level='free', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Kick Drum'),\n",
       " Row(userId='98', firstname='Sawyer', page='Downgrade', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Give It To Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Horn Concerto No. 4 in E flat K495: II. Romance (Andante cantabile)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Oh Lord_ give us more money'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Ravenous (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Third Party'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Angie (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Beautiful'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Hotel California'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='La Guerra'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='2 Late 4 Love'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Tempest (album version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Tension'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fools'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Iron Maiden (Live In Dortmund)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Almaz'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"No_ It Isn't\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Who is the Black Angel?'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Revelry'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Redneck Woman'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fly (The Face Of Love Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"'Till I Collapse\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='White Jetta'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Hail To Whatever You Found In The Sunlight That Surrounds You'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Sometimes I Feel Like A Motherless Child (2004 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='En La Ciudad De La Furia'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='What Are You Doing The Rest Of Your Life?'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Le Courage Des Oiseaux'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='We Could Run Away (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Rocket Queen'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Calyx'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Transcendence'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"(I'm A) Donkey for Your Love\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Two Little Girls From Little Rock'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Nutcracker - Suite Op. 71a (2007 Digital Remaster): I.    Miniature Overture'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Pump It'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Damelo'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Adrenalin'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Dumb It Down (feat. GemStones and Graham Burris) (Amended Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Not An Addict'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Get Original'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"That's How Country Boys Roll\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Way Things Go'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Jamaica Roots II(Agora E Sempre)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='For Once In My Life (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Catacombs'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Lovin' You Is Sweeter Than Ever feat Donna Gardier)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Cream'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Negative Creep'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Lonely (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Opened Once'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Hold You In My Arms'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"This Ain't A Love Song\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"I'm Saving My Love\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Stare At The Sun'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='I Kissed A Girl'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Dog Days Are Over (Radio Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Run Through The Jungle'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Wizards'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Heist Manoeuvre'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Masquerade (Reality Check Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Hemma FÃ\\x83Â¶re MÃ\\x83Â¶rkret'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Catch You Baby (Steve Pitron & Max Sanna Radio Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='O Segundo Sol'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Straight To The Bank'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='One More Time (Short Radio Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='I Need A Man To Love'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Rabbit Heart (Raise It Up)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Interactive Halloween Bear'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Pick Up'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Float On'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='A Certain Romance'),\n",
       " Row(userId='98', firstname='Sawyer', page='Settings', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Pretender'),\n",
       " Row(userId='98', firstname='Sawyer', page='Help', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Abrazame'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='World Turning (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"I CAN'T GET STARTED\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='These Arms Of Mine'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fireflies'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Paradigm Shift'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Losing My Edge'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Wish'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Revelry'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Yeah!'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Too Fine To Do Time'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Little Motel'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"I Guess I'm Floating\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Magnetising'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='I Am Civil Service'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Concera'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Per Te (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Rest Stop (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Jewel'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Party All Day'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='YÃ\\x83Â¶n hetket'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Do Me Again (2007 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"When I'm Gone (Acoustic Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Lean Back [Remix feat. Lil Jon_ Eminem_ Mase & Remy Martin] (Album Version - Amended)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='16 Candles'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Tears In Heaven'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"I Don't Need A Man\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Swordfish Hotkiss Night'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Frisch und g'sund\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Help', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Habalim'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"That's Why I Love You So\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Frisch und g'sund\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Angel Face'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"It's Never Too Late (Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Veleta (En Vivo Auditorio Nacional)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Undantag'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='One People'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='A Save Situation (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='A Beggar On A Beach Of Gold'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Neon Knights'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='I Gotta Feeling'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Apple'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Shadow Captain (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Phantom Lord'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Politik'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Land Of Confusion (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Nothin' On You [feat. Bruno Mars] (Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Beauty*2'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Orange TrÃ\\x83Â¤gt Nur Die MÃ\\x83Â¼llabfuhr (Go West)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Atlantic'),\n",
       " Row(userId='98', firstname='Sawyer', page='Settings', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mas Y Mas (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Sometimes When We Touch'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Be Somebody'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='SEQUENCE'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='My Savior My God (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Extra Ordinary (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Unknown Soldier'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Ain't Misbehavin\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Do You Know'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Strut (1993 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"You Don't Belong\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Glorious'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Transatlanticism'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Indian Deli'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Music'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Never There'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Sugar_ We're Goin Down\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Unbreakable'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Bloodline'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Don\\x19t Touch Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='City Love'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Piggy'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Sin Documentos'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Use Somebody'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Commotion'),\n",
       " Row(userId='98', firstname='Sawyer', page='Downgrade', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Adam's Song\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Length Of Love'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Suspended From Class'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='True Spirit (K-klassic Mix)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Rebels of the Sacred Heart'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Gonna Make You Love Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='O Pato'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Te LastimÃ\\x83Â©'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Farewell'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Aghia Sophia (2008 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Road To Zion'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Make Her Say'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Behind The Sea [Live In Chicago]'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Before I Forget (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Daydreaming'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Ghosts of Ocean Gardens'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Whiskey Girl'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Somebody Told Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Heartbeat'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Please Please Please Let Me Get What I Want'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Speak'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='La vida es mas compleja de lo que parece'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Pa'l Norte\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Debajo del mar'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Love Ja Ja Children Version'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Aerodynamic (Daft Punk Remix)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Settings', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Paul McCartney'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Little Love (Radio Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Due'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Feeling This'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Rule Of Thumb (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Line & Sinker (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Piano Music'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Secrets'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Silence'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='What A Waster'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='All-O-Gistics'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Feathers and Down'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Me Da Miedo'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Recado Falado (MetrÃ\\x83Â´ Da Saudade)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Nothin' On You [feat. Bruno Mars] (Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Goin' Steady\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Don't Dance\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Undo'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Ry Ry's Song (Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Please_ Before I Go'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Shaving Your Head'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Love The Way You Love Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Nightly Journey'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Prince Caspian'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Master Of Puppets'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Halfway There'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='So Wrong (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"You're The One\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Everyone's At It\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Black & Blue'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Leader Of The Pack'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Gears'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Osaka Loop Line'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Who Can Compare'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fight'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Smile Version Revisited (Mark Ronson Remix) (Explicit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Senhor Do Tempo'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Someone Take The Wheel'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Silhouette'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Going Under'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fix You'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Just Stop (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Billy Breathes'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Have I Been A Fool'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Glamour Girl'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Cerdo'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Le Jardin d'Hiver\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mudfly'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Tive Sim'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Sei Un Mito'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Runaway Love'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Speechless'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Quando_ Quando_ Quando (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Until The Morning'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Don't Wake Me Up\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Hole In The Bucket'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Cross [Acoustic]'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Around The World (Radio Edit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Shit On You'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Unite (2009 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='All I Could Bleed (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Creep (Explicit)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Wine Red (Tommie Sunshine's Brooklyn Fire Retouch)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Won't Go Home Without You\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Unite (2009 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Le Bagad De Lann Bihoue (Live)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Soldier Side'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Half Acre'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='My Life Would Suck Without You'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Give A Little Bit (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='OPROSTI'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Tighten Up'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Un Aro En La Nariz'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Up Out My Face'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"You're Beautiful (Album Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mercy:The Laundromat'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Rockin' Robin\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Party Like A Rock Star'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Modern Guilt'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Unholy Confessions'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Don't Stop The Music\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Like A Boss'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Walk-Don't Run\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='40 Days'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Say No More'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Killer Is Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='More Than A Feeling'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='To Build A Home'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Crazy Beautiful'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Planet Claire'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mysterious Ways'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Winding Road'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Forgive Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='What You Know'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Dead And Gone [feat. Justin Timberlake] (Explicit Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Pretender'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Tout petit la planÃ\\x83Â¨te'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Plastic Gift'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Bullet Theory'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Shine On'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Bentley's Gonna Sort You Out\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Make It Mine (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Seafloor'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Beach Party'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fortunate Son'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Green Shirt'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Green Hell'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Tremble (Original Club Mix)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mouth Shut [Live] (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='After The Gold Rush (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fall Into Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fireflies'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='High School Musical'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Alone'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Distinto'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Straight In At 101'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Dick Lips'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Thousand Names of Lord Shiva (Part 1)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Wicked Garden (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mr Bojangles'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Feel Like Makin' Love (LP Version)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Small People'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Survive'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Ambrosia'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Party Like A Rock Star'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Get It On'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Nursery_ Academy'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Heartbeats'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Revenga'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Latin Static'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='This Godless Endeavor'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Hey Daddy (Daddy's Home)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mushaboom'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='SinceritÃ\\x83Â© Et Jalousie'),\n",
       " Row(userId='98', firstname='Sawyer', page='Settings', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Knife Prty (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Down', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Letters From Madrid'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fast Car (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='There Goes My Baby'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Signs'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Crazy In Love'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='De Madrugada'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Undo'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Miss Murder'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mary'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Holiday'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"It's Raining Men\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Help', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='When I Said I Do'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Statue Of Liberty'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Good Time Women'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Monkey Man'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='I Kissed A Girl'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Number One'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='All My Life'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Ray Gun'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Lo Importante EstÃ\\x83Â¡ En Tu Cabeza'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Taking Us Home'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Last Thing I Needed (First Thing This Morning)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Bleed It Out [Live At Milton Keynes]'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Funkytown'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='My Moon My Man'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"I Bet She Doesn't Feel It\"),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Home'),\n",
       " Row(userId='98', firstname='Sawyer', page='Downgrade', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Fool'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Le Courage Des Oiseaux'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"It's A Sin (2001 Digital Remaster)\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='So Long'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='For Whom The Bell Tolls'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Stronger'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Hell Alright (Bassface Sascha & Franksen Remix)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Tabaco Y Chanel'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Turn Me Up (Sfaction Version / Feat. Sandy)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Time No More'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Lovin' You is Murder\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Red Socks Pugie'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Horn Concerto No. 4 in E flat K495: II. Romance (Andante cantabile)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Malkaus - Alap'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Sehr kosmisch'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Gli Amici Miei'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='El Chubasco'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Holiday [feat. Green Day & The Cast Of American Idiot] (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"I'll Be Doggone\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='I Say No'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Rorol'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mirror Song'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Love Is Gone (Fred Riester & Joachim Garraud Rm)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Dirt Off Your Shoulder'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Better Living Through Chemistry'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Before I Let You Go'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Where Corn Don't Grow\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Ooh La'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Maestro'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Let Me Be Myself'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Freedom For The Stallion'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Mama Said Knock You Out'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Amanda'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Long As I Can See The Light'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='House Of Pain Anthem (Original Mix)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Am I Evil'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='When You Were Young'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Blinded By The Lights [Nero Remix]'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add to Playlist', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Pearl Vision'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Under Your Spell'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Inolvidable'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Living In The Shadows (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Lull'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Crash'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Tangled Up In Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Los Aviones'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Ni TÃ\\x83Âº Ni Nadie (VersiÃ\\x83Â³n Demo)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Liberty Line'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Cold Desert'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Scotty (explicit album version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"It's My Own Fault\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Have You Ever Felt?'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='You Got Me'),\n",
       " Row(userId='98', firstname='Sawyer', page='Thumbs Up', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Alpha Shallows'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Heard Them Stirring'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Swing Javanaise'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Sing Along Forever'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Undo'),\n",
       " Row(userId='98', firstname='Sawyer', page='Add Friend', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Sinequanon'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Leave The Bourbon On The Shelf'),\n",
       " Row(userId='98', firstname='Sawyer', page='Logout', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"I CAN'T GET STARTED\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Shop Around'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Fire Flood And Plague'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Invalid'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='UnBirthday'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Lottery Spin'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Laser Speed (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Day In_ Day Out (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Shiver'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Secrets'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Step Up'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='One Way Mule (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"I'll Be Right Back\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Overture'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Joker'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Meteor'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Too Daze Gone'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The Gift'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='The City Sleeps'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Orbit'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Strong Enough'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Stupid Love Letter (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Verde Que Te Quero Rosa'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Wherever We Go'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Double Dare Ya'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Super 8'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Romance To The Grave'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Exodus (Intro)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Secret Crowds'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Bambino (2006 Digital Remaster)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Trunk'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Comforter'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Revolver'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Hijo Del Sol Luminoso (vivo)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='These Four Walls'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='DIA (Album Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Keep You Much Longer'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Sugar Ray (LP Version)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song=\"Who'd You Kill Now?\"),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Pregon Para Iluminarse'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Cowboys'),\n",
       " Row(userId='98', firstname='Sawyer', page='Home', level='paid', song=None),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Wild Wind'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Last Of Days'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Satelite (Live)'),\n",
       " Row(userId='98', firstname='Sawyer', page='NextSong', level='paid', song='Kelma'),\n",
       " ...]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check a random user\n",
    "df.select([\"userId\", \"firstname\", \"page\", \"level\", \"song\"]) \\\n",
    "    .where(df.userId == 98).collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a udf to flag Churned customers\n",
    "flag_downgrade_event = udf(lambda x: 1 if x == \"Cancellation Confirmation\" else 0, IntegerType())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new column \"churn\"\n",
    "df = df.withColumn(\"churn\", flag_downgrade_event(\"page\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(artist=None, auth='Logged In', firstName='Daelin', gender='M', itemInSession=0, lastName='Turner', length=None, level='free', location='New York-Newark-Jersey City, NY-NJ-PA', method='GET', page='Home', registration=1538227408000, sessionId=125, song=None, status=200, ts=1538387004000, userAgent='\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36\"', userId='126', hour=12, day=1, workday_='Monday', workday=1, month=10, churn=0)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+\n",
      "|userId|churn|\n",
      "+------+-----+\n",
      "|100001|    1|\n",
      "|100003|    1|\n",
      "|100005|    1|\n",
      "|100006|    1|\n",
      "|100007|    1|\n",
      "|100009|    1|\n",
      "|100011|    1|\n",
      "|100012|    1|\n",
      "|100013|    1|\n",
      "|100014|    1|\n",
      "|100015|    1|\n",
      "|100017|    1|\n",
      "|100019|    1|\n",
      "|100021|    1|\n",
      "|100022|    1|\n",
      "|100023|    1|\n",
      "|100024|    1|\n",
      "|100025|    1|\n",
      "|   101|    1|\n",
      "|   103|    1|\n",
      "+------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select('userId', 'churn').where(df.churn == 1).sort(\"userId\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add a staging: stage=1 if the user is paid, stage=2 if the user churned\n",
    "windowval = Window.partitionBy(\"userId\").orderBy(desc(\"ts\")).rangeBetween(Window.unboundedPreceding, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create new column stage to assign values\n",
    "df = df.withColumn(\"phase\", Fsum(\"churn\").over(windowval))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "User 39 played the biggest number of songs. Let's check if that matters by going through it's records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='39', firstname='Payton', ts=1538375748000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538375752000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538375952000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538376332000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538376376000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538376557000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538376754000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538377032000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538377079000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538377451000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538377604000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538377780000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538377824000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538378060000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538378336000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538378353000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538378726000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538378953000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538379219000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538379381000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538379802000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538379803000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538379896000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538380026000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538380263000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538380294000, page='About', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538380531000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538380560000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538380879000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538381061000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538381345000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538381435000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538381607000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538381875000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538382162000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538382596000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538382900000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538383177000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538383446000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538383622000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538383787000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538384057000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538384448000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538384686000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538384860000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538384860000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538385114000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538385114000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538385311000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538385562000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538385847000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538386135000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538386419000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538386420000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538386644000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538386924000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538387107000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538387139000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538387315000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538387562000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538387789000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538388091000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538388306000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538388520000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538388701000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538389007000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538389467000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538389616000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538389718000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538389723000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538390032000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538390033000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538390177000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538390224000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538390472000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538411191000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538411217000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538411603000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538411870000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538412057000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538412058000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538419890000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538419934000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538420111000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538420296000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538420494000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538420723000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538420791000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538421112000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538421324000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538421535000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538421761000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538422022000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538422023000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538422245000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538422458000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538422659000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538422695000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538436291000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538436307000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538436715000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538436957000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538437114000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538437362000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538437640000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538437642000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538437850000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538438065000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538438317000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538438541000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538438824000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538438842000, page='Upgrade', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538463562000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538463799000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538463870000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538464022000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538464042000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538464257000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538464489000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538464714000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538464965000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538465220000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538465451000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538465452000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538465505000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538465665000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538465703000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538466252000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538466554000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538466555000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538466593000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538481410000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538481468000, page='Settings', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538481542000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538481657000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538481889000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538481921000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538482146000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538482364000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538482386000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538482728000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538482964000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538483141000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538483309000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538483547000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538483801000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538483896000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538484089000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538484355000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538484545000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538484552000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538484747000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538484932000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538485280000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538485620000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538486005000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538486217000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538486907000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538486919000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538492497000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538492507000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538492508000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538492577000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538492719000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538492852000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538493163000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538493343000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538493595000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538493880000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538494102000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538494342000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538494484000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538494663000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538494829000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538494934000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538495067000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538495196000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538495468000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538495473000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538495681000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538495730000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538495925000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538495954000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538496188000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538496453000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538496755000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538496756000, page='Add Friend', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538496983000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538497021000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538499696000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538499696000, page='Upgrade', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538499949000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538500229000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538500468000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538500516000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538500649000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538500650000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538500868000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538501098000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538501283000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538501284000, page='Add Friend', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538501285000, page='Add Friend', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538501594000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538501595000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538501863000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538502195000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538502280000, page='Settings', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538502281000, page='Save Settings', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538511952000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538511953000, page='Add Friend', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538511954000, page='Add Friend', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538511979000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538512169000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538512387000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538512624000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538512868000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538512872000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538513237000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538513464000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538513658000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538513907000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538514082000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538514304000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538514500000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538514675000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538514875000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538515121000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538515176000, page='Error', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538515366000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538515574000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538515751000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538516002000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538516240000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538516631000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538516819000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538517054000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538517171000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538517332000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538517510000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538517707000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538517979000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538518160000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538518583000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538518858000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538518941000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538519125000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538519489000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538519770000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538519980000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538520191000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538520508000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538520772000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538520773000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538551821000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538551826000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538552086000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538552388000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538552646000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538552647000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538552665000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538567480000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538567579000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538567615000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538567796000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538568007000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538568128000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538568397000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538568584000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538568851000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538569076000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538569342000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538569343000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538569522000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538569790000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538569791000, page='Thumbs Down', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538569994000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538570078000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538570079000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538570384000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538570560000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538570561000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538570811000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538570882000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538571108000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538571293000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538571319000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538613460000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538613766000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538613992000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538614240000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538614420000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538614421000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538614458000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538614710000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538614893000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538615170000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538615431000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538615716000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538615931000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538616160000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538616161000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538616458000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538616666000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538616893000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538617074000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538617251000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538617300000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538617548000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538617848000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538618026000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538618258000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538618350000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538618558000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538618828000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538619093000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538619378000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538619567000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538652466000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538652656000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538652863000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538653056000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538653217000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538653494000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538653646000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538653996000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538654173000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538654263000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538654309000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538654344000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538654344000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538654375000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538654473000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538654698000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538654965000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538655310000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538655450000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538655637000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538655747000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538655841000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538656049000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538656050000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538656240000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538656445000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538656695000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538656890000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538656913000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538657166000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538657391000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538657401000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538657620000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538657873000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538658034000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538658046000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538658260000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538658269000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538658515000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538658721000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538658937000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538667730000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538668037000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538668293000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538668553000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538668721000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538668722000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538668919000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538669194000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538669400000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538669628000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538669973000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538670368000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538670369000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538670592000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538670593000, page='Thumbs Down', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538670849000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538671149000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538671361000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538671562000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538671575000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538671784000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538672024000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538672060000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538672205000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538672206000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538672263000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538672411000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538672412000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538672803000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538673105000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538673343000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538673561000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538673896000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538673942000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538754753000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538754952000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538755108000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538755358000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538755515000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538755516000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538755772000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538756050000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538756388000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538756760000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538757064000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538757358000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538757581000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538757791000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538757982000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538757983000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538758030000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538758228000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538758359000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538758360000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538758762000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538759136000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538759352000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538759611000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538759811000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538759813000, page='Upgrade', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538760092000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538760396000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538760589000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538760798000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538761002000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538761276000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538761560000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538761572000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538761836000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538761990000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538762261000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538762519000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538762520000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538762827000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538763058000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538763297000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538763588000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538763604000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538764232000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538764329000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538764363000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538764511000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538764730000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538764731000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538783038000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538783070000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538783282000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538783572000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538783807000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538784039000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538784214000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538784475000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538784667000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538784949000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538784950000, page='Add Friend', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538785227000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538785416000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538785677000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538785942000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538786006000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538786248000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538786831000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538787072000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538787363000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538787613000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538787816000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538788074000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538788249000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538788292000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538788494000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538788723000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538788732000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538788820000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538788993000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538789249000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538789701000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538789909000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538789910000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538790198000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538790511000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538790676000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538790898000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538791388000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538791655000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538791890000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538792226000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538792377000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538792597000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538792918000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538793127000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538793448000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538793651000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538793876000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538794077000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538794078000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538794278000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538794678000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538794809000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538829345000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538829548000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538829806000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538830030000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538830076000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538830221000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538830470000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538830791000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538830792000, page='Add Friend', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538831001000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538831267000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538831613000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538831856000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538832057000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538832262000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538832746000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538833079000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538833291000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538833610000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538833878000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538834212000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538834395000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538834670000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538834933000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538835026000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538835204000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538835489000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538835907000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538836138000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538836367000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538836617000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538836627000, page='Upgrade', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538836673000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538836739000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538836741000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538836798000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538836806000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538837028000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538837311000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538837646000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538837837000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538838088000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538838133000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538838136000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538838294000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538838451000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538838619000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538838820000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538838821000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538930832000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538930855000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538930856000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538930999000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538931210000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538931401000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538931422000, page='Help', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538931529000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538977366000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538977738000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538977747000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538977978000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538978264000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538978322000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538978521000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538978863000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538979044000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538979212000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538979213000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538979475000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538979476000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538979753000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538979996000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538980305000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1538980306000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539004222000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539004268000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539004570000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539004872000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539005057000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539005100000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539005359000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539005483000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539005722000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539005863000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539005972000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539006173000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539006330000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539006528000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539006739000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539007007000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539016417000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539016468000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539016469000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539018367000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539018436000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539018437000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539018620000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539018626000, page='Settings', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539018918000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539019032000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539019122000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539019352000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539019375000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539019596000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539019607000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539019623000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539019815000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539020795000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539021100000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539021101000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539021153000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539021439000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539021440000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539021694000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539022052000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539022082000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539054211000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539054212000, page='Add Friend', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539054459000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539054631000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539054845000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539055081000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539055317000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539055624000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539055916000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539056151000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539056152000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539056443000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539059467000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539059472000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539059727000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539060005000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539060449000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539060633000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539060634000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539060947000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539061132000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539061409000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539061636000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539061654000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539061844000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062156000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062162000, page='Roll Advert', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062357000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062358000, page='Logout', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062505000, page='Home', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062709000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062753000, page='Add to Playlist', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062754000, page='Thumbs Up', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062963000, page='NextSong', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062993000, page='Upgrade', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539062994000, page='Submit Upgrade', level='free', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539063069000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539063172000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539063404000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539063840000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539064122000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539064383000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539064592000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539064878000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539065101000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539065102000, page='Thumbs Up', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539065356000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539065356000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539065525000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539065761000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539066038000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539066213000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539066432000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539066433000, page='Thumbs Down', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539066584000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539066817000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539067067000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539067402000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539067846000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539068048000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539068339000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539068572000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539068806000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539069055000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539069056000, page='Thumbs Up', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539069068000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539069270000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539069463000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539069740000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539069977000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539070161000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539070396000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539070803000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539071104000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539071274000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539071501000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539071912000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539072135000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539072285000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539072562000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539072821000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539073057000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539073318000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539073381000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539073446000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539073623000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539073865000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539074049000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539074317000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539074558000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539074731000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539074894000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539075022000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539075250000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539075499000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539075773000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539075996000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539076290000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539076419000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539076720000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539076956000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539077279000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539077477000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539077478000, page='Thumbs Down', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539077685000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539077917000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539078192000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539078564000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539078937000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539079269000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539079626000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539079962000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539080238000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539080258000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539080447000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539080721000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539081141000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539081444000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539081585000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539081871000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539082169000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539082402000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539082626000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539082919000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539082965000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539083192000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539083375000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539083695000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539083865000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539084174000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539084458000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539084637000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539084808000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539084955000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539084981000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539085164000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539085689000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539085988000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539086271000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539086477000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539086526000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539086747000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539087104000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539087226000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539087569000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539087886000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539087984000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539088257000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539088475000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539088816000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539089042000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539089282000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539089493000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539089699000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539089716000, page='Downgrade', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539089830000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539089839000, page='Settings', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539089886000, page='Help', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539089887000, page='Add Friend', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539090502000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539090944000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539091219000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539091499000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539091718000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539091982000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539092220000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539092244000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539092508000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539092799000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539093006000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539093277000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539093457000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539093696000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539094011000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539094241000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539094485000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539094682000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539094921000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539095066000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539095319000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539095389000, page='Settings', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539095607000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539095653000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539095787000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539096198000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539096423000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539096448000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539096596000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539096839000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539097199000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539097580000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539097805000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539098003000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539098004000, page='Thumbs Up', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539098207000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539098367000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539098619000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539098870000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539099092000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539099314000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539099462000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539099750000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539099932000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539100137000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539100358000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539100557000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539100621000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539100940000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539101176000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539101512000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539101714000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539101908000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539102042000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539102230000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539102374000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539102542000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539102688000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539102689000, page='Logout', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539102792000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539102869000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539103078000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539103178000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539103426000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539103628000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539103930000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539104160000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539104451000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539104661000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539105033000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539105272000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539105530000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539105878000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539106076000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539106447000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539106654000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539106789000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539107033000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539107414000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539107455000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539107643000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539108298000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539108631000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539108896000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539109114000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539109321000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539109650000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539109882000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539110120000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539110435000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539110691000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539110711000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539110874000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539111043000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539111332000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539111484000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539111704000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539111996000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539112338000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539112848000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539112849000, page='Thumbs Down', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539113026000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539113237000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539113443000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539113642000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539113843000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539113844000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539114007000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539114008000, page='Thumbs Up', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539114276000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539114442000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539114725000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539114969000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539115207000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539115448000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539115615000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539115854000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539116098000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539116540000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539116725000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539116726000, page='Add Friend', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539116727000, page='Add Friend', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539116973000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539116993000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539117194000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539117463000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539117649000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539117997000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539118200000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539118431000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539118801000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539118817000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539119117000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539119465000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539119582000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539119704000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539119926000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539120196000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539120522000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539120728000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539121150000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539121400000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539121588000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539121865000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539122168000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539122498000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539122743000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539123207000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539123472000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539124127000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539124151000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539124375000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539124560000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539124833000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539125133000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539125151000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539125372000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539125390000, page='About', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539125524000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539125756000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539125963000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539126178000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539126183000, page='Add to Playlist', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539126550000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539126699000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539126955000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539127092000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539127102000, page='Settings', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539127363000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539127678000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539127679000, page='Thumbs Down', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539127796000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539127964000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539128137000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539128643000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539128866000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539129188000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539129393000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539129620000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539129621000, page='Thumbs Up', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539129829000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539130008000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539130195000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539130463000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539130843000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539131096000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539131403000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539131668000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539131896000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539132183000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539132414000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539132602000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539132773000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539133048000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539133168000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539133415000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539133563000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539133838000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539134288000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539134612000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539134880000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539135036000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539135037000, page='Thumbs Down', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539135331000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539135332000, page='Thumbs Down', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539135497000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539135758000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539135960000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539135961000, page='Thumbs Up', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539136230000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539136725000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539136737000, page='Home', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539136964000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539137344000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539137345000, page='Thumbs Up', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539137601000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539137939000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539137940000, page='Thumbs Up', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539138172000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539138462000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539138868000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539139059000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539139474000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539139617000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539140000000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539140245000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539140458000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539140575000, page='NextSong', level='paid', phase=0),\n",
       " Row(userId='39', firstname='Payton', ts=1539140576000, page='Add Friend', level='paid', phase=0),\n",
       " ...]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check result\n",
    "df.select([\"userId\", \"firstname\", \"ts\", \"page\", \"level\", \"phase\"]) \\\n",
    "    .where(df.userId == \"39\").sort(\"ts\").collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "User 39 started as free, but after some decent amount of songs listened it became a paid member. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore Data\n",
    "Once you've defined churn, perform some exploratory data analysis to observe the behavior for users who stayed vs users who churned. You can start by exploring aggregates on these two groups of users, observing how much of a specific action they experienced per a certain time unit or number of songs played."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+-----+\n",
      "|gender|churn|count|\n",
      "+------+-----+-----+\n",
      "|     F|    0|  104|\n",
      "|     M|    0|  121|\n",
      "+------+-----+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# split by gender \n",
    "df.dropDuplicates(['userId']).groupBy(['gender','churn']).count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----+-----+\n",
      "|                song|churn|count|\n",
      "+--------------------+-----+-----+\n",
      "|Ten Foot Pole (LP...|    0|    1|\n",
      "|        Crush On You|    0|    1|\n",
      "|      You're The One|    0|    2|\n",
      "|                Rise|    0|    1|\n",
      "|     The Time Is Now|    0|    1|\n",
      "|             Jam 292|    0|    1|\n",
      "|       Octavo DÃÂ­a|    0|    1|\n",
      "|Ganas De Ti (Albu...|    0|    1|\n",
      "|I'd Hate To Be Yo...|    0|    1|\n",
      "|Dialogue 14: Hono...|    0|    1|\n",
      "| Out At The Pictures|    0|    1|\n",
      "|      Fancy Footwork|    0|    1|\n",
      "|      My Moon My Man|    0|    1|\n",
      "|Rise To The Chall...|    0|    1|\n",
      "|              Galaxy|    0|    1|\n",
      "|                  II|    0|    1|\n",
      "|  If I Ain't Got You|    0|    1|\n",
      "|         Me Pregunto|    0|    1|\n",
      "|Easy Eats Or Dirt...|    0|    1|\n",
      "|      Teach Me Tiger|    0|    1|\n",
      "+--------------------+-----+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.dropDuplicates(['userId']).groupBy(['song','churn']).count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+------+-----------+\n",
      "|churn|userId|SongsPlayed|\n",
      "+-----+------+-----------+\n",
      "|    0|    39|       8002|\n",
      "|    0|    92|       5945|\n",
      "|    0|   140|       5664|\n",
      "|    0|300011|       4619|\n",
      "|    0|   124|       4079|\n",
      "|    0|300021|       3816|\n",
      "|    0|300017|       3632|\n",
      "|    0|    85|       3616|\n",
      "|    0|    42|       3573|\n",
      "|    0|     6|       3159|\n",
      "|    0|    29|       3028|\n",
      "|    0|200023|       2955|\n",
      "|    0|    54|       2841|\n",
      "|    0|   100|       2682|\n",
      "|    0|     9|       2676|\n",
      "|    0|    91|       2580|\n",
      "|    0|   126|       2577|\n",
      "|    0|300015|       2524|\n",
      "|    0|    98|       2401|\n",
      "|    0|    74|       2400|\n",
      "+-----+------+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Show the number of Songs Played per user ID\n",
    "df.where(df.song != 'null').groupBy(['churn','userId']) \\\n",
    "    .agg(count(df.song).alias('SongsPlayed')) \\\n",
    "    .sort('SongsPlayed', ascending=False).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- artist: string (nullable = true)\n",
      " |-- auth: string (nullable = true)\n",
      " |-- firstName: string (nullable = true)\n",
      " |-- gender: string (nullable = true)\n",
      " |-- itemInSession: long (nullable = true)\n",
      " |-- lastName: string (nullable = true)\n",
      " |-- length: double (nullable = true)\n",
      " |-- level: string (nullable = true)\n",
      " |-- location: string (nullable = true)\n",
      " |-- method: string (nullable = true)\n",
      " |-- page: string (nullable = true)\n",
      " |-- registration: long (nullable = true)\n",
      " |-- sessionId: long (nullable = true)\n",
      " |-- song: string (nullable = true)\n",
      " |-- status: long (nullable = true)\n",
      " |-- ts: long (nullable = true)\n",
      " |-- userAgent: string (nullable = true)\n",
      " |-- userId: string (nullable = true)\n",
      " |-- hour: integer (nullable = true)\n",
      " |-- day: integer (nullable = true)\n",
      " |-- workday_: string (nullable = true)\n",
      " |-- workday: integer (nullable = true)\n",
      " |-- month: integer (nullable = true)\n",
      " |-- churn: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Try to aggregate based on songs and month\n",
    "2. Plot some EDA based on notes\n",
    "3. Clean data set for modeling (refer to previous projects with ETL pipelines)\n",
    "4. Model based on DataCamp course (if possible|)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "location=%{x}<br>count=%{y}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "Los Angeles-Long Beach-Anaheim, CA",
          "New York-Newark-Jersey City, NY-NJ-PA",
          "Boston-Cambridge-Newton, MA-NH",
          "Houston-The Woodlands-Sugar Land, TX",
          null,
          "Charlotte-Concord-Gastonia, NC-SC",
          "Dallas-Fort Worth-Arlington, TX",
          "Louisville/Jefferson County, KY-IN",
          "Philadelphia-Camden-Wilmington, PA-NJ-DE-MD",
          "Chicago-Naperville-Elgin, IL-IN-WI",
          "St. Louis, MO-IL",
          "Phoenix-Mesa-Scottsdale, AZ",
          "Vineland-Bridgeton, NJ",
          "Wilson, NC",
          "Denver-Aurora-Lakewood, CO",
          "Ionia, MI",
          "San Antonio-New Braunfels, TX",
          "Danville, VA",
          "Atlanta-Sandy Springs-Roswell, GA",
          "New Haven-Milford, CT",
          "Jackson, MS",
          "Miami-Fort Lauderdale-West Palm Beach, FL",
          "Spokane-Spokane Valley, WA",
          "Tallahassee, FL",
          "Washington-Arlington-Alexandria, DC-VA-MD-WV",
          "Pittsburgh, PA",
          "Columbus, GA-AL",
          "Baltimore-Columbia-Towson, MD",
          "Syracuse, NY",
          "San Francisco-Oakland-Hayward, CA",
          "Kansas City, MO-KS",
          "Concord, NH",
          "Tampa-St. Petersburg-Clearwater, FL",
          "Lexington-Fayette, KY",
          "Riverside-San Bernardino-Ontario, CA",
          "Detroit-Warren-Dearborn, MI",
          "Morgantown, WV",
          "Manchester-Nashua, NH",
          "Atlantic City-Hammonton, NJ",
          "Colorado Springs, CO",
          "Minneapolis-St. Paul-Bloomington, MN-WI",
          "Billings, MT",
          "Jacksonville, NC",
          "Truckee-Grass Valley, CA",
          "Fairbanks, AK",
          "Las Vegas-Henderson-Paradise, NV",
          "Selma, AL",
          "Kingsport-Bristol-Bristol, TN-VA",
          "Oklahoma City, OK",
          "Hartford-West Hartford-East Hartford, CT",
          "Sacramento--Roseville--Arden-Arcade, CA",
          "New Philadelphia-Dover, OH",
          "Bakersfield, CA",
          "Austin-Round Rock, TX",
          "Cookeville, TN",
          "San Jose-Sunnyvale-Santa Clara, CA",
          "Orlando-Kissimmee-Sanford, FL",
          "Anchorage, AK",
          "Buffalo-Cheektowaga-Niagara Falls, NY",
          "Cleveland-Elyria, OH",
          "Milwaukee-Waukesha-West Allis, WI",
          "Gainesville, FL",
          "Sterling, IL",
          "Greensboro-High Point, NC",
          "Price, UT",
          "Napa, CA",
          "North Wilkesboro, NC",
          "Salinas, CA",
          "Quincy, IL-MO",
          "London, KY",
          "Hagerstown-Martinsburg, MD-WV",
          "Indianapolis-Carmel-Anderson, IN",
          "Alexandria, LA",
          "Providence-Warwick, RI-MA",
          "Oxnard-Thousand Oaks-Ventura, CA",
          "Raleigh, NC",
          "Bridgeport-Stamford-Norwalk, CT",
          "Boulder, CO",
          "Myrtle Beach-Conway-North Myrtle Beach, SC-NC",
          "Winston-Salem, NC",
          "Jacksonville, FL",
          "Allentown-Bethlehem-Easton, PA-NJ",
          "Laurel, MS",
          "San Diego-Carlsbad, CA",
          "Greenville-Anderson-Mauldin, SC",
          "Dubuque, IA",
          "Troy, AL",
          "Kankakee, IL",
          "Little Rock-North Little Rock-Conway, AR",
          "Albany-Schenectady-Troy, NY",
          "Flint, MI",
          "Roanoke, VA",
          "Stockton-Lodi, CA",
          "McAllen-Edinburg-Mission, TX",
          "Logan, UT-ID",
          "Pontiac, IL",
          "Muncie, IN",
          "Dayton, OH",
          "Bozeman, MT",
          "Seattle-Tacoma-Bellevue, WA",
          "Virginia Beach-Norfolk-Newport News, VA-NC",
          "Santa Maria-Santa Barbara, CA",
          "Monroe, LA",
          "Omaha-Council Bluffs, NE-IA",
          "Montgomery, AL",
          "Portland-Vancouver-Hillsboro, OR-WA",
          "Port St. Lucie, FL",
          "Duluth, MN-WI",
          "Cincinnati, OH-KY-IN",
          "Birmingham-Hoover, AL",
          "Deltona-Daytona Beach-Ormond Beach, FL",
          "Memphis, TN-MS-AR",
          "Albany, OR",
          "Corpus Christi, TX",
          "Scranton--Wilkes-Barre--Hazleton, PA",
          "New York-Newark-Jersey City, NY-NJ-PA",
          "Los Angeles-Long Beach-Anaheim, CA",
          "Spokane-Spokane Valley, WA",
          "Flint, MI",
          "Miami-Fort Lauderdale-West Palm Beach, FL",
          "Philadelphia-Camden-Wilmington, PA-NJ-DE-MD",
          "Phoenix-Mesa-Scottsdale, AZ",
          "Jackson, MS",
          "Quincy, IL-MO",
          "Indianapolis-Carmel-Anderson, IN",
          "Birmingham-Hoover, AL",
          "Detroit-Warren-Dearborn, MI",
          "Greenville-Anderson-Mauldin, SC",
          "Orlando-Kissimmee-Sanford, FL",
          "Lexington-Fayette, KY",
          "Virginia Beach-Norfolk-Newport News, VA-NC",
          "North Wilkesboro, NC",
          "London, KY",
          "Selma, AL",
          "Washington-Arlington-Alexandria, DC-VA-MD-WV",
          "Colorado Springs, CO",
          "Little Rock-North Little Rock-Conway, AR",
          "Houston-The Woodlands-Sugar Land, TX",
          "San Francisco-Oakland-Hayward, CA",
          "Chicago-Naperville-Elgin, IL-IN-WI",
          "San Diego-Carlsbad, CA",
          "Kansas City, MO-KS",
          "St. Louis, MO-IL",
          "Oxnard-Thousand Oaks-Ventura, CA",
          "Bozeman, MT",
          "Albany-Schenectady-Troy, NY",
          "Cleveland-Elyria, OH",
          "Albany, OR",
          "Denver-Aurora-Lakewood, CO",
          "Corpus Christi, TX",
          "Monroe, LA",
          "Dallas-Fort Worth-Arlington, TX",
          "Baltimore-Columbia-Towson, MD",
          "Memphis, TN-MS-AR",
          "New Philadelphia-Dover, OH"
         ],
         "xaxis": "x",
         "y": [
          30128,
          23679,
          13873,
          9498,
          8346,
          7780,
          7604,
          6880,
          5888,
          5113,
          4857,
          4844,
          4825,
          4659,
          4452,
          4428,
          4373,
          4257,
          4236,
          4007,
          3837,
          3762,
          3524,
          3216,
          3089,
          2917,
          2716,
          2709,
          2677,
          2646,
          2561,
          2544,
          2475,
          2463,
          2404,
          2331,
          2278,
          2220,
          2176,
          2167,
          2134,
          2132,
          2130,
          2073,
          2067,
          2042,
          2026,
          1863,
          1842,
          1820,
          1796,
          1780,
          1775,
          1682,
          1672,
          1603,
          1497,
          1496,
          1402,
          1391,
          1342,
          1229,
          1198,
          1131,
          1102,
          1091,
          1049,
          1026,
          1002,
          997,
          982,
          969,
          955,
          927,
          915,
          899,
          893,
          872,
          837,
          819,
          818,
          815,
          795,
          753,
          667,
          651,
          630,
          589,
          519,
          456,
          454,
          394,
          337,
          324,
          317,
          315,
          292,
          259,
          253,
          246,
          245,
          218,
          215,
          187,
          125,
          119,
          117,
          107,
          88,
          74,
          73,
          61,
          22,
          10,
          6,
          5,
          3,
          2,
          2,
          2,
          2,
          2,
          2,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "location"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"b49bab78-cfe1-4fe6-b21b-434f3050302b\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"b49bab78-cfe1-4fe6-b21b-434f3050302b\")) {                    Plotly.newPlot(                        \"b49bab78-cfe1-4fe6-b21b-434f3050302b\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"location=%{x}<br>count=%{y}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [\"Los Angeles-Long Beach-Anaheim, CA\", \"New York-Newark-Jersey City, NY-NJ-PA\", \"Boston-Cambridge-Newton, MA-NH\", \"Houston-The Woodlands-Sugar Land, TX\", null, \"Charlotte-Concord-Gastonia, NC-SC\", \"Dallas-Fort Worth-Arlington, TX\", \"Louisville/Jefferson County, KY-IN\", \"Philadelphia-Camden-Wilmington, PA-NJ-DE-MD\", \"Chicago-Naperville-Elgin, IL-IN-WI\", \"St. Louis, MO-IL\", \"Phoenix-Mesa-Scottsdale, AZ\", \"Vineland-Bridgeton, NJ\", \"Wilson, NC\", \"Denver-Aurora-Lakewood, CO\", \"Ionia, MI\", \"San Antonio-New Braunfels, TX\", \"Danville, VA\", \"Atlanta-Sandy Springs-Roswell, GA\", \"New Haven-Milford, CT\", \"Jackson, MS\", \"Miami-Fort Lauderdale-West Palm Beach, FL\", \"Spokane-Spokane Valley, WA\", \"Tallahassee, FL\", \"Washington-Arlington-Alexandria, DC-VA-MD-WV\", \"Pittsburgh, PA\", \"Columbus, GA-AL\", \"Baltimore-Columbia-Towson, MD\", \"Syracuse, NY\", \"San Francisco-Oakland-Hayward, CA\", \"Kansas City, MO-KS\", \"Concord, NH\", \"Tampa-St. Petersburg-Clearwater, FL\", \"Lexington-Fayette, KY\", \"Riverside-San Bernardino-Ontario, CA\", \"Detroit-Warren-Dearborn, MI\", \"Morgantown, WV\", \"Manchester-Nashua, NH\", \"Atlantic City-Hammonton, NJ\", \"Colorado Springs, CO\", \"Minneapolis-St. Paul-Bloomington, MN-WI\", \"Billings, MT\", \"Jacksonville, NC\", \"Truckee-Grass Valley, CA\", \"Fairbanks, AK\", \"Las Vegas-Henderson-Paradise, NV\", \"Selma, AL\", \"Kingsport-Bristol-Bristol, TN-VA\", \"Oklahoma City, OK\", \"Hartford-West Hartford-East Hartford, CT\", \"Sacramento--Roseville--Arden-Arcade, CA\", \"New Philadelphia-Dover, OH\", \"Bakersfield, CA\", \"Austin-Round Rock, TX\", \"Cookeville, TN\", \"San Jose-Sunnyvale-Santa Clara, CA\", \"Orlando-Kissimmee-Sanford, FL\", \"Anchorage, AK\", \"Buffalo-Cheektowaga-Niagara Falls, NY\", \"Cleveland-Elyria, OH\", \"Milwaukee-Waukesha-West Allis, WI\", \"Gainesville, FL\", \"Sterling, IL\", \"Greensboro-High Point, NC\", \"Price, UT\", \"Napa, CA\", \"North Wilkesboro, NC\", \"Salinas, CA\", \"Quincy, IL-MO\", \"London, KY\", \"Hagerstown-Martinsburg, MD-WV\", \"Indianapolis-Carmel-Anderson, IN\", \"Alexandria, LA\", \"Providence-Warwick, RI-MA\", \"Oxnard-Thousand Oaks-Ventura, CA\", \"Raleigh, NC\", \"Bridgeport-Stamford-Norwalk, CT\", \"Boulder, CO\", \"Myrtle Beach-Conway-North Myrtle Beach, SC-NC\", \"Winston-Salem, NC\", \"Jacksonville, FL\", \"Allentown-Bethlehem-Easton, PA-NJ\", \"Laurel, MS\", \"San Diego-Carlsbad, CA\", \"Greenville-Anderson-Mauldin, SC\", \"Dubuque, IA\", \"Troy, AL\", \"Kankakee, IL\", \"Little Rock-North Little Rock-Conway, AR\", \"Albany-Schenectady-Troy, NY\", \"Flint, MI\", \"Roanoke, VA\", \"Stockton-Lodi, CA\", \"McAllen-Edinburg-Mission, TX\", \"Logan, UT-ID\", \"Pontiac, IL\", \"Muncie, IN\", \"Dayton, OH\", \"Bozeman, MT\", \"Seattle-Tacoma-Bellevue, WA\", \"Virginia Beach-Norfolk-Newport News, VA-NC\", \"Santa Maria-Santa Barbara, CA\", \"Monroe, LA\", \"Omaha-Council Bluffs, NE-IA\", \"Montgomery, AL\", \"Portland-Vancouver-Hillsboro, OR-WA\", \"Port St. Lucie, FL\", \"Duluth, MN-WI\", \"Cincinnati, OH-KY-IN\", \"Birmingham-Hoover, AL\", \"Deltona-Daytona Beach-Ormond Beach, FL\", \"Memphis, TN-MS-AR\", \"Albany, OR\", \"Corpus Christi, TX\", \"Scranton--Wilkes-Barre--Hazleton, PA\", \"New York-Newark-Jersey City, NY-NJ-PA\", \"Los Angeles-Long Beach-Anaheim, CA\", \"Spokane-Spokane Valley, WA\", \"Flint, MI\", \"Miami-Fort Lauderdale-West Palm Beach, FL\", \"Philadelphia-Camden-Wilmington, PA-NJ-DE-MD\", \"Phoenix-Mesa-Scottsdale, AZ\", \"Jackson, MS\", \"Quincy, IL-MO\", \"Indianapolis-Carmel-Anderson, IN\", \"Birmingham-Hoover, AL\", \"Detroit-Warren-Dearborn, MI\", \"Greenville-Anderson-Mauldin, SC\", \"Orlando-Kissimmee-Sanford, FL\", \"Lexington-Fayette, KY\", \"Virginia Beach-Norfolk-Newport News, VA-NC\", \"North Wilkesboro, NC\", \"London, KY\", \"Selma, AL\", \"Washington-Arlington-Alexandria, DC-VA-MD-WV\", \"Colorado Springs, CO\", \"Little Rock-North Little Rock-Conway, AR\", \"Houston-The Woodlands-Sugar Land, TX\", \"San Francisco-Oakland-Hayward, CA\", \"Chicago-Naperville-Elgin, IL-IN-WI\", \"San Diego-Carlsbad, CA\", \"Kansas City, MO-KS\", \"St. Louis, MO-IL\", \"Oxnard-Thousand Oaks-Ventura, CA\", \"Bozeman, MT\", \"Albany-Schenectady-Troy, NY\", \"Cleveland-Elyria, OH\", \"Albany, OR\", \"Denver-Aurora-Lakewood, CO\", \"Corpus Christi, TX\", \"Monroe, LA\", \"Dallas-Fort Worth-Arlington, TX\", \"Baltimore-Columbia-Towson, MD\", \"Memphis, TN-MS-AR\", \"New Philadelphia-Dover, OH\"], \"xaxis\": \"x\", \"y\": [30128, 23679, 13873, 9498, 8346, 7780, 7604, 6880, 5888, 5113, 4857, 4844, 4825, 4659, 4452, 4428, 4373, 4257, 4236, 4007, 3837, 3762, 3524, 3216, 3089, 2917, 2716, 2709, 2677, 2646, 2561, 2544, 2475, 2463, 2404, 2331, 2278, 2220, 2176, 2167, 2134, 2132, 2130, 2073, 2067, 2042, 2026, 1863, 1842, 1820, 1796, 1780, 1775, 1682, 1672, 1603, 1497, 1496, 1402, 1391, 1342, 1229, 1198, 1131, 1102, 1091, 1049, 1026, 1002, 997, 982, 969, 955, 927, 915, 899, 893, 872, 837, 819, 818, 815, 795, 753, 667, 651, 630, 589, 519, 456, 454, 394, 337, 324, 317, 315, 292, 259, 253, 246, 245, 218, 215, 187, 125, 119, 117, 107, 88, 74, 73, 61, 22, 10, 6, 5, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"location\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('b49bab78-cfe1-4fe6-b21b-434f3050302b');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot location\n",
    "plot = df.groupBy('location', \"churn\").count().dropDuplicates().sort('count', ascending=False).toPandas()\n",
    "fig = px.bar(plot, x='location', y='count')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a pattern in the locaiton entries that we can exploit by getting the state abbreviation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create state column\n",
    "df = df.withColumn(\"state\", df.location.substr(-2,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+\n",
      "|state|\n",
      "+-----+\n",
      "|   PA|\n",
      "|   CA|\n",
      "| null|\n",
      "|   OK|\n",
      "|   WI|\n",
      "|   WV|\n",
      "|   WA|\n",
      "|   CA|\n",
      "|   CA|\n",
      "|   CA|\n",
      "|   TX|\n",
      "|   KS|\n",
      "|   KY|\n",
      "|   CO|\n",
      "|   IN|\n",
      "|   IN|\n",
      "|   MT|\n",
      "|   IN|\n",
      "|   CO|\n",
      "|   SC|\n",
      "+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select('state').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----+\n",
      "|state|count|\n",
      "+-----+-----+\n",
      "|   CA|46771|\n",
      "|   PA|26607|\n",
      "|   TX|23494|\n",
      "|   NH|18637|\n",
      "|   FL|13190|\n",
      "+-----+-----+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Show the top 5 states \n",
    "df.groupBy(\"state\").count().sort(\"count\", ascending=False).show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "state=%{x}<br>count=%{y}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "CA",
          "PA",
          "TX",
          "NH",
          "FL",
          "NC",
          "WI",
          "MD",
          "SC",
          null,
          "IN",
          "NJ",
          "CO",
          "MI",
          "IL",
          "CT",
          "VA",
          "WV",
          "AL",
          "AZ",
          "MS",
          "NY",
          "GA",
          "WA",
          "AK",
          "KY",
          "OH",
          "KS",
          "MT",
          "NV",
          "OK",
          "TN",
          "LA",
          "UT",
          "MO",
          "MA",
          "IA",
          "AR",
          "ID",
          "OR",
          "CA",
          "PA",
          "TX",
          "MD",
          "MI",
          "FL",
          "AZ",
          "KY",
          "MS",
          "WA",
          "NC",
          "AR",
          "CO",
          "OH",
          "AL",
          "WI",
          "MT",
          "IN",
          "MO",
          "KS",
          "WV",
          "NY",
          "OR",
          "IL",
          "SC",
          "LA"
         ],
         "xaxis": "x",
         "y": [
          46765,
          26602,
          23491,
          18637,
          13187,
          11769,
          8696,
          8597,
          8447,
          8346,
          8229,
          7816,
          7491,
          7213,
          6959,
          6720,
          6514,
          6349,
          5571,
          4844,
          4632,
          4535,
          4236,
          3889,
          3563,
          3460,
          3430,
          2561,
          2385,
          2042,
          1842,
          1672,
          1170,
          1102,
          1002,
          927,
          838,
          580,
          317,
          22,
          6,
          5,
          3,
          3,
          3,
          3,
          2,
          2,
          2,
          2,
          2,
          2,
          2,
          2,
          2,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "state"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"13eac41b-bed5-4af0-b5c9-06d3d419e2e0\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"13eac41b-bed5-4af0-b5c9-06d3d419e2e0\")) {                    Plotly.newPlot(                        \"13eac41b-bed5-4af0-b5c9-06d3d419e2e0\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"state=%{x}<br>count=%{y}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [\"CA\", \"PA\", \"TX\", \"NH\", \"FL\", \"NC\", \"WI\", \"MD\", \"SC\", null, \"IN\", \"NJ\", \"CO\", \"MI\", \"IL\", \"CT\", \"VA\", \"WV\", \"AL\", \"AZ\", \"MS\", \"NY\", \"GA\", \"WA\", \"AK\", \"KY\", \"OH\", \"KS\", \"MT\", \"NV\", \"OK\", \"TN\", \"LA\", \"UT\", \"MO\", \"MA\", \"IA\", \"AR\", \"ID\", \"OR\", \"CA\", \"PA\", \"TX\", \"MD\", \"MI\", \"FL\", \"AZ\", \"KY\", \"MS\", \"WA\", \"NC\", \"AR\", \"CO\", \"OH\", \"AL\", \"WI\", \"MT\", \"IN\", \"MO\", \"KS\", \"WV\", \"NY\", \"OR\", \"IL\", \"SC\", \"LA\"], \"xaxis\": \"x\", \"y\": [46765, 26602, 23491, 18637, 13187, 11769, 8696, 8597, 8447, 8346, 8229, 7816, 7491, 7213, 6959, 6720, 6514, 6349, 5571, 4844, 4632, 4535, 4236, 3889, 3563, 3460, 3430, 2561, 2385, 2042, 1842, 1672, 1170, 1102, 1002, 927, 838, 580, 317, 22, 6, 5, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"state\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('13eac41b-bed5-4af0-b5c9-06d3d419e2e0');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Graph state results\n",
    "state = df.groupBy(\"state\", \"churn\").count().sort(\"count\", ascending=False).toPandas()\n",
    "fig = px.bar(state, x='state', y='count')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----+-----+\n",
      "|state|churn|count|\n",
      "+-----+-----+-----+\n",
      "|   CA|    1|    6|\n",
      "|   PA|    1|    5|\n",
      "|   FL|    1|    3|\n",
      "|   MD|    1|    3|\n",
      "|   MI|    1|    3|\n",
      "+-----+-----+-----+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check state by churn\n",
    "churn = df.filter(df.churn == 1)\n",
    "churn.groupBy(\"state\", \"churn\").count().sort(\"count\", ascending=False).show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "state=%{x}<br>count=%{y}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "textposition": "auto",
         "type": "bar",
         "x": [
          "CA",
          "PA",
          "TX",
          "MD",
          "MI",
          "FL",
          "AZ",
          "KY",
          "MS",
          "WA",
          "NC",
          "AR",
          "CO",
          "OH",
          "AL",
          "WI",
          "MT",
          "IN",
          "MO",
          "KS",
          "WV",
          "NY",
          "OR",
          "IL",
          "SC",
          "LA"
         ],
         "xaxis": "x",
         "y": [
          6,
          5,
          3,
          3,
          3,
          3,
          2,
          2,
          2,
          2,
          2,
          2,
          2,
          2,
          2,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "state"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"63059bc9-43fc-4c06-81b1-986ebf95803a\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"63059bc9-43fc-4c06-81b1-986ebf95803a\")) {                    Plotly.newPlot(                        \"63059bc9-43fc-4c06-81b1-986ebf95803a\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"state=%{x}<br>count=%{y}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [\"CA\", \"PA\", \"TX\", \"MD\", \"MI\", \"FL\", \"AZ\", \"KY\", \"MS\", \"WA\", \"NC\", \"AR\", \"CO\", \"OH\", \"AL\", \"WI\", \"MT\", \"IN\", \"MO\", \"KS\", \"WV\", \"NY\", \"OR\", \"IL\", \"SC\", \"LA\"], \"xaxis\": \"x\", \"y\": [6, 5, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"state\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('63059bc9-43fc-4c06-81b1-986ebf95803a');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot churn by state\n",
    "churned_state = churn.groupBy(\"state\", \"churn\").count().sort(\"count\", ascending=False).toPandas()\n",
    "fig = px.bar(churned_state, x='state', y='count')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "workday_=%{x}<br>count=%{text}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "text": [
          22140,
          31137,
          41785,
          45147,
          47827,
          48927,
          49537
         ],
         "textposition": "auto",
         "type": "bar",
         "x": [
          "Sunday",
          "Saturday",
          "Monday",
          "Thursday",
          "Tuesday",
          "Wednesday",
          "Friday"
         ],
         "xaxis": "x",
         "y": [
          22140,
          31137,
          41785,
          45147,
          47827,
          48927,
          49537
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "What are the most active days?"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "workday_"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"3344153d-5d72-4b26-b107-11745f18e70c\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"3344153d-5d72-4b26-b107-11745f18e70c\")) {                    Plotly.newPlot(                        \"3344153d-5d72-4b26-b107-11745f18e70c\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"workday_=%{x}<br>count=%{text}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"text\": [22140.0, 31137.0, 41785.0, 45147.0, 47827.0, 48927.0, 49537.0], \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [\"Sunday\", \"Saturday\", \"Monday\", \"Thursday\", \"Tuesday\", \"Wednesday\", \"Friday\"], \"xaxis\": \"x\", \"y\": [22140, 31137, 41785, 45147, 47827, 48927, 49537], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"What are the most active days?\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"workday_\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('3344153d-5d72-4b26-b107-11745f18e70c');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# What are the most active workdays?\n",
    "active_workdays = df.groupBy('workday_').count().sort('count').toPandas()\n",
    "fig = px.bar(active_workdays, x='workday_', y='count', text=\"count\")\n",
    "fig.update_layout(title_text='What are the most active days?')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "workday_=%{x}<br>count=%{text}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "text": [
          5,
          5,
          6,
          6,
          7,
          9,
          14
         ],
         "textposition": "auto",
         "type": "bar",
         "x": [
          "Thursday",
          "Sunday",
          "Tuesday",
          "Saturday",
          "Wednesday",
          "Monday",
          "Friday"
         ],
         "xaxis": "x",
         "y": [
          5,
          5,
          6,
          6,
          7,
          9,
          14
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "What are the most active workdays for churn?"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "workday_"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"a69419e5-0f4a-4087-8af3-3a930febcea8\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"a69419e5-0f4a-4087-8af3-3a930febcea8\")) {                    Plotly.newPlot(                        \"a69419e5-0f4a-4087-8af3-3a930febcea8\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"workday_=%{x}<br>count=%{text}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"text\": [5.0, 5.0, 6.0, 6.0, 7.0, 9.0, 14.0], \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [\"Thursday\", \"Sunday\", \"Tuesday\", \"Saturday\", \"Wednesday\", \"Monday\", \"Friday\"], \"xaxis\": \"x\", \"y\": [5, 5, 6, 6, 7, 9, 14], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"What are the most active workdays for churn?\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"workday_\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('a69419e5-0f4a-4087-8af3-3a930febcea8');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot churn by workday\n",
    "churned_workdays = churn.groupBy('workday_').count().sort('count').toPandas()\n",
    "fig = px.bar(churned_workdays , x='workday_', y='count', text=\"count\")\n",
    "fig.update_layout(title_text='What are the most active workdays for churn?')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firday and Monday are the churn days!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+-----+-----+\n",
      "|song|churn|count|\n",
      "+----+-----+-----+\n",
      "|null|    1|   52|\n",
      "+----+-----+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check if any songs records are present for churn users\n",
    "churn.groupBy('song', \"churn\").count().dropDuplicates().sort('count', ascending=False).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**No songs available for churned users**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "hour=%{x}<br>count=%{text}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "text": [
          12853,
          12334,
          11818,
          11086,
          10462,
          10392,
          10318,
          10098,
          10116,
          10160,
          10254,
          10327,
          10725,
          10863,
          11632,
          12025,
          12699,
          13946,
          14376,
          14731,
          14219,
          13867,
          13633,
          13566
         ],
         "textposition": "auto",
         "type": "bar",
         "x": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23
         ],
         "xaxis": "x",
         "y": [
          12853,
          12334,
          11818,
          11086,
          10462,
          10392,
          10318,
          10098,
          10116,
          10160,
          10254,
          10327,
          10725,
          10863,
          11632,
          12025,
          12699,
          13946,
          14376,
          14731,
          14219,
          13867,
          13633,
          13566
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "What are the most active hours?"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "hour"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"7617f496-edb9-4d4d-9f96-7a469af10a68\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"7617f496-edb9-4d4d-9f96-7a469af10a68\")) {                    Plotly.newPlot(                        \"7617f496-edb9-4d4d-9f96-7a469af10a68\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"hour=%{x}<br>count=%{text}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"text\": [12853.0, 12334.0, 11818.0, 11086.0, 10462.0, 10392.0, 10318.0, 10098.0, 10116.0, 10160.0, 10254.0, 10327.0, 10725.0, 10863.0, 11632.0, 12025.0, 12699.0, 13946.0, 14376.0, 14731.0, 14219.0, 13867.0, 13633.0, 13566.0], \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23], \"xaxis\": \"x\", \"y\": [12853, 12334, 11818, 11086, 10462, 10392, 10318, 10098, 10116, 10160, 10254, 10327, 10725, 10863, 11632, 12025, 12699, 13946, 14376, 14731, 14219, 13867, 13633, 13566], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"What are the most active hours?\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"hour\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('7617f496-edb9-4d4d-9f96-7a469af10a68');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# What are the most active hours?\n",
    "active_hours = df.groupBy('hour').count().sort('hour').toPandas()\n",
    "fig = px.bar(active_hours, x='hour', y='count', text='count')\n",
    "fig.update_layout(title_text='What are the most active hours?')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "hour=%{x}<br>count=%{text}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "text": [
          3,
          2,
          2,
          3,
          1,
          3,
          2,
          2,
          5,
          1,
          2,
          1,
          2,
          1,
          1,
          5,
          2,
          3,
          5,
          3,
          3
         ],
         "textposition": "outside",
         "texttemplate": "%{text:.2s}",
         "type": "bar",
         "x": [
          0,
          1,
          2,
          3,
          5,
          6,
          7,
          9,
          10,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23
         ],
         "xaxis": "x",
         "y": [
          3,
          2,
          2,
          3,
          1,
          3,
          2,
          2,
          5,
          1,
          2,
          1,
          2,
          1,
          1,
          5,
          2,
          3,
          5,
          3,
          3
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "What are the most active hours for churn?"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "hour"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"c699b421-c20d-4ac3-a753-4cba4c217420\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"c699b421-c20d-4ac3-a753-4cba4c217420\")) {                    Plotly.newPlot(                        \"c699b421-c20d-4ac3-a753-4cba4c217420\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"hour=%{x}<br>count=%{text}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"text\": [3.0, 2.0, 2.0, 3.0, 1.0, 3.0, 2.0, 2.0, 5.0, 1.0, 2.0, 1.0, 2.0, 1.0, 1.0, 5.0, 2.0, 3.0, 5.0, 3.0, 3.0], \"textposition\": \"outside\", \"texttemplate\": \"%{text:.2s}\", \"type\": \"bar\", \"x\": [0, 1, 2, 3, 5, 6, 7, 9, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23], \"xaxis\": \"x\", \"y\": [3, 2, 2, 3, 1, 3, 2, 2, 5, 1, 2, 1, 2, 1, 1, 5, 2, 3, 5, 3, 3], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"What are the most active hours for churn?\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"hour\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('c699b421-c20d-4ac3-a753-4cba4c217420');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# What are the most active hours for churn?\n",
    "churned_hours = churn.groupBy('hour').count().sort('hour').toPandas()\n",
    "fig = px.bar(churned_hours, x='hour', y='count', text='count')\n",
    "fig.update_layout(title_text='What are the most active hours for churn?')\n",
    "fig.update_traces(texttemplate='%{text:.2s}', textposition='outside')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "day=%{x}<br>count=%{text}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "text": [
          11107,
          11989,
          9097,
          7136,
          8572,
          8773,
          7596,
          8734,
          11628,
          9716,
          9165,
          9092,
          8208,
          7427,
          10552,
          11475,
          8350,
          7667,
          9878,
          11559,
          9826,
          8235,
          9894,
          8722,
          8375,
          9920,
          10096,
          7472,
          9639,
          11241,
          5359
         ],
         "textposition": "auto",
         "type": "bar",
         "x": [
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31
         ],
         "xaxis": "x",
         "y": [
          11107,
          11989,
          9097,
          7136,
          8572,
          8773,
          7596,
          8734,
          11628,
          9716,
          9165,
          9092,
          8208,
          7427,
          10552,
          11475,
          8350,
          7667,
          9878,
          11559,
          9826,
          8235,
          9894,
          8722,
          8375,
          9920,
          10096,
          7472,
          9639,
          11241,
          5359
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "What are the most active days during the month?"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "day"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"8c5e83b9-ee12-4216-b464-c2678cd16c1e\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"8c5e83b9-ee12-4216-b464-c2678cd16c1e\")) {                    Plotly.newPlot(                        \"8c5e83b9-ee12-4216-b464-c2678cd16c1e\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"day=%{x}<br>count=%{text}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"text\": [11107.0, 11989.0, 9097.0, 7136.0, 8572.0, 8773.0, 7596.0, 8734.0, 11628.0, 9716.0, 9165.0, 9092.0, 8208.0, 7427.0, 10552.0, 11475.0, 8350.0, 7667.0, 9878.0, 11559.0, 9826.0, 8235.0, 9894.0, 8722.0, 8375.0, 9920.0, 10096.0, 7472.0, 9639.0, 11241.0, 5359.0], \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31], \"xaxis\": \"x\", \"y\": [11107, 11989, 9097, 7136, 8572, 8773, 7596, 8734, 11628, 9716, 9165, 9092, 8208, 7427, 10552, 11475, 8350, 7667, 9878, 11559, 9826, 8235, 9894, 8722, 8375, 9920, 10096, 7472, 9639, 11241, 5359], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"What are the most active days during the month?\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"day\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('8c5e83b9-ee12-4216-b464-c2678cd16c1e');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# What are the most active days during the month?\n",
    "active_days = df.groupBy('day').count().sort('day').toPandas()\n",
    "fig = px.bar(active_days , x='day', y='count', text='count')\n",
    "fig.update_layout(title_text='What are the most active days during the month?')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "day=%{x}<br>count=%{text}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "text": [
          2,
          6,
          1,
          3,
          1,
          1,
          1,
          2,
          1,
          4,
          3,
          4,
          1,
          5,
          2,
          4,
          2,
          2,
          3,
          1,
          1,
          1,
          1
         ],
         "textposition": "auto",
         "type": "bar",
         "x": [
          1,
          2,
          4,
          5,
          6,
          7,
          8,
          9,
          11,
          12,
          14,
          15,
          16,
          17,
          19,
          20,
          22,
          23,
          24,
          25,
          27,
          29,
          30
         ],
         "xaxis": "x",
         "y": [
          2,
          6,
          1,
          3,
          1,
          1,
          1,
          2,
          1,
          4,
          3,
          4,
          1,
          5,
          2,
          4,
          2,
          2,
          3,
          1,
          1,
          1,
          1
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "What are the most active days during the month?"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "day"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"0bde767d-0a49-4b0e-8d92-a843afc1a3c8\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"0bde767d-0a49-4b0e-8d92-a843afc1a3c8\")) {                    Plotly.newPlot(                        \"0bde767d-0a49-4b0e-8d92-a843afc1a3c8\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"day=%{x}<br>count=%{text}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"text\": [2.0, 6.0, 1.0, 3.0, 1.0, 1.0, 1.0, 2.0, 1.0, 4.0, 3.0, 4.0, 1.0, 5.0, 2.0, 4.0, 2.0, 2.0, 3.0, 1.0, 1.0, 1.0, 1.0], \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [1, 2, 4, 5, 6, 7, 8, 9, 11, 12, 14, 15, 16, 17, 19, 20, 22, 23, 24, 25, 27, 29, 30], \"xaxis\": \"x\", \"y\": [2, 6, 1, 3, 1, 1, 1, 2, 1, 4, 3, 4, 1, 5, 2, 4, 2, 2, 3, 1, 1, 1, 1], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"What are the most active days during the month?\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"day\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('0bde767d-0a49-4b0e-8d92-a843afc1a3c8');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# What are the most active days during the month for churn?\n",
    "churned_days = churn.groupBy('day').count().sort('day').toPandas()\n",
    "fig = px.bar(churned_days, x='day', y='count', text='count')\n",
    "fig.update_layout(title_text='What are the most active days during the month?')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "gender=%{x}<br>count=%{text}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "text": [
          8346,
          154578,
          123576
         ],
         "textposition": "auto",
         "type": "bar",
         "x": [
          null,
          "F",
          "M"
         ],
         "xaxis": "x",
         "y": [
          8346,
          154578,
          123576
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Gender split"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "gender"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"13a553b9-0571-4f11-bc08-4e445b5881f2\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"13a553b9-0571-4f11-bc08-4e445b5881f2\")) {                    Plotly.newPlot(                        \"13a553b9-0571-4f11-bc08-4e445b5881f2\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"gender=%{x}<br>count=%{text}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"text\": [8346.0, 154578.0, 123576.0], \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [null, \"F\", \"M\"], \"xaxis\": \"x\", \"y\": [8346, 154578, 123576], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Gender split\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"gender\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('13a553b9-0571-4f11-bc08-4e445b5881f2');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# How people are split per gender?\n",
    "gender_split = df.groupBy('gender').count().sort('gender').toPandas()\n",
    "fig = px.bar(gender_split, x='gender', y='count', text='count')\n",
    "fig.update_layout(title_text='Gender split')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "alignmentgroup": "True",
         "hovertemplate": "gender=%{x}<br>count=%{text}<extra></extra>",
         "legendgroup": "",
         "marker": {
          "color": "#636efa"
         },
         "name": "",
         "offsetgroup": "",
         "orientation": "v",
         "showlegend": false,
         "text": [
          20,
          32
         ],
         "textposition": "auto",
         "type": "bar",
         "x": [
          "F",
          "M"
         ],
         "xaxis": "x",
         "y": [
          20,
          32
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "barmode": "relative",
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Gender split (churn)"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "gender"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "count"
         }
        }
       }
      },
      "text/html": [
       "<div>                            <div id=\"5c492ea5-e9d0-466f-949b-5816254bcec8\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"5c492ea5-e9d0-466f-949b-5816254bcec8\")) {                    Plotly.newPlot(                        \"5c492ea5-e9d0-466f-949b-5816254bcec8\",                        [{\"alignmentgroup\": \"True\", \"hovertemplate\": \"gender=%{x}<br>count=%{text}<extra></extra>\", \"legendgroup\": \"\", \"marker\": {\"color\": \"#636efa\"}, \"name\": \"\", \"offsetgroup\": \"\", \"orientation\": \"v\", \"showlegend\": false, \"text\": [20.0, 32.0], \"textposition\": \"auto\", \"type\": \"bar\", \"x\": [\"F\", \"M\"], \"xaxis\": \"x\", \"y\": [20, 32], \"yaxis\": \"y\"}],                        {\"barmode\": \"relative\", \"legend\": {\"tracegroupgap\": 0}, \"margin\": {\"t\": 60}, \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Gender split (churn)\"}, \"xaxis\": {\"anchor\": \"y\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"gender\"}}, \"yaxis\": {\"anchor\": \"x\", \"domain\": [0.0, 1.0], \"title\": {\"text\": \"count\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('5c492ea5-e9d0-466f-949b-5816254bcec8');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                });            </script>        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# How people whor churn are split per gender?\n",
    "churned_gender = churn.groupBy('gender').count().sort('gender').toPandas()\n",
    "fig = px.bar(churned_gender, x='gender', y='count', text='count')\n",
    "fig.update_layout(title_text='Gender split (churn)')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eventhough more users are female, it turns out that Men are churning the most."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Based on the EDA I would choose the below features:\n",
    "- state\n",
    "- workday\n",
    "- day\n",
    "- songs played\n",
    "- gender\n",
    "- level\n",
    "\n",
    "# Feature Engineering\n",
    "Once you've familiarized yourself with the data, build out the features you find promising to train your model on. To work with the full dataset, you can follow the following steps.\n",
    "- Write a script to extract the necessary features from the smaller subset of data\n",
    "- Ensure that your script is scalable, using the best practices discussed in Lesson 3\n",
    "- Try your script on the full data set, debugging your script if necessary\n",
    "\n",
    "If you are working in the classroom workspace, you can just extract features based on the small subset of data contained here. Be sure to transfer over this work to the larger dataset when you work on your Spark cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- artist: string (nullable = true)\n",
      " |-- auth: string (nullable = true)\n",
      " |-- firstName: string (nullable = true)\n",
      " |-- gender: string (nullable = true)\n",
      " |-- itemInSession: long (nullable = true)\n",
      " |-- lastName: string (nullable = true)\n",
      " |-- length: double (nullable = true)\n",
      " |-- level: string (nullable = true)\n",
      " |-- location: string (nullable = true)\n",
      " |-- method: string (nullable = true)\n",
      " |-- page: string (nullable = true)\n",
      " |-- registration: long (nullable = true)\n",
      " |-- sessionId: long (nullable = true)\n",
      " |-- song: string (nullable = true)\n",
      " |-- status: long (nullable = true)\n",
      " |-- ts: long (nullable = true)\n",
      " |-- userAgent: string (nullable = true)\n",
      " |-- userId: string (nullable = true)\n",
      " |-- hour: integer (nullable = true)\n",
      " |-- day: integer (nullable = true)\n",
      " |-- workday_: string (nullable = true)\n",
      " |-- workday: integer (nullable = true)\n",
      " |-- month: integer (nullable = true)\n",
      " |-- churn: integer (nullable = true)\n",
      " |-- state: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+------+-------+---+-----+\n",
      "|userId|churn|gender|workday|day|state|\n",
      "+------+-----+------+-------+---+-----+\n",
      "|100001|    1|     F|      2|  2|   FL|\n",
      "|100003|    1|     F|      4| 11|   FL|\n",
      "|100005|    1|     M|      5| 19|   LA|\n",
      "|100006|    1|     F|      5|  5|   MI|\n",
      "|100007|    1|     F|      4| 29|   AR|\n",
      "|100009|    1|     M|      6| 27|   CA|\n",
      "|100011|    1|     M|      1|  1|   OR|\n",
      "|100012|    1|     M|      4|  1|   WI|\n",
      "|100013|    1|     F|      5|  2|   OH|\n",
      "|100014|    1|     M|      2| 20|   PA|\n",
      "|100015|    1|     F|      6| 24|   NC|\n",
      "|100017|    1|     M|      6| 20|   AL|\n",
      "|100019|    1|     M|      1|  8|   WA|\n",
      "|100021|    1|     M|      7| 25|   TX|\n",
      "|100022|    1|     F|      1| 12|   TX|\n",
      "|100023|    1|     M|      7| 14|   SC|\n",
      "|100024|    1|     M|      5|  5|   PA|\n",
      "|100025|    1|     F|      2| 23|   PA|\n",
      "|   101|    1|     M|      3| 17|   CO|\n",
      "|   103|    1|     F|      7|  4|   PA|\n",
      "+------+-----+------+-------+---+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get feauture candidates for modeling\n",
    "df.select('userId', 'churn', 'gender', 'workday', 'day', 'state') \\\n",
    "    .where(df.churn != 0).sort('userId').show(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+------+---------+---+\n",
      "|userId|churn|gender| workday_|day|\n",
      "+------+-----+------+---------+---+\n",
      "|   126|    0|     M|   Monday|  1|\n",
      "|    39|    0|     F|   Monday|  1|\n",
      "|      |    0|  null|   Monday|  1|\n",
      "|   120|    0|     F|   Monday|  1|\n",
      "|    36|    0|     F|   Monday|  1|\n",
      "|    15|    0|     M|   Monday|  1|\n",
      "|    54|    0|     F|   Monday|  1|\n",
      "|    61|    0|     M|  Tuesday|  2|\n",
      "|    61|    0|     M|  Tuesday|  2|\n",
      "|    46|    0|     F|  Tuesday|  2|\n",
      "|   127|    0|     F|  Tuesday|  2|\n",
      "|    18|    0|     M|  Tuesday|  2|\n",
      "|    51|    0|     M|  Tuesday|  2|\n",
      "|   101|    0|     M|Wednesday|  3|\n",
      "|   105|    0|     M|Wednesday|  3|\n",
      "|   105|    0|     M|Wednesday|  3|\n",
      "|     3|    0|     M|Wednesday|  3|\n",
      "|   105|    0|     M|Wednesday|  3|\n",
      "|   101|    0|     M|Wednesday|  3|\n",
      "|   136|    0|     F|Wednesday|  3|\n",
      "+------+-----+------+---------+---+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get feauture candidates for modeling\n",
    "df.select('userId', 'churn', 'gender', 'workday_', 'day') \\\n",
    "    .where(df.churn == 0).show(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create model data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = df.select('userId', 'churn', 'gender', 'ts', 'day', 'state', 'song').where(df.churn != 0).sort('userId')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = df.select('userId', 'churn', 'gender', 'workday', 'day', 'state')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:37169)\n",
      "Traceback (most recent call last):\n",
      "  File \"/home/freemo/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\", line 977, in _get_connection\n",
      "    connection = self.deque.pop()\n",
      "IndexError: pop from an empty deque\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"/home/freemo/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\", line 1115, in start\n",
      "    self.socket.connect((self.address, self.port))\n",
      "ConnectionRefusedError: [Errno 111] Connection refused\n"
     ]
    },
    {
     "ename": "Py4JNetworkError",
     "evalue": "An error occurred while trying to connect to the Java server (127.0.0.1:37169)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36m_get_connection\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    976\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 977\u001b[0;31m             \u001b[0mconnection\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdeque\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    978\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mIndexError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIndexError\u001b[0m: pop from an empty deque",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mConnectionRefusedError\u001b[0m                    Traceback (most recent call last)",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36mstart\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1114\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1115\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maddress\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mport\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1116\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstream\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmakefile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"rb\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mConnectionRefusedError\u001b[0m: [Errno 111] Connection refused",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mPy4JNetworkError\u001b[0m                          Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-59-ff98313fef17>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdtypes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/sql/dataframe.py\u001b[0m in \u001b[0;36mdtypes\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1184\u001b[0m         \u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'age'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'int'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m'name'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'string'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1185\u001b[0m         \"\"\"\n\u001b[0;32m-> 1186\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataType\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msimpleString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mf\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mschema\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfields\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1187\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1188\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/sql/dataframe.py\u001b[0m in \u001b[0;36mschema\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    274\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_schema\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    275\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 276\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_schema\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_parse_datatype_json_string\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mschema\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    277\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mAttributeError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    278\u001b[0m                 raise Exception(\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1301\u001b[0m             \u001b[0mproto\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mEND_COMMAND_PART\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1302\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1303\u001b[0;31m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1304\u001b[0m         return_value = get_return_value(\n\u001b[1;32m   1305\u001b[0m             answer, self.gateway_client, self.target_id, self.name)\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36msend_command\u001b[0;34m(self, command, retry, binary)\u001b[0m\n\u001b[1;32m   1029\u001b[0m          \u001b[0;32mif\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mbinary\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0;32mis\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1030\u001b[0m         \"\"\"\n\u001b[0;32m-> 1031\u001b[0;31m         \u001b[0mconnection\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_connection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1032\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1033\u001b[0m             \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconnection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36m_get_connection\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    977\u001b[0m             \u001b[0mconnection\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdeque\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    978\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mIndexError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 979\u001b[0;31m             \u001b[0mconnection\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_connection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    980\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mconnection\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    981\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36m_create_connection\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    983\u001b[0m         connection = GatewayConnection(\n\u001b[1;32m    984\u001b[0m             self.gateway_parameters, self.gateway_property)\n\u001b[0;32m--> 985\u001b[0;31m         \u001b[0mconnection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    986\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mconnection\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    987\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36mstart\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1125\u001b[0m                 \u001b[0;34m\"server ({0}:{1})\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maddress\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mport\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1126\u001b[0m             \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1127\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mPy4JNetworkError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1128\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1129\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_authenticate_connection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mPy4JNetworkError\u001b[0m: An error occurred while trying to connect to the Java server (127.0.0.1:37169)"
     ]
    }
   ],
   "source": [
    "model.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model.replace(['M','F'], ['1', '0'], 'gender')\n",
    "model = model.withColumn('gender', model.gender.cast(\"int\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+------+-------+---+-----+\n",
      "|userId|churn|gender|workday|day|state|\n",
      "+------+-----+------+-------+---+-----+\n",
      "|   126|    0|     1|      1|  1|   PA|\n",
      "|    39|    0|     0|      1|  1|   CA|\n",
      "|      |    0|  null|      1|  1| null|\n",
      "|   120|    0|     0|      1|  1|   OK|\n",
      "|    36|    0|     0|      1|  1|   WI|\n",
      "|    15|    0|     1|      1|  1|   WV|\n",
      "|    54|    0|     0|      1|  1|   WA|\n",
      "|    61|    0|     1|      2|  2|   CA|\n",
      "|    61|    0|     1|      2|  2|   CA|\n",
      "|    46|    0|     0|      2|  2|   CA|\n",
      "|   127|    0|     0|      2|  2|   TX|\n",
      "|    18|    0|     1|      2|  2|   KS|\n",
      "|    51|    0|     1|      2|  2|   KY|\n",
      "|   101|    0|     1|      3|  3|   CO|\n",
      "|   105|    0|     1|      3|  3|   IN|\n",
      "|   105|    0|     1|      3|  3|   IN|\n",
      "|     3|    0|     1|      3|  3|   MT|\n",
      "|   105|    0|     1|      3|  3|   IN|\n",
      "|   101|    0|     1|      3|  3|   CO|\n",
      "|   136|    0|     0|      3|  3|   SC|\n",
      "+------+-----+------+-------+---+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model = model.select('userId', 'churn', 'gender', 'workday', 'day', 'state')\n",
    "model.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+------+-------+---+-----+\n",
      "|userId|churn|gender|workday|day|state|\n",
      "+------+-----+------+-------+---+-----+\n",
      "|100001|    1|     0|      2|  2|   FL|\n",
      "|100003|    1|     0|      4| 11|   FL|\n",
      "|100005|    1|     1|      5| 19|   LA|\n",
      "|100006|    1|     0|      5|  5|   MI|\n",
      "|100007|    1|     0|      4| 29|   AR|\n",
      "|100009|    1|     1|      6| 27|   CA|\n",
      "|100011|    1|     1|      1|  1|   OR|\n",
      "|100012|    1|     1|      4|  1|   WI|\n",
      "|100013|    1|     0|      5|  2|   OH|\n",
      "|100014|    1|     1|      2| 20|   PA|\n",
      "|100015|    1|     0|      6| 24|   NC|\n",
      "|100017|    1|     1|      6| 20|   AL|\n",
      "|100019|    1|     1|      1|  8|   WA|\n",
      "|100021|    1|     1|      7| 25|   TX|\n",
      "|100022|    1|     0|      1| 12|   TX|\n",
      "|100023|    1|     1|      7| 14|   SC|\n",
      "|100024|    1|     1|      5|  5|   PA|\n",
      "|100025|    1|     0|      2| 23|   PA|\n",
      "|   101|    1|     1|      3| 17|   CO|\n",
      "|   103|    1|     0|      7|  4|   PA|\n",
      "+------+-----+------+-------+---+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model.select('userId', 'churn', 'gender', 'workday', 'day', 'state') \\\n",
    "    .where(df.churn != 0) \\\n",
    "    .sort('userId').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_songs = df.where(df.song!='null').groupby('userId') \\\n",
    "    .agg(count(df.song).alias('SongsPlayed')) \\\n",
    "    .orderBy('userId') \\\n",
    "    .select(['userId','SongsPlayed'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('userId', 'string'), ('SongsPlayed', 'bigint')]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_songs.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model.join(df_songs,'userId')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model.select('userId', 'churn', 'gender', 'workday', 'day', 'state', 'SongsPlayed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+------+-------+---+-----+-----------+\n",
      "|userId|churn|gender|workday|day|state|SongsPlayed|\n",
      "+------+-----+------+-------+---+-----+-----------+\n",
      "|100010|    0|     0|      6| 20|   CT|        275|\n",
      "|100010|    0|     0|      1|  5|   CT|        275|\n",
      "|100010|    0|     0|      3| 21|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      2| 16|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      2| 16|   CT|        275|\n",
      "|100010|    0|     0|      5|  9|   CT|        275|\n",
      "|100010|    0|     0|      1| 15|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      6| 20|   CT|        275|\n",
      "|100010|    0|     0|      1|  5|   CT|        275|\n",
      "|100010|    0|     0|      1|  5|   CT|        275|\n",
      "|100010|    0|     0|      1|  5|   CT|        275|\n",
      "|100010|    0|     0|      5|  9|   CT|        275|\n",
      "|100010|    0|     0|      5|  9|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      5|  9|   CT|        275|\n",
      "|100010|    0|     0|      2| 16|   CT|        275|\n",
      "|100010|    0|     0|      2| 16|   CT|        275|\n",
      "+------+-----+------+-------+---+-----+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = model.withColumnRenamed(\"churn\", \"label\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+------+-------+---+-----+-----------+\n",
      "|userId|label|gender|workday|day|state|SongsPlayed|\n",
      "+------+-----+------+-------+---+-----+-----------+\n",
      "|100001|    1|     0|      2|  2|   FL|        133|\n",
      "|100003|    1|     0|      4| 11|   FL|         51|\n",
      "|100005|    1|     1|      5| 19|   LA|        154|\n",
      "|100006|    1|     0|      5|  5|   MI|         26|\n",
      "|100007|    1|     0|      4| 29|   AR|        423|\n",
      "|100009|    1|     1|      6| 27|   CA|        518|\n",
      "|100011|    1|     1|      1|  1|   OR|         11|\n",
      "|100012|    1|     1|      4|  1|   WI|        476|\n",
      "|100013|    1|     0|      5|  2|   OH|       1131|\n",
      "|100014|    1|     1|      2| 20|   PA|        257|\n",
      "|100015|    1|     0|      6| 24|   NC|        800|\n",
      "|100017|    1|     1|      6| 20|   AL|         52|\n",
      "|100019|    1|     1|      1|  8|   WA|         62|\n",
      "|100021|    1|     1|      7| 25|   TX|        230|\n",
      "|100022|    1|     0|      1| 12|   TX|        681|\n",
      "|100023|    1|     1|      7| 14|   SC|        407|\n",
      "|100024|    1|     1|      5|  5|   PA|         22|\n",
      "|100025|    1|     0|      2| 23|   PA|        490|\n",
      "|   101|    1|     1|      3| 17|   CO|       1797|\n",
      "|   103|    1|     0|      7|  4|   PA|       1073|\n",
      "+------+-----+------+-------+---+-----+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model.select('userId', 'label', 'gender', 'workday', 'day', 'state', 'SongsPlayed') \\\n",
    "    .where(df.churn != 0) \\\n",
    "    .sort('userId').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert column to integer\n",
    "model = model.withColumn(\"userId\", model.userId.cast(\"integer\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('userId', 'int'),\n",
       " ('label', 'int'),\n",
       " ('gender', 'int'),\n",
       " ('workday', 'int'),\n",
       " ('day', 'int'),\n",
       " ('state', 'string'),\n",
       " ('SongsPlayed', 'bigint')]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove any missing values\n",
    "model = model.filter(\"userId is not NULL \\\n",
    "and label is not NULL \\\n",
    "and gender is not NULL \\\n",
    "and workday is not NULL \\\n",
    "and day is not NULL \\\n",
    "and state is not NULL \\\n",
    "and SongsPlayed is not NULL\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+------+-------+---+-----+-----------+\n",
      "|userId|label|gender|workday|day|state|SongsPlayed|\n",
      "+------+-----+------+-------+---+-----+-----------+\n",
      "|100010|    0|     0|      6| 20|   CT|        275|\n",
      "|100010|    0|     0|      1|  5|   CT|        275|\n",
      "|100010|    0|     0|      3| 21|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      2| 16|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      2| 16|   CT|        275|\n",
      "|100010|    0|     0|      5|  9|   CT|        275|\n",
      "|100010|    0|     0|      1| 15|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      6| 20|   CT|        275|\n",
      "|100010|    0|     0|      1|  5|   CT|        275|\n",
      "|100010|    0|     0|      1|  5|   CT|        275|\n",
      "|100010|    0|     0|      1|  5|   CT|        275|\n",
      "|100010|    0|     0|      5|  9|   CT|        275|\n",
      "|100010|    0|     0|      5|  9|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      5|  9|   CT|        275|\n",
      "|100010|    0|     0|      2| 16|   CT|        275|\n",
      "|100010|    0|     0|      2| 16|   CT|        275|\n",
      "+------+-----+------+-------+---+-----+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[userId: int, label: int, gender: int, workday: int, day: int, state: string, SongsPlayed: bigint]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop duplicates\n",
    "model.dropDuplicates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modeling\n",
    "Split the full dataset into train, test, and validation sets. Test out several of the machine learning methods you learned. Evaluate the accuracy of the various models, tuning parameters as necessary. Determine your winning model based on test accuracy and report results on the validation set. Since the churned users are a fairly small subset, I suggest using F1 score as the metric to optimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a StringIndexer\n",
    "state_indexer = StringIndexer(inputCol =\"state\", outputCol =\"state_index\")\n",
    "\n",
    "# Create a OneHotEncoder\n",
    "state_encoder = OneHotEncoder(inputCol=\"state_index\", outputCol=\"state_fact\")\n",
    "\n",
    "# Make a VectorAssembler\n",
    "vec_assembler = VectorAssembler(inputCols=[\"gender\", \"state_fact\", \"workday\", \"day\", \"SongsPlayed\"], \\\n",
    "                                outputCol=\"features\")\n",
    "\n",
    "# Make the pipeline\n",
    "churn_pipe = Pipeline(stages=[state_indexer, state_encoder, vec_assembler])\n",
    "\n",
    "# Fit and transform the data\n",
    "piped_data = churn_pipe.fit(model).transform(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the data into training and test sets\n",
    "training, test = piped_data.randomSplit([.6, .4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a LogisticRegression Estimator\n",
    "lr = LogisticRegression()\n",
    "\n",
    "# Create a RandomForest Classifier\n",
    "rf = RandomForestClassifier()\n",
    "\n",
    "# Create Decision Tree Classifier\n",
    "dt = DecisionTreeClassifier()\n",
    "\n",
    "# Create Gradiant Boosting Classifier\n",
    "gbt = GBTClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a BinaryClassificationEvaluator ROC\n",
    "evaluator_ROC = evals.BinaryClassificationEvaluator(metricName=\"areaUnderROC\")\n",
    "\n",
    "# Create a BinaryClassificationEvaluator F1 score\n",
    "evaluator_f1 = evals.BinaryClassificationEvaluator(metricName=\"f1\")\n",
    "\n",
    "# Create a BinaryClassificationEvaluator Precision\n",
    "evaluator_PR = evals.BinaryClassificationEvaluator(metricName=\"precisionByLabel\")\n",
    "\n",
    "# Create a BinaryClassificationEvaluator Recall\n",
    "evaluator_RC = evals.BinaryClassificationEvaluator(metricName=\"recallByLabel\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the parameter grid\n",
    "grid = tune.ParamGridBuilder()\n",
    "\n",
    "# Add the hyperparameter\n",
    "grid = grid.addGrid(lr.regParam, np.arange(0, .1, .01))\n",
    "grid = grid.addGrid(lr.elasticNetParam, [0, 1])\n",
    "\n",
    "# Build the grid\n",
    "grid = grid.build()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the CrossValidator\n",
    "cv = tune.CrossValidator(estimator=lr,\n",
    "            estimatorParamMaps=grid,\n",
    "            evaluator=evaluator_ROC\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit cross validation models\n",
    "models = cv.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the best model\n",
    "best_lr = models.bestModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the model to predict the test set\n",
    "test_results = best_lr.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5754628261939931\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the predictions\n",
    "print(evaluator_ROC.evaluate(test_results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m~/anaconda3/lib/python3.7/multiprocessing/pool.py\u001b[0m in \u001b[0;36mnext\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    732\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 733\u001b[0;31m                 \u001b[0mitem\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_items\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpopleft\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    734\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mIndexError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIndexError\u001b[0m: pop from an empty deque",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-58-3c9cc2c303b9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m \u001b[0;31m# Run cross validations.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m \u001b[0mrfcvModel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrfcv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtraining\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     22\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrfcvModel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/base.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, dataset, params)\u001b[0m\n\u001b[1;32m    159\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    160\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 161\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    162\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    163\u001b[0m             raise ValueError(\"Params must be either a param map or a list/tuple of param maps, \"\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/tuning.py\u001b[0m in \u001b[0;36m_fit\u001b[0;34m(self, dataset)\u001b[0m\n\u001b[1;32m    685\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    686\u001b[0m             \u001b[0mtasks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_parallelFitTasks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meva\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepm\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcollectSubModelsParam\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 687\u001b[0;31m             \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmetric\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msubModel\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mpool\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimap_unordered\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtasks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    688\u001b[0m                 \u001b[0mmetrics\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmetric\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mnFolds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    689\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mcollectSubModelsParam\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/multiprocessing/pool.py\u001b[0m in \u001b[0;36mnext\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    735\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_index\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_length\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    736\u001b[0m                     \u001b[0;32mraise\u001b[0m \u001b[0mStopIteration\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 737\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_cond\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    738\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    739\u001b[0m                     \u001b[0mitem\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_items\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpopleft\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/threading.py\u001b[0m in \u001b[0;36mwait\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    294\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m    \u001b[0;31m# restore state no matter what (e.g., KeyboardInterrupt)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    295\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 296\u001b[0;31m                 \u001b[0mwaiter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0macquire\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    297\u001b[0m                 \u001b[0mgotit\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    298\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Create an initial RandomForest model.\n",
    "rf = RandomForestClassifier(labelCol=\"label\", featuresCol=\"features\")\n",
    "\n",
    "# Evaluate model\n",
    "rfevaluator = BinaryClassificationEvaluator()\n",
    "\n",
    "# Create ParamGrid for Cross Validation\n",
    "rfparamGrid = (ParamGridBuilder()\n",
    "             .addGrid(rf.maxDepth, [2, 5, 10, 20, 30])\n",
    "             .addGrid(rf.maxBins, [10, 20, 40, 80, 100])\n",
    "             .addGrid(rf.numTrees, [5, 20, 50, 100, 500])\n",
    "             .build())\n",
    "\n",
    "# Create 5-fold CrossValidator\n",
    "rfcv = CrossValidator(estimator = rf,\n",
    "                      estimatorParamMaps = rfparamGrid,\n",
    "                      evaluator = rfevaluator,\n",
    "                      numFolds = 5)\n",
    "\n",
    "# Run cross validations.\n",
    "rfcvModel = rfcv.fit(training)\n",
    "print(rfcvModel)\n",
    "\n",
    "# Use test set here so we can measure the accuracy of our model on new data\n",
    "rfpredictions = rfcvModel.transform(test)\n",
    "\n",
    "# cvModel uses the best model found from the Cross Validation\n",
    "# Evaluate best model\n",
    "print('Accuracy:', rfevaluator.evaluate(rfpredictions))\n",
    "print('AUC:', BinaryClassificationMetrics(rfpredictions['label','prediction'].rdd).areaUnderROC)\n",
    "#ComputeModelStatistics().transform(rfpredictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the CrossValidator RandomForest\n",
    "cv1 = tune.CrossValidator(estimator=rf,\n",
    "            estimatorParamMaps=grid,\n",
    "            evaluator=evaluator\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the CrossValidator RandomForest\n",
    "cv_DT = tune.CrossValidator(estimator=dt,\n",
    "            estimatorParamMaps=grid,\n",
    "            evaluator=evaluator\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the CrossValidator RandomForest\n",
    "cv_GBT = tune.CrossValidator(estimator=gbt,\n",
    "            estimatorParamMaps=grid,\n",
    "            evaluator=evaluator\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit cross validation models\n",
    "models = cv.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "models1 = cv1.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "models1 = cv_DT.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "models3 = cv_GBT.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the best model\n",
    "best_lr = models.bestModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the best model\n",
    "best_lr = models3.bestModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the model to predict the test set\n",
    "test_results = best_lr.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the best model\n",
    "test_results1 = best_lr.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6595653631970981\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the predictions\n",
    "print(evaluator.evaluate(test_results))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6511118233361021\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the predictions\n",
    "print(evaluator.evaluate(test_results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6511118233361021\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the predictions\n",
    "evaluator1 = evals.BinaryClassificationEvaluator()\n",
    "print(evaluator1.evaluate(test_results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n"
     ]
    }
   ],
   "source": [
    "#RF\n",
    "print(evaluator.evaluate(test_results1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n"
     ]
    }
   ],
   "source": [
    "#DT\n",
    "print(evaluator.evaluate(test_results1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n"
     ]
    }
   ],
   "source": [
    "#DTG\n",
    "print(evaluator.evaluate(test_results1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the below for model features:\n",
    "1. gender\n",
    "2. workday\n",
    "3. day\n",
    "4. state\n",
    "5. SongsPlayed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature 1 - gender - APPROAHC2!!!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='10', gender=1)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create gender data frame\n",
    "gender = df.dropDuplicates(['userId']).sort('userId').select(['userId','gender'])\n",
    "gender = gender.filter(gender[\"userId\"] != \"\")\n",
    "gender = gender.replace(['M','F'], ['1', '0'], 'gender')\n",
    "gender = gender.withColumn('gender', gender.gender.cast(\"int\"))\n",
    "\n",
    "gender.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[userId: string, gender: int]"
      ]
     },
     "execution_count": 243,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "\n",
    "df_gender = df.dropDuplicates(['userId']).sort('userId').select(['userId','gender'])\n",
    "df_gender = df_gender.replace(['F','M'], ['1', '0'], 'gender')\n",
    "df_gender.withColumn('gender', df_gender.gender.cast(\"int\"))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "df_songs = df.where(df.song!='null').groupby('userId') \\\n",
    "    .agg(count(df.song).alias('SongsPlayed')).orderBy('userId') \\\n",
    "    .select(['userId','SongsPlayed'])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final = df.dropDuplicates(['userId']).sort('userId').select(['userId','churn', 'workday', 'day', 'state'])\n",
    "for feature in [df_gender, df_songs]:\n",
    "    df_model = df_final.join(feature,'userId')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+-------+---+-----+-----------+\n",
      "|userId|churn|workday|day|state|SongsPlayed|\n",
      "+------+-----+-------+---+-----+-----------+\n",
      "|100010|    0|      1|  8|   CT|        275|\n",
      "|200002|    0|      1|  1|   WI|        387|\n",
      "|   125|    0|      5| 12|   TX|          8|\n",
      "|   124|    0|      1|  1|   NJ|       4079|\n",
      "|    51|    0|      1|  1|   KY|       2111|\n",
      "|     7|    0|      3|  3|   WA|        150|\n",
      "|    15|    0|      1|  1|   WV|       1914|\n",
      "|    54|    0|      1|  1|   WA|       2841|\n",
      "|   155|    0|      5|  2|   NH|        820|\n",
      "|100014|    0|      3| 10|   PA|        257|\n",
      "|   132|    0|      2|  2|   CO|       1928|\n",
      "|   154|    0|      5|  2|   PA|         84|\n",
      "|   101|    0|      1|  1|   CO|       1797|\n",
      "|    11|    0|      5|  5|   MA|        647|\n",
      "|   138|    0|      6|  6|   PA|       2070|\n",
      "|300017|    0|      1|  1|   MI|       3632|\n",
      "|100021|    0|      4| 11|   TX|        230|\n",
      "|    29|    0|      2|  2|   CA|       3028|\n",
      "|    69|    0|      3|  3|   WI|       1125|\n",
      "|   112|    0|      4|  4|   IN|        215|\n",
      "+------+-----+-------+---+-----+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "df_model.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----+\n",
      "|churn|count|\n",
      "+-----+-----+\n",
      "|    0|  225|\n",
      "+-----+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_model.groupby(\"churn\").count().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature 2 - workday"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(artist='Martha Tilston', auth='Logged In', firstName='Colin', gender='M', itemInSession=50, lastName='Freeman', length=277.89016, level='paid', location='Bakersfield, CA', method='PUT', page='NextSong', registration=1538173362000, sessionId=29, song='Rockpools', status=200, ts=1538352117000, userAgent='Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0', userId='30', hour=3, day=1, workday_='Monday', month=10, churn=0, state='CA', workday=1)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert workdays to digits\n",
    "day_ind = [1,2,3,4,5,6,7]\n",
    "get_wkday = udf(lambda x: day_ind[datetime.datetime.fromtimestamp(x/1000).weekday()], IntegerType() )\n",
    "df = df.withColumn('workday', get_wkday('ts'))\n",
    "\n",
    "df.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='10', workday=1)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create workday data frame\n",
    "workday = df.dropDuplicates(['userId']).sort('userId').select(['userId','workday'])\n",
    "workday = workday.filter(workday[\"userId\"] != \"\")\n",
    "\n",
    "workday.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='30', workday=1)]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workday = df.select(['userId','workday'])\n",
    "workday = workday.filter(workday[\"userId\"] != \"\")\n",
    "\n",
    "workday.take(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature 3 - day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='10', day=8)]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create day data frame\n",
    "day = df.dropDuplicates(['userId']).sort('userId').select(['userId','day'])\n",
    "day = day.filter(day[\"userId\"] != \"\")\n",
    "\n",
    "day.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='30', day=1)]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "day = df.select(['userId','day'])\n",
    "day = day.filter(day[\"userId\"] != \"\")\n",
    "\n",
    "day.take(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature 4 - state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='10', state='MS')]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create state data frame\n",
    "state = df.dropDuplicates(['userId']).sort('userId').select(['userId','state'])\n",
    "state = state.filter(state[\"userId\"] != \"\")\n",
    "\n",
    "state.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='30', state='CA')]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state = df.select(['userId','state'])\n",
    "state = state.filter(state[\"userId\"] != \"\")\n",
    "\n",
    "state.take(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature 5 - SongsPlayed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='10', SongsPlayed=673)]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create songs played data frame\n",
    "songs_played = df.where(df.song!='null').groupby('userId') \\\n",
    "    .agg(count(df.song).alias('SongsPlayed')) \\\n",
    "    .orderBy('userId').select(['userId','SongsPlayed'])\n",
    "\n",
    "songs_played = songs_played.filter(songs_played[\"userId\"] != \"\")\n",
    "\n",
    "songs_played.take(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Features data frame - gets churn only with 0!!!! - used DropDuplicates() on userID which removed any churn data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='100010', churn=0)]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create features data frame\n",
    "#model_data = df.select(['userId','churn'])\n",
    "#model_data = model_data.filter(state[\"userId\"] != \"\")\n",
    "model_data = df.dropDuplicates(['userId']).select(['userId','churn'])\n",
    "\n",
    "\n",
    "\n",
    "model_data.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+------+-----+\n",
      "|churn|userId|count|\n",
      "+-----+------+-----+\n",
      "+-----+------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check if churn has value of 1 records\n",
    "model_data.where(model_data.churn != 0).groupBy(['churn','userId']).count()\\\n",
    "    .sort('userId', ascending=False).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId='100010', label=0)]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert column churn to label\n",
    "model_data = model_data.withColumnRenamed(\"churn\", \"label\")\n",
    "\n",
    "model_data.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+------+-------+---+------+-------+---+-----+-----------+------+-------+---+-----+-----------+\n",
      "|userId|label|gender|workday|day|gender|workday|day|state|SongsPlayed|gender|workday|day|state|SongsPlayed|\n",
      "+------+-----+------+-------+---+------+-------+---+-----+-----------+------+-------+---+-----+-----------+\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|     0|      1|  8|   CT|        275|     0|      1|  8|   CT|        275|\n",
      "+------+-----+------+-------+---+------+-------+---+-----+-----------+------+-------+---+-----+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Join all features into one data frame\n",
    "model_data = model_data.join(gender, on='userId', how=\"left\")\n",
    "model_data = model_data.join(workday, on='userId', how=\"left\")\n",
    "model_data = model_data.join(day, on='userId', how=\"left\")\n",
    "model_data = model_data.join(state, on='userId', how=\"left\")\n",
    "model_data = model_data.join(songs_played, on='userId', how=\"left\")\n",
    "\n",
    "model_data.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('userId', 'string'),\n",
       " ('label', 'int'),\n",
       " ('gender', 'int'),\n",
       " ('workday', 'int'),\n",
       " ('day', 'int'),\n",
       " ('state', 'string'),\n",
       " ('SongsPlayed', 'bigint')]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-91-decd9c4f6f6b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mmodel_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwhere\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlabel\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroupBy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'label'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'userId'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcount\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0;34m.\u001b[0m\u001b[0msort\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'userId'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mascending\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/sql/dataframe.py\u001b[0m in \u001b[0;36mshow\u001b[0;34m(self, n, truncate, vertical)\u001b[0m\n\u001b[1;32m    482\u001b[0m         \"\"\"\n\u001b[1;32m    483\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtruncate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mtruncate\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 484\u001b[0;31m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshowString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m20\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvertical\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    485\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    486\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshowString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtruncate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvertical\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1301\u001b[0m             \u001b[0mproto\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mEND_COMMAND_PART\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1302\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1303\u001b[0;31m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1304\u001b[0m         return_value = get_return_value(\n\u001b[1;32m   1305\u001b[0m             answer, self.gateway_client, self.target_id, self.name)\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36msend_command\u001b[0;34m(self, command, retry, binary)\u001b[0m\n\u001b[1;32m   1031\u001b[0m         \u001b[0mconnection\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_connection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1032\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1033\u001b[0;31m             \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconnection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1034\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1035\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_connection_guard\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconnection\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36msend_command\u001b[0;34m(self, command)\u001b[0m\n\u001b[1;32m   1198\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1199\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1200\u001b[0;31m             \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msmart_decode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstream\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1201\u001b[0m             \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Answer received: {0}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0manswer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1202\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0manswer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mproto\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRETURN_MESSAGE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    587\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    588\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 589\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    590\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    591\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "model_data.where(model_data.label != 0).groupBy(['label','userId']).count()\\\n",
    "    .sort('userId', ascending=False).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+------+-----+\n",
      "|label|userId|count|\n",
      "+-----+------+-----+\n",
      "|    1|    87|    1|\n",
      "|    1|    73|    1|\n",
      "|    1|    70|    1|\n",
      "|    1|    58|    1|\n",
      "|    1|    54|    1|\n",
      "|    1|    53|    1|\n",
      "|    1|    51|    1|\n",
      "|    1|    32|    1|\n",
      "|    1|300007|    1|\n",
      "|    1|300001|    1|\n",
      "|    1|     3|    1|\n",
      "|    1|    29|    1|\n",
      "|    1|    28|    1|\n",
      "|    1|200024|    1|\n",
      "|    1|200021|    1|\n",
      "|    1|200020|    1|\n",
      "|    1|200018|    1|\n",
      "|    1|200017|    1|\n",
      "|    1|200016|    1|\n",
      "|    1|200015|    1|\n",
      "+-----+------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check if churn has value of 1 records\n",
    "model_data.where(model_data.label != 0).groupBy(['label','userId']).count()\\\n",
    "    .sort('userId', ascending=False).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "model1 = model_data.dropDuplicates(['userId'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-93-4fb763c2e2b8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mmodel1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwhere\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlabel\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroupBy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'label'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'userId'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcount\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;31m\\\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0;34m.\u001b[0m\u001b[0msort\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'userId'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mascending\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/sql/dataframe.py\u001b[0m in \u001b[0;36mshow\u001b[0;34m(self, n, truncate, vertical)\u001b[0m\n\u001b[1;32m    482\u001b[0m         \"\"\"\n\u001b[1;32m    483\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtruncate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mtruncate\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 484\u001b[0;31m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshowString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m20\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvertical\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    485\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    486\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshowString\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtruncate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvertical\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1301\u001b[0m             \u001b[0mproto\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mEND_COMMAND_PART\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1302\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1303\u001b[0;31m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1304\u001b[0m         return_value = get_return_value(\n\u001b[1;32m   1305\u001b[0m             answer, self.gateway_client, self.target_id, self.name)\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36msend_command\u001b[0;34m(self, command, retry, binary)\u001b[0m\n\u001b[1;32m   1031\u001b[0m         \u001b[0mconnection\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_connection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1032\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1033\u001b[0;31m             \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconnection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1034\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1035\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_connection_guard\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconnection\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36msend_command\u001b[0;34m(self, command)\u001b[0m\n\u001b[1;32m   1198\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1199\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1200\u001b[0;31m             \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msmart_decode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstream\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1201\u001b[0m             \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Answer received: {0}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0manswer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1202\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0manswer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mproto\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRETURN_MESSAGE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    587\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    588\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 589\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    590\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    591\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "model1.where(model1.label != 0).groupBy(['label','userId']).count()\\\n",
    "    .sort('userId', ascending=False).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('userId', 'string'),\n",
       " ('label', 'int'),\n",
       " ('gender', 'int'),\n",
       " ('workday', 'int'),\n",
       " ('day', 'int'),\n",
       " ('state', 'string'),\n",
       " ('SongsPlayed', 'bigint')]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check column types\n",
    "model_data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Covert columns to integers for modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert string columns to integers\n",
    "model_data = model_data.withColumn(\"userId\", model_data.userId.cast(\"integer\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('userId', 'int'),\n",
       " ('label', 'int'),\n",
       " ('gender', 'int'),\n",
       " ('workday', 'int'),\n",
       " ('day', 'int'),\n",
       " ('state', 'string'),\n",
       " ('SongsPlayed', 'bigint')]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove any missing values\n",
    "model_data = model_data.filter(\"userId is not NULL \\\n",
    "and label is not NULL \\\n",
    "and gender is not NULL \\\n",
    "and workday is not NULL \\\n",
    "and day is not NULL \\\n",
    "and state is not NULL \\\n",
    "and SongsPlayed is not NULL\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(userId=100010, label=0, gender=0, workday=1, day=8, state='CT', SongsPlayed=275)]"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_data.take(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ETL pipeline\n",
    "\n",
    "1. Load data\n",
    "2. Create workdays/days columns\n",
    "3. Create features\n",
    "4. Join data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modeling\n",
    "Split the full dataset into train, test, and validation sets. Test out several of the machine learning methods you learned. Evaluate the accuracy of the various models, tuning parameters as necessary. Determine your winning model based on test accuracy and report results on the validation set. Since the churned users are a fairly small subset, I suggest using F1 score as the metric to optimize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+------+-------+---+-----+-----------+\n",
      "|userId|label|gender|workday|day|state|SongsPlayed|\n",
      "+------+-----+------+-------+---+-----+-----------+\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "|100010|    0|     0|      1|  8|   CT|        275|\n",
      "+------+-----+------+-------+---+-----+-----------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_data.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a StringIndexer\n",
    "state_indexer = StringIndexer(inputCol =\"state\", outputCol =\"state_index\")\n",
    "\n",
    "# Create a OneHotEncoder\n",
    "state_encoder = OneHotEncoder(inputCol=\"state_index\", outputCol=\"state_fact\")\n",
    "\n",
    "# Make a VectorAssembler\n",
    "vec_assembler = VectorAssembler(inputCols=[\"gender\", \"state_fact\", \"workday\", \"day\", \"SongsPlayed\"], \\\n",
    "                                outputCol=\"features\")\n",
    "\n",
    "# Make the pipeline\n",
    "churn_pipe = Pipeline(stages=[state_indexer, state_encoder, vec_assembler])\n",
    "\n",
    "# Fit and transform the data\n",
    "piped_data = churn_pipe.fit(model_data).transform(model_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the data into training and test sets\n",
    "training, test = piped_data.randomSplit([.6, .4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a LogisticRegression Estimator\n",
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a BinaryClassificationEvaluator\n",
    "evaluator = evals.BinaryClassificationEvaluator(metricName=\"areaUnderROC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the parameter grid\n",
    "grid = tune.ParamGridBuilder()\n",
    "\n",
    "# Add the hyperparameter\n",
    "grid = grid.addGrid(lr.regParam, np.arange(0, .1, .01))\n",
    "grid = grid.addGrid(lr.elasticNetParam, [0, 1])\n",
    "\n",
    "# Build the grid\n",
    "grid = grid.build()\n",
    "\n",
    "# Create the CrossValidator\n",
    "cv = tune.CrossValidator(estimator=lr,\n",
    "            estimatorParamMaps=grid,\n",
    "            evaluator=evaluator\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13min 6s ± 2min 42s per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "# Fit cross validation models\n",
    "models = cv.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'models' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-84-9784c1427948>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'timeit'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'# Extract the best model\\nbest_lr = models.bestModel\\n'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m   2397\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2398\u001b[0m                 \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2399\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2400\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2401\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-53>\u001b[0m in \u001b[0;36mtimeit\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m    185\u001b[0m     \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    186\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m         \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    189\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/IPython/core/magics/execution.py\u001b[0m in \u001b[0;36mtimeit\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n\u001b[1;32m   1167\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mindex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1168\u001b[0m                 \u001b[0mnumber\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m10\u001b[0m \u001b[0;34m**\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1169\u001b[0;31m                 \u001b[0mtime_number\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtimer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimeit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnumber\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1170\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mtime_number\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0;36m0.2\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1171\u001b[0m                     \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/IPython/core/magics/execution.py\u001b[0m in \u001b[0;36mtimeit\u001b[0;34m(self, number)\u001b[0m\n\u001b[1;32m    167\u001b[0m         \u001b[0mgc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    168\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 169\u001b[0;31m             \u001b[0mtiming\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minner\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mit\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    170\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    171\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mgcold\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<magic-timeit>\u001b[0m in \u001b[0;36minner\u001b[0;34m(_it, _timer)\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'models' is not defined"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "# Extract the best model\n",
    "best_lr = models.bestModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'best_lr' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-81-7614efce2b26>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Use the model to predict the test set\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtest_results\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbest_lr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'best_lr' is not defined"
     ]
    }
   ],
   "source": [
    "# Use the model to predict the test set\n",
    "test_results = best_lr.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the predictions\n",
    "print(evaluator.evaluate(test_results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Train Test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 402,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split the data\n",
    "train, test = model_data.randomSplit([0.6, 0.4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Build Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 298,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a StringIndexer\n",
    "state_indexer = StringIndexer(inputCol =\"state\", outputCol =\"state_index\")\n",
    "\n",
    "# Create a OneHotEncoder\n",
    "state_encoder = OneHotEncoder(inputCol=\"state_index\", outputCol=\"state_fact\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 406,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "39.5 ms ± 6.77 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "\n",
    "# vector for features\n",
    "#vec_assembler = VectorAssembler(inputCols=[\"gender\", \"workday\", \"day\", \"state\", \"SongsPlayed\"], outputCol=\"features\")\n",
    "vec_assembler = VectorAssembler(inputCols=model_data.columns[2:],outputCol='features')\n",
    "\n",
    "# standard scaler\n",
    "scaler = StandardScaler(inputCol=\"features\", outputCol=\"features_scaled\")\n",
    "\n",
    "# build indexer\n",
    "indexer = StringIndexer(inputCol=\"churn\", outputCol=\"label\")\n",
    "\n",
    "# Call Logistic regression\n",
    "lr =  LogisticRegression(maxIter=10, regParam=0.0, elasticNetParam=0)\n",
    "\n",
    "stringIndexer = StringIndexer(inputCol=\"label\", outputCol=\"indexed\")\n",
    "\n",
    "rf = RandomForestClassifier(numTrees=3, maxDepth=2, labelCol=\"indexed\", seed=42)\n",
    "\n",
    "# Build pipeline\n",
    "#pipeline_lr = Pipeline(stages=[state_indexer, state_encoder, vec_assembler, scaler, indexer, lr])\n",
    "pipeline_lr = Pipeline(stages=[vec_assembler, scaler, indexer, lr])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Tune model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 404,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build parameter grid \n",
    "paramGrid_lr = ParamGridBuilder() \\\n",
    "    .addGrid(lr.regParam,[0.0, 0.1, 0.01]) \\\n",
    "    .build()\n",
    "\n",
    "# Build Cross Validator\n",
    "crossval_lr = CrossValidator(estimator=pipeline_lr,\n",
    "                          estimatorParamMaps=paramGrid_lr,\n",
    "                          evaluator=BinaryClassificationEvaluator(),\n",
    "                          numFolds=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 407,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "ename": "Py4JJavaError",
     "evalue": "An error occurred while calling o40045.fit.\n: org.apache.spark.SparkException: Input column state does not exist.\n\tat org.apache.spark.ml.feature.StringIndexerBase.$anonfun$validateAndTransformSchema$2(StringIndexer.scala:128)\n\tat scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)\n\tat scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)\n\tat scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)\n\tat scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)\n\tat scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)\n\tat org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema(StringIndexer.scala:123)\n\tat org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema$(StringIndexer.scala:115)\n\tat org.apache.spark.ml.feature.StringIndexer.validateAndTransformSchema(StringIndexer.scala:145)\n\tat org.apache.spark.ml.feature.StringIndexer.transformSchema(StringIndexer.scala:252)\n\tat org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:71)\n\tat org.apache.spark.ml.feature.StringIndexer.fit(StringIndexer.scala:237)\n\tat sun.reflect.GeneratedMethodAccessor960.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mPy4JJavaError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-407-c6caac924b65>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# vector problem\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mcvModel_lr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcrossval_lr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/base.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, dataset, params)\u001b[0m\n\u001b[1;32m    159\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    160\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 161\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    162\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    163\u001b[0m             raise ValueError(\"Params must be either a param map or a list/tuple of param maps, \"\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/tuning.py\u001b[0m in \u001b[0;36m_fit\u001b[0;34m(self, dataset)\u001b[0m\n\u001b[1;32m    685\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    686\u001b[0m             \u001b[0mtasks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_parallelFitTasks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meva\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepm\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcollectSubModelsParam\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 687\u001b[0;31m             \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmetric\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msubModel\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mpool\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimap_unordered\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtasks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    688\u001b[0m                 \u001b[0mmetrics\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmetric\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mnFolds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    689\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mcollectSubModelsParam\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/multiprocessing/pool.py\u001b[0m in \u001b[0;36mnext\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    746\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0msuccess\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    747\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 748\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    749\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    750\u001b[0m     \u001b[0m__next__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnext\u001b[0m                    \u001b[0;31m# XXX\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/multiprocessing/pool.py\u001b[0m in \u001b[0;36mworker\u001b[0;34m(inqueue, outqueue, initializer, initargs, maxtasks, wrap_exception)\u001b[0m\n\u001b[1;32m    119\u001b[0m         \u001b[0mjob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtask\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    120\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 121\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    122\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    123\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mwrap_exception\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfunc\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0m_helper_reraises_exception\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/tuning.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f)\u001b[0m\n\u001b[1;32m    685\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    686\u001b[0m             \u001b[0mtasks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_parallelFitTasks\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meva\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalidation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepm\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcollectSubModelsParam\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 687\u001b[0;31m             \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmetric\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msubModel\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mpool\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimap_unordered\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtasks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    688\u001b[0m                 \u001b[0mmetrics\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmetric\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mnFolds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    689\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mcollectSubModelsParam\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/tuning.py\u001b[0m in \u001b[0;36msingleTask\u001b[0;34m()\u001b[0m\n\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0msingleTask\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 69\u001b[0;31m         \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodelIter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     70\u001b[0m         \u001b[0;31m# TODO: duplicate evaluator to take extra params from input\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     71\u001b[0m         \u001b[0;31m#  Note: Supporting tuning params in evaluator need update method\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/base.py\u001b[0m in \u001b[0;36m__next__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     67\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"No models remaining.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcounter\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 69\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfitSingleModel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     70\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     71\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/base.py\u001b[0m in \u001b[0;36mfitSingleModel\u001b[0;34m(index)\u001b[0m\n\u001b[1;32m    124\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    125\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mfitSingleModel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 126\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparamMaps\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    127\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    128\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0m_FitMultipleIterator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfitSingleModel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparamMaps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/base.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, dataset, params)\u001b[0m\n\u001b[1;32m    157\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    158\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 159\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    160\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    161\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/pipeline.py\u001b[0m in \u001b[0;36m_fit\u001b[0;34m(self, dataset)\u001b[0m\n\u001b[1;32m    112\u001b[0m                     \u001b[0mdataset\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtransform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    113\u001b[0m                 \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# must be an Estimator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 114\u001b[0;31m                     \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    115\u001b[0m                     \u001b[0mtransformers\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    116\u001b[0m                     \u001b[0;32mif\u001b[0m \u001b[0mi\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0mindexOfLastEstimator\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/base.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, dataset, params)\u001b[0m\n\u001b[1;32m    159\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    160\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 161\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    162\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    163\u001b[0m             raise ValueError(\"Params must be either a param map or a list/tuple of param maps, \"\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/wrapper.py\u001b[0m in \u001b[0;36m_fit\u001b[0;34m(self, dataset)\u001b[0m\n\u001b[1;32m    333\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    334\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 335\u001b[0;31m         \u001b[0mjava_model\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fit_java\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    336\u001b[0m         \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjava_model\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    337\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_copyValues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/ml/wrapper.py\u001b[0m in \u001b[0;36m_fit_java\u001b[0;34m(self, dataset)\u001b[0m\n\u001b[1;32m    330\u001b[0m         \"\"\"\n\u001b[1;32m    331\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_transfer_params_to_java\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 332\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_java_obj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jdf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    333\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    334\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_fit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdataset\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1303\u001b[0m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1304\u001b[0m         return_value = get_return_value(\n\u001b[0;32m-> 1305\u001b[0;31m             answer, self.gateway_client, self.target_id, self.name)\n\u001b[0m\u001b[1;32m   1306\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1307\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mtemp_arg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtemp_args\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/pyspark/sql/utils.py\u001b[0m in \u001b[0;36mdeco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m    109\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mdeco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    110\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 111\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    112\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mpy4j\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPy4JJavaError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    113\u001b[0m             \u001b[0mconverted\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconvert_exception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjava_exception\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.7/site-packages/py4j/protocol.py\u001b[0m in \u001b[0;36mget_return_value\u001b[0;34m(answer, gateway_client, target_id, name)\u001b[0m\n\u001b[1;32m    326\u001b[0m                 raise Py4JJavaError(\n\u001b[1;32m    327\u001b[0m                     \u001b[0;34m\"An error occurred while calling {0}{1}{2}.\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 328\u001b[0;31m                     format(target_id, \".\", name), value)\n\u001b[0m\u001b[1;32m    329\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    330\u001b[0m                 raise Py4JError(\n",
      "\u001b[0;31mPy4JJavaError\u001b[0m: An error occurred while calling o40045.fit.\n: org.apache.spark.SparkException: Input column state does not exist.\n\tat org.apache.spark.ml.feature.StringIndexerBase.$anonfun$validateAndTransformSchema$2(StringIndexer.scala:128)\n\tat scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)\n\tat scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)\n\tat scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)\n\tat scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)\n\tat scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)\n\tat scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)\n\tat scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)\n\tat org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema(StringIndexer.scala:123)\n\tat org.apache.spark.ml.feature.StringIndexerBase.validateAndTransformSchema$(StringIndexer.scala:115)\n\tat org.apache.spark.ml.feature.StringIndexer.validateAndTransformSchema(StringIndexer.scala:145)\n\tat org.apache.spark.ml.feature.StringIndexer.transformSchema(StringIndexer.scala:252)\n\tat org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:71)\n\tat org.apache.spark.ml.feature.StringIndexer.fit(StringIndexer.scala:237)\n\tat sun.reflect.GeneratedMethodAccessor960.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)\n\tat py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)\n\tat py4j.Gateway.invoke(Gateway.java:282)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:748)\n"
     ]
    }
   ],
   "source": [
    "# vector problem\n",
    "cvModel_lr = crossval_lr.fit(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Compute Accuracy of the best model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Spark it's important to make sure you split the data after all the transformations. This is because operations like StringIndexer don't always produce the same index even when given the same list of strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6369735662079051\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Steps\n",
    "Clean up your code, adding comments and renaming variables to make the code easier to read and maintain. Refer to the Spark Project Overview page and Data Scientist Capstone Project Rubric to make sure you are including all components of the capstone project and meet all expectations. Remember, this includes thorough documentation in a README file in a Github repository, as well as a web app or blog post."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}