sklearn/5-sklearn-classify-nycflights.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scikit-learn Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import sklearn as sklearn\n",
    "from sklearn import linear_model, cross_validation, metrics, svm, ensemble\n",
    "from sklearn.metrics import classification_report, confusion_matrix, precision_recall_fscore_support, accuracy_score\n",
    "from sklearn.model_selection  import train_test_split, cross_val_score, ShuffleSplit\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "flights = pd.read_csv('https://elephantscale-public.s3.amazonaws.com/data/nycflights13/flights.csv.gz')\n",
    "weather = pd.read_csv('https://elephantscale-public.s3.amazonaws.com/data/nycflights13/weather.csv.gz')\n",
    "airports = pd.read_csv('https://elephantscale-public.s3.amazonaws.com/data/nycflights13/airports.csv.gz')\n",
    "\n",
    "df_withweather = pd.merge(flights, weather, how='left', on=['year','month', 'day', 'hour', 'origin'])\n",
    "df = pd.merge(df_withweather, airports, how='left', left_on='dest', right_on='faa')\n",
    "\n",
    "df = df.dropna()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "pred = 'dep_delay'\n",
    "features =  ['month','day','dep_time','arr_time','carrier','dest','air_time','distance', \n",
    "             'lat', 'lon', 'alt',  'dewp', 'humid', 'wind_speed', 'wind_gust', \n",
    "             'precip', 'pressure', 'visib' ]\n",
    "\n",
    "features_v = df[features]\n",
    "pred_v = df[pred]\n",
    "\n",
    "how_late_is_late = 15.0;\n",
    "\n",
    "pd.options.mode.chained_assignment = None  # default='warn'\n",
    "\n",
    "\n",
    "# carrier is not a number, so transform it into an number\n",
    "features_v['carrier'] = pd.factorize(features_v['carrier'])[0]\n",
    "\n",
    "# dest is not a number, so transform it into a number\n",
    "features_v['dest'] = pd.factorize(features_v['dest'])[0]\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaled_features_v = scaler.fit_transform(features_v)\n",
    "\n",
    "features_train, features_test, pred_train, pred_test = train_test_split(\n",
    "    scaled_features_v, pred_v, test_size=0.30, random_state=0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doing the classification with Logistic Regression\n",
    "\n",
    "We will first attempt to do classification with logistic regression.  This likely will not give us the best results because logistic regression is a linear model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Perform logistic regression for classification\n",
    "\n",
    "clf_lr = sklearn.linear_model.LogisticRegression(penalty='l2', \n",
    "                                                 class_weight='balanced')\n",
    "logistic_fit=clf_lr.fit(features_train, \n",
    "                        np.where(pred_train >= how_late_is_late,1,0))\n",
    "\n",
    "predictions = clf_lr.predict(features_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Summary Report\n",
    "\n",
    "# Confusion Matrix\n",
    "cm_lr = confusion_matrix(np.where(pred_test >= how_late_is_late,1,0), \n",
    "                         predictions)\n",
    "print(\"Confusion matrix\")\n",
    "print(pd.DataFrame(cm_lr))\n",
    "\n",
    "# Get accuracy\n",
    "report_lr = precision_recall_fscore_support(\n",
    "    list(np.where(pred_test >= how_late_is_late,1,0)), \n",
    "    list(predictions), average='binary')\n",
    "\n",
    "#Print Accuracy\n",
    "print (\"\\nprecision = %0.2f, recall = %0.2f, F1 = %0.2f, accuracy = %0.2f\"\n",
    "       % (report_lr[0], report_lr[1], report_lr[2],                                                                         \n",
    "          accuracy_score(list(np.where(pred_test >= how_late_is_late,1,0)), \n",
    "                                                                                             list(predictions))))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An accuracy of 67% is not particularly good.  A bigger concern, however is the relatively low precision and F1 scores, which indicate that our model is better at predicting negatives (not late) than positives (late).  \n",
    "\n",
    "However, predicting flight delays from the data we have is not easy. \n",
    "\n",
    "### Another Attempt: A Random Forest Classifier\n",
    "\n",
    "The low precision should concern us. This indicates that potentially we are less able to predict flights that are actuallly late (which is more important) than those that are not late.  This is due to an unbalanced training set.\n",
    "\n",
    "Perhaps a Random Forest Classifier could help us.  Let's try that.  We'll do 40 trees, and we'll scramble the input set of features (bagging) so that we'll get 40 different trees. Let's see what that does to our precision.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform Random Forest Model for classification\n",
    "# 40 Trees.\n",
    "\n",
    "clf_rf = sklearn.ensemble.RandomForestClassifier(n_estimators=40)\n",
    "rf_fit=clf_rf.fit(features_train, \n",
    "                        np.where(pred_train >= how_late_is_late,1,0))\n",
    "\n",
    "predictions_rf = clf_rf.predict(features_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Summary Report\n",
    "\n",
    "# Confusion Matrix\n",
    "cm_rf = confusion_matrix(np.where(pred_test >= how_late_is_late,1,0), \n",
    "                         predictions_rf)\n",
    "print(\"Confusion matrix\")\n",
    "print(pd.DataFrame(cm_rf))\n",
    "\n",
    "# Get accuracy\n",
    "report_rf = precision_recall_fscore_support(\n",
    "    list(np.where(pred_test >= how_late_is_late,1,0)), \n",
    "    list(predictions_rf), average='binary')\n",
    "\n",
    "#Print Accuracy\n",
    "print (\"\\nprecision = %0.2f, recall = %0.2f, F1 = %0.2f, accuracy = %0.2f\"\n",
    "       % (report_rf[0], report_rf[1], report_rf[2],                                                                         \n",
    "          accuracy_score(list(np.where(pred_test >= how_late_is_late,1,0)), \n",
    "                                                                                             list(predictions))))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the Results\n",
    "\n",
    "Our accuracy is about the same, but precision is now quite good at 82%. That's good.  But the low recall score should give us pause, and possibly tell us that we need to further tune our model.\n",
    "\n",
    "### Feature Importances\n",
    "\n",
    "Random forest classifiers can also tell us how important each feature is.  This is because we use different sets of input features for each of the 40 trees, and we can use this to evaluate which features appear to be the most predictive. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Print the feature ranking\n",
    "print(\"Feature ranking:\")\n",
    "\n",
    "indices = np.argsort(clf_rf.feature_importances_)[::-1]\n",
    "std = np.std([clf_rf.feature_importances_ for tree in clf_rf.estimators_],\n",
    "             axis=0)\n",
    "\n",
    "for f in range(features_train.shape[1]):\n",
    "    print(\"%d. feature %d: %s (%f)\" % (f + 1, indices[f], features[f], clf_rf.feature_importances_[indices[f]]))\n",
    "\n",
    "# Plot the feature importances of the forest\n",
    "plt.figure()\n",
    "plt.title(\"Feature importances\")\n",
    "plt.bar(range(features_train.shape[1]), clf_rf.feature_importances_[indices],\n",
    "       color=\"r\", yerr=std[indices], align=\"center\")\n",
    "plt.xticks(range(features_train.shape[1]), indices)\n",
    "plt.xlim([-1, features_train.shape[1]])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating Feature Importances\n",
    "\n",
    "What does the relative weight of feature importances say about\n",
    "our dataset? Are there any features we might want to omit?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}