part2/Part 2 - Data Transformation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "id": "TKLpHW8UcdmR",
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Data Transformation\n",
    "\n",
    "- **Objective**: Learn about integrating data transformations into a chart definition.\n",
    "- **Key Topics**:\n",
    "  - Binning and aggregation\n",
    "  - Advanced data transformations\n",
    "\n",
    "In the previous section we learned how to use marks and visual encodings to represent individual data records. In this section, we will explore methods for *transforming* data, including the use of aggregates to summarize multiple records. Data transformation is an integral part of visualization: choosing the variables to show and their level of detail is just as important as choosing appropriate visual encodings. After all, it doesn't matter how well chosen your visual encodings are if you are showing the wrong information!\n",
    "\n",
    "As we work through this module, it may be helpful to open the [Altair Data Transformations](https://altair-viz.github.io/user_guide/transform/index.html) documentation in another tab. It will be a useful resource if at any point you'd like more details or want to see what other transformations are available.\n",
    "\n",
    "## Credits\n",
    "This section is largely based on the [Visualization Curriculum](https://idl.uw.edu/visualization-curriculum/intro.html) developed at the University of Washington by Jeffrey Heer, Dominik Moritz, Jake VanderPlas, and Brock Craft.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bptagN7weJAt"
   },
   "source": [
    "## Imports\n",
    "Perform the package imports that will be needed for this section of the tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 753,
     "status": "ok",
     "timestamp": 1714994301504,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "__PnOOh2zPY7"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import altair as alt\n",
    "from vega_datasets import data\n",
    "import vegafusion\n",
    "\n",
    "print(\"The installed Vega-Altair version is \" + alt.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uIecrVj2epeR"
   },
   "source": [
    "## The Movies Dataset\n",
    "We will be working with a table of data about motion pictures, taken from the vega-datasets collection which is provided by the `vega_datasets` Python package. The dataset includes variables such as the film name, director, genre, release date, ratings, and gross revenues. However, *be careful when working with this data*: the films are from unevenly sampled years, using data combined from multiple sources. If you dig in you will find issues with missing values and even some subtle errors! Nevertheless, the data should prove interesting to explore…\n",
    "\n",
    "\n",
    "Let's load the dataset into a pandas DataFrame using the `vega_datasets` package so that we can inspect its contents.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 180,
     "status": "ok",
     "timestamp": 1714994301681,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "gnqEaZJNbvVD"
   },
   "outputs": [],
   "source": [
    "movies = data.movies()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M5EHV_yUgABM"
   },
   "source": [
    "How many rows (records) and columns (fields) are in the movies dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 162,
     "status": "ok",
     "timestamp": 1714994301840,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "-yaixVCif_yn",
    "outputId": "6552f9c4-aaa7-4154-c23a-2b0290441264"
   },
   "outputs": [],
   "source": [
    "movies.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bArXYSPegFe2"
   },
   "source": [
    "Now let’s peek at the first 5 rows of the table to get a sense of the fields and data types…"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "executionInfo": {
     "elapsed": 296,
     "status": "ok",
     "timestamp": 1714994302134,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "3U1plChWgJHw",
    "outputId": "fa7cd428-0f4e-4101-e7f7-b69454936972"
   },
   "outputs": [],
   "source": [
    "movies.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6lXjiQwXgZ2D"
   },
   "source": [
    "## Histograms\n",
    "We’ll start our transformation tour by *binning* data into discrete groups and *counting* records to summarize those groups. The resulting plots are known as [*histograms*](https://en.wikipedia.org/wiki/Histogram).\n",
    "\n",
    "Let’s first look at the unaggregated data: a scatter plot showing movie ratings from Rotten Tomatoes versus ratings from IMDB users. We'll provide data to Altair by passing the `movies` DataFrame to the `Chart` constructor.  We can then encode the Rotten Tomatoes and IMDB ratings fields using the `x` and `y` channels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 1156,
     "status": "ok",
     "timestamp": 1714994303288,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "qHQ_jeeehFU3",
    "outputId": "5dea7c13-7ad6-4485-9182-334622aa3a3a"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_circle().encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating'),\n",
    "    alt.Y('IMDB_Rating')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-kWGMSh4isVA"
   },
   "source": [
    "To summarize this dataset, we can bin a data field to group numeric values into discrete groups. Here we bin along the x-axis by adding `.bin()` to the `x` encoding channel. The result is a set of ten bins of equal step size, each corresponding to a span of ten ratings points.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 852,
     "status": "ok",
     "timestamp": 1714994304138,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "Bupp8Mysi1x7",
    "outputId": "5d0d365d-6c13-4981-f2ae-f7e4bc5f68fa"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_circle().encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating').bin(),\n",
    "    alt.Y('IMDB_Rating:Q')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r4j-Ew_NjHHm"
   },
   "source": [
    "Adding `.bin()` uses default binning settings, but we can exercise more control if desired. Let's instead set the maximum bin count (`maxbins`) to 20, which has the effect of doubling the number of bins. Now each bin corresponds to a span of five ratings points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 1204,
     "status": "ok",
     "timestamp": 1714994305340,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "Zm8t-bQpjNMi",
    "outputId": "36e459cf-0b17-484d-f460-49a0eaba62f6"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_circle().encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating').bin(maxbins=20),\n",
    "    alt.Y('IMDB_Rating:Q')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FpeHDfo0j5_h"
   },
   "source": [
    "It's important to note that binning, by itself, does not change the number of displayed points, it only discretizes the values of a dimension of the data, typically causing points to overlap with each other.\n",
    "\n",
    "With the data binned, let's now summarize the distribution of Rotten Tomatoes ratings. We will drop the IMDB ratings for now and instead use the `y` encoding channel to show an aggregate `count` of records, so that the vertical position of each point indicates the number of movies per Rotten Tomatoes rating bin.\n",
    "\n",
    "As the `count` aggregate counts the number of total records in each bin regardless of the field values, we do not need to include a field name in the `y` encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 9,
     "status": "ok",
     "timestamp": 1714994305340,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "MxDju_XZkLT4",
    "outputId": "78db49a1-43c3-46fc-c345-0f18ca94e7af"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_circle().encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating').bin(maxbins=20),\n",
    "    alt.Y('count()')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7n0JOq0VkUBV"
   },
   "source": [
    "To arrive at a standard histogram, let's change the mark type from `circle` to `bar`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 603,
     "status": "ok",
     "timestamp": 1714994305940,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "9iHuTM-PkROp",
    "outputId": "70425c48-15fe-4f18-c86b-e2d08590c796"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating').bin(maxbins=20),\n",
    "    alt.Y('count()')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H_kFdnFdkgL4"
   },
   "source": [
    "> **Observations:** We can now examine the distribution of ratings more clearly. we can see fewer movies on the negative end, and a bit more movies on the high end, but a generally uniform distribution overall. Rotten Tomatoes ratings are determined by taking “thumbs up” and “thumbs down” judgments from film critics and calculating the percentage of positive reviews. It appears this approach does a good job of utilizing the full range of rating values.\n",
    "\n",
    "\n",
    "Similarly, we can create a histogram for IMDB ratings by changing the field in the x encoding channel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 1216,
     "status": "ok",
     "timestamp": 1714994307153,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "RromFYukk1_i",
    "outputId": "a6cd3d0c-ae92-4b98-eca8-c26dca15751e"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('IMDB_Rating').bin(maxbins=20),\n",
    "    alt.Y('count()')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9CYU2_Xck64N"
   },
   "source": [
    "> **Observation:** In contrast to the more uniform distribution we saw before, IMDB ratings exhibit a bell-shaped (though [negatively skewed](https://en.wikipedia.org/wiki/Skewness)) distribution. IMDB ratings are formed by averaging scores (ranging from 1 to 10) provided by the site's users. We can see that this form of measurement leads to a different shape than the Rotten Tomatoes ratings. We can also see that the mode of the distribution is between 6.5 and 7: people generally enjoy watching movies, potentially explaining the positive bias!\n",
    "\n",
    "Now let's turn back to our scatter plot of Rotten Tomatoes and IMDB ratings. Here's what happens if we bin both axes of our original plot.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 1224,
     "status": "ok",
     "timestamp": 1714994308375,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "i7o3gzQDlVw6",
    "outputId": "7f559667-692c-4b9a-c60b-9d8b482377f5"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_circle().encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating').bin(maxbins=20),\n",
    "    alt.Y('IMDB_Rating').bin(maxbins=20)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g9JhX02Ilqvm"
   },
   "source": [
    "Detail is lost due to *overplotting*, with many points drawn directly on top of each other.\n",
    "\n",
    "To form a two-dimensional histogram we can add a `count` aggregate as before. As both the `x` and `y` encoding channels are already taken, we must use a different encoding channel to convey the counts. Here is the result of using circular area by adding a size encoding channel.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 957,
     "status": "ok",
     "timestamp": 1714994309330,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "mZn7OEbXl0dx",
    "outputId": "25810268-03f4-4375-b47e-5c7633d0e1a0"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_circle().encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating').bin(maxbins=20),\n",
    "    alt.Y('IMDB_Rating').bin(maxbins=20),\n",
    "    alt.Size('count()')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dUBExQxEl59a"
   },
   "source": [
    "Alternatively, we can encode counts using the `color` channel and change the mark type to `rect`. The result is a two-dimensional histogram in the form of a [heatmap](https://en.wikipedia.org/wiki/Heat_map)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 1342,
     "status": "ok",
     "timestamp": 1714994310670,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "z6s4agGBmGkE",
    "outputId": "6772b587-df63-4251-a74c-627e69f2e9ab"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_rect().encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating').bin(maxbins=20),\n",
    "    alt.Y('IMDB_Rating').bin(maxbins=20),\n",
    "    alt.Color('count()')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aPadkMg3mQzO"
   },
   "source": [
    "Compare the size and color-based 2D histograms above. Which encoding do you think should be preferred? Why? In which plot can you more precisely compare the magnitude of individual values? In which plot can you more accurately see the overall density of ratings?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6wuIaX4TmXnv"
   },
   "source": [
    "## Aggregation\n",
    "Counts are just one type of aggregate. We might also calculate summaries using measures such as the `average`, `median`, `min`, or `max`. The Altair documentation includes the [full set of available aggregation functions](https://altair-viz.github.io/user_guide/encodings/index.html#aggregation-functions).\n",
    "\n",
    "Let's look at some more examples!\n",
    "\n",
    "### Averages and Sorting\n",
    "> **Question:** Do different genres of films receive consistently different ratings from critics?\n",
    "\n",
    "As a first step towards answering this question, we might examine the *average* (a.k.a. the [arithmetic mean](https://en.wikipedia.org/wiki/Arithmetic_mean)) rating for each genre of movie.\n",
    "\n",
    "Let's visualize genre along the `y` axis and plot `average` Rotten Tomatoes ratings along the `x` axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 323
    },
    "executionInfo": {
     "elapsed": 2261,
     "status": "ok",
     "timestamp": 1714994312929,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "GmqxZ9ovm73S",
    "outputId": "a3335c55-0dc2-4677-baa0-1c15d06d6f0b"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('average(Rotten_Tomatoes_Rating)'),\n",
    "    alt.Y('Major_Genre')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rGplcf59nCOZ"
   },
   "source": [
    "> **Observation:** There does appear to be some interesting variation, but looking at the data as an alphabetical list is not very helpful for ranking critical reactions to the genres.\n",
    "\n",
    "For a tidier picture, let's sort the genres in descending order of average rating. To do so, we will use the `.sort()` method on the y encoding channel, stating that we wish to sort by the bar x positions in descending order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 323
    },
    "executionInfo": {
     "elapsed": 14,
     "status": "ok",
     "timestamp": 1714994312929,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "BtHfaZ00nTEm",
    "outputId": "950307ba-3e2e-48b0-d799-39899014eb11"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('average(Rotten_Tomatoes_Rating)'),\n",
    "    alt.Y('Major_Genre').sort('-x')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i3qtIWVnn_ki"
   },
   "source": [
    "> **Observation:** The sorted plot suggests that critics think highly of documentaries, musicals, westerns, and dramas, but look down upon romantic comedies and horror films… and who doesn’t love `null` movies!?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eQRyvAe4oI8z"
   },
   "source": [
    "### Medians and the Inter-Quartile Range\n",
    "\n",
    "While averages are a common way to summarize data, they can sometimes mislead. For example, very large or very small values ([*outliers*](https://en.wikipedia.org/wiki/Outlier)) might skew the average. To be safe, we can compare the genres according to the [*median*](https://en.wikipedia.org/wiki/Median) ratings as well.\n",
    "\n",
    "The median is a point that splits the data evenly, such that half of the values are less than the median and the other half are greater. The median is less sensitive to outliers and so is referred to as a [*robust statistic*](https://en.wikipedia.org/wiki/Robust_statistics). For example, arbitrarily increasing the largest rating value will not cause the median to change.\n",
    "\n",
    "Let’s update our plot to use a `median` aggregate and sort by those values:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 323
    },
    "executionInfo": {
     "elapsed": 8,
     "status": "ok",
     "timestamp": 1714994312929,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "sSMPwmqWovi7",
    "outputId": "047dbf32-826b-4e15-d41f-b1b368e1a955"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('median(Rotten_Tomatoes_Rating)'),\n",
    "    alt.Y('Major_Genre').sort('-x')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Csrf-on_ozx5"
   },
   "source": [
    "> **Observation:** We can see that some of the genres with similar averages have swapped places (films of unknown genre, or `null`, are now rated highest!), but the overall groups have stayed stable. Horror films continue to get little love from professional film critics.\n",
    "\n",
    "It's a good idea to stay skeptical when viewing aggregate statistics. So far we've only looked at *point estimates*. We have not examined how ratings vary within a genre.\n",
    "\n",
    "Let's visualize the variation among the ratings to add some nuance to our rankings. Here we will encode the [inter-quartile range (IQR)](https://en.wikipedia.org/wiki/Interquartile_range) for each genre. The IQR is the range in which the middle half of data values reside. A quartile contains 25% of the data values. The inter-quartile range consists of the two middle quartiles, and so contains the middle 50%.\n",
    "\n",
    "To visualize ranges, we can use the `x` and `x2` encoding channels to indicate the starting and ending points. We use the aggregate functions `q1` (the lower quartile boundary) and `q3` (the upper quartile boundary) to provide the inter-quartile range. (In case you are wondering, *q2* would be the median.)\n",
    "\n",
    "Rather than sort the bars by the `x` encoding, we'll use a more general form of sorting that allows us to sort by a specific aggregation `op` on a specific `field` in a specific `order`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 323
    },
    "executionInfo": {
     "elapsed": 1279,
     "status": "ok",
     "timestamp": 1714994314205,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "U2F0I4gXpZad",
    "outputId": "919927e4-d92f-43eb-809d-0dfb53b66204"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('q1(Rotten_Tomatoes_Rating)'),\n",
    "    alt.X2('q3(Rotten_Tomatoes_Rating)'),\n",
    "    alt.Y('Major_Genre').sort(\n",
    "        op='median',\n",
    "        field='Rotten_Tomatoes_Rating',\n",
    "        order='descending'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we add a `tick` mark to indicate the median for each genre.  Note that the second chart defined below automatically inherits the y-encoding (and data) from the `base` chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base = alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('q1(Rotten_Tomatoes_Rating)'),\n",
    "    alt.X2('q3(Rotten_Tomatoes_Rating)'),\n",
    "    alt.Y('Major_Genre').sort(\n",
    "        op='median',\n",
    "        field='Rotten_Tomatoes_Rating',\n",
    "        order='descending'\n",
    "    )\n",
    ")\n",
    "\n",
    "tick = base.mark_tick(color=\"red\").encode(\n",
    "    alt.X('median(Rotten_Tomatoes_Rating)')\n",
    ")\n",
    "\n",
    "base+tick"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BPD_SNzRqv1F"
   },
   "source": [
    "### Time Units\n",
    "> **Question:** Do box office returns vary by season?\n",
    "\n",
    "To get an initial answer, let’s plot the median U.S. gross revenue by month.\n",
    "\n",
    "To make this chart, use the `timeUnit` transform to map release dates to the `month` of the year. The result is similar to binning, but using meaningful time intervals. Other valid time units include `year`, `quarter`, `date` (numeric day in month), `day` (day of the week), and `hours`, as well as compound units such as `yearmonth` or `hoursminutes`. See the Vega-Altair documentation for a [complete list of time units](https://altair-viz.github.io/user_guide/transform/timeunit.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 24,
     "status": "ok",
     "timestamp": 1714994314205,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "EQNBdb9XrXpI",
    "outputId": "ae745dbb-b422-4ab2-b73f-6ba8f9041ce2"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_area().encode(\n",
    "    alt.X('month(Release_Date)'),\n",
    "    alt.Y('median(US_Gross)')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7L_gROJsrcQ9"
   },
   "source": [
    "> **Observation:** Looking at the resulting plot, median movie sales in the U.S. appear to spike around the summer blockbuster season and the end of year holiday period. Of course, people around the world (not just the U.S.) go out to the movies.\n",
    "\n",
    "---\n",
    "\n",
    "> **Question**: Does a similar pattern arise for worldwide gross revenue?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 18,
     "status": "ok",
     "timestamp": 1714994314205,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "SUexhD2mriMJ",
    "outputId": "a2f38b1a-2ba3-4f6a-9ed4-2962ffa44f31"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_area().encode(\n",
    "    alt.X('month(Release_Date)'),\n",
    "    alt.Y('median(Worldwide_Gross)')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bL29eZ_Crmcr"
   },
   "source": [
    "> **Observation:** Yes!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y77xknJyuTkY"
   },
   "source": [
    "## Data Transform Architecture\n",
    "By default, the entire input DataFrame is serialized to JSON and transfered from the Python Kernel to the browser, before being transformed and displayed by the Vega JavaScript library.\n",
    "\n",
    "We can see this by creating a Histogram chart and then selecting \"Open in Vega Editor\" from the dropdown menu. The Vega-Lite JSON specification contains every row and column from the movies DataFrame.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 12,
     "status": "ok",
     "timestamp": 1714994314205,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "r4oYr8RgvZpg",
    "outputId": "a1fe49f5-6f61-46df-837d-7c9e49337b4c"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('IMDB_Rating').bin(),\n",
    "    alt.Y('count()')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vXrAd5JYwa1c"
   },
   "source": [
    "For large datasets with many columns, this approach can result in very large chart specifications, which can be slow to serialize to JSON, slow to display, and lead to large notebooks. Because of these downsides, Vega-Altair enforces a default limit of 5000 rows.\n",
    "\n",
    "We can improve the situation using the `\"vegafusion\"` data transformer, which uses the optional VegaFusion dependency to evaluate most Vega-Altair transforms in the Python kernel before inlining the transformed data into the resulting chart specification.\n",
    "\n",
    "Let's repeat the process above after enabling this data transformer. Now when selecting \"Open in Vega Editor, we see the transformed dataset is included in the spec. This transformed dataset includes one row per histogram bar with fields for the bar height and bin edges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 6,
     "status": "ok",
     "timestamp": 1714994314205,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "WCqFLs_cxIxw",
    "outputId": "6a2b9fd1-5e3e-4eab-ac58-250c28ae8886"
   },
   "outputs": [],
   "source": [
    "alt.data_transformers.enable(\"vegafusion\")\n",
    "alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('IMDB_Rating').bin(),\n",
    "    alt.Y('count()')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WPOEhaHyk4mv"
   },
   "source": [
    "> **Note:** The VegaFusion data transformer is also the recommended way to overcome Vega-Altair's default limit of 5000 rows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zkIaXXyus0pS"
   },
   "source": [
    "## Extracting Transformed Data\n",
    "Transformed data can be extracted in Python using the `.transformed_data()` chart method. This is also powered by VegaFusion, and can be very helpful for analysis as it avoids the need to repeat calcultions in pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 332
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1714994314206,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "nf_crQ7vxtja",
    "outputId": "b53574af-1cce-4696-9fa6-909e89dee83f"
   },
   "outputs": [],
   "source": [
    "chart = alt.Chart(movies).mark_bar().encode(\n",
    "    alt.X('IMDB_Rating').bin(),\n",
    "    alt.Y('count()')\n",
    ")\n",
    "chart.transformed_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jbTJpDTjXG5J"
   },
   "source": [
    "## Advanced Transforms\n",
    "\n",
    "The examples above all use transformations (*bin*, *timeUnit*, *aggregate*, *sort*) that are defined relative to an encoding channel. However, at times you may want to apply a chain of multiple transformations prior to visualization, or use transformations that don't integrate into encoding definitions. For such cases, Vega-Altair supports data transformations defined separately from encodings. These transformations are applied to the data before any encodings are considered.\n",
    "\n",
    "We *could* also perform transformations using pandas directly, and then visualize the result. However, using the built-in transforms allows our visualizations to be published more easily in other contexts; for example, exporting the Vega-Lite JSON to use in a stand-alone web interface. It also makes it possible to adjust transformations based in Vega-Altair interactive features (which we will cover in the next section)\n",
    "\n",
    "Let's look at some of the built-in transforms supported by Vega-Altair, such as `calculate`, `filter`, `aggregate`, and `window`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VbPbecExnSFk"
   },
   "source": [
    "### Calculate\n",
    "\n",
    "> **Question:** Think back to our comparison of U.S. gross and worldwide gross. Doesn't worldwide revenue include the U.S.? (Indeed it does.) How might we get a better sense of trends outside the U.S.?\n",
    "\n",
    "With the `calculate` transform we can derive new fields. Here we want to subtract U.S. gross from worldwide gross. The `calculate` transform takes a [Vega expression string](https://vega.github.io/vega/docs/expressions/) to define a formula over a single row. Vega expressions use JavaScript syntax (but only support a small subset of JavaScript). The `datum.` prefix accesses a field value on the input row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1714994314206,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "KwMsO4HTnyDi",
    "outputId": "8356c272-61da-4ffb-fa31-8563639693d9"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_area().transform_calculate(\n",
    "    NonUS_Gross='datum.Worldwide_Gross - datum.US_Gross'\n",
    ").encode(\n",
    "    alt.X('month(Release_Date):T'),\n",
    "    alt.Y('median(NonUS_Gross):Q')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fTLz4MW5oW0p"
   },
   "source": [
    "> **Observation:** We can see that seasonal trends hold outside the U.S., but with a more pronounced decline in the non-peak months."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ykcVmRokoln9"
   },
   "source": [
    "### Filter\n",
    "The `filter` transform creates a new table with a subset of the original data, removing rows that fail to meet a provided [*predicate*](https://en.wikipedia.org/wiki/Predicate_%28mathematical_logic%29) test. Similar to the `calculate` transform, filter predicates are expressed using the [Vega expression language](https://vega.github.io/vega/docs/expressions/).\n",
    "\n",
    "Below we add a filter to limit our initial scatter plot of IMDB vs. Rotten Tomatoes ratings to only films in the major genre of “Romantic Comedy”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 379,
     "status": "ok",
     "timestamp": 1714994614443,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "5okg5cdun4yj",
    "outputId": "7a8a6411-1360-4ff6-fb79-47dfdf0d8750"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_circle().transform_filter(\n",
    "    'datum.Major_Genre == \"Romantic Comedy\"'\n",
    ").encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating'),\n",
    "    alt.Y('IMDB_Rating')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LfUstjibpS66"
   },
   "source": [
    "> **Question:** How does the plot change if we filter to view other genres? Edit the filter expression to find out!\n",
    "\n",
    "Now let's filter to look at films released before 1970.\n",
    "\n",
    "**Note:** The Vega expression language provides a convenient `year` function that we can use to extract the year from a datetime column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 368
    },
    "executionInfo": {
     "elapsed": 1525,
     "status": "ok",
     "timestamp": 1714994677638,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "ejCMEXFqo9Au",
    "outputId": "c22fe008-7b70-469e-96a6-395193fcee20"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_circle().transform_filter(\n",
    "    'year(datum.Release_Date) < 1970'\n",
    ").encode(\n",
    "    alt.X('Rotten_Tomatoes_Rating'),\n",
    "    alt.Y('IMDB_Rating')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lM7r17vKp3U1"
   },
   "source": [
    "> **Observation:** They seem to score unusually high! Are older films simply better, or is there a selection bias towards more highly-rated older films in this dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qVQo69Nep_fd"
   },
   "source": [
    "### Aggregate\n",
    "\n",
    "We have already seen `aggregate` transforms such as `count` and `average` in the context of encoding channels. We can also specify aggregates separately, as a pre-processing step for other transforms (as in the window transform examples below). The output of an `aggregate` transform is a new data table with records that contain both the `groupby` fields and the computed `aggregate` measures.\n",
    "\n",
    "Let's recreate our plot of average ratings by genre, but this time using a separate aggregate transform. The output table from the `aggregate` transform contains 13 rows, one for each genre.\n",
    "\n",
    "As before, we sort the bars in descending order by their x encoding using the `-x` shortcut."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 323
    },
    "executionInfo": {
     "elapsed": 157,
     "status": "ok",
     "timestamp": 1714994971907,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "btL57Z5ipgB5",
    "outputId": "a560e513-4d80-46ca-eac1-502068f4543a"
   },
   "outputs": [],
   "source": [
    "chart = alt.Chart(movies).mark_bar().transform_aggregate(\n",
    "    Average_Rating='average(Rotten_Tomatoes_Rating)',\n",
    "    groupby=['Major_Genre'],\n",
    ").encode(\n",
    "    alt.X('Average_Rating:Q'),\n",
    "    alt.Y('Major_Genre:N').sort('-x')\n",
    ")\n",
    "chart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RMlKdJrArWAT"
   },
   "source": [
    "Now we can use the `.transformed_data()` method to view the aggregated data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 457
    },
    "executionInfo": {
     "elapsed": 468,
     "status": "ok",
     "timestamp": 1714994981126,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "SBrwYMVVqjQW",
    "outputId": "5a0d73e3-ac97-4ab2-f30d-82a182269d22"
   },
   "outputs": [],
   "source": [
    "chart.transformed_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DX4OAGTxrgX2"
   },
   "source": [
    "Unlike pandas, Vega-Altair treats `null` as a grouping category, rather than automatically dropping `null` values.  The pandas behavior can be implemented in Vega-Altair if desired using a combination of the `filter` and `aggregate` transforms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 303
    },
    "executionInfo": {
     "elapsed": 315,
     "status": "ok",
     "timestamp": 1714995434688,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "QkKOjyE8qqZO",
    "outputId": "c0ecdd36-376c-413f-dcb4-03c309d44aca"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).mark_bar().transform_filter(\n",
    "    'isValid(datum.Major_Genre)'\n",
    ").transform_aggregate(\n",
    "    Average_Rating='average(Rotten_Tomatoes_Rating)',\n",
    "    groupby=['Major_Genre'],\n",
    ").encode(\n",
    "    alt.X('Average_Rating:Q'),\n",
    "    alt.Y('Major_Genre:N').sort('-x')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S4b4-gWdsgEr"
   },
   "source": [
    "### Window\n",
    "\n",
    "The `window` transform performs calculations over sorted groups of rows. Window transforms are quite powerful, supporting tasks such as ranking, lead/lag analysis, cumulative totals, and running sums or averages. Values calculated by a `window` transform are written back to the input data table as new fields. Window operations include the aggregate operations we've seen earlier, as well as specialized operations such as `rank`, `row_number`, `lead`, and `lag`. The Vega-Lite documentation lists [all valid window operations](https://vega.github.io/vega-lite/docs/window.html#ops).\n",
    "\n",
    "One use case for a window transform is to calculate top-k lists. Let's plot the top 20 directors in terms of total worldwide gross.\n",
    "\n",
    "We first use a filter transform to remove records for which we don't know the director. Otherwise, the director null would dominate the list!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 739
    },
    "executionInfo": {
     "elapsed": 191,
     "status": "ok",
     "timestamp": 1714995932942,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "dcbrlFVYrmtY",
    "outputId": "90dbffaa-ed21-48c4-9122-be65c682b63f"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).transform_filter(\n",
    "    'isValid(datum.Director)'\n",
    ").transformed_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "a5zVNsNOtPux"
   },
   "source": [
    "We then apply an aggregate to sum up the worldwide gross for all films, grouped by director."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "executionInfo": {
     "elapsed": 365,
     "status": "ok",
     "timestamp": 1714995928796,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "pltfGcyDtIDg",
    "outputId": "13571612-5fff-4fca-9cd7-304e71cf0a83"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).transform_filter(\n",
    "    'isValid(datum.Director)'\n",
    ").transform_aggregate(\n",
    "    Gross='sum(Worldwide_Gross)',\n",
    "    groupby=['Director']\n",
    ").transformed_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1wxTddgZtpN8"
   },
   "source": [
    "At this point we could plot a sorted bar chart, but we'd end up with hundreds and hundreds of directors. How can we limit the display to the top 20?\n",
    "\n",
    "The `window` transform allows us to determine the top directors by calculating their rank order. Within our `window` transform definition we can `sort` by gross and use the `rank` operation to calculate rank scores according to that sort order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "executionInfo": {
     "elapsed": 180,
     "status": "ok",
     "timestamp": 1714995924401,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "w7bedCsGtZOk",
    "outputId": "2b67a5e7-b75a-4abc-bbee-aa860c4b6a2f"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).transform_filter(\n",
    "    'isValid(datum.Director)'\n",
    ").transform_aggregate(\n",
    "    Gross='sum(Worldwide_Gross)',\n",
    "    groupby=['Director']\n",
    ").transform_window(\n",
    "    Rank='rank()',\n",
    "    sort=[alt.SortField('Gross', order='descending')]\n",
    ").transformed_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mASMxIwMuEX6"
   },
   "source": [
    "We can then add a subsequent `filter` transform to limit the data to only records with a rank value less than or equal to 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 645
    },
    "executionInfo": {
     "elapsed": 136,
     "status": "ok",
     "timestamp": 1714995919988,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "UnBo1LeMuHe3",
    "outputId": "977f431c-72d8-400b-913a-c394b7092557"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).transform_filter(\n",
    "    'isValid(datum.Director)'\n",
    ").transform_aggregate(\n",
    "    Gross='sum(Worldwide_Gross)',\n",
    "    groupby=['Director']\n",
    ").transform_window(\n",
    "    Rank='rank()',\n",
    "    sort=[alt.SortField('Gross', order='descending')]\n",
    ").transform_filter(\n",
    "    'datum.Rank <= 20'\n",
    ").transformed_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-npCynoIt_Mv"
   },
   "source": [
    "Now that we've build up the transformed data, we can build our chart by replacing the `.transformed_data()` method call above with our mark and encoding configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 443
    },
    "executionInfo": {
     "elapsed": 135,
     "status": "ok",
     "timestamp": 1714996013569,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "3EErxBuGt8QX",
    "outputId": "0a7efbf0-eacf-4069-b9cc-66e30d89660b"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).transform_filter(\n",
    "    'isValid(datum.Director)'\n",
    ").transform_aggregate(\n",
    "    Gross='sum(Worldwide_Gross)',\n",
    "    groupby=['Director']\n",
    ").transform_window(\n",
    "    Rank='rank()',\n",
    "    sort=[alt.SortField('Gross', order='descending')]\n",
    ").transform_filter(\n",
    "    'datum.Rank <= 20'\n",
    ").mark_bar().encode(\n",
    "    alt.X('Gross:Q'),\n",
    "    alt.Y('Director:N').sort('-x')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rD7QAcqqurgt"
   },
   "source": [
    "> **Observation:** We can see that Steven Spielberg has been quite successful in his career! However, showing sums might favor directors who have had longer careers, and so have made more movies and thus more money. What happens if we change the choice of aggregate operation? Who is the most successful director in terms of `average` or `median` gross per film? Modify the aggregate transform above!\n",
    "\n",
    "Earlier in this section we looked at histograms, which approximate the [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) of a set of values. A complementary approach is to look at the [cumulative distribution](https://en.wikipedia.org/wiki/Cumulative_distribution_function). For example, think of a histogram in which each bin includes not only its own count but also the counts from all previous bins — the result is a *running total*, with the last bin containing the total number of records. A cumulative chart directly shows us, for a given reference value, how many data values are less than or equal to that reference.\n",
    "\n",
    "As a concrete example, let's look at the cumulative distribution of films by running time (in minutes). Only a subset of records actually include running time information, so we first filter down to the subset of films for which we have running times.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 687
    },
    "executionInfo": {
     "elapsed": 814,
     "status": "ok",
     "timestamp": 1714996223183,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "QDEZy4o1umhZ",
    "outputId": "4e033ae8-46d9-40da-84d1-4f279ad641ca"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).transform_filter(\n",
    "    'isValid(datum.Running_Time_min)'\n",
    ").transformed_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8X5RoMhOvT2X"
   },
   "source": [
    "Next, we apply an `aggregate` to count the number of films per duration (implicitly using \"bins\" of 1 minute each)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "executionInfo": {
     "elapsed": 128,
     "status": "ok",
     "timestamp": 1714996268227,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "hiugluNyvg1e",
    "outputId": "aefd0d40-0a74-4945-e911-2f6c22a4fd57"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).transform_filter(\n",
    "    'isValid(datum.Running_Time_min)'\n",
    ").transform_aggregate(\n",
    "    groupby=['Running_Time_min'],\n",
    "    Count='count()',\n",
    ").transformed_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_HGXvRwgvhGs"
   },
   "source": [
    "We then use a window transform to compute a running total of counts across bins, sorted by increasing running time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 424
    },
    "executionInfo": {
     "elapsed": 149,
     "status": "ok",
     "timestamp": 1714996295112,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "etr-ROj5vpR2",
    "outputId": "4232448b-2f3d-438a-b8a9-beaf25fe8457"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).transform_filter(\n",
    "    'isValid(datum.Running_Time_min)'\n",
    ").transform_aggregate(\n",
    "    groupby=['Running_Time_min'],\n",
    "    Count='count()',\n",
    ").transform_window(\n",
    "    Cumulative_Sum='sum(Count)',\n",
    "    sort=[alt.SortField('Running_Time_min', order='ascending')]\n",
    ").transformed_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5Gzt803svtPa"
   },
   "source": [
    "Finally, let's encode the transformed data as a line mark, with \"step-before\" interpolation to make the cumulate distribution steps visible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "executionInfo": {
     "elapsed": 317,
     "status": "ok",
     "timestamp": 1714996511355,
     "user": {
      "displayName": "Jon Mease",
      "userId": "14424318106531239228"
     },
     "user_tz": 240
    },
    "id": "m_ejdKwivrN3",
    "outputId": "f296a698-7fbe-4bd6-dd63-cbbcd46fc5d9"
   },
   "outputs": [],
   "source": [
    "alt.Chart(movies).transform_filter(\n",
    "    'isValid(datum.Running_Time_min)'\n",
    ").transform_aggregate(\n",
    "    groupby=['Running_Time_min'],\n",
    "    Count='count()',\n",
    ").transform_window(\n",
    "    Cumulative_Sum='sum(Count)',\n",
    "    sort=[alt.SortField('Running_Time_min', order='ascending')]\n",
    ").mark_line(interpolate=\"step-before\").encode(\n",
    "    alt.X('Running_Time_min:Q').axis(title='Duration (min)'),\n",
    "    alt.Y('Cumulative_Sum:Q').axis(title='Cumulative Count of Films')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BYphRauAwbhr"
   },
   "source": [
    "> **Observation:** Let’s examine the cumulative distribution of film lengths. We can see that films under 110 minutes make up about half of all the films for which we have running times. We see a steady accumulation of films between 90 minutes and 2 hours, after which the distribution begins to taper off. Though rare, the dataset does contain multiple films more than 3 hours long!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SsjLaWMxwryS"
   },
   "source": [
    "## Summary\n",
    "\n",
    "We've only scratched the surface of what data transformations can do! For more details, including all the available transformations and their parameters, see the Vega-Altair [data transformation documentation](https://vega.github.io/vega-lite/docs/transform.html)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyPzM+kwOxkG1chHntF36fSV",
   "collapsed_sections": [
    "uIecrVj2epeR"
   ],
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}