Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "628a6b78-7bd5-4e2d-b268-784cb0f7d915",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Using `thisnotthat` for examining clustered data and editing labels\n",
    "\n",
    "Consider a dataset composed of a sequence of _records_, each consisting in a loose set of named _fields_. A great process for auditing and understanding such a dataset is to figure out the correlations between records: which are similar and why, as opposed to which are dissimilar, and why. While somebody with a monk's patience and tireless eyeballs can work such correlations out of a spreadsheet view of the data, there exists a more interesting approach:\n",
    "\n",
    "1. Embed the records in a vector space.\n",
    "1. Reduce the dimension of the record vectors to 2 and visualize them.\n",
    "    - Bonus: use hover-like tooling to keep a description of the records handy during visualization.\n",
    "\n",
    "Some of the natural record correlations will take the form of _clusters_ in the plot display, which can be visually appreciated. A small upgrade to this process involves exploring the similarities quantitatively, using a clustering algorithm. The exploration process then goes:\n",
    "\n",
    "1. Embed the records in a vector space.\n",
    "1. _Maybe_ reduce the dimension a little, just enough to compress the data a bit?\n",
    "1. Run the clustering algorithm to discover the groups of similar records, according to a chosen _distance function_. These group identifiers become the groups' respective _labels_.\n",
    "1. Reduce the dimension of the record vectors to 2 and visualize them, colored by label.\n",
    "    - Bonus: use hover-like tooling to keep a description of the records handy during visualization.\n",
    "\n",
    "In both cases, the plotting of the dataset grants insights into the correlations and the reasons behind them. These insights can drive the scientist towards matching or breaking assumptions or prior knowledge regarding phenomena the data is supposed to describe. Thus, a given cluster can correspond to one of these phenomena. Two or more clusters may describe the same phenomenon, even if the algorithm has marked them as distinct. Alternatively, a correlation between some records might be spurious with respect to a phenomenon of interest, so one may want to split a cluster into two or more. All these cases involve _editing labels_, a process by which one exchanges knowledge between information present in data and prior knowledge one possesses.\n",
    "\n",
    "`thisnotthat` provides a data viewer and label editor as an ipywidget."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a617476a-cfd1-43bf-9b07-246c74d5f3c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import thisnotthat as tnt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8aa69d1-678e-416a-8824-38a6b9a7f1dc",
   "metadata": {},
   "source": [
    "### Brief digression: getting \"clustered\" data to plot\n",
    "\n",
    "Let's simulate a few Gaussian clusters centered respectively at regular intervals on the circumference of a circle."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d334f2e9-1fbc-4f3d-8f79-c56a24338a90",
   "metadata": {},
   "source": [
    "Number of points to split among clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "66ecffa9-58dd-452c-9e38-72837e1f1e10",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_POINTS = 1000\n",
    "assert NUM_POINTS > 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c29f2016-306f-4d9b-a704-75ed3c33ff0b",
   "metadata": {},
   "source": [
    "Cluster \"distribution\": this is a list with one number for each cluster to simulate. The number corresponds to the \"weight\" of the cluster: for instance, a cluster of weight 3 will be associated to approximately three times as many points as a cluster of weight 1. Under the hood, this list is $l_1$-normalized to yield the probabilities that make up a multinomial distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d0f8151c-c54f-4bcd-853b-6d47fcc98bef",
   "metadata": {},
   "outputs": [],
   "source": [
    "DISTRIBUTION_CLUSTERS = [10, 10, 20, 30]\n",
    "assert len(DISTRIBUTION_CLUSTERS) > 0\n",
    "assert all(p > 0.0 for p in DISTRIBUTION_CLUSTERS)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "694c9994-b4bd-487e-953a-b20cfb7756ea",
   "metadata": {},
   "source": [
    "The clusters will be respectively centered around a circle centered at the origin. This determines the radius of this circle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "978fbd66-f2dc-45f9-995d-ee120cc84738",
   "metadata": {},
   "outputs": [],
   "source": [
    "RADIUS = 5\n",
    "assert RADIUS > 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c092ec47-fd58-46e4-a8de-6874f32e58e6",
   "metadata": {},
   "source": [
    "This _scale_ parameters determines how wide the Gaussian clusters will tend to be. The value is mostly relevant in comparison to the circle's radius: a high radius/scale ratio will more likely yield well-separated clusters; a low radius/scale ratio will yield confusion and chaos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b27c424c-56d5-40d7-8775-45d98489221b",
   "metadata": {},
   "outputs": [],
   "source": [
    "SCALE_CLUSTERS = 3\n",
    "assert SCALE_CLUSTERS > 0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "02024e2f-37b1-4101-ade5-5f56e364f0f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "import numpy.linalg as npl\n",
    "from scipy.spatial.transform import Rotation\n",
    "import scipy.stats as st"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcfa326a-b87d-417f-979e-8acb5aa96ecf",
   "metadata": {},
   "source": [
    "Put all the ingredients in a put, bring to a boil, simmer for 30 ms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d5a723db-17d7-4454-9576-7cf4e7832ee8",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_clusters = len(DISTRIBUTION_CLUSTERS)\n",
    "sizes = st.multinomial(\n",
    "    n=NUM_POINTS,\n",
    "    p=np.array(DISTRIBUTION_CLUSTERS) / npl.norm(DISTRIBUTION_CLUSTERS, ord=1)\n",
    ").rvs().squeeze()\n",
    "\n",
    "seeds = 2 * np.pi * np.arange(num_clusters) / num_clusters + st.uniform(scale=np.pi).rvs()\n",
    "means = RADIUS * np.hstack([np.cos(seeds).reshape((-1, 1)), np.sin(seeds).reshape((-1, 1))])\n",
    "stds = st.expon(scale=SCALE_CLUSTERS * np.outer(np.ones((num_clusters,)), np.ones((2,)))).rvs() / 3\n",
    "angles = st.uniform(scale=np.pi * np.ones((num_clusters,))).rvs()\n",
    "\n",
    "cluster = []\n",
    "ez = np.array([0, 0, 1])\n",
    "for i in range(num_clusters):\n",
    "    S = np.diag(stds[i, :] ** 2)\n",
    "    R = Rotation.from_rotvec(angles[i] * ez).as_matrix()[:2, :2]\n",
    "    cluster.append(\n",
    "        st.multivariate_normal(mean=means[i, :], cov=R @ S @ R.transpose()).rvs((sizes[i],))\n",
    "    )\n",
    "data = np.vstack(cluster)\n",
    "assert data.shape == (NUM_POINTS, 2)\n",
    "hover = list(range(NUM_POINTS))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d52434f-2a83-4ff6-a512-9237e792b59f",
   "metadata": {},
   "source": [
    "We determine a random word as label for each cluster, using the word file included in this repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a2976838-53d0-4bdc-8199-a037802e1c3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"words.txt\", \"r\", encoding=\"utf-8\") as file:\n",
    "    words = [line.strip() for line in file]\n",
    "\n",
    "labels = sum(\n",
    "    [[random.choice(words)] * s for s in sizes],\n",
    "    []\n",
    ")\n",
    "assert len(labels) == NUM_POINTS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98a7177e-2318-40b5-b529-c226fbf6d95d",
   "metadata": {},
   "source": [
    "## Here's the labeler!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0139c86f-38d8-4f93-acb1-151a5c160825",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f32db76e2d3452bad7a82124c7bf247",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Labeler(children=(Figure(axes=[Axis(scale=LinearScale()), Axis(orientation='vertical', scale=LinearScale())], …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lb = tnt.Labeler(data, labels, hover=hover)\n",
    "lb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed8c296-0e42-4bfc-b2bf-a470d65347f8",
   "metadata": {},
   "source": [
    "It starts with the **Pick** tool selected. This is useful to roam around the plot and get information about each point (as a tooltip shown when hovering over the point). To look at a certain part of the plot more closely, use the **Pan/Zoom** tool: point somewhere and roll the mouse wheel to zoom in and out; drag the mouse to move the center of the plot. Clicking the **Reset** button will restore the original perspective."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2df2716c-f8a6-44a5-beab-129e5aef37e0",
   "metadata": {},
   "source": [
    "On the upper right corner of the display, you have an editable legend of the plot: each legend element corresponds to the label associated to the points of the corresponding color. You can click on the colors to change them if they don't contrast to your taste. You can also edit the label names so they correspond to your understanding of the correlations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd2cbac6-50a3-41d6-ac46-b3ea8f5996b2",
   "metadata": {},
   "source": [
    "The fun things occur when you _select_ some of the points. There are three ways to do so:\n",
    "    \n",
    "1. With the **Pick** tool, click on points that interest you (click again to deselect).\n",
    "1. With the **Lasso** tool, draw a line around the points that interest you.\n",
    "1. Click on the ☑ button by the name of a cluster to select all the points of that cluster (click again to deselect).\n",
    "\n",
    "When you have points selected, all other points are faded out. You can perform multiple selection actions in most orders: only Lassoing points after picking will reset the selection (likely a bug in the BQPlot library). You can also clear the selection by clicking the **Reset** button (click a second time to restore the perspective).\n",
    "\n",
    "With points selected, two new actions become possible. The first consists in creating a new cluster composed of the points that are selected: do this by clicking the **Split** button. You will see the legend modified to describe your new cluster, newly drawn in its own color.\n",
    "\n",
    "The second action consists in merging the selected points to an existing cluster. You do this by choosing the label of this cluster in the **Merge to** drop-down menu. Note that if _all_ the points of a cluster are selected, it suddenly no longer count any point to itself; we then delete it from the display."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "927121be-cc6f-42a2-9af9-9d893cb52152",
   "metadata": {},
   "source": [
    "The goal of all this is effectively to label the points to your heart's content. At any point during the process, you can look up the labels associated to your various data points (rows of the `data` matrix), as a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c1c8b474-6fbc-406a-b4b2-354b98c8d6cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['palstave',\n",
       " 'palstave',\n",
       " 'palstave',\n",
       " 'palstave',\n",
       " 'palstave',\n",
       " 'palstave',\n",
       " 'palstave',\n",
       " 'palstave',\n",
       " 'palstave',\n",
       " 'palstave']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lb.labels_named[:10]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "thisnotthat (Python)",
   "language": "python",
   "name": "conda-env-thisnotthat-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}