tokenizing-text-files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/CC_BY.png\"><br />\n",
    "\n",
    "Created by [Nathan Kelber](http://nkelber.com) and Ted Lawless for [JSTOR Labs](https://labs.jstor.org/) under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/)<br />\n",
    "For questions/comments/improvements, email tdm@ithaka.org.<br />\n",
    "____\n",
    "# Tokenizing Text Files\n",
    "\n",
    "**Description:**\n",
    "You may have text files and metadata that you want to tokenize into ngrams with Python. \n",
    "\n",
    "This notebook takes as input:\n",
    "\n",
    "* Plain text files (.txt) in a folder \n",
    "* A metadata CSV file called 'metadata.csv'\n",
    "\n",
    "and outputs a single JSON-L file containing the unigrams, bigrams, trigrams, full-text, and metadata. \n",
    "\n",
    "**Use Case:** For Researchers (Mostly code without explanation, not ideal for learners)\n",
    "\n",
    "**Difficulty:** Intermediate\n",
    "\n",
    "**Completion time:** 10-15 minutes\n",
    "\n",
    "**Knowledge Required:** \n",
    "* Python Basics ([Start Python Basics I](./python-basics-1.ipynb))\n",
    "\n",
    "**Knowledge Recommended:**\n",
    "\n",
    "* [Working with Dataset Files](./working-with-dataset-files.ipynb)\n",
    "\n",
    "**Data Format:** .txt, .csv, .jsonl\n",
    "\n",
    "**Libraries Used:**\n",
    "* json\n",
    "* collections\n",
    "* pandas\n",
    "\n",
    "**Research Pipeline:**\n",
    "\n",
    "1. Scan documents\n",
    "2. OCR files\n",
    "3. Clean up texts\n",
    "4. **Tokenize text files** (this notebook)\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import gzip\n",
    "import json\n",
    "import os\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and inspect sample files\n",
    "\n",
    "For purposes of this tutorial, we will download a set of sample files from Project Gutenberg using a helper function from the tdm_client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from tdm_client import download_gutenberg_sample\n",
    "\n",
    "text_file_directory = download_gutenberg_sample()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now have sample text files and a CSV of metadata in your data directory. \n",
    "\n",
    "You can list the contents of this directory with this command. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -lt ~/data/gutenberg-sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the first 20 lines of a sample file with this command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -n 20 ~/data/gutenberg-sample/205-0.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a tokenizing function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def constellate_ngrams(text, n=1):\n",
    "    # Define a Counter object to hold our ngrams.\n",
    "    c = Counter()\n",
    "    # Replace line breaks in the text.\n",
    "    t = text.replace(\"\\r\", \" \").replace(\"\\n\", \"\")\n",
    "    # Convert the text to a list of words.\n",
    "    words = t.split()\n",
    "    # Slice the words into ngrams.\n",
    "    for grams in zip(*[words[i:] for i in range(n)]):\n",
    "        g = \" \".join(grams)\n",
    "        c[g] += 1\n",
    "    return c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize a text\n",
    "\n",
    "Let's tokenize one of the sample files using our function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Read in one of the texts. See note about file paths.\n",
    "with open(f\"{text_file_directory}{os.sep}205-0.txt\") as input_file:\n",
    "    text = input_file.read()\n",
    "    \n",
    "unigrams = constellate_ngrams(text)\n",
    "unigrams.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can create bigrams or trigrams (or n grams) by changing the `n` keyword argument passed to the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bigrams = constellate_ngrams(text, n=2)\n",
    "bigrams.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Constellate JSONL file\n",
    "\n",
    "For your analysis, you may want to create files that conform to the same data specification as the files provided by Constellate. The following steps show you how to load metadata and the raw text, create ngrams and output a JSONL (JSON lines) file that matches, in format, what you download from the Constellate web application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(text_file_directory + os.sep + \"metadata.csv\")\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loop through the dataframe and print out some of the metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for item in df.itertuples():\n",
    "    print(item.title, item.author, item.url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now convert the metadata to the Constellate schema as defined [here](https://docs.constellate.org/what-format-are-jstor-portico-datasets/) by mapping the column names from the source csv to the corresponding Constellate schema attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a list to hold our documents.\n",
    "documents = []\n",
    "\n",
    "for item in df.itertuples():\n",
    "    document = {\n",
    "        \"id\": item.url,\n",
    "        \"title\": item.title,\n",
    "        \"creator\": [item.author],\n",
    "        \"docType\": \"book\",\n",
    "        \"publicationYear\": item.published,\n",
    "        \"url\": item.url,\n",
    "        \"language\": [item.language]\n",
    "    }\n",
    "    documents.append(document)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our metadata stored in a list, let's revise our function to capture the full text of the documents and generate ngrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Create a list to hold our documents.\n",
    "documents = []\n",
    "\n",
    "for item in df.itertuples():\n",
    "    document = {\n",
    "        \"id\": item.url,\n",
    "        \"title\": item.title,\n",
    "        # A document can have authors/creators, so map as a list.\n",
    "        \"creator\": [item.author],\n",
    "        \"docType\": \"book\",\n",
    "        \"publicationYear\": item.published,\n",
    "        \"url\": item.url,\n",
    "        # A document can have multiple languages, so map as a list.\n",
    "        \"language\": [item.language]\n",
    "    }\n",
    "    # Read in the full text\n",
    "    with open(text_file_directory + \"/\" + item.file) as text_file:\n",
    "        text = text_file.read()\n",
    "    \n",
    "    # Split the text into pages. See note below.\n",
    "    document[\"fullText\"] = text.split(\"\\n\\n\\n\")\n",
    "    # Generate ngrams\n",
    "    document[\"unigramCount\"] = constellate_ngrams(text, n=1)\n",
    "    document[\"bigramCount\"] = constellate_ngrams(text, n=2)\n",
    "    document[\"trigramCount\"] = constellate_ngrams(text, n=3)\n",
    "    # Add our document to the list of documents\n",
    "    documents.append(document)\n",
    "    print(f\"{item.title} procesed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the first document and print some of the metadata and content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "first_doc = documents[0]\n",
    "print(first_doc[\"title\"], first_doc[\"publicationYear\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the twenty five most common trigrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for term, count in first_doc[\"trigramCount\"].most_common(25):\n",
    "    print(term, count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the first 500 characters of \"page\" 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(first_doc[\"fullText\"][20][:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate a Constellate gzip file\n",
    "\n",
    "You may now want to create a gzip file so that it matches what you have downloaded from Constellate. You could also then use the `gzip_reader` that's part of the `tdm_client` to read it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_file = text_file_directory + os.sep + \"sample_gutenberg_dataset.json.gzip\"\n",
    "\n",
    "with gzip.open(output_file, \"wb\") as handle:\n",
    "    for doc in documents:\n",
    "        # Convert the document to a string and add the line separator\n",
    "        raw = json.dumps(doc) + \"\\n\"\n",
    "        handle.write(raw.encode())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now use the dataset reader to read your file back in and verify it is what we expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tdm_client import dataset_reader\n",
    "\n",
    "for doc in dataset_reader(output_file):\n",
    "    print(doc[\"title\"], doc[\"creator\"], doc[\"publicationYear\"])\n",
    "    # See note about assert\n",
    "    assert(doc[\"unigramCount\"] is not None)\n",
    "    assert(doc[\"fullText\"] is not None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes\n",
    "\n",
    "* File paths - in Unix based systems (including Linux and MacOS), a file is separated with a `/`. On Windows the separator is a `\\`. Python includes the helpful `os.sep` to find the correct file separator for your system. This allows the notebook to run just fine on multiple operating systems.\n",
    "\n",
    "* Pagination - the plain text files from Project Gutenberg aren't paginated. Here we are using a simple rule of thumb: if there are three consecutive line breaks, treat this as a page break. This is unlikely to work well across all Project Gutenberg content but should be sufficient for demonstration purposes. You may be curious about more sophisticated attempts to format Project Gutenberg books, such a [chapterize](https://github.com/JonathanReeve/chapterize) by [Jonathan Reeve](https://github.com/JonathanReeve).\n",
    "\n",
    "* `assert` - Python's `assert` statement can be a quick and useful way to validate your logic. By using assert, you are guaranteeing that the program won't run if the statement is false. So in this usage, we are guaranteeing that each of our documents have a `fullText` and a `unigramCount` attribute. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}