06_OSEMN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OSEMN\n",
    "\n",
    "According to a popular model, the elements of data science are:\n",
    "\n",
    "* Obtaining data\n",
    "* Scrubbing data\n",
    "* Exploring data\n",
    "* Modeling data\n",
    "* iNterpreting data\n",
    "\n",
    "and hence the acronym OSEMN, pronounced as “Awesome”.\n",
    "\n",
    "We will start with the **O**, and let's have a quick look at what it all boils down to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# O: obtaining data\n",
    "#!wget https://www.dropbox.com/s/ebe1cnyd2gm836a/populations.txt -P data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# S: scrubbing data\n",
    "import numpy as np\n",
    "data = np.loadtxt('data/populations.txt')\n",
    "year, hares, lynxes, carrots = data.T # trick: columns to variables\n",
    "# this is not real scrubbing, rather a simple operation to better handle the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# E: exploring data\n",
    "!cat data/populations.txt\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.axes([0.2, 0.1, 0.5, 0.8]) \n",
    "plt.plot(year, hares, year, lynxes, year, carrots) \n",
    "plt.legend(('Hare', 'Lynx', 'Carrot'), loc=(1.05, 0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# M: modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By plotting the data a clear (and reasonable) correlations between pray and predator becomes evident.\n",
    "\n",
    " - How can it be quantified?\n",
    " - Is that statistical significant?\n",
    " - What about the correlation between carrots and hares?\n",
    " \n",
    " We will deal with data modeling in few weeks, when we will learn to fit the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# N: interpreting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding correlations in data is one of the main goals of data science, though that is not the end of the story: as this interesting [site](http://tylervigen.com/spurious-correlations) demonstrates, **correlations does not imply causation**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Optional exercise**: write an algorithm that determines and quantifies a correlation between two time series. Use as an example the hare-lynx-carrot dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtaining and processing data\n",
    "\n",
    "Accessing data may not be as easy as it seems. The previous example, where the data is stored on your own machine, represents the simplest (but alas, less common) case. Data is usually stored on remote machines, which can be either *publicly accessible* (everyone can access the data, even without credentials) rather than *private*. In the case of the former, things may be straightforward, whereas in the latter case you need to worry about a few things.\n",
    "\n",
    "In both cases, depending on the size of the dataset, the managment of the dataset can become extremely complicated. We won't deal here with large datasets (which require a whole course *per se*), but still we should pay attention to few basic things: for instance, *it is not wise to keep (and even worse commit) data into a git repository*!\n",
    "\n",
    "The suggestion is then to create a directory somewhere and copy the example datasets there. From a terminal:\n",
    "\n",
    "```bash\n",
    "\n",
    "# create a data directory in your home directory\n",
    "mkdir data/\n",
    "\n",
    "# check the content (it's empty now of course)\n",
    "ls -ltr data/\n",
    "\n",
    "# in the case you need to move there:\n",
    "cd data/\n",
    "\n",
    "# if you need to copy a file\n",
    "cp data/data_original.txt data/data_copy.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Public data\n",
    "\n",
    "An increasingly number of institutions, reaserch centers, experiments, ... are making their data public.\n",
    "\n",
    "A nice set of interesting datasets can be found on this [server](https://archive.ics.uci.edu/ml/datasets.php) that collects training/test data for machine learning developments. Several of those belong to physical sciences, and may be worth a look. Since they are public, you can freely download any of those. However, they usually come with a license (e.g. you won't be allowed to make profit from them), and deserve a reference in your paper, if you publish some result.\n",
    "\n",
    "Sometimes, they are used as the input to machine learning challenges, where different groups compete to achieve the best result.\n",
    "\n",
    "In the following, we consider a dataset from the MAGIC experiment. We will get it with the `wget` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download the dataset and its description to the data/ directory\n",
    "#!wget https://archive.ics.uci.edu/ml/machine-learning-databases/magic/magic04.data -P data/\n",
    "#!wget https://archive.ics.uci.edu/ml/machine-learning-databases/magic/magic04.names -P data/    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read the description file. This can (and should) be done from a terminal\n",
    "!cat data/magic04.names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing the data without downloading the file(s)\n",
    "\n",
    "It is possible to download and load remote files via their url from within python (and thus on a jupyter session). This is a rather powerful tool as it allows http communications, I/O streaming and so on.\n",
    "\n",
    "Care should be put as the dataset is stored in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/magic/magic04.names'\n",
    "with urllib.request.urlopen(url) as data_file:\n",
    "    #print(data_file.read(300))\n",
    "    for line in data_file:\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Copy data from a remote machine\n",
    "\n",
    "Often datasets are not available on websites, but rather they are stored on some remote machine. Several tools have been developed allow you to get remote data, even from within python (e.g. [paramiko](https://www.paramiko.org/)), but it's often enough to get a local copy.\n",
    "\n",
    "The unix `scp` command is of great help in this case. Its syntax resembles the one of the `cp` command, e.g. from a terminal:\n",
    "\n",
    "```bash\n",
    "scp username@machinename.unipd.it:/path/to/the/file/filename.* path/to/the/target/\n",
    "```\n",
    "\n",
    "The `scp` works provided that you have the necessary permissions and authorization to log in to the remote machine, and you know the location of the files within that machine. This seems trivial, but it's one of the most difficult aspects you will have to deal when trying to obtain the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, it could be worse. Especially in large reaserch centers or companies, the remote machine `R` is often not accessible from \"outside\", meaning that it is hidden behind a firewall. In that case you may need to use to access to a *gate* machine `G` first (with very limited disk space and functionality), and only then you can connect to the machine. This also means that you cannot run the `scp` command twice to get the data.\n",
    "\n",
    "The simplest solution is to create an `ssh` tunnel:\n",
    "\n",
    "``` bash\n",
    "ssh -L port:<address of R known to G>:22 <user at G>@<address of G> \n",
    "\n",
    "scp -P port <user at R>@localhost:/path/to/the/file file-name-to-be-copied\n",
    "```\n",
    "\n",
    "where `port` is the port number which is a number between 1025 and 65535, `R` is the machine where the files are located, `G` is the gate, and `user` is the username.\n",
    "\n",
    "The first command redirects your local port `port` to port 22 of `R`. It could happen that the chosen port number is already in use, in that case just pick another unused port.\n",
    "\n",
    "In summary, just getting the data could be complicated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Formats\n",
    "\n",
    "Datasets can be stored in several different ways. Sometimes, they have formats which are application-dependent, even though more and more standards are being established.\n",
    "\n",
    "#### Proprietary formats\n",
    "\n",
    "In some of those cases, the format of the data is proprietary, meaning that the software house does not provide the documentation needed to read and write the data in that format. We won't cover those cases in this course, and there are a number of reasons to avoid them.\n",
    "\n",
    "#### Open formats\n",
    "\n",
    "In case an open, widely-accepted data formats, Python has \"readers\" for most of these, another reason for being the optimal programming language for data analysis. In the follwing, we will explore the most common ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text files \n",
    "\n",
    "Plain text files are very common, and are used for \"readibility\", at the price of a poor storing efficiency. [UTF-8](https://en.wikipedia.org/wiki/UTF-8) is the most common encoding.\n",
    "\n",
    "Reading (and writing) text files in Python is straightforward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = \"data/magic04.data\"\n",
    "n = 0\n",
    "\n",
    "# mode can be specified for writing, reading or both\n",
    "with open(file_name, mode = 'r') as f:\n",
    "    # print-out the whole file\n",
    "    # print(f.read()) \n",
    "    for line in f:\n",
    "        print(n, \":\")\n",
    "        # print line by line\n",
    "        print(line)\n",
    "        # each line is a string, you need to split it yourself\n",
    "        #for c in line.split(','): print(\"  \", c) # check the functionalities of the split() method\n",
    "        n += 1\n",
    "        if n > 10: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, it's possible to write files to a txt file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_file_name = \"data/magic04.out\"\n",
    "\n",
    "with open(out_file_name, 'a') as outfile:\n",
    "    outfile.write(line + '\\n') # write last line to a file\n",
    "\n",
    "!cat data/magic04.out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSV files\n",
    "\n",
    "Text files are already framed into a defined structured, in a \"table-like\" manner. If the values/features/attributes are separated with a comma, these files are colled \"comma separated values\" (CSV), even though the separator may well not be the \",\" symbol. It could also happen that the separator is a whitespace, or a tab character.\n",
    "\n",
    "Python has a package to deal with those, named `csv`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "\n",
    "n = 0\n",
    "\n",
    "with open('data/magic04.data') as data_file:\n",
    "    for line in csv.reader(data_file, delimiter=','): # the delimiter is often guessed by the reader\n",
    "        # again note that elements of each line are treated as strings\n",
    "        # if you need to convert them into numbers, you need to to that yourself\n",
    "        fLength, fWidth, fSize, fConc, fConc1, fAsym, fM3Long, fM3Trans, fAlpha, fDist = map(float, line[:-1])\n",
    "        category = line[-1]\n",
    "        print(fLength, fWidth, fSize, fConc, fConc1, fAsym, fM3Long, fM3Trans, fAlpha, fDist, category)\n",
    "        n += 1\n",
    "        if n > 10: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, csv files have comments (e.g. starting with '#'), which cannot be interpreted by the reader. Tricks like:\n",
    "\n",
    "```python\n",
    "csv.reader(row for row in f if not row.startswith('#'))\n",
    "```\n",
    "\n",
    "may be useful to skip those lines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON files\n",
    "\n",
    "JSON stands for JavaScript Object Notation, and is a format widely used for web-based resource sharing. It is very similar in structure to a Python nested dictionary. It is humand readable (then human-editable), even if readability may be difficult for complex data structures. Its use is convenient in Python, as its items can be accessed by key. Here is an example http://json.org/example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file example.json\n",
    "{\n",
    "    \"glossary\": {\n",
    "        \"title\": \"example glossary\",\n",
    "            \"GlossDiv\": {\n",
    "            \"title\": \"S\",\n",
    "                    \"GlossList\": {\n",
    "                \"GlossEntry\": {\n",
    "                    \"ID\": \"SGML\",\n",
    "                                    \"SortAs\": \"SGML\",\n",
    "                                    \"GlossTerm\": \"Standard Generalized Markup Language\",\n",
    "                                    \"Acronym\": \"SGML\",\n",
    "                                    \"Abbrev\": \"ISO 8879:1986\",\n",
    "                                    \"GlossDef\": {\n",
    "                        \"para\": \"A meta-markup language, used to create markup languages such as DocBook.\",\n",
    "                                            \"GlossSeeAlso\": [\"GML\", \"XML\"]\n",
    "                    },\n",
    "                                    \"GlossSee\": \"markup\"\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat example.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json # import the JSON module\n",
    "data = json.load(open('example.json'))\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and can be parsed using standard key lookups\n",
    "data['glossary']['GlossDiv']['GlossList']['GlossEntry']['GlossTerm']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pickle\n",
    "\n",
    "Sometimes, you may want to save and read more complex Python data structures than int, floats and strings, as for example dictionaries, tuples, lists, or even more complex objects.\n",
    "\n",
    "In these cases, the `pkl` format is very handful. Pickle is used for serializing and de-serializing Python object structures, also called flattening. Serialization consists of converting an object in memory to a byte stream that can be stored on disk. Later on, this character stream can then be retrieved and de-serialized back to a Python object.\n",
    "\n",
    "There are fundamental differences between the pickle protocols and JSON (JavaScript Object Notation):\n",
    "\n",
    "- JSON is a text serialization format (it outputs unicode text, although most of the time it is then encoded to utf-8), while pickle is a binary serialization format;\n",
    "\n",
    "- JSON is human-readable, while pickle is not;\n",
    "\n",
    "- JSON is interoperable and widely used outside of the Python ecosystem, while pickle is Python-specific;\n",
    "\n",
    "- JSON, by default, can only represent a subset of the Python built-in types, and no custom classes; pickle can represent an extremely large number of Python types (many of them automatically, by clever usage of Python’s introspection facilities; complex cases can be tackled by implementing specific object APIs);\n",
    "\n",
    "- Unlike pickle, deserializing untrusted JSON does not in itself create an arbitrary code execution vulnerability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "# define the data structure\n",
    "ages_dict = {'Fido' : 3, 'Laika' : 16, 'Skipper' : 10, 'Balou' : 9}\n",
    "\n",
    "filename = \"data/dog_ages.pkl\"\n",
    "\n",
    "outfile = open(filename, 'wb') # note: no filename extension. 'wb' stands for w: write and b: binary\n",
    "pickle.dump(ages_dict, outfile) # \"dump\" the data to pkl\n",
    "outfile.close() # remember to close the file\n",
    "\n",
    "!ls data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read pickle\n",
    "infile = open(filename, 'rb')\n",
    "new_dict = pickle.load(infile)\n",
    "infile.close()\n",
    "\n",
    "print(new_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HDF5\n",
    "\n",
    "HDF5 is a hierarchical format often used to store complex scientific data. For instance, Matlab now saves its data to HDF5. It is particularly useful to store complex hierarchical data sets with associated metadata, for example, the results of a computer simulation experiment.\n",
    "\n",
    "The main concepts associated with HDF5 are:\n",
    "\n",
    "* file: container for hierachical data - serves as ‘root’ for tree\n",
    "* group: a node for a tree\n",
    "* dataset: array for numeric data - can be huge\n",
    "* attribute: small pieces of metadata that provide additional context\n",
    "\n",
    "Now let's create a dummy dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# creating a HDF5 file\n",
    "\n",
    "filename = \"data/example.hdf5\"\n",
    "\n",
    "if os.path.exists(filename): # check if file exists, and if it does, delete it\n",
    "    os.remove(filename)\n",
    "\n",
    "with h5py.File(filename, 'w') as f:\n",
    "    project = f.create_group('project')\n",
    "    expt1 = project.create_group('expt1')\n",
    "    expt2 = project.create_group('expt2')\n",
    "    expt1.create_dataset('counts', (100,), dtype='i')\n",
    "    expt2.create_dataset('values', (1000,), dtype='f')\n",
    "\n",
    "    expt1['counts'][:] = np.arange(100)\n",
    "    expt2['values'][:] = np.arange(1000)/10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read HDF5 file\n",
    "\n",
    "with h5py.File(filename) as f:\n",
    "    project = f['project']\n",
    "    print(project['expt1']['counts'][:10])\n",
    "    print(project['expt2']['values'][:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy and Pandas\n",
    "\n",
    "Numpy provides the useful methdod `np.loadtxt()` to read a txt files, as we have seen in the introduction.\n",
    "\n",
    "Pandas itself provides integrated and convenient tools to read and write CSV files with the `read_csv()` method of DataFrames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy the unpacked version\n",
    "#!wget https://www.dropbox.com/s/ga9wi6b40cakgae/data_000637.txt -P data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "file_name= \"data/data_000637.txt\"\n",
    "data = pd.read_csv(file_name, nrows=10, skiprows=range(1,1))\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that Pandas is smart enough to determine that the first row consist of the headers of each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data.columns)\n",
    "#!cat data/data_000637.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Columns can be specified when reading the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "file_name = \"data/magic04.data\"\n",
    "data = pd.read_csv(file_name, nrows=1000, names=['fLength', 'fWidth', 'fSize', 'fConc', 'fConc1', 'fAsym', 'fM3Long', 'fM3Trans', 'fAlpha', 'fDist', 'category'])\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "data.plot.scatter(\"fLength\", \"fWidth\",)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.hist(\"fAlpha\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROOT\n",
    "\n",
    "[ROOT](https://root.cern.ch/) needs a special mention. It is still nowadays, and by far, the most convenient tool to store and manage complex datasets pertaining physics experiments where \"events\" are recorded, in particular High Energy, Nuclear, Astro physics.\n",
    "\n",
    "ROOT allows a nested structure, with complex data objects (classes) and references between them. For instance, a variable number of attributes/features/variables can be set on a per-event basis, which is something that other data structures are not really optimized to do.\n",
    "\n",
    "ROOT is the C++ implementation of PAW (Fortran), which was the plotting and I/O tool for CERN experiments. Later on, ROOT evolved in a much more complete and complex tool, adding new packages like PyROOT (finally!), which provided an (almost) fully functional Python interpreter which made ROOT much more user friendly. Its core is still in C++ and uses a debatable memory management method, which is however hidden from the common user.\n",
    "\n",
    "In spite of the many flaws, its I/O and the statistical analysis tools included are formidable.\n",
    "\n",
    "Installing ROOT is [not trivial](https://root.cern/install/).\n",
    "\n",
    "Fortunately, ROOT files can be opened with a non-ROOT library, [uproot](https://github.com/scikit-hep/uproot). See the [documentation](https://uproot.readthedocs.io/en/latest/basic.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://scikit-hep.org/uproot3/examples/Zmumu.root -P data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uproot\n",
    "events = uproot.open(\"data/Zmumu.root:events\") # pass file name : name of the TTree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "events.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e1 = events['E1'].array(library=\"np\") # get numpy array - default library uses 'awkward'\n",
    "\n",
    "mass = events['M'].array(library=\"pd\") # \"pd\" returns pandas Series\n",
    "mass.hist(bins=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binary (hexadecimal) files\n",
    "\n",
    "The raw output of sensors often consists of hexadecimal files. Information is packed in a well defined format (similarly to how floating point numbers are formatted)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://www.dropbox.com/s/9nu2i111if55135/data_000637.dat -P ./data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read and process hexadecimal files in python you need to use the `\"b\"` option of `open` and progress along the file at step of defined length (depending on the size of the words information is packed into)\n",
    "\n",
    "There are several tool to display and edit hex/bin files:\n",
    "- from command line, using the `hexdump` command:\n",
    "- online, using online tools, like the [hexed.it](https://hexed.it/) website"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!hexdump data/data_000637.dat # warning, the file is large"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following is an example from data collected from an FPGA implementing a TDC. Relevant infomation are the coordinates of the TDC channels and their time measurements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image(\"images/data_format.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import struct, time\n",
    "\n",
    "data = {}\n",
    "columns = ['HEAD', 'FPGA', 'CHANNEL', 'ORBIT_CNT', 'BX_CNT', 'TDC_MEAS']\n",
    "df = pd.DataFrame({}, columns=columns)\n",
    "\n",
    "with open('data/data_000637.dat', 'rb') as file:\n",
    "    file_content = file.read()\n",
    "    word_counter = 0\n",
    "    word_size = 8 # size of the word in bytes\n",
    "    for i in range(0, len(file_content), word_size):\n",
    "        word_counter += 1\n",
    "        if word_counter > 10: break\n",
    "        word = struct.unpack('<q', file_content[i : i + word_size])[0] # get an 8-byte word\n",
    "        head     = (word >> 62) & 0x3\n",
    "        fpga     = (word >> 58) & 0xF\n",
    "        tdc_chan = (word >> 49) & 0x1FF\n",
    "        orb_cnt  = (word >> 17) & 0xFFFFFFFF\n",
    "        bx       = (word >> 5 ) & 0xFFF\n",
    "        tdc_meas = (word >> 0 ) & 0x1F\n",
    "        #if i == 0: print ('{0}\\t{1}\\t{2}\\t{3}\\t{4}\\t{5}'.format('HEAD', 'FPGA', 'CHANNEL', 'ORBIT_CNT', 'BX_CNT', 'TDC_MEAS'))\n",
    "        #print('{0}\\t{1}\\t{2}\\t{3}\\t{4}\\t{5}'.format(head, fpga, tdc_chan, orb_cnt, bx, tdc_meas))\n",
    "        entry = {'HEAD' : head, 'FPGA' : fpga, 'CHANNEL' : tdc_chan, 'ORBIT_CNT' : orb_cnt, 'BX_CNT' : bx, 'TDC_MEAS' : tdc_meas}\n",
    "        df = df.append(entry, ignore_index=True)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}