Analysis-scheme.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Inclusive B-tagging\n",
    "\n",
    "\n",
    "\n",
    "### Authors: \n",
    "* Tatiana Likhomanenko (contact)\n",
    "* Alexey Rogozhnikov\n",
    "* Denis Derkach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data: \n",
    "\n",
    "* real data $B^+ \\to J/\\psi K^+$, $B^- \\to J/\\psi K^-$ (RECO 14), 2012\n",
    "* apply sPlot to obtain sWeight ~ P(B)\n",
    "* set `event_id` - EventNumber & RunNumber"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "import pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Opposite side tagging (OS) tagging (Current version)\n",
    "\n",
    "https://github.com/tata-antares/tagging_LHCb/blob/master/old-tagging.ipynb\n",
    "\n",
    "We first tested the current algorithm (OS taggers: muon, electron, kaon, vertex). \n",
    "TMVA original method was compared with XGBoost.\n",
    "\n",
    "* isotonic symmetric calibration\n",
    "* use different train-test divisions to calculate $D^2$\n",
    "* compute mean and std \n",
    "* detail see below (the same formulas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Electron, muon, kaon and vertex taggers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>$\\epsilon_{tag}, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon_{tag}, \\%$</th>\n",
       "      <th>$D^2$</th>\n",
       "      <th>$\\Delta D^2$</th>\n",
       "      <th>$\\epsilon, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon, \\%$</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vtx_xgboost</td>\n",
       "      <td>18.1978</td>\n",
       "      <td>0.0495</td>\n",
       "      <td>0.0506</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.9212</td>\n",
       "      <td>0.0198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>vtx_tmva</td>\n",
       "      <td>18.1978</td>\n",
       "      <td>0.0495</td>\n",
       "      <td>0.0420</td>\n",
       "      <td>0.0009</td>\n",
       "      <td>0.7649</td>\n",
       "      <td>0.0171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>$K$_xgboost</td>\n",
       "      <td>17.0796</td>\n",
       "      <td>0.0479</td>\n",
       "      <td>0.0546</td>\n",
       "      <td>0.0011</td>\n",
       "      <td>0.9332</td>\n",
       "      <td>0.0182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>$K$_tmva</td>\n",
       "      <td>17.0796</td>\n",
       "      <td>0.0479</td>\n",
       "      <td>0.0476</td>\n",
       "      <td>0.0013</td>\n",
       "      <td>0.8129</td>\n",
       "      <td>0.0219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>$e$_xgboost</td>\n",
       "      <td>1.6424</td>\n",
       "      <td>0.0149</td>\n",
       "      <td>0.1745</td>\n",
       "      <td>0.0063</td>\n",
       "      <td>0.2865</td>\n",
       "      <td>0.0107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>$e$_tmva</td>\n",
       "      <td>1.6424</td>\n",
       "      <td>0.0149</td>\n",
       "      <td>0.1716</td>\n",
       "      <td>0.0072</td>\n",
       "      <td>0.2818</td>\n",
       "      <td>0.0120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>$\\mu$_xgboost</td>\n",
       "      <td>5.4385</td>\n",
       "      <td>0.0271</td>\n",
       "      <td>0.1709</td>\n",
       "      <td>0.0036</td>\n",
       "      <td>0.9294</td>\n",
       "      <td>0.0202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>$\\mu$_tmva</td>\n",
       "      <td>5.4385</td>\n",
       "      <td>0.0271</td>\n",
       "      <td>0.1649</td>\n",
       "      <td>0.0030</td>\n",
       "      <td>0.8967</td>\n",
       "      <td>0.0168</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name  $\\epsilon_{tag}, \\%$  $\\Delta \\epsilon_{tag}, \\%$   $D^2$  \\\n",
       "0    vtx_xgboost               18.1978                       0.0495  0.0506   \n",
       "1       vtx_tmva               18.1978                       0.0495  0.0420   \n",
       "2    $K$_xgboost               17.0796                       0.0479  0.0546   \n",
       "3       $K$_tmva               17.0796                       0.0479  0.0476   \n",
       "4    $e$_xgboost                1.6424                       0.0149  0.1745   \n",
       "5       $e$_tmva                1.6424                       0.0149  0.1716   \n",
       "6  $\\mu$_xgboost                5.4385                       0.0271  0.1709   \n",
       "7     $\\mu$_tmva                5.4385                       0.0271  0.1649   \n",
       "\n",
       "   $\\Delta D^2$  $\\epsilon, \\%$  $\\Delta \\epsilon, \\%$  \n",
       "0        0.0011          0.9212                 0.0198  \n",
       "1        0.0009          0.7649                 0.0171  \n",
       "2        0.0011          0.9332                 0.0182  \n",
       "3        0.0013          0.8129                 0.0219  \n",
       "4        0.0063          0.2865                 0.0107  \n",
       "5        0.0072          0.2818                 0.0120  \n",
       "6        0.0036          0.9294                 0.0202  \n",
       "7        0.0030          0.8967                 0.0168  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 4)\n",
    "pandas.read_csv('img/old-tagging-parts.csv').drop(['AUC, with untag', '$\\Delta$ AUC, with untag'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Combined taggers\n",
    "\n",
    "We then tested a combination with two calibrations for individual taggers: \n",
    "\n",
    "* isotonic regression \n",
    "* logistic regression. \n",
    "\n",
    "Combination was calibrated using isotonic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>$\\epsilon_{tag}, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon_{tag}, \\%$</th>\n",
       "      <th>$D^2$</th>\n",
       "      <th>$\\Delta D^2$</th>\n",
       "      <th>$\\epsilon, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon, \\%$</th>\n",
       "      <th>AUC, with untag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>iso-xgb_combined</td>\n",
       "      <td>32.735</td>\n",
       "      <td>0.0664</td>\n",
       "      <td>0.0703</td>\n",
       "      <td>0.0018</td>\n",
       "      <td>2.3003</td>\n",
       "      <td>0.0583</td>\n",
       "      <td>56.7023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>iso-tmva_combined</td>\n",
       "      <td>32.735</td>\n",
       "      <td>0.0664</td>\n",
       "      <td>0.0665</td>\n",
       "      <td>0.0019</td>\n",
       "      <td>2.1778</td>\n",
       "      <td>0.0611</td>\n",
       "      <td>56.5846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>log-xgb_combined</td>\n",
       "      <td>32.735</td>\n",
       "      <td>0.0664</td>\n",
       "      <td>0.0736</td>\n",
       "      <td>0.0008</td>\n",
       "      <td>2.4104</td>\n",
       "      <td>0.0277</td>\n",
       "      <td>56.6686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>log-tmva_combined</td>\n",
       "      <td>32.735</td>\n",
       "      <td>0.0664</td>\n",
       "      <td>0.0682</td>\n",
       "      <td>0.0008</td>\n",
       "      <td>2.2317</td>\n",
       "      <td>0.0278</td>\n",
       "      <td>56.5878</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name  $\\epsilon_{tag}, \\%$  $\\Delta \\epsilon_{tag}, \\%$  \\\n",
       "0   iso-xgb_combined                32.735                       0.0664   \n",
       "1  iso-tmva_combined                32.735                       0.0664   \n",
       "2   log-xgb_combined                32.735                       0.0664   \n",
       "3  log-tmva_combined                32.735                       0.0664   \n",
       "\n",
       "    $D^2$  $\\Delta D^2$  $\\epsilon, \\%$  $\\Delta \\epsilon, \\%$  \\\n",
       "0  0.0703        0.0018          2.3003                 0.0583   \n",
       "1  0.0665        0.0019          2.1778                 0.0611   \n",
       "2  0.0736        0.0008          2.4104                 0.0277   \n",
       "3  0.0682        0.0008          2.2317                 0.0278   \n",
       "\n",
       "   AUC, with untag  \n",
       "0          56.7023  \n",
       "1          56.5846  \n",
       "2          56.6686  \n",
       "3          56.5878  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 4)\n",
    "pandas.read_csv('img/old-tagging.csv').drop(['$\\Delta$ AUC, with untag'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Additional information\n",
    "\n",
    "Details see in the previous presentation: https://indico.cern.ch/event/369520/contribution/3/attachments/1178333/1704665/15.10.28.Tagging.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### $\\epsilon_{tag}$ calculation\n",
    "\n",
    "$$N (\\text{B events, passed selection}) = \\sum_{\\text{B events, passed selection}} sw_i$$\n",
    "\n",
    "$$N (\\text{all B events}) = \\sum_{\\text{all B events}} sw_i,$$\n",
    "\n",
    "where $sw_i$ - sPLot weight\n",
    "\n",
    "$$\\epsilon_{tag} = \\frac{N (\\text{passed selection})} {N (\\text{all events})}$$\n",
    "\n",
    "$$\\Delta\\epsilon_{tag} = \\frac{\\sqrt{N (\\text{passed selection})}} {N (\\text{all events})}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\n",
    "### Data for training\n",
    "\n",
    "* `data_sw_passed` - tracks/vertices with **B-sWeight > 1**, are used for training\n",
    "* `data_sw_not_passed` - tracks/vertices with **B-sWeight <= 1**, are tagged after training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Training "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Tracks Features (sig = signal, part = tagger track):\n",
    "\n",
    "* `cos_diff_phi` = $\\cos(\\phi^{sig} - \\phi^{\\rm part})$\n",
    "* `diff_pt` = $\\max(p_T)^{part} - p_T(B^{sig})$\n",
    "* `partPt`= $p_T^{part}$\n",
    "* `max_PID_e_mu` = $\\max(PIDNN(e), PIDNN(\\mu))^{part}$\n",
    "* `partP` = $p^{part}$\n",
    "* `nnkrec` = Number of reconstructed vertices\n",
    "* `diff_eta` = $(\\eta^{sig} - \\eta^{\\rm part})$\n",
    "* `EOverP` = E/P (from CALO)\n",
    "* `sum_PID_k_mu` = $\\sum\\limits_{i\\in part}(PIDNN(K)+PIDNN(\\mu))$\n",
    "* `ptB` = $p_T^{sig}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "* `sum_PID_e_mu` = $\\sum\\limits_{i\\in part}(PIDNN(e)+PIDNN(\\mu))$\n",
    "* `sum_PID_k_e` = $\\sum\\limits_{i\\in part}(PIDNN(K)+PIDNN(e))$\n",
    "* `proj` = $(\\vec{p}^{sig},\\vec{p}^{part})$\n",
    "* `PIDNNe` = $PIDNN(e)$\n",
    "* `PIDNNk` = $PIDNN(K)$\n",
    "* `PIDNNm` = $PIDNN(\\mu)$\n",
    "* `phi` = $\\phi^{sig}$\n",
    "* `IP` = number of IPs in the event"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "* `max_PID_k_mu` = $max(PIDNN(K)+PIDNN(\\mu))$\n",
    "* `IPerr` = error of IP\n",
    "* `IPs` = IP/IPerr\n",
    "* `veloch` = dE/dx track charge from the VELO system\n",
    "* `max_PID_k_e`  = $max(PIDNN(K)+PIDNN(e))$\n",
    "* `diff_phi`  = $(\\phi^{sig} - \\phi^{\\rm part})$\n",
    "* `ghostProb` = ghost probability \n",
    "* `IPPU` =  impact parameter with respect to any other reconstructed primary vertex.\n",
    "* `eta` = pseudorapity of signal particle\n",
    "* `partlcs` = chi2PerDoF for a track "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Vertex Selections\n",
    "\n",
    "* All selection are removed except DaVinci probability cuts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Vertex Features:\n",
    "\n",
    "* `mult` = multiplicity in the event\n",
    "* `nnkrec` = number of reconstructed vertices \n",
    "* `ptB` = signal B transverse momentum \n",
    "* `vflag` = number of tracks in the vertex\n",
    "* `ipsmean` = mean distance to IP of the tracks\n",
    "* `ptmean`  = mean pt of the tracks\n",
    "* `vcharge` = charge of the vertex weigthed by pt\n",
    "* `svm` = mass of the vertex \n",
    "* `svp` = momentum of the vertex\n",
    "* `BDphiDir` = distance betwen B and vertex direction\n",
    "* `svtau`  = lifetime of the vertex\n",
    "* `docamax` = mean DOCA of the tracks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Classifier\n",
    "\n",
    "Try to define B sign using track/vertex sign (to define they have the same signs or opposite).\n",
    "\n",
    "`target` = `signB` * `signTrack/signVertex` > 0\n",
    "\n",
    "* classifier returns \n",
    "$$P(\\text{track/vertex same sign as B| B sign}) = $$\n",
    "$$ =P(\\text{B same sign as track/vertex| track sign})$$\n",
    "* 2-folding training on the full training sample to use full sample for further analysis (folding scheme provides not overfitted model, details: http://yandex.github.io/rep/metaml.html#module-rep.metaml.folding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Calibration of $P(\\text{track/vertex same sign as B| B sign})$\n",
    "\n",
    "* use 2-folding logistic/isotonic calibration for track/vertex classifier's prediction\n",
    "* compare with isotonic/logistic calibration\n",
    "* compare with absent calibration (bad, have shift predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Computation of $p(B^+)$ using $P(\\text{track/vertex same sign as B| B sign})$\n",
    "\n",
    "Compute $p(B^+)$ using this probabilistic model representation (similar to the previous tagging combination):\n",
    "\n",
    "$$ \\frac{P(B^+)}{P(B^-)} = \\prod_{track, vertex} \\frac{P(\\text{track/vertex}|B^+)} {P(\\text{track/vertex} |B^-)} = \\alpha\n",
    "\\qquad $$\n",
    "$$\\Rightarrow\\qquad P(B^+) = \\frac {\\alpha}{1+\\alpha},   \\qquad \\qquad    [1] $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "where\n",
    "\n",
    "$$\n",
    "\\frac{P(B^+)}{P(B^-)} = \\prod_{track, vertex} \n",
    "\\begin{cases}\n",
    "\\frac{P(\\text{track/vertex same sign as } B| B)}{P(\\text{track/vertex opposite sign as } B| B)}, \\text{if track/vertex}^+ \\\\ \\\\\n",
    "\\frac{P(\\text{track/vertex opposite sign as } B| B)}{P(\\text{track/vertex same sign as } B| B)}, \\text{if track/vertex}^- \n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "$$p_{mistag} = min(p(B^+), p(B^-))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Intermediate estimation $<D^2>$ for tracking\n",
    "\n",
    "Do calibration of $p(B^+)$ and compute $<D^2>$ :\n",
    "\n",
    "* use Isotonic calibration (generalization of bins fitting by linear function) - piecewise-constant monotonic function\n",
    "* randomly divide events into two parts (1-train, 2-calibrate)\n",
    "* symmetric isotonic fitting on train and $<D^2>$ computation on test\n",
    "* take mean and std for computed $<D^2>$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "$<D^2>$  formula for sample:\n",
    "$$<D^2> =  \\frac{\\sum_i[2(p^{mistag}_i - 0.5)]^2 * sw_i}{\\sum_i sw_i} = $$\n",
    "$$ =  \\frac{\\sum_i[2(p_i(B^+) - 0.5)]^2 * sw_i}{\\sum_i sw_i}$$\n",
    "\n",
    "Formula is symmetric and it is not necessary to compute mistag probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Preliminary estimation\n",
    "\n",
    "### $\\epsilon$ calculation\n",
    "\n",
    "$$\\epsilon = <D^2> * \\epsilon_{tag}$$\n",
    "\n",
    "$$\\Delta \\epsilon = \\sqrt{ \\left(\\frac{\\Delta <D^2>}{<D^2>}\\right)^2 + \\left(\\frac{\\Delta \\epsilon_{tag} }{\\epsilon_{tag}} \\right)^2 }$$\n",
    "\n",
    "* Combine track-based and vertex-based tagging using formula [1]\n",
    "* symmetric isotonic calibration on random subsample with $D^2$ calculation\n",
    "* take mean and std for computed $<D^2>$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Full estimation of systematic error\n",
    "\n",
    "* set random state\n",
    "* train the best model (track and vertex taggers with 2-folding with fixed random state)\n",
    "* do calibration for track and vertex taggers with 2-folding with fixed random state\n",
    "* compute $p(B^+)$\n",
    "* do calibration with isotonic 2-folding (random state is fixed)\n",
    "* compute $<D^2>$\n",
    "\n",
    "This procedure is repeated (from the scratch) for 30 different random states and then we compute **mean** and **std** for these 30 values of $<D^2>$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Check calibration of mistag\n",
    "* axis x: predicted mistag probability \n",
    "$$p_{mistag} = min(p(B^+), p(B^-))$$\n",
    "* axis y: true mistag probability (computed for bin)\n",
    "$$p_{mistag} = \\frac{N_{wrong}} {N_{wrong} + N_{right}}$$\n",
    "\n",
    "$$\\Delta p_{mistag} = \\frac{\\sqrt{N_{wrong} N_{right}}} {(N_{wrong} + N_{right})^{1.5}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Stability of calibration\n",
    "Add random noise after isotonic calibration of $p(B^+)$ for stability:\n",
    "\n",
    "$$ 0.001 * normal(0, 1)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Inclusive tagging (NEW)\n",
    "\n",
    "* Check \"OS\" and \"SS\" regions separately (to check that tagging includes \"SS\" and \"OS\")\n",
    "* Check dependences on lifetime, lifetime error, number of tracks, momentum, transverse momentum, mass\n",
    "* Asymmetry of charges in events: understanding of high tagging quality or what information we use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tracking \"OS\" tagging \n",
    "https://github.com/tata-antares/tagging_LHCb/blob/master/track-based-tagging-OS.ipynb\n",
    "\n",
    "\n",
    "Take all possible tracks for all B-events.\n",
    "\n",
    "Apply:\n",
    "\n",
    "* (IPs > 3) & ((abs(diff_eta) > 0.6) | (abs(diff_phi) > 0.825)) - geometrical cuts\n",
    "* (PIDNNp < 0.5) & (PIDNNpi < 0.5) & (ghostProb < 0.4)\n",
    "* ((PIDNNk > trk) | (PIDNNm > trm) | (PIDNNe > tre)), trk=0., trm=0., tre=0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### B mass before sWeight cut\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bmass_OS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### B mass after sWeight cut\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bmass_selected_OS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Number of tracks in event\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/tracks_number_OS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### PIDNN distributions after selection\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/PID_selected_OS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Preliminary estimation (track OS + vertex OS)\n",
    "\n",
    "https://github.com/tata-antares/tagging_LHCb/blob/master/combined-tagging-OS.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>$\\epsilon_{tag}, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon_{tag}, \\%$</th>\n",
       "      <th>$D^2$</th>\n",
       "      <th>$\\Delta D^2$</th>\n",
       "      <th>$\\epsilon, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon, \\%$</th>\n",
       "      <th>AUC, with untag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Inclusive tagging, PID less</td>\n",
       "      <td>86.70309</td>\n",
       "      <td>0.10803</td>\n",
       "      <td>0.02494</td>\n",
       "      <td>0.00033</td>\n",
       "      <td>2.16214</td>\n",
       "      <td>0.02875</td>\n",
       "      <td>57.20222</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          name  $\\epsilon_{tag}, \\%$  \\\n",
       "0  Inclusive tagging, PID less              86.70309   \n",
       "\n",
       "   $\\Delta \\epsilon_{tag}, \\%$    $D^2$  $\\Delta D^2$  $\\epsilon, \\%$  \\\n",
       "0                      0.10803  0.02494       0.00033         2.16214   \n",
       "\n",
       "   $\\Delta \\epsilon, \\%$  AUC, with untag  \n",
       "0                0.02875         57.20222  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 5)\n",
    "pandas.read_csv('img/eff_OS.csv').drop(['$\\Delta$ AUC, with untag'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Check calibration of mistag\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_iso_calibrated_OS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### before calibration\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_calibration_check_percentile_OS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Symmetric isotonic calibration + random noise * 0.001 (noise for stability of bins)\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_calibration_check_iso_percentile_OS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tracking \"SS\" tagging\n",
    "https://github.com/tata-antares/tagging_LHCb/blob/master/track-based-tagging-SS.ipynb\n",
    "\n",
    "\n",
    "Take all possible tracks for all B-events.\n",
    "\n",
    "Apply:\n",
    "\n",
    "* (IPs < 3) & (abs(diff_eta) < 0.6) & (abs(diff_phi) < 0.825) & (ghostProb < 0.4)\n",
    "* ((PIDNNk > {trk}) | (PIDNNm > {trm}) | (PIDNNe > {tre}) | (PIDNNpi > {trpi}) | (PIDNNp > {trp})), trk=0, trm=0, tre=0, trpi=0, trp=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### B mass before sWeight cut\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bmass_SS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### B mass after sWeight cut\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bmass_selected_SS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Number of tracks in event\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/tracks_number_SS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### PIDNN distributions after selection\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/PID_selected_SS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Preliminary estimation (track \"SS\" only)\n",
    "https://github.com/tata-antares/tagging_LHCb/blob/master/combined-tagging-SS.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>$\\epsilon_{tag}, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon_{tag}, \\%$</th>\n",
       "      <th>$D^2$</th>\n",
       "      <th>$\\Delta D^2$</th>\n",
       "      <th>$\\epsilon, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon, \\%$</th>\n",
       "      <th>AUC, with untag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Inclusive tagging, PID less</td>\n",
       "      <td>72.39764</td>\n",
       "      <td>0.09872</td>\n",
       "      <td>0.03077</td>\n",
       "      <td>0.00035</td>\n",
       "      <td>2.22756</td>\n",
       "      <td>0.02573</td>\n",
       "      <td>57.419</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          name  $\\epsilon_{tag}, \\%$  \\\n",
       "0  Inclusive tagging, PID less              72.39764   \n",
       "\n",
       "   $\\Delta \\epsilon_{tag}, \\%$    $D^2$  $\\Delta D^2$  $\\epsilon, \\%$  \\\n",
       "0                      0.09872  0.03077       0.00035         2.22756   \n",
       "\n",
       "   $\\Delta \\epsilon, \\%$  AUC, with untag  \n",
       "0                0.02573           57.419  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 5)\n",
    "pandas.read_csv('img/eff_tracking_SS.csv').drop(['$\\Delta$ AUC, with untag'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Check calibration of mistag\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_iso_calibrated_SS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### before calibration\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_calibration_check_percentile_SS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Symmetric isotonic calibration + random noise * 0.001 (noise for stability of bins)\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_calibration_check_iso_percentile_SS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tracking inclusive tagging\n",
    "https://github.com/tata-antares/tagging_LHCb/blob/master/track-based-tagging-PID-less.ipynb\n",
    "\n",
    "\n",
    "Take all possible tracks for all B-events.\n",
    "\n",
    "Apply:\n",
    "\n",
    "* (ghostProb < 0.4)\n",
    "* ((PIDNNk > {trk}) | (PIDNNm > {trm}) | (PIDNNe > {tre}) | (PIDNNpi > {trpi}) | (PIDNNp > {trp})), trk=0, trm=0, tre=0, trpi=0, trp=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### B mass before sWeight cut\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bmass_less_PID.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### B mass after sWeight cut\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bmass_selected_less_PID.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Number of tracks in event\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/tracks_number_less_PID.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### PIDNN distributions after selection\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/PID_selected_less_PID.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Dependence on PIDNN cuts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "* `(PIDNNp < 0.6) & (PIDNNpi < 0.6) & (ghostProb < 0.4)`\n",
    "* `( (PIDNNk > 0.7) | (PIDNNm > 0.4) | (PIDNNe > 0.6) ) `"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>$\\epsilon_{tag}, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon_{tag}, \\%$</th>\n",
       "      <th>$D^2$</th>\n",
       "      <th>$\\Delta D^2$</th>\n",
       "      <th>$\\epsilon, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon, \\%$</th>\n",
       "      <th>AUC, with untag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Inclusive tagging</td>\n",
       "      <td>77.78995</td>\n",
       "      <td>0.10233</td>\n",
       "      <td>0.03449</td>\n",
       "      <td>0.00046</td>\n",
       "      <td>2.68331</td>\n",
       "      <td>0.03576</td>\n",
       "      <td>57.92576</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name  $\\epsilon_{tag}, \\%$  $\\Delta \\epsilon_{tag}, \\%$  \\\n",
       "0  Inclusive tagging              77.78995                      0.10233   \n",
       "\n",
       "     $D^2$  $\\Delta D^2$  $\\epsilon, \\%$  $\\Delta \\epsilon, \\%$  \\\n",
       "0  0.03449       0.00046         2.68331                0.03576   \n",
       "\n",
       "   AUC, with untag  \n",
       "0         57.92576  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 5)\n",
    "pandas.read_csv('img/new-tagging.csv').drop(['$\\Delta$ AUC, with untag'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* `(PIDNNp < 0.6) & (PIDNNpi < 0.6) & (ghostProb < 0.4)`\n",
    "* `( (PIDNNk > 0.1) | (PIDNNm > 0.1) | (PIDNNe > 0.1) ) `"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>$\\epsilon_{tag}, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon_{tag}, \\%$</th>\n",
       "      <th>$D^2$</th>\n",
       "      <th>$\\Delta D^2$</th>\n",
       "      <th>$\\epsilon, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon, \\%$</th>\n",
       "      <th>AUC, with untag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Inclusive tagging</td>\n",
       "      <td>97.0983</td>\n",
       "      <td>0.1143</td>\n",
       "      <td>0.0384</td>\n",
       "      <td>0.0003</td>\n",
       "      <td>3.7256</td>\n",
       "      <td>0.0306</td>\n",
       "      <td>60.5811</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name  $\\epsilon_{tag}, \\%$  $\\Delta \\epsilon_{tag}, \\%$  \\\n",
       "0  Inclusive tagging               97.0983                       0.1143   \n",
       "\n",
       "    $D^2$  $\\Delta D^2$  $\\epsilon, \\%$  $\\Delta \\epsilon, \\%$  \\\n",
       "0  0.0384        0.0003          3.7256                 0.0306   \n",
       "\n",
       "   AUC, with untag  \n",
       "0          60.5811  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 5)\n",
    "pandas.read_csv('img/new-tagging_relax1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* `(PIDNNpi < 0.6) & (ghostProb < 0.4)`\n",
    "* `( (PIDNNk > 0.) | (PIDNNm > 0.) | (PIDNNe > 0.) ) `"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>$\\epsilon_{tag}, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon_{tag}, \\%$</th>\n",
       "      <th>$D^2$</th>\n",
       "      <th>$\\Delta D^2$</th>\n",
       "      <th>$\\epsilon, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon, \\%$</th>\n",
       "      <th>AUC, with untag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Inclusive tagging</td>\n",
       "      <td>99.208</td>\n",
       "      <td>0.1156</td>\n",
       "      <td>0.0408</td>\n",
       "      <td>0.0004</td>\n",
       "      <td>4.05</td>\n",
       "      <td>0.0356</td>\n",
       "      <td>61.2362</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name  $\\epsilon_{tag}, \\%$  $\\Delta \\epsilon_{tag}, \\%$  \\\n",
       "0  Inclusive tagging                99.208                       0.1156   \n",
       "\n",
       "    $D^2$  $\\Delta D^2$  $\\epsilon, \\%$  $\\Delta \\epsilon, \\%$  \\\n",
       "0  0.0408        0.0004            4.05                 0.0356   \n",
       "\n",
       "   AUC, with untag  \n",
       "0          61.2362  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 5)\n",
    "pandas.read_csv('img/new-tagging_relax2.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Preliminary estimation (track: OS+SS, OS vertex)\n",
    "\n",
    "https://github.com/tata-antares/tagging_LHCb/blob/master/combined-tagging-PID-less.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>$\\epsilon_{tag}, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon_{tag}, \\%$</th>\n",
       "      <th>$D^2$</th>\n",
       "      <th>$\\Delta D^2$</th>\n",
       "      <th>$\\epsilon, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon, \\%$</th>\n",
       "      <th>AUC, with untag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Inclusive tagging, PID less</td>\n",
       "      <td>99.98595</td>\n",
       "      <td>0.11601</td>\n",
       "      <td>0.05873</td>\n",
       "      <td>0.00043</td>\n",
       "      <td>5.87239</td>\n",
       "      <td>0.04359</td>\n",
       "      <td>64.08899</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          name  $\\epsilon_{tag}, \\%$  \\\n",
       "0  Inclusive tagging, PID less              99.98595   \n",
       "\n",
       "   $\\Delta \\epsilon_{tag}, \\%$    $D^2$  $\\Delta D^2$  $\\epsilon, \\%$  \\\n",
       "0                      0.11601  0.05873       0.00043         5.87239   \n",
       "\n",
       "   $\\Delta \\epsilon, \\%$  AUC, with untag  \n",
       "0                0.04359         64.08899  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 5)\n",
    "pandas.read_csv('img/new-tagging-PID-less.csv').drop(['$\\Delta$ AUC, with untag'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Preliminary estimation (track: OS+SS, no vertex)\n",
    "\n",
    "https://github.com/tata-antares/tagging_LHCb/blob/master/track-based-tagging-PID-less.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>$\\epsilon_{tag}, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon_{tag}, \\%$</th>\n",
       "      <th>$D^2$</th>\n",
       "      <th>$\\Delta D^2$</th>\n",
       "      <th>$\\epsilon, \\%$</th>\n",
       "      <th>$\\Delta \\epsilon, \\%$</th>\n",
       "      <th>AUC, with untag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Inclusive tagging</td>\n",
       "      <td>99.98595</td>\n",
       "      <td>0.11601</td>\n",
       "      <td>0.06303</td>\n",
       "      <td>0.00051</td>\n",
       "      <td>6.30254</td>\n",
       "      <td>0.05125</td>\n",
       "      <td>64.43919</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                name  $\\epsilon_{tag}, \\%$  $\\Delta \\epsilon_{tag}, \\%$  \\\n",
       "0  Inclusive tagging              99.98595                      0.11601   \n",
       "\n",
       "     $D^2$  $\\Delta D^2$  $\\epsilon, \\%$  $\\Delta \\epsilon, \\%$  \\\n",
       "0  0.06303       0.00051         6.30254                0.05125   \n",
       "\n",
       "   AUC, with untag  \n",
       "0         64.43919  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 5)\n",
    "pandas.read_csv('img/new-tagging_full_tracks.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Checks on track: OS+SS, OS vertex model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Check calibration of mistag\n",
    "\n",
    "#### for signal (B-like events)\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_iso_calibrated_PID_less.png)\n",
    "\n",
    "#### for background\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_iso_calibrated_PID_less_bck.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### before calibration\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_calibration_check_percentile_PID_less.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Symmetric isotonic calibration + random noise * 0.001 (noise for stability of bins)\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/Bprob_calibration_check_iso_percentile_PID_less.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Tagging power dependency on ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "* For B mass, B momentum, B transverse momentum, B lifetime use sidebands as bck and peak region as signal:\n",
    "    * `mask_signal = ((Bmass > 5.27) & (Bmass < 5.3))`\n",
    "    * `mask_bck = ((Bmass < 5.25) | (Bmass > 5.32))`\n",
    "* For B lifetime error and number of tracks use sWeights "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Procedure:\n",
    "* divide variable into 5 percentile bins\n",
    "* for each bin plot mistag vs true mistag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Signal dependence \n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_sig_B_mass.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_sig_B_P.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_sig_B_Pt.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_sig_life_time.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_sig_life_time_error.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_sig_N_tracks.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Bck dependence\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_bck_B_mass.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_bck_B_P.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_bck_B_Pt.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_bck_life_time.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_bck_life_time_error.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/dependence_bck_N_tracks.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why effective efficiency is so high for this model (model of tracks probability combination to obtain B probability)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see on the following characteristic of the event:\n",
    "\n",
    "$$ -\\sum_{track} charge_{track}$$\n",
    "\n",
    "It seems, that for $B^+$ event it should be around +1 + constant (because we exclude signal part)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Regions:\n",
    "\n",
    "* 'OS' region: (IP > 3)  & ((abs(diff_eta) > 0.6) | (abs(diff_phi) > 0.825))\n",
    "* 'SS' region: (IP < 3)  & (abs(diff_eta) < 0.6) & (abs(diff_phi) < 0.825)\n",
    "* full data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### \"OS\" data \n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_OS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### \"SS\" data \n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_SS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Full sample\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_full.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Add signal track"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### \"OS\" data \n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_with_sig_part_OS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### \"SS\" data \n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_with_sig_part_SS.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Full sample\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_with_sig_part_full.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Means of distributions (with signal track and without it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>$B^+$</th>\n",
       "      <th>$B^+$, with signal part</th>\n",
       "      <th>$B^-$</th>\n",
       "      <th>$B^-$, with signal part</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>ROC AUC, with signal part</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>full</th>\n",
       "      <td>0.44341</td>\n",
       "      <td>-0.55659</td>\n",
       "      <td>-0.57216</td>\n",
       "      <td>0.42784</td>\n",
       "      <td>0.57158</td>\n",
       "      <td>0.56915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OS</th>\n",
       "      <td>0.11117</td>\n",
       "      <td>-0.88883</td>\n",
       "      <td>-0.15727</td>\n",
       "      <td>0.84273</td>\n",
       "      <td>0.52953</td>\n",
       "      <td>0.68460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SS</th>\n",
       "      <td>0.17597</td>\n",
       "      <td>-0.82403</td>\n",
       "      <td>-0.17810</td>\n",
       "      <td>0.82190</td>\n",
       "      <td>0.56770</td>\n",
       "      <td>0.77769</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        $B^+$  $B^+$, with signal part    $B^-$  $B^-$, with signal part  \\\n",
       "name                                                                       \n",
       "full  0.44341                 -0.55659 -0.57216                  0.42784   \n",
       "OS    0.11117                 -0.88883 -0.15727                  0.84273   \n",
       "SS    0.17597                 -0.82403 -0.17810                  0.82190   \n",
       "\n",
       "      ROC AUC  ROC AUC, with signal part  \n",
       "name                                      \n",
       "full  0.57158                    0.56915  \n",
       "OS    0.52953                    0.68460  \n",
       "SS    0.56770                    0.77769  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 5)\n",
    "pandas.read_csv('img/track_signs_assymetry_means.csv', index_col='name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**This ROC AUC score is similar to the current tagging implementation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Charges asymmetry checks on MC sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### \"OS\" sample\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_OS_mc.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### \"SS\" sample\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_SS_mc.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Full sample\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_full_mc.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Means of distributions for MC and data (with signal track and without it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>$B^+$</th>\n",
       "      <th>$B^+$, with signal part</th>\n",
       "      <th>$B^-$</th>\n",
       "      <th>$B^-$, with signal part</th>\n",
       "      <th>ROC AUC</th>\n",
       "      <th>ROC AUC, with signal part</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>full</th>\n",
       "      <td>0.44341</td>\n",
       "      <td>-0.55659</td>\n",
       "      <td>-0.57216</td>\n",
       "      <td>0.42784</td>\n",
       "      <td>0.57158</td>\n",
       "      <td>0.56915</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OS</th>\n",
       "      <td>0.11117</td>\n",
       "      <td>-0.88883</td>\n",
       "      <td>-0.15727</td>\n",
       "      <td>0.84273</td>\n",
       "      <td>0.52953</td>\n",
       "      <td>0.68460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SS</th>\n",
       "      <td>0.17597</td>\n",
       "      <td>-0.82403</td>\n",
       "      <td>-0.17810</td>\n",
       "      <td>0.82190</td>\n",
       "      <td>0.56770</td>\n",
       "      <td>0.77769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>full_mc</th>\n",
       "      <td>0.31778</td>\n",
       "      <td>-0.68222</td>\n",
       "      <td>-0.77006</td>\n",
       "      <td>0.22994</td>\n",
       "      <td>0.58490</td>\n",
       "      <td>0.57159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OS_mc</th>\n",
       "      <td>0.04728</td>\n",
       "      <td>-0.95272</td>\n",
       "      <td>-0.28497</td>\n",
       "      <td>0.71503</td>\n",
       "      <td>0.54030</td>\n",
       "      <td>0.69439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SS_mc</th>\n",
       "      <td>0.15520</td>\n",
       "      <td>-0.84480</td>\n",
       "      <td>-0.18414</td>\n",
       "      <td>0.81586</td>\n",
       "      <td>0.56807</td>\n",
       "      <td>0.78832</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           $B^+$  $B^+$, with signal part    $B^-$  $B^-$, with signal part  \\\n",
       "name                                                                          \n",
       "full     0.44341                 -0.55659 -0.57216                  0.42784   \n",
       "OS       0.11117                 -0.88883 -0.15727                  0.84273   \n",
       "SS       0.17597                 -0.82403 -0.17810                  0.82190   \n",
       "full_mc  0.31778                 -0.68222 -0.77006                  0.22994   \n",
       "OS_mc    0.04728                 -0.95272 -0.28497                  0.71503   \n",
       "SS_mc    0.15520                 -0.84480 -0.18414                  0.81586   \n",
       "\n",
       "         ROC AUC  ROC AUC, with signal part  \n",
       "name                                         \n",
       "full     0.57158                    0.56915  \n",
       "OS       0.52953                    0.68460  \n",
       "SS       0.56770                    0.77769  \n",
       "full_mc  0.58490                    0.57159  \n",
       "OS_mc    0.54030                    0.69439  \n",
       "SS_mc    0.56807                    0.78832  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.set_option('display.precision', 5)\n",
    "pandas.concat([pandas.read_csv('img/track_signs_assymetry_means.csv', index_col='name'),\n",
    "               pandas.read_csv('img/track_signs_assymetry_means_mc.csv', index_col='name')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Algorithm uses this information during tracks probabilities combination:\n",
    "\n",
    "$$\n",
    "\\frac{P(B^+)}{P(B^-)} = \\prod_{track, vertex} \n",
    "\\begin{cases}\n",
    "\\frac{P(\\text{track/vertex same sign as } B| B)}{P(\\text{track/vertex opposite sign as } B| B)}, \\text{if track/vertex}^+ \\\\ \\\\\n",
    "\\frac{P(\\text{track/vertex opposite sign as } B| B)}{P(\\text{track/vertex same sign as } B| B)}, \\text{if track/vertex}^- \n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "* #### Can we use it indeed?\n",
    "\n",
    "\n",
    "* #### What is the source of this asymmetry?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Current tagging algorithm implicitly also use this information! \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** The asymmery plays a discriminative role even if we choose the random track in the event! **\n",
    "\n",
    "![](https://raw.githubusercontent.com/tata-antares/tagging_LHCb/master/img/assymetry_tracks_random.png)\n",
    "\n",
    "**Random is not random!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checked\n",
    "\n",
    "* modify loss function during vertex training to use track tagging output as baseline (try to correct track predictions using vertex information): doesn't help, but maybe need all vertices for event\n",
    "* different calibrations: bins, logistic, isotonic\n",
    "* normalize number of positive and negative tracks for $B^+$ and $B^-$ separately to avoid track charges asymmetry, the quality drops:\n",
    "    * ROC AUC: 0.62\n",
    "    * $\\epsilon: 4.5$\n",
    "    \n",
    "* use track sign as feature during training (bad quality):\n",
    "    * ROC AUC: 0.611\n",
    "    * $\\epsilon: 3.6$\n",
    "* check as discriminative variable sum of weighted with Pt charges of tracks:\n",
    "\n",
    "$$ -\\frac{\\sum_{track} charge_{track} Pt_{track}} {\\sum_{track} Pt_{track}}$$\n",
    "   \n",
    "    * ROC AUC for all regions (\"OS\", \"SS\", full sample) < 0.5006\n",
    "    * doesn't discriminate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## TODO \n",
    "\n",
    "* check efficiency for inclusive tagging on the other decays (please, send us your tuples with Flavour tagging checker info)\n",
    "* understand the asymmetry of sum of charges (maybe somebody understands?) \n",
    "* get all vertices from DaVinci (to check if we indeed need vertex to improve tagger)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}